DataJobs.io
← Back to all jobs

Job Description

BMC Software’s AMI team is hiring a Data Scientist III to help build and improve enterprise AI capabilities. This hybrid role in Los Angeles, CA focuses on hands-on machine learning, statistical modeling, and Generative AI, including RAG evaluation and experimentation, with an emphasis on reproducible workflows and MLOps/LLMOps.

The work blends model development with rigorous measurement: designing evaluation datasets and golden test cases, benchmarking models (including open-weight options), and partnering with engineering to support production readiness and continuous quality.

What you’ll do

  • Design, develop, evaluate, and improve AI/ML solutions to address enterprise business and technical challenges.
  • Use statistical rigor, reproducible experimentation, and MLOps/LLMOps practices to improve model quality, RAG effectiveness, and overall AI application performance.
  • Own defined data science initiatives and lead moderately complex or ambiguous modeling and evaluation tasks.
  • Collaborate with engineering teams to support production readiness, packaging, deployment handoffs, monitoring, and lifecycle management.
  • Mentor less experienced Data Scientists.
  • Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases.
  • Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques using traditional ML, embeddings, and LLM-based approaches.
  • Analyze execution latency, token usage, inference costs, workload trends, and resource requirements using statistical modeling.
  • Use clustering and anomaly detection to identify recurring intents, unusual agent behavior, capability gaps, and execution regressions.
  • Perform exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experimentation.
  • Evaluate commercial and self-hosted open-weight models for accuracy, instruction adherence, structured outputs, latency, and cost.
  • Create and test prompt templates, few-shot examples, tool-calling prompts, and context assembly approaches.
  • Compare model alternatives and document quality, performance, and cost tradeoffs.
  • Collaborate with Senior Data Scientists and AI Engineering on model performance analysis and inference optimization.
  • Develop and evaluate document chunking, metadata, embedding, retrieval, reranking, and context selection strategies.
  • Compare vector, keyword/BM25, hybrid, and semantic retrieval approaches.
  • Measure retrieval effectiveness using Recall@K, Precision@K, nDCG, groundedness, faithfulness, and answer quality.
  • Investigate retrieval and generation failures, then recommend improvements based on evaluation results.
  • Benchmark embedding and reranking models using representative enterprise datasets.
  • Support evaluation of prompting, RAG, supervised instruction fine-tuning, and parameter-efficient adaptation approaches.
  • Prepare and validate instruction datasets, training examples, and holdout evaluation datasets, including participation in SFT, LoRA, QLoRA, or comparable fine-tuning experiments under appropriate technical guidance.
  • Compare adapted models against baseline models for accuracy, generalization, regressions, and operational cost.
  • Collaborate on validation and production handoff of fine-tuned model artifacts.
  • Apply modeling and evaluation techniques to intent classification, agent/skill discovery, tool selection, and routing effectiveness.
  • Evaluate task completion, tool-call correctness, parameter extraction, delegation quality, and recovery behavior.
  • Understand agentic AI concepts including planning, tool calling, MCP tools, A2A Agent Cards, skills, and task delegation.
  • Contribute model-based decision signals and evaluation approaches related to agentic workflows, without owning agent runtime, MCP server, or A2A protocol implementation.
  • Develop evaluation datasets, golden test cases, holdout sets, and automated evaluation workflows.
  • Track accuracy, relevance, groundedness, instruction following, task completion, latency, and cost; investigate evaluation failures using experimental evidence.
  • Support automated regression testing and production readiness validation, including experiment tracking and versioning across datasets, code, model configurations, prompts, retrieval settings, metrics, and artifacts.
  • Develop and maintain reproducible model training, benchmarking, and evaluation workflows, including model registries and automated evaluation pipelines.
  • Contribute to model quality gates, regression testing, and release-readiness checks, including analysis of performance, quality drift, routing accuracy, latency, and inference cost signals.

What you’ll bring

  • 5-9 years of relevant Data Science, Machine Learning, or Applied AI experience.
  • Bachelor’s degree in Machine Learning, Statistics, Data Science, Physics, Computer Science, or a related field, or equivalent education and industry experience.
  • Strong Python proficiency with Pandas, NumPy, scikit-learn, and PyTorch or TensorFlow; SQL and practical data preparation skills.
  • Solid understanding of statistical analysis, experimental design, hypothesis testing, ML validation, classification, regression, ranking, clustering, forecasting, and anomaly detection.
  • Practical experience with LLM prompting, embeddings, model comparison, and RAG concepts, including chunking, retrieval, and reranking.
  • Familiarity with instruction for fine-tuning and parameter-efficient adaptation or model customization techniques.
  • Experience developing evaluation datasets, automated quality metrics, and reproducible ML/LLM experiments.
  • Hands-on experience with MLOps/LLMOps, including experiment tracking, dataset/model/prompt versioning, reproducible workflows, evaluation automation, and monitoring.
  • Working knowledge of agentic AI concepts including planning, routing, tool calling, MCP, A2A, and multi-agent patterns.
  • Understanding of model latency, throughput, token usage, inference cost, and model performance tradeoffs.
  • Familiarity with AWS and/or Kubernetes or Red Hat OpenShift (OCP), plus collaboration with engineering teams.
  • Clear communication, problem-solving, cross-functional collaboration, and ability to manage multiple priorities.
  • Demonstrated ability to deliver moderately complex AI/ML initiatives, make sound technical decisions, and mentor junior Data Scientists.
  • Experience contributing reusable modeling, evaluation, or MLOps/LLMOps practices within project teams.

Tools and technologies

  • Python, Pandas, NumPy, scikit-learn, PyTorch, TensorFlow, SQL
  • MLOps, LLMOps, MCP, A2A, AWS, Kubernetes, Red Hat OpenShift (OCP)
  • BM25, Recall@K, Precision@K, nDCG, LoRA, QLoRA, SFT
  • FastAPI, Flask, OpenTelemetry, OpenSearch, Elasticsearch, FAISS, Milvus, pgvector
  • LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, MLflow, Kubeflow
  • Ragas, DeepEval, OpenJudge, LangSmith
  • vLLM, NVIDIA Triton, quantization, GPU memory

Compensation and benefits

  • Annual base salary range $131,700 - $219,500 (actual salaries depend on skill sets; experience and training; licensure/certifications; and other business/organizational needs).
  • Other rewards may include a variable plan and country specific benefits.

Additional job details

  • Location: Los Angeles, CA, United States (hybrid). Travel may be required occasionally for business.
  • Remote: This role can be based remotely in United States, and is also listed as #LI-Hybrid and #LI-Remote.
  • Additional locations: San Francisco - Los Angeles - San Diego - Sacramento - Anaheim - San Francisco.
  • Job ID: 47815; Published 05-Nov-2026.
  • Job name: Data Scientist III - USA (B).

Nice-to-have skills

  • Experience with OpenSearch, Elasticsearch, FAISS, Milvus, pgvector, or similar retrieval platforms.
  • Exposure to LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, or comparable agentic frameworks.
  • Familiarity with MLflow, Kubeflow, Ragas, DeepEval, OpenJudge, LangSmith, or similar MLOps/LLMOps and evaluation tools.
  • Exposure to instruction fine-tuning, SFT, LoRA, QLoRA, or domain-specific model adaptation.
  • Familiarity with vLLM, NVIDIA Triton, quantization, GPU memory considerations, or inference optimization.
  • Python backend APIs (FastAPI, Flask, or similar); OpenTelemetry and AI observability.
  • AI safety, prompt injection evaluation, privacy, responsible AI, and model governance.
  • Time-series forecasting, causal inference, or anomaly detection in enterprise systems data.
  • Agile practices and Atlassian Jira/Confluence.
  • Master’s degree in a related field.

Similar Jobs