Data Scientist III
Job Description
BMC Software’s AMI team is hiring a Data Scientist III to help build and improve enterprise AI capabilities. This hybrid role in Los Angeles, CA focuses on hands-on machine learning, statistical modeling, and Generative AI, including RAG evaluation and experimentation, with an emphasis on reproducible workflows and MLOps/LLMOps.
The work blends model development with rigorous measurement: designing evaluation datasets and golden test cases, benchmarking models (including open-weight options), and partnering with engineering to support production readiness and continuous quality.
What you’ll do
- Design, develop, evaluate, and improve AI/ML solutions to address enterprise business and technical challenges.
- Use statistical rigor, reproducible experimentation, and MLOps/LLMOps practices to improve model quality, RAG effectiveness, and overall AI application performance.
- Own defined data science initiatives and lead moderately complex or ambiguous modeling and evaluation tasks.
- Collaborate with engineering teams to support production readiness, packaging, deployment handoffs, monitoring, and lifecycle management.
- Mentor less experienced Data Scientists.
- Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases.
- Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques using traditional ML, embeddings, and LLM-based approaches.
- Analyze execution latency, token usage, inference costs, workload trends, and resource requirements using statistical modeling.
- Use clustering and anomaly detection to identify recurring intents, unusual agent behavior, capability gaps, and execution regressions.
- Perform exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experimentation.
- Evaluate commercial and self-hosted open-weight models for accuracy, instruction adherence, structured outputs, latency, and cost.
- Create and test prompt templates, few-shot examples, tool-calling prompts, and context assembly approaches.
- Compare model alternatives and document quality, performance, and cost tradeoffs.
- Collaborate with Senior Data Scientists and AI Engineering on model performance analysis and inference optimization.
- Develop and evaluate document chunking, metadata, embedding, retrieval, reranking, and context selection strategies.
- Compare vector, keyword/BM25, hybrid, and semantic retrieval approaches.
- Measure retrieval effectiveness using Recall@K, Precision@K, nDCG, groundedness, faithfulness, and answer quality.
- Investigate retrieval and generation failures, then recommend improvements based on evaluation results.
- Benchmark embedding and reranking models using representative enterprise datasets.
- Support evaluation of prompting, RAG, supervised instruction fine-tuning, and parameter-efficient adaptation approaches.
- Prepare and validate instruction datasets, training examples, and holdout evaluation datasets, including participation in SFT, LoRA, QLoRA, or comparable fine-tuning experiments under appropriate technical guidance.
- Compare adapted models against baseline models for accuracy, generalization, regressions, and operational cost.
- Collaborate on validation and production handoff of fine-tuned model artifacts.
- Apply modeling and evaluation techniques to intent classification, agent/skill discovery, tool selection, and routing effectiveness.
- Evaluate task completion, tool-call correctness, parameter extraction, delegation quality, and recovery behavior.
- Understand agentic AI concepts including planning, tool calling, MCP tools, A2A Agent Cards, skills, and task delegation.
- Contribute model-based decision signals and evaluation approaches related to agentic workflows, without owning agent runtime, MCP server, or A2A protocol implementation.
- Develop evaluation datasets, golden test cases, holdout sets, and automated evaluation workflows.
- Track accuracy, relevance, groundedness, instruction following, task completion, latency, and cost; investigate evaluation failures using experimental evidence.
- Support automated regression testing and production readiness validation, including experiment tracking and versioning across datasets, code, model configurations, prompts, retrieval settings, metrics, and artifacts.
- Develop and maintain reproducible model training, benchmarking, and evaluation workflows, including model registries and automated evaluation pipelines.
- Contribute to model quality gates, regression testing, and release-readiness checks, including analysis of performance, quality drift, routing accuracy, latency, and inference cost signals.
What you’ll bring
- 5-9 years of relevant Data Science, Machine Learning, or Applied AI experience.
- Bachelor’s degree in Machine Learning, Statistics, Data Science, Physics, Computer Science, or a related field, or equivalent education and industry experience.
- Strong Python proficiency with Pandas, NumPy, scikit-learn, and PyTorch or TensorFlow; SQL and practical data preparation skills.
- Solid understanding of statistical analysis, experimental design, hypothesis testing, ML validation, classification, regression, ranking, clustering, forecasting, and anomaly detection.
- Practical experience with LLM prompting, embeddings, model comparison, and RAG concepts, including chunking, retrieval, and reranking.
- Familiarity with instruction for fine-tuning and parameter-efficient adaptation or model customization techniques.
- Experience developing evaluation datasets, automated quality metrics, and reproducible ML/LLM experiments.
- Hands-on experience with MLOps/LLMOps, including experiment tracking, dataset/model/prompt versioning, reproducible workflows, evaluation automation, and monitoring.
- Working knowledge of agentic AI concepts including planning, routing, tool calling, MCP, A2A, and multi-agent patterns.
- Understanding of model latency, throughput, token usage, inference cost, and model performance tradeoffs.
- Familiarity with AWS and/or Kubernetes or Red Hat OpenShift (OCP), plus collaboration with engineering teams.
- Clear communication, problem-solving, cross-functional collaboration, and ability to manage multiple priorities.
- Demonstrated ability to deliver moderately complex AI/ML initiatives, make sound technical decisions, and mentor junior Data Scientists.
- Experience contributing reusable modeling, evaluation, or MLOps/LLMOps practices within project teams.
Tools and technologies
- Python, Pandas, NumPy, scikit-learn, PyTorch, TensorFlow, SQL
- MLOps, LLMOps, MCP, A2A, AWS, Kubernetes, Red Hat OpenShift (OCP)
- BM25, Recall@K, Precision@K, nDCG, LoRA, QLoRA, SFT
- FastAPI, Flask, OpenTelemetry, OpenSearch, Elasticsearch, FAISS, Milvus, pgvector
- LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, MLflow, Kubeflow
- Ragas, DeepEval, OpenJudge, LangSmith
- vLLM, NVIDIA Triton, quantization, GPU memory
Compensation and benefits
- Annual base salary range $131,700 - $219,500 (actual salaries depend on skill sets; experience and training; licensure/certifications; and other business/organizational needs).
- Other rewards may include a variable plan and country specific benefits.
Additional job details
- Location: Los Angeles, CA, United States (hybrid). Travel may be required occasionally for business.
- Remote: This role can be based remotely in United States, and is also listed as #LI-Hybrid and #LI-Remote.
- Additional locations: San Francisco - Los Angeles - San Diego - Sacramento - Anaheim - San Francisco.
- Job ID: 47815; Published 05-Nov-2026.
- Job name: Data Scientist III - USA (B).
Nice-to-have skills
- Experience with OpenSearch, Elasticsearch, FAISS, Milvus, pgvector, or similar retrieval platforms.
- Exposure to LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, or comparable agentic frameworks.
- Familiarity with MLflow, Kubeflow, Ragas, DeepEval, OpenJudge, LangSmith, or similar MLOps/LLMOps and evaluation tools.
- Exposure to instruction fine-tuning, SFT, LoRA, QLoRA, or domain-specific model adaptation.
- Familiarity with vLLM, NVIDIA Triton, quantization, GPU memory considerations, or inference optimization.
- Python backend APIs (FastAPI, Flask, or similar); OpenTelemetry and AI observability.
- AI safety, prompt injection evaluation, privacy, responsible AI, and model governance.
- Time-series forecasting, causal inference, or anomaly detection in enterprise systems data.
- Agile practices and Atlassian Jira/Confluence.
- Master’s degree in a related field.