Machine Learning Engineer 4 - Manager IC
Job Description
Capital One’s Enterprise ML Platform team builds and evolves a high-throughput, multi-tenant training platform that supports model workflows across the organization, from classical machine learning to GenAI fine-tuning and distributed computing using tools like Ray, Spark, and Dask. In this role, you’ll help run and extend an ML platform that powers critical, regulated workloads, while owning the Kubeflow Pipelines v2 application layer on Amazon EKS.
If you care about shipping reliable systems at scale, you’ll also contribute to agentic AI integration and work toward Responsible and Explainable AI through strong governance and security practices. The position is based in McLean, VA (onsite), with a salary range of USD 197,300 to 225,100 per year.
What you’ll do
- Design, build, and deliver ML models and components that address real business problems in collaboration with Product and Data Science teams
- Use ML expertise to guide infrastructure and implementation decisions, covering model choice, data and feature selection, training, hyperparameter tuning, dimensionality, and validation
- Solve complex problems by writing and testing application code, developing and validating ML models, and automating tests and deployment
- Collaborate in a cross-functional Agile team to create and enhance software for big data and ML applications
- Retrain, maintain, and monitor models in production
- Leverage or build cloud-based architectures and platforms to deliver optimized ML models at scale
- Construct optimized data pipelines to feed ML models
- Apply CI/CD best practices, including test automation and monitoring, to support successful deployment of models and application code
- Ensure code quality to reduce vulnerabilities, support risk governance for models, and apply best practices in Responsible and Explainable AI
How you’ll contribute to the platform
- Own and evolve the Kubeflow Pipelines v2 application layer on Amazon EKS
- Develop internal Python and Go SDKs, create custom Kubernetes operators, and build developer tooling to abstract complex infrastructure
- Integrate agentic AI into modern engineering lifecycles and contribute to upstream open-source Kubeflow
Key requirements
- Bachelor’s Degree or higher in Computer Science, Machine Learning, or a related quantitative field (Statistics, Economics, Operations Research, Analytics, Mathematics, Engineering)
- At least 4 years of experience programming with Python, Java, Golang, or C++
- At least 4 years of Machine Learning experience using PyTorch or Tensorflow and libraries including Pandas, NumPy, Scikit-learn
- At least 4 years operating large-scale distributed systems such as Spark and Ray to prepare AI or ML data
- At least 2 years deploying and operating ML solutions in production services in the cloud (AWS, GCP, Azure), using Kubernetes
Tools and technologies
Python, Scala, Java, Golang, C++, PyTorch, Tensorflow, Pandas, NumPy, Scikit-learn, Spark, Ray, Dask, Kubeflow Pipelines v2, Amazon EKS, Kubernetes, AWS, GCP, Azure, Docker, Kubeflow
Preferred qualifications
- Master’s or Doctoral Degree in Computer Science, Electrical Engineering, Mathematics, or related field
- 3+ years designing, training, optimizing, and evaluating modern ML models across architectures (e.g., Transformers, CNNs, RNNs) and paradigms (supervised, unsupervised, reinforcement learning), including loss design, hyperparameter tuning, and diagnostic debugging
- 3+ years architecting and scaling distributed, end-to-end ML systems, including production-ready training/evaluation data pipelines and algorithm or infrastructure optimization
- 3+ years managing ML workloads using Docker, Kubernetes, and pipeline orchestration tools (Kubeflow is a strong plus)
- 3+ years applying modern software best practices (CI/CD, version control, automated testing, code reviews) and deploying resilient systems with progressive rollout strategies (blue/green, canary/one-box), monitoring, alerting, and incident response planning
- 1+ years of experience as a technical lead developing ML solutions using industry best practices, patterns, and automation
- Authored/co-authored a paper on an ML technique, model, or proof of concept
Benefits
Comprehensive, competitive, and inclusive set of health, financial and other benefits that support your total well-being.