Senior Machine Learning Engineer, Agentic Evals and Self-Improvement
Agentic Ai
Ai Agent
Ai Agent Platform
Ai Ml
Ai Platform
Artificial Intelligence
Artificial Intelligence Engineer
Data Analysis
Data Engineer
Data Pipeline
Data Processing
Data Science
Engineer
Gen Ai Evaluation
Generative AI
Generative Ai Agents
Generative Ai Engineering
Machine Learning Engineering
Machine Learning Evaluation
Machine Learning Optimization
Machine Learning Platform
Machine Learning Systems
Machine Learning Training
Programming
Software Development
Software Engineering
Job Description
Iterable is hiring a Senior Machine Learning Engineer for agentic evaluation and self-improvement work on Iterable Nova in San Francisco.
Responsibilities
- Lead evaluation and automatic optimization for Nova’s models and agents
- Build evaluation systems including benchmarks, LLM judges, simulations, trajectory analysis, and regression tests
- Create a feedback loop that moves from diagnosing failures to implementing fixes and measuring improvement
- Translate evaluation outcomes into automatic optimization for prompts, context, tools, rubrics, and agent harnesses
- Develop synthetic data pipelines for evaluation and post-training, with focus on quality, privacy, and reproducibility
- Apply recent research approaches such as LLM-as-judge, reward modeling, test-time search, and post-training in production settings
- Measure model and agent performance, identify where systems fail, and use findings to improve prompts, context, tools, and agent harnesses automatically
- Partner with Nova retrieval and product teams to connect real user interactions to measurable improvements
- Mentor engineers and help set technical direction for Nova
Requirements
- 5+ years of hands-on MLE experience deploying complex ML systems into production at scale (for example production ML pipelines or large-scale experimentation platforms)
- Master’s or PhD in Computer Science, Machine Learning, AI, or a related field
- Ability to stay current with the latest research and focus on shipping reliable changes that measurably improve the customer experience
- Strong Python skills with emphasis on readable, modular, and maintainable code, plus experience with modern ML frameworks
- Deep experience evaluating LLMs or agents, including multi-turn behavior, tool use, and reasoning
- Experience building automated evaluation such as LLM judges, reward models, or benchmark infrastructure
- Experience with at least one of: prompt or context optimization, post-training, preference optimization, or reinforcement learning
- Strong communication and cross-team collaboration
- Experience with LLM APIs from multiple providers including Anthropic, OpenAI, and Google, including tool calling and prompt caching
- Experience with automatic prompt optimization and error analysis
Technologies
- Python
- LLM-as-judge
- Reward modeling
- Test-time search
- Post-training
- Anthropic
- OpenAI
Location
- San Francisco, CA (onsite)
- US
Compensation
- USD 150,000 - 250,000 per year
Similar Jobs
F