DataJobs.io
← Back to all jobs

Job Description

Iterable is hiring a Senior Machine Learning Engineer for agentic evaluation and self-improvement work on Iterable Nova in San Francisco.

Responsibilities

  • Lead evaluation and automatic optimization for Nova’s models and agents
  • Build evaluation systems including benchmarks, LLM judges, simulations, trajectory analysis, and regression tests
  • Create a feedback loop that moves from diagnosing failures to implementing fixes and measuring improvement
  • Translate evaluation outcomes into automatic optimization for prompts, context, tools, rubrics, and agent harnesses
  • Develop synthetic data pipelines for evaluation and post-training, with focus on quality, privacy, and reproducibility
  • Apply recent research approaches such as LLM-as-judge, reward modeling, test-time search, and post-training in production settings
  • Measure model and agent performance, identify where systems fail, and use findings to improve prompts, context, tools, and agent harnesses automatically
  • Partner with Nova retrieval and product teams to connect real user interactions to measurable improvements
  • Mentor engineers and help set technical direction for Nova

Requirements

  • 5+ years of hands-on MLE experience deploying complex ML systems into production at scale (for example production ML pipelines or large-scale experimentation platforms)
  • Master’s or PhD in Computer Science, Machine Learning, AI, or a related field
  • Ability to stay current with the latest research and focus on shipping reliable changes that measurably improve the customer experience
  • Strong Python skills with emphasis on readable, modular, and maintainable code, plus experience with modern ML frameworks
  • Deep experience evaluating LLMs or agents, including multi-turn behavior, tool use, and reasoning
  • Experience building automated evaluation such as LLM judges, reward models, or benchmark infrastructure
  • Experience with at least one of: prompt or context optimization, post-training, preference optimization, or reinforcement learning
  • Strong communication and cross-team collaboration
  • Experience with LLM APIs from multiple providers including Anthropic, OpenAI, and Google, including tool calling and prompt caching
  • Experience with automatic prompt optimization and error analysis

Technologies

  • Python
  • LLM-as-judge
  • Reward modeling
  • Test-time search
  • Post-training
  • Anthropic
  • OpenAI
  • Google

Location

  • San Francisco, CA (onsite)
  • US

Compensation

  • USD 150,000 - 250,000 per year

Similar Jobs