AI Engineer, Evaluation

Responsibilities

• Craft and deploy evaluation frameworks that support Evaluation-Driven Development for AI systems operating in customer environments
• Specify domain specific quality metrics so evaluation signals reflect real user needs, constraints, and business goals
• Create and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture key behaviors and edge cases
• Treat these test suites as core system components that evolve alongside the AI system
• Build and sustain offline and online evaluation pipelines that integrate into system iteration loops
• Use evaluation results to inform prompt design, agent logic, model selection, and release readiness to drive measurable improvements
• Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments
• Investigate divergences between evaluation signals and real-world outcomes and refine grading approaches as systems and domains evolve
• Collaborate with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts to ensure evaluation frameworks guide production development and deployment

Requirements

• 2+ years of software engineering experience
• Strong Python engineering skills: write clean, maintainable Python and build production-grade evaluation and experimentation pipelines
• Experience with Evaluation-Driven or Experiment-Driven Development: familiarity with structured evaluation or experimentation frameworks to drive iteration and awareness of overfitting risks
• Ability to translate human judgment into code: collaborate with subject matter experts to encode high-quality judgments into test cases, scoring functions, and scalable graders
• Systems-oriented mindset: understand how evaluation interacts with prompts, agents, data, and deployment; enable fast iteration while preserving trust and safety in production
• AI-native working style: leverage AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging
• Travel: 10–50% of the time, depending on project and role

Technologies

• Python

Benefits

• Base salary range of USD 150K–250K plus meaningful equity and comprehensive benefits
• 100% coverage of medical, dental, and vision insurance for employees and their dependents
• Flexible time off policy
• Retirement and financial planning benefits, including pre-tax HSA, FSA, commuter accounts, 401(k), and financial coaching
• Wellness benefits covering physical and mental health, plus fertility and family-building support via Carrot
• Complimentary in-office lunches and snacks
• Access to state-of-the-art AI models and tools to tackle real-world business problems
• Opportunity to own high-impact projects across major enterprises
• Fast-paced, mission-driven culture that values curiosity, pragmatism, and excellence