AI Engineer, Evaluation

Distyl AI builds evaluation systems that help AI teams ship measurable improvements into customer environments. In this hybrid San Francisco role, you will help create the tooling and signal-quality foundations for Evaluation-Driven Development, so iteration is guided by outcomes you can verify. The position comes with meaningful equity, comprehensive benefits, and the opportunity to work on high-impact projects with access to state-of-the-art AI models and modern AI tooling.

Responsibilities

• Design and implement evaluation frameworks that support Evaluation-Driven Development for AI systems deployed in customer environments.
• Define how system quality is measured across domains, ensuring evaluation signals reflect user needs, domain constraints, and business objectives.
• Build and maintain golden test cases and regression suites in Python, using human-authored and AI-assisted test generation to capture critical behaviors and edge cases.
• Treat test suites as first-class system components that evolve alongside the AI system.
• Develop and maintain evaluation pipelines that run offline and online, integrating results into system iteration loops.
• Ensure evaluation results inform prompt design, agent logic, model selection, and release readiness using measurable improvements instead of intuition.
• Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments and refining grading approaches when signals diverge from real-world outcomes.
• Collaborate with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts to ensure evaluation frameworks guide development and deployment in production.

Requirements

• 2+ years of software engineering experience.
• Strong Python engineering skills, including the ability to build evaluation and experimentation pipelines that run in production with the same rigor as application code.
• Experience with Evaluation-Driven Development or Experiment-Driven Development, including awareness of pitfalls like overfitting to metrics that do not reflect real outcomes.
• Ability to translate human judgment into code, working with subject matter experts to encode judgments into test cases, scoring functions, and graders that scale.
• Systems-oriented mindset for designing evaluation systems that connect prompts, agents, data, and deployment while supporting fast iteration and trust and safety in production.
• AI-native working style, using AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging and iteration.
• Travel between 10–50% of the time, depending on the project and role.

Benefits

• Meaningful equity plus a comprehensive benefits package.
• 100% coverage of medical, dental, and vision insurance for employees and dependents.
• Flexible time off.
• Retirement and financial planning benefits, including access to pre-tax HSA, FSA, and commuter accounts, 401(k), and financial coaching resources.
• Comprehensive wellness benefits, including physical fitness, mental well-being, and fertility and family-building benefits through Carrot.
• Complimentary in-office lunches and snacks provided.
• Access to state-of-the-art AI models, generous usage of modern AI tools, and real-world business problems.
• Ownership of high-impact projects across top enterprises.
• A mission-driven, fast-moving culture that values curiosity, pragmatism, and excellence.

Work model: Hybrid collaboration with 3+ days per week (Tuesday–Thursday) in-office.

Compensation: Base salary range is $150,000 to $250,000 per year, depending on experience, location, and level.

Technology focus: Python, LLM-based graders, Evaluation-Driven Development, Experiment-Driven Development, HSA, FSA, 401(k), Carrot.