Model Quality Data Jobs

1 job found

Today

FurtherAI San Francisco, CA

San Francisco, CA

Today

$300k - $350k/yr Full time Senior Agent Evaluation Ai Evaluation

FurtherAI is hiring a Senior/Staff AI Engineer to lead evaluations and continuously raise the quality of AI agents across the platform.

Own evaluations across FurtherAI by building evaluation frameworks and infrastructure to measure accuracy, quality, and reliability across workflows, Assistant, document processing, and computer use.
Build and maintain high-quality datasets, including golden datasets, benchmarks, and regression suites, in collaboration with Forward Deployed Engineers and domain experts to cover real-world insurance tasks and edge cases.
Design evaluations for complex agent behavior, including multi-step trajectories, tool use, reasoning, and end-to-end task completion beyond single-response quality.
Develop automated grading systems using LLM-as-judge approaches, deterministic graders, and human evaluation workflows with rubrics aligned to real-world quality.
Make evaluation results central to engineering by integrating offline and online evaluations into development and release processes to measure improvements, catch regressions, and support confident shipping.
Drive model evaluation and selection by continuously benchmarking frontier and open-weight models for accuracy, latency, cost, and reliability, using results to guide model choices and production releases.
Build continuous improvement loops by converting production failures, customer feedback, and agent traces into new evaluation cases and measurable quality gains.

5+ years of software engineering experience with hands-on experience building production AI/ML systems.
Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.
Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.
Strong Python engineering skills to build reliable, production-grade systems.
Ability to independently own ambiguous technical problems from research and experimentation through production.
Clear communicator in person and in writing.
Bonus: Experience with LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking.
Bonus: Experience with open-weight models, fine-tuning, post-training, or model routing.
Bonus: Background in B2B SaaS and 0-to-1 experience.
Above all: drive, grit, and ownership.

Build an end-to-end evaluation framework for FurtherAI agents measuring success across multi-step workflows, tool calls, and complex insurance tasks.
Create a library of golden datasets and benchmarks for common insurance workflows, reusable across customer implementations and out-of-the-box AI skills.
Develop an automated benchmarking system that evaluates new frontier and open-weight model releases and identifies the best models for different tasks.
Implement production evaluation pipelines to detect quality regressions, identify failure patterns, and turn them into actionable improvements.

Location: San Francisco, CA (onsite)

Salary: USD 300,000 - 350,000 per year

Experience: 5+ years

New Model Quality Data Jobs, daily.