Agent Observability Data Jobs

1 job found

Today

AI Engineer - Data and Evals

Glean Technologies San Francisco Bay Area, CA

San Francisco Bay Area, CA

Today

$200k - $280k/yr Full time Agent Observability Ai Evals Artificial Intelligence

Apply Now

Job Description

Glean Technologies builds AI products and agents, and this role focuses on making them more reliable and useful through disciplined measurement. As an AI Engineer - Data and Evals, you will design and run evaluation systems that connect offline benchmarks to production behavior, then translate results into engineering improvements.

This is a hybrid position in the San Francisco Bay Area, CA, with 4 days per week in the Mountain View or San Francisco office. The base salary range is USD 200,000 - 280,000 per year, with compensation determined by factors such as location, level, and relevant experience.

What you’ll do

Build evalsets and benchmarks from expert-curated examples, production data, synthetic data, and past failures, ensuring coverage of edge cases and key user slices.
Create and scale an eval platform and agent observability, including experiment pipelines, trace capture and replay, and dashboards that support reproducible, affordable runs.
Develop agentic LLM verifiers for correctness, groundedness, citation quality, and task success, then calibrate them against human labels using clear rubrics.
Run evaluations across the lifecycle, from CI and pre-release suites to sampled production traffic, and communicate outcomes with confidence intervals.
Build enterprise simulation environments using simulated users and realistic company data to test multi-step agent workflows.
Diagnose failures by slice and trace, then convert eval results into concrete fixes while balancing quality against latency and cost.

What you bring

5+ years of software engineering experience with strong Go and/or Python and backend fundamentals, including APIs, data modeling, testing, and distributed pipelines. TypeScript is a plus.
Experience creating datasets and benchmarks for LLM use cases such as RAG or agents, including evalset curation and synthetic data generation.
Familiarity with eval methodologies and tradeoffs, including deterministic metrics, LLM-as-a-judge, pointwise vs pairwise grading, human labeling, and judge calibration.
Experience building eval or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tooling such as OpenTelemetry.
Strong analytical skills: ability to work through traces and failure cases, write SQL, and use data and metrics to explain quality movement.
Nice to have: experience with simulation environments and/or model post-training techniques.

Tools you may use

Go, Python, TypeScript, SQL
OpenTelemetry, CI/CD

Compensation and benefits

Base salary range: $200,000 - $280,000 annually
Medical, Vision, and Dental coverage
Generous time-off policy
401(k) contribution opportunity
Home office improvement stipend
Annual education and wellness stipends
Healthy lunches daily
Certain roles may be eligible for variable compensation, equity, and benefits

AI-first mindset at Glean

During the interview process, you will complete a brief AI-focused exercise or discussion to help Glean understand how you think about, design, and apply AI to drive impact in your role.

View full job page →

Get Agent Observability Data Jobs alerts

New Agent Observability Data Jobs, daily.