Agent Evaluation Data Jobs

1 job found

Today

Member of Technical Staff, AI Engineer

Parameter ai San Francisco, CA

San Francisco, CA

Today

$180k - $240k/yr Full time Backend Developer Agent Evaluation Agentic Ai

Apply Now

Job Description

Parameter ai is hiring a Member of Technical Staff, AI Engineer in San Francisco to build LangGraph/Temporal-based agent systems for autonomous offensive security testing.

Responsibilities

Design and ship agent workflows in LangGraph and Temporal to plan, execute, and verify multi-step exploitation chains
Build the agent tool layer, including HTTP clients, authentication handling, browser control, cloud enumeration, and payload generation
Own the evaluation loop by defining what “good” means for agent runs using Braintrust and Langfuse, with the goal of improving outcomes over time
Work to reduce false positives, identified as the hardest problem in this space and a top customer priority
Manage inference cost and latency at scale, including prompt caching, context strategy, and model selection across Claude and AWS Bedrock

Requirements

3+ years of software engineering experience
At least one year building production LLM systems (not prototypes or demos)
Strong TypeScript
Comfort with async development, distributed systems, and long-running workflow orchestration
Intuition for context engineering, tool design, and why agents fail in real environments
Data-driven evaluation: built or maintained an eval harness, including understanding why offline metrics can diverge from production behavior
Bonus: security background, CTF experience, bug bounty history, or a track record of breaking things you were not supposed to break

Technologies

TypeScript, LangGraph, Temporal
GCP
AWS Bedrock, Anthropic API
Langfuse, Braintrust
GitHub Actions, Linear, Graphite

Interview Process

Intro call with a founder (30 minutes)
Technical deep dive on your past work (60 minutes)
Paid work trial or take-home, scoped to roughly one day
Onsite in San Francisco with the team
References and offer

Location and Work Authorization

Onsite in San Francisco, CA
US work authorization required; visa sponsorship is not available

Compensation: USD 180,000 - 240,000 per year

View full job page →

Get Agent Evaluation Data Jobs alerts

New Agent Evaluation Data Jobs, daily.