Llm As A Judge Data Jobs

1 job found

Today

AI Engineer / Generative AI Engineer

TMS LLC Los Angeles, CA

Los Angeles, CA

Today

Contract Agentic Ai Ai Agent Ai Evals

Apply Now

Job Description

TMS LLC is seeking an AI Engineer / Generative AI Engineer to help build production-grade AI agents and bring them through deployment, monitoring, and continuous improvement. This contract role is based in Los Angeles, CA with a hybrid work model, and focuses on agent development, evaluation, and observability in real environments.

The work centers on developing and operating AI agents using Python and the Claude Agent SDK, deploying via AWS Bedrock AgentCore, and strengthening quality through golden datasets, LLM-as-a-Judge evaluations, and CI/CD quality gates.

What you will do

Develop AI agents using Python and Claude Agent SDK.
Deploy agents using AWS Bedrock AgentCore Runtime.
Design and maintain golden datasets for reliable evaluation coverage.
Implement LLM-as-a-Judge evaluations, including calibration against human labels.
Integrate automated evaluation workflows into CI/CD pipelines as quality gates.
Set up Langfuse and/or OpenTelemetry for AI observability, tracing, and monitoring.
Evaluate agent outcomes including accuracy, task completion, tool execution, and performance.
Support RAG pipelines and AWS data processing workflows.

Required skills

Experience in Python and agentic AI development.
Hands-on work with AWS Bedrock AgentCore Evaluations (including Bedrock evaluation).
Ability to create, curate, and use golden datasets.
Experience designing LLM-as-a-Judge systems and calibrating them against human labels.
Automated evaluation implementation and integration into CI gates.
Experience with Langfuse and/or OpenTelemetry.

Technology stack

Python, Claude Agent SDK
AWS Bedrock AgentCore, AWS Bedrock AgentCore Runtime
LLM-as-a-Judge, CI/CD
Langfuse, OpenTelemetry
RAG, AWS data processing workflows
Bedrock evaluation

Nice to have

RAGAS or equivalent evaluation frameworks.
Trajectory evaluation and agentic red-teaming.
Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.

Qualifications

15+ years of experience (minimum).
6–8+ years of software engineering, cloud engineering, or AI/ML experience.
Hands-on experience with production AI agents and continuous evaluation.
Strong problem-solving and troubleshooting skills.

View full job page →

Get Llm As A Judge Data Jobs alerts

New Llm As A Judge Data Jobs, daily.