AI Engineer / Generative AI Engineer

TMS LLC is seeking an AI Engineer / Generative AI Engineer to help build production-grade AI agents and bring them through deployment, monitoring, and continuous improvement. This contract role is based in Los Angeles, CA with a hybrid work model, and focuses on agent development, evaluation, and observability in real environments.

The work centers on developing and operating AI agents using Python and the Claude Agent SDK, deploying via AWS Bedrock AgentCore, and strengthening quality through golden datasets, LLM-as-a-Judge evaluations, and CI/CD quality gates.

What you will do

• Develop AI agents using Python and Claude Agent SDK.
• Deploy agents using AWS Bedrock AgentCore Runtime.
• Design and maintain golden datasets for reliable evaluation coverage.
• Implement LLM-as-a-Judge evaluations, including calibration against human labels.
• Integrate automated evaluation workflows into CI/CD pipelines as quality gates.
• Set up Langfuse and/or OpenTelemetry for AI observability, tracing, and monitoring.
• Evaluate agent outcomes including accuracy, task completion, tool execution, and performance.
• Support RAG pipelines and AWS data processing workflows.

Required skills

• Experience in Python and agentic AI development.
• Hands-on work with AWS Bedrock AgentCore Evaluations (including Bedrock evaluation).
• Ability to create, curate, and use golden datasets.
• Experience designing LLM-as-a-Judge systems and calibrating them against human labels.
• Automated evaluation implementation and integration into CI gates.
• Experience with Langfuse and/or OpenTelemetry.

Technology stack

• Python, Claude Agent SDK
• AWS Bedrock AgentCore, AWS Bedrock AgentCore Runtime
• LLM-as-a-Judge, CI/CD
• Langfuse, OpenTelemetry
• RAG, AWS data processing workflows
• Bedrock evaluation

Nice to have

• RAGAS or equivalent evaluation frameworks.
• Trajectory evaluation and agentic red-teaming.
• Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.

Qualifications

• 15+ years of experience (minimum).
• 6–8+ years of software engineering, cloud engineering, or AI/ML experience.
• Hands-on experience with production AI agents and continuous evaluation.
• Strong problem-solving and troubleshooting skills.