Model & Prompt Evaluation

Model & Prompt Evaluation Data Jobs

2 jobs found

Today

Senior Applied AI Engineer

Curai Remote 🌍 Remote

Remote

Today

$175k - $200k/yr Full time Senior Artificial Intelligence Fine Tuning

Apply Now

Job Description

Curai is seeking a Senior Applied AI Engineer to design, build, and ship machine learning and LLM systems that influence how clinicians and patients engage with its healthcare platform. This role covers the full lifecycle, from problem framing and data exploration to training, evaluation, production inference, and measurement of real-world clinical and product impact.

Key Responsibilities

Lead the technical execution of complex AI initiatives, owning solution design and delivery within a product or technical domain, while collaborating with senior engineers on broader architectural direction.
Design, build, train, evaluate, and improve advanced machine learning and LLM-based systems for patient and provider-facing products, including use cases such as conversational AI, personalization, user understanding, clinical decision support, and chronic care management.
Own end-to-end problem work: scope with clinicians and product partners, create datasets and evaluations, iterate on modeling, and ship to production with appropriate monitoring and guardrails.
Develop robust evaluation frameworks, including offline benchmarks, human-in-the-loop review, and online experiments, to support confidence in model safety, accuracy, and ongoing improvement.
Build and enhance the team’s platform to accelerate delivery, including data pipelines, training and inference infrastructure, prompt and model management, and tooling for clinical reviewers.
Partner across clinicians, product, and engineering to translate medical and operational requirements into ML work and deliver measurable improvements to patient and clinician experience.
Provide technical direction for your area, mentor other engineers, and strengthen engineering and scientific rigor, with leadership scope that increases with seniority.
Stay current with the literature and the fast-moving AI ecosystem, and bring forward the most relevant ideas for patients and the team.

Required Qualifications

Bachelor’s degree in Computer Science, Software Engineering, Math, or a related technical degree.
3+ years of hands-on engineering experience, including 1+ years building and deploying machine learning systems with generative AI (LLMs), and a clear track record of impact.
Strong software engineering fundamentals, with the ability to ship reliable, well-tested Python (or comparable language) code in production.
Practical knowledge of modern LLM methods, including prompting, retrieval-augmented generation, fine-tuning, and evaluation, plus awareness of their trade-offs.
Ability to work with messy, real-world data and design evaluations that determine whether a system is actually working.
Strong written and verbal communication skills, including cross-collaboration with clinicians, product managers, and engineers.
A bias toward action and ownership, including the ability to take an ambiguous problem to a result and bring others along.
Motivation to align work with improved health outcomes for real patients.

Technologies

Python
LLMs
Retrieval-augmented generation
Prompting
Fine-tuning
Evaluation

Preferred Qualifications

Experience applying ML or LLMs in healthcare, life sciences, or another regulated, high-stakes domain.
Experience with clinical NLP, medical knowledge representation, or working with electronic health record data.
Experience building agentic systems and tool-using LLMs in production.
Experience scaling ML infrastructure, including training pipelines, distributed inference, and evaluation platforms, for a small, fast-moving team.
Track record of technical leadership, such as setting direction across teams, mentoring engineers, or publishing influential work.

Benefits

High-ownership work on meaningful problems with a tight feedback loop from real clinicians and patients.
A small, senior team where work shows up in the product quickly.
Competitive compensation, meaningful equity, and comprehensive benefits.
Remote-first, flexible work environment across the U.S.
Comprehensive medical, dental, and vision coverage.
Flexible spending plans.
Generous and flexible Paid Time Off (PTO), floating holidays, and parental leave.
401k plan with employer matching.
100% remote, work from home.

Compensation

USD 175,000 - 200,000 per year. Actual base salary will depend on qualifications and years of relevant experience.

Location

Remote (remote), across the U.S.

View full job page →

1d ago

Senior AI Engineer

Mitratech Remote 🌍 Remote

Remote

1d ago

$170k - $190k/yr Full time Backend Developer Senior Agentic Ai

Apply Now

Job Description

Mitratech is looking for a Senior AI Engineer to design, build, and operate intelligent agent systems end-to-end on an AWS-native GenAI platform. This role spans engineering delivery and stable operations, with hands-on work across agent services, evaluation, and reliability for internal tooling, customer-facing experiences, and workflow automation. The position is remote and offers a salary range of USD 170,000 to 190,000 per year.

What you’ll build

You will own LLM-powered agents end-to-end on Amazon Bedrock, using frameworks and tooling such as LangChain and LangGraph, alongside the langchain-aws, langchain-anthropic, and langchain-core stack. Your work includes defining agent workflows and RAG architectures for team-level reuse, and ensuring that agent services remain reliable through production operations.

Responsibilities

Design and own LLM-powered agents end-to-end for internal tooling, customer-facing features, and workflow automation on Bedrock using LangChain/LangGraph and the langchain-aws / langchain-anthropic / langchain-core stack
Design agent workflows and RAG architectures for team-level use, influencing design decisions beyond your individual services
Build and operate MCP servers that expose company data and services as structured tools for AI models
Define testing, reliability, and operational practices for your areas, and own delivery and on-call quality for major services
Stand up evaluation pipelines using Langfuse and eval harnesses, implement prompt-versioning practices, and use results to improve reliability and cost/quality
Lead design reviews, mentor engineers, and drive improvements in engineering practice
Collaborate with product, data, and platform teams to scope and deliver agent-based solutions while applying platform patterns and governance hooks

Requirements

5+ years of software engineering experience, including multiple years building and operating LLM or AI agent systems in production
Expert-level Python and service development (async, FastAPI, Pydantic v2), with a history of owning services end-to-end
Deep GenAI/LLM application experience, including RAG architectures and agent workflows (ideally with Claude via Amazon Bedrock or the Anthropic API)
Strong AWS engineering skills across Bedrock, Lambda, ECS, API Gateway, and related services
Hands-on experience with LangChain/LangGraph and MCP (building and operating tool/resource servers)
Experience with Terraform / IaC and containerized delivery through CI/CD
Data-layer experience with Redis, Snowflake, DynamoDB, and Postgres/Aurora using async SQLAlchemy
Identity and application security fundamentals including JWT/OAuth and secrets management
Production observability and LLM evaluation experience, including tracing and eval metrics such as latency/cost/quality
Demonstrated ability to mentor engineers, lead design reviews, and set engineering practices

Benefits

Health, Dental & Vision Insurance
401(k) + Employer Match
Open PTO + 11 Paid Holidays + 4 Annual Paid Global Wellness Days Off
STD, LTD & Group Life Insurance
Paid Parental Leave
Pet Insurance
FSA & HSA Options
Employee Assistance Program
Remote Work
Career Advancement & Professional Development Opportunities
Employee Recognition
LinkedIn Learning Platform

Technologies you’ll work with

AWS-native GenAI platform, Amazon Bedrock, LangChain, LangGraph, langchain-aws, langchain-anthropic, langchain-core, FastAPI, Terraform, MCP, Langfuse, Python, Pydantic v2, async, Lambda, ECS, API Gateway, CI/CD, Redis, Snowflake, DynamoDB, Postgres/Aurora, async SQLAlchemy, JWT, OAuth, secrets management, Claude, Anthropic API, tracing, evals, latency/cost/quality metrics.

View full job page →

Model & Prompt Evaluation Data Jobs

Senior Applied AI Engineer

Senior AI Engineer

Get Model & Prompt Evaluation Data Jobs alerts