Llm Evaluation

Llm Evaluation Data Jobs

3 jobs found

7d ago

Machine Learning Engineer, Human Centered AI - Evaluations & Insights

Apple Seattle, WA

Seattle, WA

7d ago

$142k - $263k/yr Full time Adversarial Ai Evaluation Ai Evaluation Artificial Intelligence

Apply Now

Job Description

Apple’s Human-Centered AI team focuses on building trustworthy generative AI experiences. In this role, you will evaluate and improve foundation models by creating measurement frameworks, scalable MLOps pipelines, and automated insights that connect human perception to model performance.

Responsibilities

Architect and execute comprehensive evaluation suites for LLMs and multimodal models, identifying edge cases across multi-step reasoning, factuality, adversarial robustness, safety, and alignment.
Build deterministic, heuristic, and LLM-assisted evaluation frameworks (including LLM-as-a-judge and reward modeling) to quantify human-perceived quality metrics such as helpfulness and hallucination rates.
Convert qualitative failure modes into quantifiable loss patterns, programmatic guardrails, and actionable data-mixture adjustments for both training and inference.
Partner with engineering teams to refine model behavior using evaluation telemetry to guide prompt engineering, Retrieval-Augmented Generation (RAG) strategies, and model fine-tuning.
Apply advanced ML methods (such as embedding-based clustering, representation learning, and perturbation analysis) to map error taxonomies and latent failure manifolds in model outputs.
Develop robust MLOps workflows that codify evaluation metrics, automate regression testing across model checkpoints, and incorporate human-centric assessments into ML CI/CD pipelines.
Architect scalable, distributed inference and processing pipelines (for example, Ray and vLLM) to support high-throughput evaluation, automated annotation, and large-scale output analysis.
Define quantitative evaluation frameworks that capture nuanced human factors, including trust calibration, conversational state tracking, and interpretability.
Create automated evaluation pipelines using LLMs to assess outputs at scale, optimizing for high correlation with human baseline annotations.
Collaborate with ML researchers, software developers, and product managers across Apple to convert product requirements into scalable, reliable, and efficient model evaluation infrastructure.

Requirements

5+ years of relevant industry experience in ML Engineering or Applied Research.
Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.
Advanced proficiency in Python and modern deep learning ecosystems, including PyTorch, JAX, and Hugging Face.
Experience building scalable ML inference pipelines, model-evaluation workflows, and structured rating frameworks for large-scale AI systems.
Strong ability to interpret unstructured model outputs (text, transcripts, and embedding spaces) and translate qualitative findings into actionable engineering guidance and training objectives.
Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems.
Deep familiarity with AI quality metrics, hallucination detection techniques (including SelfCheckGPT), model alignment methods (including RLHF and DPO), and LLM-as-a-judge frameworks such as G-Eval and DeepEval.
Experience building internal tools or automated pipelines for ML workflows using tools such as MLflow, Weights & Biases, or similar platforms.
Strong familiarity with advanced prompt engineering, RAG architectures (vector databases and semantic search), and fine-tuning.
Bachelor’s or Master’s degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field.

Technologies

Python, PyTorch, JAX, Hugging Face
Ray, vLLM
MLflow, Weights & Biases
SelfCheckGPT, RLHF, DPO
G-Eval, DeepEval
LLM-as-a-judge
Retrieval-Augmented Generation (RAG), vector databases, semantic search

Benefits

Comprehensive medical and dental coverage.
Retirement benefits.
Range of discounted products and free services.
Reimbursement for certain educational expenses (including tuition) for formal education related to advancing your career.
Opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs.
Eligible for discretionary restricted stock unit awards.
Can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan.
May be eligible for discretionary bonuses or commission payments and relocation.

Role Location and Salary

This position is based in Seattle, WA and is onsite. The salary range is USD 142,300 - 263,300 per year.

Additional Notes

The role will help bridge human perception and algorithmic performance to evaluate and optimize foundation models and generative AI systems. Responsibilities include architecting evaluation frameworks, designing scalable MLOps pipelines for model assessment, and working cross-functionally with Software Engineering, Product, Research, and Responsible AI teams to support reliable, safe, and human-aligned AI experiences.

Preferred Qualifications

Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.

View full job page →

25d ago

Machine Learning Engineer, GAI Search Platform

ServiceNow Mountain View, CA

Mountain View, CA

25d ago

$140k - $217k/yr Full time Agentic Ai AI Ai Ml

Apply Now

Job Description

ServiceNow’s GenAI Search Platform team is hiring Machine Learning Engineers to advance enterprise search-based question answering. The work combines traditional machine learning with generative AI and LLMs, with a focus on building LLM-enabled search capabilities and establishing systematic evaluation for ranking and summarization grounded in heterogeneous knowledge sources.

Responsibilities

Drive development of the search platform using LLM-enabled generative AI features in a fast-paced environment, supporting product teams as they build new applications and features quickly.
Ship prototypes to customers at scale, evaluate and iterate on ranking improvements in a systematic manner, and enforce guarantees related to quality, security, privacy, and scalability.
Create systematic evaluation metrics and methodologies for search models, using LLMs to address enterprise-specific challenges such as security and privacy.
Collaborate with ranking, product, design, infrastructure, and data science teams to develop next-generation agentic search capabilities with an emphasis on performance and enterprise readiness.
Partner with product, user experience, customer success, and engineering teams to identify, define, and build solutions.

Required Qualifications

3+ years of engineering experience.
First-hand experience relevant to search platforms, including traditional information retrieval, search ranking and relevance, recommendation systems, or question answering systems.
Background in text processing, information retrieval, natural language understanding, or machine learning.
Strong interest in helping product and ranking teams move faster and scale solutions effectively while maintaining customer impact and success as priorities.

Technologies

LLMs (large language models)
Natural language understanding
Retrieval augmented generation (RAG)

Nice to Have

Experience with, or interest in learning, large language models, retrieval augmented generation (RAG), and agents.
Experience with Python, including contributing to mature Python libraries or platform code, including open-source repositories.
Experience in Golang (not required).

Compensation and Benefits

Base pay: $139,700 - $216,500 per year.
Equity (when applicable), variable or incentive compensation, and benefits.
Health plans, including flexible spending accounts.
401(k) plan with company match.
ESPP.
Matching donations.
Flexible time away plan.
Family leave programs.

Work Location and Work Persona

The role is based in Mountain View, CA and is onsite.

ServiceNow assigns work personas (flexible, remote, or required in office) based on the nature of the work and the assigned work location. To determine eligibility for a work persona, ServiceNow may confirm the distance between your primary residence and the closest ServiceNow office using a third-party service.

View full job page →

1mo ago

Lead AI Engineer

Capital One New York, NY

New York, NY

1mo ago Closed

$215k - $246k/yr Full time Ai Governance Ai Ml Ai Observability

This position is no longer accepting applications (closed Sep 4, 2026).

Job Description

Capital One is hiring a Lead AI Engineer to build responsible, reliable AI systems and LLM-enabled capabilities for production use.

Partner with cross-functional teams across engineering, research science, technical program management, and product management to deliver AI-powered products for associates and customers
Design, develop, test, deploy, and support AI software components, including:
- Foundation model training
- Large language model inference
- Similarity search
- Guardrails
- Model evaluation
- Experimentation
- Governance and observability
Apply Open Source and SaaS AI technologies including AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, and related tooling
Develop state-of-the-art LLM optimization techniques to improve production performance including scalability, cost, latency, and throughput
Help shape the technical vision and long-term roadmap for foundational AI systems at Capital One

Requirements

Bachelor’s or Master’s degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or a related field, plus:
- At least 4 years experience developing AI and ML algorithms or technologies, OR
- At least 2 years experience developing AI and ML algorithms or technologies
At least 4 years of programming experience with Python, Go, Scala, or Java

Preferred Qualifications

At least 6 years of experience deploying scalable and responsible AI solutions on cloud platforms (e.g., AWS, Google Cloud, Azure, or equivalent private cloud)
Experience designing, developing, delivering, and supporting AI services
Experience developing AI and ML algorithms or technologies (e.g., LLM inference, similarity search and VectorDBs, guardrails, memory) using Python, C++, C#, Java, or Golang
Experience developing and applying state-of-the-art techniques for optimizing training and inference software to improve hardware utilization, latency, throughput, and cost
Interest in staying current with AI research and applying novel techniques judiciously in production

Technologies

AWS Ultraclusters
Huggingface
VectorDBs
Nemo Guardrails
PyTorch
Foundation model training
Large language model inference
Similarity search
Guardrails
Model evaluation
Experimentation
Governance
Observability
Python, Go, Scala, Java
AWS, Google Cloud, Azure
C++, C#, Golang

Team

The Intelligent Foundations and Experiences (IFX) team brings Capital One’s AI vision to life
IFX partners across the company to advance science and AI engineering and deploy proprietary solutions central to the business
AI models and platforms help teams enhance products with responsible, scalable AI for high-leverage impact

Benefits

Comprehensive, competitive, and inclusive health, financial, and other benefits supporting total well-being
Performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)

Location

New York, NY (onsite)

Salary

USD 215,200 - 245,600 per year

Additional Information

Capital One may consider sponsoring a new qualified applicant for employment authorization for this position
Application window: expected to accept applications for a minimum of 5 business days
No agencies please
Capital One is an equal opportunity employer (EOE, including disability/vet) committed to non-discrimination
Capital One promotes a drug-free workplace
Capital One will consider for employment qualified applicants with a criminal history consistent with applicable laws
Accommodation: contact Capital One Recruiting at 1-800-304-9102 or RecruitingAccommodation@capitalone.com
Recruiting process support: Careers@capitalone.com
Capital One does not provide, endorse, or guarantee third-party products, services, educational tools, or information on this site
Capital One Financial includes multiple entities; postings in other countries correspond to specific entities

View full job page →

Llm Evaluation Data Jobs

Machine Learning Engineer, Human Centered AI - Evaluations & Insights

Machine Learning Engineer, GAI Search Platform

Lead AI Engineer

Get Llm Evaluation Data Jobs alerts