Evaluation Set Design Data Jobs

1 job found

Today

Data Scientist

Apple Sunnyvale, CA

Sunnyvale, CA

Today

$185k - $325k/yr Full time Artificial Intelligence Computer Vision + Llm Data Analysis

Apply Now

Job Description

Apple is building evaluation capabilities to measure the quality of generative imaging features in ways that engineering and product teams can trust. In this role, you will design evaluation sets, run human annotation, develop automated metrics including LLM-as-judge systems, and convert the findings into decision-ready quality signals that guide model development and release decisions.

What you will do

Own the quality evaluation lifecycle end to end, from evaluation set design through annotation, scoring, analysis, and recommendations that influence what engineering and product build next.
Establish the measurement foundation by designing and maintaining golden sets and evaluation sets, analyzing data distributions to keep results representative, and ensuring known defects remain covered.
Run human evaluation by writing and iterating annotation guidelines and label schemas, building or adapting lightweight labeling tools for image and video review, monitoring annotator quality, and resolving disagreements and ambiguity in guidelines.
Automate repeatable work by developing automated metrics and LLM-as-judge systems, calibrating them against human labels, and tracking where automated outputs diverge.
Make results decision-ready through model-vs-model comparisons with confidence intervals, significance testing, and breakdowns by hotspot, content type, and failure mode, then translate outputs into root-cause findings and prioritized recommendations.
Partner across ML, engineering, and design to define quality standards and communicate evaluation results clearly to both technical and non-technical audiences.

Minimum qualifications

Education: MS
Experience: Minimum 5 years of relevant industry experience
5+ years of experience in data science, machine learning evaluation, algorithm analysis, or related technical disciplines
Experience designing and analyzing evaluations and experiments for ML or generative model quality, including sampling design, significance testing, and confidence intervals
Strong Python skills and comfort with SQL, with the ability to build and maintain evaluation pipelines and small tools
Familiarity with AI-assisted analysis workflows, foundation models, agentic systems, or intelligent automation approaches for technical problem solving

Technologies you may work with

Python, SQL
LLM-as-judge systems, LLMs, VLM
3D Gaussian splats, diffusion models

Preferred qualifications

Experience evaluating 3D Gaussian splats or diffusion models, or assessing perceptual quality
Experience building LLM or VLM judges and validating them against human ratings
Experience building web-based labeling or review tools
Ability to run several work streams in parallel and prioritize them
Proven cross-functional collaboration and the ability to learn new tools quickly
Strong written and verbal communication, including explaining statistical tradeoffs to engineers and product partners

Location and compensation

Location: Sunnyvale, CA (onsite)
Salary: USD 184,700 - 324,800 per year

Benefits

Discretionary employee stock programs (Apple shareholder opportunity)
Discretionary restricted stock unit awards
Employee Stock Purchase Plan (purchase Apple stock at a discount, if voluntarily participating)
Comprehensive medical and dental coverage
Retirement benefits
Range of discounted products and free services
Reimbursement for certain educational expenses, including tuition
Discretionary bonuses or commission payments as well as relocation (may be eligible)
Base pay range between $184,700 and $324,800 (depends on skills, qualifications, experience, and location)

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

View full job page →

Get Evaluation Set Design Data Jobs alerts

New Evaluation Set Design Data Jobs, daily.