Ai Testing And Evaluation

Ai Testing And Evaluation Data Jobs

2 jobs found

Today

Evaluation & Insights Machine Learning Engineer

Apple Cupertino, CA

Cupertino, CA

Today

$185k - $325k/yr Full time Ai Evaluation Ai Testing And Evaluation Artificial Intelligence

Apply Now

Job Description

As part of Apple’s Human-Centered AI team, you will evaluate and improve AI systems through data science, model behavior analysis, and qualitative insights.

Responsibilities

Architect and run comprehensive evaluation suites for LLMs and multimodal models, surfacing edge cases across multi-step reasoning, factuality, adversarial robustness, safety, and alignment.
Build deterministic, heuristic, and LLM-assisted evaluation frameworks (for example, LLM-as-a-judge and reward modeling) to measure human-perceived quality metrics such as helpfulness and hallucination rates.
Convert qualitative failure modes into quantified loss patterns, programmatic guardrails, and actionable data-mixture adjustments for model training and inference.
Work with engineering teams to refine model behavior using evaluation telemetry to guide prompt engineering, Retrieval-Augmented Generation (RAG) strategy, and model fine-tuning.
Apply advanced ML methods (embedding-based clustering, representation learning, perturbation analysis) to map error taxonomies and latent failure patterns.
Implement MLOps workflows to formalize evaluation metrics, automate regression testing across model checkpoints, and connect human-centric assessments to ML CI/CD pipelines.
Design scalable, distributed inference and processing pipelines (for example, Ray and vLLM) to support high-throughput evaluation, automated annotation, and large-scale output analysis.
Define quantitative evaluation frameworks that capture human factors such as trust calibration, conversational state tracking, and interpretability.
Build automated evaluation pipelines using LLMs to score outputs at scale, optimizing for high correlation with human baseline annotations.
Collaborate with ML researchers, software developers, and product managers to translate product requirements into reliable and efficient evaluation infrastructure.

Requirements

Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design.
Bachelor’s or Master’s degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field.
8+ years of relevant industry experience in ML Engineering or Applied Research.
Advanced proficiency in Python and modern deep learning ecosystems (PyTorch, JAX, Hugging Face).
Proven experience building scalable ML inference pipelines, model-evaluation workflows, and structured rating frameworks for large-scale AI systems.
Strong ability to interpret unstructured model outputs (text, transcripts, embedding spaces) and synthesize qualitative findings into actionable engineering guidance and training objectives.
Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems.
Deep familiarity with AI quality metrics, hallucination detection techniques (including SelfCheckGPT), model alignment (including RLHF and DPO), and LLM-as-a-judge frameworks (including G-Eval and DeepEval).
Experience building internal tools or automated pipelines for ML workflows using platforms such as MLflow and Weights & Biases or similar.
Strong familiarity with advanced prompt engineering, RAG architectures (vector databases, semantic search), and fine-tuning.

Technologies

Python, PyTorch, JAX, Hugging Face
LLM-as-a-judge, reward modeling
Retrieval-Augmented Generation (RAG), vector databases, semantic search
Embedding-based clustering, representation learning, perturbation analysis
MLOps, CI/CD pipelines
Ray, vLLM
Trust calibration, SelfCheckGPT
RLHF, DPO
G-Eval, DeepEval
MLflow, Weights & Biases
Fine-Tuning

Benefits

Comprehensive medical and dental coverage
Retirement benefits
Range of discounted products and free services
Reimbursement for certain educational expenses, including tuition
Discretionary restricted stock unit awards
Opportunity to purchase Apple stock at a discount through the Employee Stock Purchase Plan
Base pay range between $184,700 and $324,800
Comprehensive total compensation package may include discretionary bonuses or commission payments as well as relocation

Pay & Benefits

Base pay range: $184,700 - $324,800 (depends on skills, qualifications, experience, and location).
Employees may participate in discretionary employee stock programs and purchase stock at a discount via the Employee Stock Purchase Plan.
Eligibility for benefits, compensation, and stock programs is subject to plan or program terms and requirements.

Location

Cupertino, CA (onsite)

Experience Level

8+ years

View full job page →

8d ago

Data Scientist (ARTIFICIAL INTELLIGENCE/MACHINE LEARNING)

US Defense Logistics Agency Battle Creek, MI

Battle Creek, MI

8d ago

$126k - $187k/yr Full time Ai Ethics Ai Ml Ai Testing And Evaluation

Apply Now

Job Description

The US Defense Logistics Agency (DLA) is hiring an AI Testing and Evaluation Data Scientist to apply data science expertise in support of testing and evaluation of artificial intelligence systems. This Streamlined Direct Hiring Authority notice is open to the public and places candidates at one of several listed locations.

Role Overview

In this onsite role located in Battle Creek, MI, the selected candidate will serve as an AI Testing and Evaluation Data Scientist, providing subject matter expertise to test and evaluate AI systems. The position supports AI testing and evaluation activities within Department of Defense parameters and includes execution of data science disciplines for complex studies in machine learning and deep learning.

Key Responsibilities

Serve as an AI Testing and Evaluation Data Scientist providing subject matter AI expertise and execution to test and evaluate AI systems.
Provide professional and scientific expertise applying data science disciplines for complex studies involving machine learning and deep learning algorithms, statistical analysis, data visualizations, programming, and computer science.
Evaluate and measure AI systems in alignment with data science lifecycle practices within DoD parameters.
Ensure AI systems are developed in accordance with applicable legal frameworks, including GDPR, and NIST.
Incorporate OMB test and evaluation requirements and DoD AI ethical principles into the DLA AI test and evaluation framework to produce an AI test strategy for modeling cases.
Coordinate integration of the DLA AI testing and evaluation program across the product lifecycle, from concept and design inception through development and integration to support effective AI implementation.

Specialized Experience (What Counts)

Conduct large, agency-wide research and development reviews of metrics, measurements, and evaluation methods for emerging and existing AI areas.
Use data science expertise to develop algorithms and tools for data manipulation and processing, including data visualization techniques to articulate high-risk findings.
Support design of AI systems for auditability to manage AI risk assessment policies and principles guiding automated decisions supporting DLA business operations.
Provide expert advice to senior leadership and AI stakeholders regarding adoption of new or revised policy and implementation plans resulting from AI test and evaluation integration.
Assess data quality and establish standards to validate quality criteria so data meets requirements for AI testing and evaluation.

Required Qualifications

U.S. citizenship is required.
Tour of Duty: Set Schedule.
Security Requirements: Non-Critical Sensitive with Secret Access.
Appointment is subject to completion of a favorable suitability or fitness determination; unfavorably adjudicated background checks may result in removal where reciprocity cannot be applied.
FLSA: Exempt.
Selective Service Requirement: Males born after 12-31-59 must be registered or exempt from Selective Service.
Recruitment Incentives: Not Authorized.
Bargaining Unit Status: No.
Pre-Employment Physical: Not Required.
Selectees must have a REAL ID or other acceptable identification documents to access certain federal facilities.
This announcement and future selections from it may be used to fill future vacancies for various shifts located anywhere within DLA Information Operations, J-6.
Degree: Mathematics, statistics, computer science, data science, or a field directly related. The degree must be in an appropriate major field of study at least at the baccalaureate level.
Experience: One year of specialized experience directly in or related to the duties of the position, demonstrating the particular competencies required to perform the role.

Technologies and Methods

Machine learning
Deep learning algorithms
Statistical analysis
Data visualization techniques
Programming
Computer science

Compensation and Location

Salary: USD 125,776 - 187,093 per year.

Location: Battle Creek, MI (onsite).

Education and Documentation Requirements

To qualify using education, transcripts or other documentation must be provided.
For graduate education, one academic year is based on the number of credits hours determined by the graduate school to represent one academic year of full-time study.
For foreign education, credentials must be evaluated by a private organization specializing in interpretation of foreign education programs, or full credit must be given for courses at a U.S. accredited college or university.

Benefits

A career with the U.S. government provides employees with a comprehensive benefits package.
Eligibility depends on the type of position held and whether the position is full-time, part-time, or intermittent.

How This Job Is Open

The public
U.S. Citizens, Nationals, or those who owe allegiance to the U.S.

Additional Information

DLA will provide Section 508 conformant ICT to employees with disabilities, as required by 5 C.F.R. § 302.106.
All ICT interfaces with the public related to vacancy announcements and career transition opportunities must conform to applicable ICT Standards and Guidelines (36 C.F.R. § 1194).
Reemployed Annuitants are not eligible for appointment to this position.
All DLA employees must refrain from illegal drug use on and off duty; DLA conducts pre-employment, reasonable suspicion, post-accident, and random drug testing.
The job will close when the agency has received 100 applications, which may occur sooner than the closing date.

Evaluation and Assessments

Candidates will be evaluated based on how well they meet the qualifications above. The listed assessments include:

FSA Behavioral Index: Accountability; Customer Service; Decisiveness; Flexibility; Interpersonal Skills; Problem Solving; Resilience
FSA Situational Inventory: Accountability; Interpersonal Skills; Problem Solving
Occupational Interaction Assessment: Attention to Detail; Customer Service; Flexibility; Influencing/negotiating; Integrity/Honesty; Interpersonal Skills; Learning; Self-Management; Stress Tolerance; Teamwork
Occupational Judgment Assessment: Decision Making
Occupational Reasoning Assessment: Reasoning
Direct Hire Evaluation: After the application process, review is conducted to ensure job requirements are met.

Required Documents to Apply

Resume: must not exceed two (2) pages.
Resume must include the work schedule, hours worked per week, dates of employment, and duties performed.
If multiple resumes are submitted, only the last resume submitted will be reviewed for qualifications and considered for selection.
Resume file size must be no more than 5MB and uploaded as a PDF to maintain formatting and page count.
PDF portfolios are not accepted.
Applicable documents supporting eligibility for the position.
If requesting reasonable accommodation for USA Hire Competency Based Assessments, submit required documentation including the Reasonable Accommodation Request Form.
Failure to provide all required information may result in an ineligible rating or affect overall rating.

Open Hiring Authority Notice

This position is listed under a Streamlined Direct Hiring Authority notice and is open to the public.

View full job page →

Ai Testing And Evaluation Data Jobs

Evaluation & Insights Machine Learning Engineer

Data Scientist (ARTIFICIAL INTELLIGENCE/MACHINE LEARNING)

Get Ai Testing And Evaluation Data Jobs alerts