Ml Experimentation Data Jobs

1 job found

Today

Machine Learning Engineer – Evals

Mango Global Technologies New York, NY

New York, NY

Today

$220k - $300k/yr Full time Artificial Intelligence Data Science Engineer

Apply Now

Job Description

The Machine Learning Engineer for Evals will own end-to-end evaluation for an AI memory and identity system, shaping how quality is measured and ensuring evaluation infrastructure supports rapid iteration across models, prompts, and representations.

Responsibilities

Define how to determine whether evolving entity representations are improving.
Build infrastructure needed to measure improvements and track evaluation outcomes over time.
Use evaluation results to identify failures and drive fixes through engineering or research changes.
Define what “better” means for representations that evolve across updates.
Design novel evaluation methodologies when traditional ground truth is unavailable or insufficient.
Develop scoring frameworks, metrics, rubrics, judge models, and diagnostic methodologies.
Continuously evolve evaluation definitions as models and product capabilities change.
Build production-quality Python pipelines and evaluation harnesses.
Manage datasets, labels, experiment versions, model versions, judges, reruns, and regression testing.
Create systems that help researchers and engineers answer new evaluation questions quickly.
Integrate experiment tracking and observability into evaluation workflows.
Analyze model outputs, agent trajectories, and execution traces.
Identify meaningful failure modes rather than optimizing for superficial metrics.
Run experiments, compare results, document findings, and execute the resulting rerun and recommendations.
Develop and operate simulation environments capable of modeling large numbers of entities.
Increase throughput and reliability of evaluation runs.
Build systems that enable rapid iteration across models, prompts, representations, and evaluation strategies.
Own the evaluation process end to end: research question → evaluation design → data → implementation → execution → analysis → diagnosis → rerun → recommendation.

Required Qualifications

3+ years of professional experience designing and owning LLM evaluations.
Demonstrated experience designing, building, and running LLM evaluations end to end.
Production-quality Python experience, including testing, maintainability, debugging, and release practices.
Experience creating evaluation methodologies, not only executing existing benchmarks.
Strong ability to investigate model or agent failures using traces, outputs, datasets, and experimental results.
Ability to work onsite in New York City five days per week.
Unrestricted U.S. work authorization; visa sponsorship is not available.

Technologies

Python
PyTorch
Hugging Face / Transformers
Weights & Biases
Hydra
SQL
Kafka

Preferred Technical Skills

Weights & Biases
Hydra
SQL
Kafka or other streaming or event-driven infrastructure
Evaluation and observability frameworks
Agent simulation and multi-agent systems
Large-scale data pipelines

Nice to Have

Experience with AI memory, identity, personalization, agent evaluation, or agentic optimization.
Research experience in LLMs, NLP, machine learning, evaluation, or related areas.
First-author or co-author research at NeurIPS, ICML, ICLR, ACL, EMNLP, or comparable venues.
Alternatively, a high-quality open-source project, benchmark, framework, or technical publication demonstrating comparable research ability.
Degree in Computer Science, Mathematics, Statistics, Machine Learning, or another quantitative discipline.

Benefits

Employee assistance program

Compensation

USD 220,000 - 300,000 base salary per year plus competitive equity. Final compensation depends on experience, technical depth, research contributions, and interview performance.

Hiring Process

The selected candidate should expect technical discussions focused on:

Designing an evaluation from scratch.
Diagnosing an LLM or agent failure from traces.
Building a production evaluation pipeline.
Research methodology and experimental design.
Python and ML engineering.
Previous research or open-source work.

Application Questions

Can you work full-time on-site in New York City 5 days per week, and do you currently have unrestricted U.S. work authorization without requiring visa sponsorship?
Have you either (a) published ML/LLM research at a recognized conference or journal or (b) released substantial open-source research or ML work of comparable quality?
Do you have hands-on experience with PyTorch AND Hugging Face or Transformers?
Do you have production-level Python experience, including writing tests, maintaining production code, debugging, and managing releases?
How many years of professional experience do you have designing and owning LLM evaluations?
Have you personally designed, built, and run an LLM evaluation system end to end, including defining the evaluation methodology or metrics rather than only executing an existing benchmark?
Share your LinkedIn URL.

Work Location and Salary Details

Location: New York, NY (onsite)
Salary: USD 220,000 - 300,000 per year
Minimum experience: 3 years

Not a Strong Fit

Candidates who have only run established benchmarks without designing their own evaluations.
Engineers whose LLM experience is limited to application development without evaluation ownership.
Candidates without substantial hands-on Python engineering experience.
Candidates without evidence of research thinking, novel experimentation, published work, or comparable open-source work.
Candidates unwilling or unable to work five days per week from the New York City office.

View full job page →

Get Ml Experimentation Data Jobs alerts

New Ml Experimentation Data Jobs, daily.