Llm Evaluation

Llm Evaluation Data Jobs

2 jobs found

1d ago

Machine Learning Engineer, GAI Search Platform

ServiceNow Mountain View, CA

Mountain View, CA

1d ago

$140k - $217k/yr Full time Agentic Ai AI Ai Ml

Apply Now

Job Description

ServiceNow’s GenAI Search Platform team is hiring Machine Learning Engineers to advance enterprise search-based question answering. The work combines traditional machine learning with generative AI and LLMs, with a focus on building LLM-enabled search capabilities and establishing systematic evaluation for ranking and summarization grounded in heterogeneous knowledge sources.

Responsibilities

Drive development of the search platform using LLM-enabled generative AI features in a fast-paced environment, supporting product teams as they build new applications and features quickly.
Ship prototypes to customers at scale, evaluate and iterate on ranking improvements in a systematic manner, and enforce guarantees related to quality, security, privacy, and scalability.
Create systematic evaluation metrics and methodologies for search models, using LLMs to address enterprise-specific challenges such as security and privacy.
Collaborate with ranking, product, design, infrastructure, and data science teams to develop next-generation agentic search capabilities with an emphasis on performance and enterprise readiness.
Partner with product, user experience, customer success, and engineering teams to identify, define, and build solutions.

Required Qualifications

3+ years of engineering experience.
First-hand experience relevant to search platforms, including traditional information retrieval, search ranking and relevance, recommendation systems, or question answering systems.
Background in text processing, information retrieval, natural language understanding, or machine learning.
Strong interest in helping product and ranking teams move faster and scale solutions effectively while maintaining customer impact and success as priorities.

Technologies

LLMs (large language models)
Natural language understanding
Retrieval augmented generation (RAG)

Nice to Have

Experience with, or interest in learning, large language models, retrieval augmented generation (RAG), and agents.
Experience with Python, including contributing to mature Python libraries or platform code, including open-source repositories.
Experience in Golang (not required).

Compensation and Benefits

Base pay: $139,700 - $216,500 per year.
Equity (when applicable), variable or incentive compensation, and benefits.
Health plans, including flexible spending accounts.
401(k) plan with company match.
ESPP.
Matching donations.
Flexible time away plan.
Family leave programs.

Work Location and Work Persona

The role is based in Mountain View, CA and is onsite.

ServiceNow assigns work personas (flexible, remote, or required in office) based on the nature of the work and the assigned work location. To determine eligibility for a work persona, ServiceNow may confirm the distance between your primary residence and the closest ServiceNow office using a third-party service.

View full job page →

8d ago

Lead AI Engineer

Capital One New York, NY

New York, NY

8d ago

$215k - $246k/yr Full time Ai Governance Ai Ml Ai Observability

Apply Now

Job Description

Capital One is hiring a Lead AI Engineer to build responsible, reliable AI systems and LLM-enabled capabilities for production use.

Partner with cross-functional teams across engineering, research science, technical program management, and product management to deliver AI-powered products for associates and customers
Design, develop, test, deploy, and support AI software components, including:
- Foundation model training
- Large language model inference
- Similarity search
- Guardrails
- Model evaluation
- Experimentation
- Governance and observability
Apply Open Source and SaaS AI technologies including AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, and related tooling
Develop state-of-the-art LLM optimization techniques to improve production performance including scalability, cost, latency, and throughput
Help shape the technical vision and long-term roadmap for foundational AI systems at Capital One

Requirements

Bachelor’s or Master’s degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or a related field, plus:
- At least 4 years experience developing AI and ML algorithms or technologies, OR
- At least 2 years experience developing AI and ML algorithms or technologies
At least 4 years of programming experience with Python, Go, Scala, or Java

Preferred Qualifications

At least 6 years of experience deploying scalable and responsible AI solutions on cloud platforms (e.g., AWS, Google Cloud, Azure, or equivalent private cloud)
Experience designing, developing, delivering, and supporting AI services
Experience developing AI and ML algorithms or technologies (e.g., LLM inference, similarity search and VectorDBs, guardrails, memory) using Python, C++, C#, Java, or Golang
Experience developing and applying state-of-the-art techniques for optimizing training and inference software to improve hardware utilization, latency, throughput, and cost
Interest in staying current with AI research and applying novel techniques judiciously in production

Technologies

AWS Ultraclusters
Huggingface
VectorDBs
Nemo Guardrails
PyTorch
Foundation model training
Large language model inference
Similarity search
Guardrails
Model evaluation
Experimentation
Governance
Observability
Python, Go, Scala, Java
AWS, Google Cloud, Azure
C++, C#, Golang

Team

The Intelligent Foundations and Experiences (IFX) team brings Capital One’s AI vision to life
IFX partners across the company to advance science and AI engineering and deploy proprietary solutions central to the business
AI models and platforms help teams enhance products with responsible, scalable AI for high-leverage impact

Benefits

Comprehensive, competitive, and inclusive health, financial, and other benefits supporting total well-being
Performance based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)

Location

New York, NY (onsite)

Salary

USD 215,200 - 245,600 per year

Additional Information

Capital One may consider sponsoring a new qualified applicant for employment authorization for this position
Application window: expected to accept applications for a minimum of 5 business days
No agencies please
Capital One is an equal opportunity employer (EOE, including disability/vet) committed to non-discrimination
Capital One promotes a drug-free workplace
Capital One will consider for employment qualified applicants with a criminal history consistent with applicable laws
Accommodation: contact Capital One Recruiting at 1-800-304-9102 or RecruitingAccommodation@capitalone.com
Recruiting process support: Careers@capitalone.com
Capital One does not provide, endorse, or guarantee third-party products, services, educational tools, or information on this site
Capital One Financial includes multiple entities; postings in other countries correspond to specific entities

View full job page →

Llm Evaluation Data Jobs

Machine Learning Engineer, GAI Search Platform

Lead AI Engineer

Get Llm Evaluation Data Jobs alerts