Generative Ai Security Evaluation

Generative Ai Security Evaluation Data Jobs

2 jobs found

Today

Machine Learning Engineer

GitHub, Inc. Remote 📍 United States Remote

Remote

Today

$108k - $286k/yr Full time Agentic Ai Ai Enabled Security Ai Security

Apply Now

Job Description

GitHub, Inc. is hiring an experienced Machine Learning Engineer to design, build, and deploy agentic LLM-based solutions that help detect and prevent fraud, abuse, and security threats across GitHub at scale. The role also involves identifying vulnerabilities tied to abuse, performing ad-hoc analysis, and assessing the impact of safety and integrity initiatives.

Responsibilities

Design, build, and deploy agentic solutions using large language models for large-scale detection and prevention of fraud, abuse, and security threats, including content classification and multi-step agentic investigations.
Build production-grade systems that operate reliably against high-volume event streams, leveraging AI coding assistants to accelerate and improve development.
Develop and operate scalable machine learning systems on cloud platforms, including Azure AI Foundry, to train, deploy, and serve models and agentic solutions in production.
Evaluate and improve existing models and agentic solutions through offline evaluations, including tool-use loops and LLM-as-judge evaluation, along with performance metrics and feedback from operational deployments.
Identify vulnerabilities in products that can lead to abuse and consult with product teams during reviews of new features.
Collaborate with cross-functional teams, including data scientists, software engineers, product managers, and content moderators, to integrate agentic solutions into production systems.
Document the systems you help build and support the technical growth of peers.

Requirements

4+ years of experience in machine learning or a related field; or a Bachelor’s degree in Computer Science, Software Development, Electrical or Computer Engineering, Mathematical Sciences, or a related field plus 2+ years of machine learning experience; or a Master’s degree in Machine Learning or related fields; or equivalent experience.
Strong understanding of large language models, including hands-on experience applying them at scale, ideally for classification and agentic workflows or agents.
Strong software engineering skills, including experience building with AI coding assistants.
Experience designing or evaluating agentic systems, such as tool-use loops, multi-step workflows, or LLM-as-judge evaluation.
Hands-on experience building and operating classification or detection systems at scale, including work with imbalanced data and precision/recall tradeoffs.
Experience in Trust and Safety, National Security, or fighting spam, malware, fraud, and threat actor activity at scale.
Experience in responsible AI and Safety-by-Design.
Experience handling user data and privacy.
Solid understanding of machine learning algorithms (supervised and unsupervised learning, anomaly detection, etc.) and practical implementation.

Technologies

Large language models (LLMs)
Azure AI Foundry
Tool-use loops
LLM-as-judge evaluation
AI coding assistants
Machine learning
Precision/recall tradeoffs

What We Value

Collaboration
Empathy
Quality
Positive Impact
Shipping

Compensation

The base salary range for this role is USD $107,700.00 to USD $285,900.00 per year.

Locations

Remote (United States)

Leadership Principles

GitHub values: Customer-obsessed, Ship to learn, Growth mindset, Own the outcome, Better together, Diverse and inclusive
Manager fundamentals: Model, Coach, Care
Leadership principles: Create clarity, Generate energy, Deliver success

View full job page →

4d ago

Lead Generative AI Engineer, VP

Citi Jacksonville, FL

Jacksonville, FL

4d ago

$126k - $189k/yr Full time Backend Developer Manager Agentic Ai

Apply Now

Job Description

Citi is hiring a Lead Generative AI Engineer (VP) to design, develop, and deploy generative AI solutions for intelligent operations and automation.

Responsibilities

Lead hands-on development and delivery of intelligent operations capabilities, including advanced chatbots, agentic workflow automation, and AI-assisted decision platforms
Fine-tune, prompt-engineer, and deploy open-source LLMs (including Llama, Mistral, Gemma, etc.) and proprietary models using Google Vertex AI
Implement RAG (Retrieval-Augmented Generation) pipelines
Build and operationalize AI safety frameworks using guardrails (for example, NeMo Guardrails and Llama Guard) plus deterministic controls to reduce hallucinations, support data privacy, and maintain regulatory compliance
Design and deploy scalable AI pipelines using Python and cloud-native architectures with containerization (Docker, Kubernetes)
Apply financial services technology expertise to align AI model usage with risk management, auditability, and financial data security requirements

Requirements

Deep understanding of LLM architectures, vector databases (Pinecone, Milvus, Chroma), embedding models, and AI orchestration frameworks (Google ADK, LangChain, etc.)
Hands-on experience with Google Vertex AI (or equivalent cloud AI suites) and deploying open-source models such as Llama, Mistral, and Gemma
Proven production experience implementing deterministic controls, semantic routing, guardrails, and bias and hallucination detection
Expert-level Python skills, including AI/ML libraries: PyTorch, Hugging Face, Pandas, NumPy; plus API development with FastAPI and Flask
Strong containerization and delivery experience with OpenShift, Docker, Kubernetes, and CI/CD pipelines
Experience with financial services technology, including financial products, compliance requirements, and the competitive AI landscape in fintech
Demonstrated ability to lead AI and automation efforts from concept through production, establishing credible senior-level technical direction

Technologies

Google Vertex AI
Python, Docker, Kubernetes, OpenShift
NeMo Guardrails, Llama Guard
LLMs: Llama, Mistral, Gemma
RAG (Retrieval-Augmented Generation)
Pinecone, Milvus, Chroma
Google ADK, LangChain
PyTorch, Hugging Face, Pandas, NumPy
FastAPI, Flask
CI/CD pipelines, containerization

Beneficial Skills & Qualifications

Experience deploying and tuning models on Google Vertex AI or equivalent enterprise-grade cloud AI platforms
Familiarity with additional AI orchestration tools and emerging open-source LLM frameworks beyond core requirements
Bachelor’s degree or equivalent experience; Master’s degree in Computer Science, Artificial Intelligence, or a related technical discipline preferred

Benefits

Hybrid working model: 3 days in the office and 2 days working remotely
Continuous learning and development opportunities
Competitive financial wellbeing and benefits package
Medical, dental & vision coverage
401(k)
Life, accident, and disability insurance
Wellness programs
Paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays
Discretionary and formulaic incentive and retention awards (for eligible employees)

Job Details

Location: Jacksonville, FL (hybrid)
Employment type: Full time
Job family group: Technology
Job family: Applications Development
Salary range: USD 125,760 - 188,640 per year
Primary location (listed): Irving Texas United States
Anticipated posting close date: Aug 15, 2026

Automated Processing

We use automated processing, including artificial intelligence, for legitimate business interests to identify and align candidate skills and abilities to the job opening. This automated processing does not involve relying on automatic or autonomous decision-making.

Additional Notes

Other relevant skills: Artificial Intelligence (AI), Containerization, Generative AI, Large Language Models (LLMs), Python (Programming Language)

View full job page →

Generative Ai Security Evaluation Data Jobs

Machine Learning Engineer

Lead Generative AI Engineer, VP

Get Generative Ai Security Evaluation Data Jobs alerts