American Express is building evaluation practices that keep AI systems high quality, safe, reliable, and ready for enterprise deployment. In this Senior Agentic AI Engineer role (onsite in New York, NY), you will design evaluation frameworks and pipelines that help teams validate LLM, RAG, agentic, and multimodal applications across quality, safety, compliance, latency, and cost.
In addition to competitive pay and bonus incentives, American Express offers 6% company match on retirement savings, comprehensive health coverage (medical, dental, vision, life insurance, and disability), and free financial coaching with financial well-being support. You will also have access to wellness centers staffed with nurses and doctors (depending on location) and confidential counseling through the Healthy Minds program. The working model is hybrid, onsite, or virtual depending on role and business need, and eligible parents can access 20+ weeks paid parental leave for pregnancy, adoption, or surrogacy, regardless of gender. Career development and training opportunities are also included.
Responsibilities
- Design, develop, and maintain evaluation frameworks for AI and machine learning solutions, including LLM, RAG, agentic, and multimodal applications
- Translate business, product, risk, and governance requirements into measurable success criteria
- Build automated and human-in-the-loop evaluation pipelines
- Partner with cross-functional teams to help AI systems meet enterprise standards for quality, safety, reliability, compliance, latency, and cost
- Execute human-in-the-loop and model-based evaluation workflows, including calibration, error analysis, and validation of evaluation quality
- Create evaluation harnesses and automated test pipelines for LLM, RAG, or agentic applications using production-representative datasets
- Integrate evaluation results into CI/CD, release readiness, production monitoring, and governance evidence
Requirements
- 5+ years of experience in software development, including hands-on work in machine learning, AI technologies, and agentic solutions
- Bachelor’s degree in computer science, machine learning, data science, and/or comparable experience (advanced degree preferred)
- Knowledge of Python and data processing technologies such as Big Data (Hadoop, Spark)
- Knowledge of NLP techniques such as fuzzy wuzzy and models including BERT and Transformer models
- Advanced knowledge of machine learning techniques and performance optimization
- Strong understanding of AI system architectures, including LLM ecosystems, retrieval systems, orchestration frameworks, and evaluation methodologies
- Experience delivering end-to-end AI systems in production, including design and operational ownership
- Experience integrating evaluation work with CI/CD and containerized deployments
- Experience building and maintaining data pipelines and ETL (extract, transform, load)
- Experience with distributed systems design, scalable APIs, and cloud-native AI infrastructure, including monitoring and optimization
- Experience with continuous integration/continuous deployment pipelines and containerized deployments
- Experience working with LLMs, especially GPT-3.5/4 or similar AI models
- Experience with programming languages such as Python, R, Java, or C++
- Experience deploying AI models and pipelines using machine learning operations tools
- Advanced expertise in a technical domain and moderate expertise in a new, different technical domain
Technologies
Python, Big Data (Hadoop, Spark), fuzzy wuzzy, BERT, Transformer models, LLMs, GPT-3.5/4, R, Java, C++, machine learning operations tools, CI/CD, containerized deployments, ETL, scalable APIs, cloud-native AI infrastructure
Employment eligibility: Employment eligibility to work with American Express in the United States is required, and the company will not pursue visa sponsorship for these positions.
US job seekers: Click to view the “Know Your Rights” poster. If the link does not work, access it here: https://www.eeoc.gov/poster
