Build production-grade AI at Caterpillar using LLMs, RAG pipelines, and autonomous agent workflows. This onsite role in East Peoria, IL or Irving, TX focuses on creating safer, more reliable, and more deterministic AI experiences through rigorous testing, evaluation, and observability.
You will work on end-to-end system design, prompt and orchestration strategies, retrieval and tool-use reliability, and the operational metrics that keep AI behaviors stable in production. You will also collaborate with cross-functional teams to translate commercial needs into working AI capabilities.
Responsibilities
- Architect, version, and refine system-, contextual-, and dynamic prompts using advanced techniques such as Chain-of-Thought, Tree-of-Thoughts, ReAct, and Few-Shot learning.
- Reduce hallucinations, jailbreaking, and prompt injections using programmatic guardrails and adversarial prompting strategies.
- Adapt, benchmark, and fine-tune prompt structures across diverse open-source and proprietary frontier model options.
- Design and orchestrate multi-agent and multi-step reasoning workflows using LangGraph and LangChain.
- Integrate Retrieval-Augmented Generation (RAG) systems, including semantic search, chunking strategies, embedding models, and vector database queries.
- Implement function-calling and tool-use prompt patterns that help LLMs interact reliably with external APIs, databases, and legacy codebases.
- Build automated testing and evaluation suites using RAGAS and Evals to validate RAG accuracy and continuously benchmark outputs across datasets and models.
- Create regression testing suites and CI/CD pipelines to verify prompt stability, agent trajectory correctness, and tool-calling accuracy.
- Set up production monitoring and observability using Azure Application Insights and Microsoft Foundry.
- Track and optimize runtime metrics such as semantic drift, token usage, cost efficiency, and end-to-end latency.
- Collaborate with Product Managers, Data Scientists, and Software Engineers to deliver deterministic AI capabilities aligned to commercial requirements.
- Keep pace with weekly LLM advancements, evaluation frameworks, and orchestration libraries to maintain competitive technical momentum.
Requirements
- Expert mastery of structured prompt formats (including XML and JSON wrappers), system-level steering, and context-window optimization.
- High proficiency in Python or TypeScript for backend AI logic, asynchronous data streams, and API integrations.
- Hands-on experience building applications using LangChain, LangGraph, and native SDKs.
- Strong experience with vector databases such as Azure AI Search and AWS Open Search, including hybrid retrieval approaches.
- Solid capabilities in data manipulation (Pandas, NumPy), data cleaning, token handling, and foundational NLP transformer understanding.
- Prior implementation experience with Application Insights, OpenLLMetry, or custom evaluation and guardrail layers.
- Knowledge of deep learning architectures, fine-tuning frameworks such as QLoRA and PEFT, and platforms including Hugging Face, PyTorch, or TensorFlow.
- Experience deploying containerized AI microservices with Docker, Kubernetes, AWS, or GCP.
- Bachelor’s or master’s degree in computer science, data science, artificial intelligence, computational linguistics, mathematics, or a highly technical discipline.
- 2+ years of dedicated professional experience developing production software integrated with GenAI/LLM pipelines.
- Portfolio showing successfully deployed RAG systems or operational AI agents in commercial environments.
Technologies
LLMs, Chain-of-Thought, Tree-of-Thoughts, ReAct, Few-Shot learning, LangGraph, LangChain, Retrieval-Augmented Generation (RAG), RAGAS, Evals, Azure Application Insights, Microsoft Foundry, Python, TypeScript, XML, JSON, Pandas, NumPy, Azure AI Search, AWS Open Search, OpenLLMetry, QLoRA, PEFT, Hugging Face, PyTorch, TensorFlow, Docker, Kubernetes, AWS, GCP, CI/CD
Benefits
- Medical, dental, and vision benefits
- Paid time off plan (Vacation, Holidays, Volunteer, etc.)
- 401(k) savings plans
- Health Savings Account (HSA)
- Flexible Spending Accounts (FSAs)
- Health Lifestyle Programs
- Employee Assistance Program
- Voluntary Benefits and Employee Discounts
- Career Development
- Incentive bonus
- Disability benefits
- Life Insurance
- Parental leave
- Adoption benefits
- Tuition Reimbursement
These benefits also apply to part-time employees.
Location: East Peoria, IL or Irving, TX (onsite). Onsite schedule: five days a week. Visa Sponsorship: not available.
Posting dates: October 27, 2026 - November 8, 2026. Salary range: USD 97,530 - 158,480 per year. Summary pay range: $97,530.00 - $158,480.00.
