AI Engineer

Join Vytalize Health as an AI Engineer in Kansas onsite to design, build, and maintain agentic systems and LLM powered healthcare applications that automate workflows and deliver data driven clinical solutions. This role emphasizes validation, reliability, and regulatory compliance while enabling collaboration across data, platform, product, and clinical teams to scale AI driven automation across care delivery.

Responsibilities

• Create and sustain agent based systems and LLM powered healthcare apps that streamline workflows, data pipelines, and clinical decision support from concept to production rollout.
• Develop and orchestrate agents with LLM APIs and agentic frameworks (such as LangChain, LangGraph, CrewAI, or custom orchestration) to tackle complex, multi step healthcare challenges.
• Build prompt libraries, agent instructions, and reusable skills to improve accuracy, consistency, and reliability across varied use cases and data domains.
• Implement validation and confidence scoring to flag low confidence decisions for human review before deployment; establish guardrails and review workflows for agent generated code and outputs.
• Lead end to end delivery of AI automated systems, from problem scoping and requirements through development, testing, and validated production rollout.
• Establish rigorous evaluation and QA frameworks for agentic systems, including golden datasets, test cases, output validation, hallucination detection, and regression testing.
• Define and monitor evaluation metrics for agent performance, reliability, and clinical appropriateness, tracking accuracy, hallucination rates, clinical validity, and real world impact.
• Set up observability, evaluation, and regression testing frameworks specific to agentic systems, including decision tracing, lineage logging, and performance tracking.
• Collaborate with data engineering and platform teams to integrate agent built outputs (dbt models, transformation logic, recommendations) into existing data architectures and clinical workflows.
• Ensure all agentic systems comply with healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices, emphasizing explainability, auditability, and clinician trust.
• Continuously evaluate new LLM models, agent frameworks, prompt engineering techniques, and tooling; recommend adoption or migration based on healthcare specific requirements such as accuracy, cost, latency, and regulatory alignment.
• Partner with data engineering to establish robust data validation and input validation layers for agents—agents are only as good as the data they operate on.
• Lead experimentation and measurement of AI automated systems impact on speed, quality, compliance, and cost across healthcare workflows.
• Document agent architectures, prompt strategies, evaluation frameworks, and best practices for both technical and non technical stakeholders.
• Mentor AI Connector Engineers and other team members on agentic development patterns, LLM powered application design, and responsible AI practices.
• Provide on call support to production agentic systems as needed, troubleshooting issues and responding to performance degradation or hallucination detection.

Requirements

• Minimum 3 years of professional experience in data engineering, backend engineering, machine learning, or a related field.
• At least 1 year of hands on experience building with LLM APIs and agentic orchestration frameworks, not just using AI coding assistants but architecting agentic systems.
• Strong Python and SQL proficiency.
• Experience with cloud data platforms such as AWS and Databricks.
• Solid understanding of data modeling, ETL/ELT patterns, and medallion architecture (Bronze/Silver/Gold).
• Experience building and consuming APIs.
• Demonstrated experience with prompt engineering, agent evaluation, and validating LLM outputs.
• Experience designing evaluation frameworks, test cases, and quality assurance for AI/ML systems.
• Ability to measure and track AI system performance using metrics and KPIs such as accuracy, precision, recall, and hallucination rates.
• Strong debugging and analytical skills, especially in ambiguous or novel technical territory.
• Excellent written and verbal communication skills, with the ability to document agent reasoning, decisions, and limitations for both technical and non technical audiences.
• Comfortable working in a fast moving environment with evolving AI/ML capabilities and incomplete information.

Technologies

• Python
• SQL
• AWS
• Databricks
• OpenAI
• Anthropic
• LangChain
• LangGraph
• CrewAI
• dbt
• Airflow
• Databricks Workflows
• LangSmith
• Langfuse
• FHIR
• HL7

Strong pluses

• Experience with dbt or similar data transformation frameworks
• Familiarity with orchestration tools (Airflow, Databricks Workflows) and workflow automation
• Experience with agent evaluation and observability tooling (LangSmith, Langfuse, or custom frameworks)
• Background in healthcare, fintech, or another regulated/high-stakes domain where AI reliability is critical
• Experience building internal developer tooling, platform capabilities, or developer facing products
• Hands on experience with RAG or other grounding techniques for LLMs
• Familiarity with healthcare data formats and standards (FHIR, HL7, claims data, clinical NLP)
• Experience with model evaluation, fairness assessment, or bias detection in ML/AI systems
• Understanding of healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices
• Experience establishing QA frameworks, test plans, and quality metrics for ML/AI systems
• Startup or high growth environment experience with rapid iteration and learning
• Published research or open source contributions in AI/agentic systems