Sr AI Engineer
Job Description
One Call Medical Inc is seeking a Sr/Principal AI Engineer to design and deliver agentic, production-ready AI services. This role focuses on LLM orchestration, multi-agent coordination, and RAG systems to improve the workers’ compensation experience in a highly governed PHI environment.
Responsibilities
- Translate product specifications into AI service architectures by decomposing business intent into agent workflows, data source integrations, and scalable service designs.
- Build agentic AI workflows using Azure AI Foundry Agent Service and Microsoft Agent Framework, including multi-agent coordination, tool integrations, and agent lifecycle management.
- Develop and iterate on LLM-based solutions, including prompt engineering, model selection, and cost or token optimization, along with RAG pipeline design.
- Create evaluation systems using rubrics, labeled datasets, and judge agents to validate agent correctness and safety before production deployment.
- Apply engineering fundamentals to agentic systems with emphasis on separation of concerns, memory architecture, and tool integration.
- Contribute across the full stack, spanning model experimentation, evaluation design, and production system deployment.
- Build and maintain APIs, microservices, CI/CD pipelines, and cloud-native deployments with attention to observability and reliability.
- Support GenAIOps processes, including automated testing, regression evaluation, latency monitoring, and continual improvement.
- Balance performance, safety, responsible AI principles, and cost across system design.
- Implement guardrails, fallbacks, red-teaming strategies, and human-in-the-loop (HITL) workflows.
- Partner with legal and ethics teams to align with OneCall Responsible AI standards while working in a highly governed PHI space.
- Build automated evaluation suites integrating user signals, continual learning cycles, and ongoing model updates.
- Design and implement evaluation frameworks covering hallucination rate and factual consistency, relevance and precision or recall, latency and throughput plus system-level performance, and cost tracking and efficiency.
- Partner with product, engineering, data science, security and ethics, and infrastructure teams to deliver robust and compliant AI systems.
- Share technical insights and communicate findings clearly to cross-functional stakeholders and, where relevant, clients.
- Share knowledge with peers and support technical learning across RAG, agents, prompt engineering, and AI safety.
Requirements
- At least 5 years in software engineering, traditional ML engineering, or applied AI roles with hands-on building responsibilities.
- Demonstrated experience shipping generative AI features or systems end-to-end, from prototyping through production.
- Working knowledge of evaluation, design, experimentation frameworks, and data labeling strategies for LLM applications.
- Experience with RAG architectures including vector-based retrieval; exposure to hybrid or graph-based approaches is a plus.
- Experience with agentic patterns such as tool use, routing, memory management; multi-agent systems experience is a plus.
- Experience with ReAct, RLAIF, and other HITL plus feedback loop approaches.
- Experience with AI-specific tools and frameworks including orchestration frameworks, vector and graph databases, and Model + API ecosystems.
- Strong background in system design, architecture, and production-grade deployment.
- Familiarity with Azure cloud services, logging, monitoring, and observability patterns.
- Healthcare, workers compensation, claims processing, or regulated industry experience is preferred.
- Solid prompt engineering and context engineering skills, including familiarity with conversation design principles.
- Clear communication skills, including the ability to explain technical concepts to non-technical collaborators and stakeholders.
- Ability to collaborate across engineering, product, and data science teams; supporting or informally mentoring peers is a plus.
- Familiarity with cost and latency tradeoffs for LLM workloads.
- Comfort operating in high-ambiguity environments with cross-functional teams.
- Eagerness to grow into a technical leadership role and support junior team members.
- Client-facing consulting or enterprise transformation experience is a strong plus.
- Ability to align with the company’s diversity, equity, inclusion, and belonging standards.
- Ability to work independently and in a team environment.
Technologies
- Azure AI Foundry Agent Service
- Microsoft Agent Framework
- LLM
- RAG
- CI/CD
- APIs
- Microservices
- GenAIOps
- HITL
- ReAct
- RLAIF
- Azure cloud services
- Vector databases
- Graph databases
- Model + API ecosystems
- PHI
Work Location and Schedule
- United States (remote)
- Work is performed in the colleague’s home.
- Ability to set up computer equipment within a home office is required.
- Occasional in-person team meetings may be required depending on the position.
Work Environment and Physical/Mental Demands
- Primarily sedentary work that may involve repetitive motions.
- Ability to remain in a stationary position for extended periods, operate a computer and office equipment, and assess information and files stored electronically.
- Ability to discern text on a monitor, input data using a keyboard, and adjust focus to distances of up to three feet.
- Ability to learn new tasks, follow established processes, maintain focus, and complete tasks independently.
- Ability to manage multiple tasks simultaneously and communicate professionally with colleagues and customers.
- Ability to handle situations with speed or productivity requirements.
- Ability to manage moderate to significant mental stress from a dynamic and heavy workload.
Compensation
USD 120,442 - 192,707 per year.