DataJobs.io
← Back to all jobs

Job Description

One Call Medical Inc is seeking a Sr/Principal AI Engineer to design and deliver agentic, production-ready AI services. This role focuses on LLM orchestration, multi-agent coordination, and RAG systems to improve the workers’ compensation experience in a highly governed PHI environment.

Responsibilities

  • Translate product specifications into AI service architectures by decomposing business intent into agent workflows, data source integrations, and scalable service designs.
  • Build agentic AI workflows using Azure AI Foundry Agent Service and Microsoft Agent Framework, including multi-agent coordination, tool integrations, and agent lifecycle management.
  • Develop and iterate on LLM-based solutions, including prompt engineering, model selection, and cost or token optimization, along with RAG pipeline design.
  • Create evaluation systems using rubrics, labeled datasets, and judge agents to validate agent correctness and safety before production deployment.
  • Apply engineering fundamentals to agentic systems with emphasis on separation of concerns, memory architecture, and tool integration.
  • Contribute across the full stack, spanning model experimentation, evaluation design, and production system deployment.
  • Build and maintain APIs, microservices, CI/CD pipelines, and cloud-native deployments with attention to observability and reliability.
  • Support GenAIOps processes, including automated testing, regression evaluation, latency monitoring, and continual improvement.
  • Balance performance, safety, responsible AI principles, and cost across system design.
  • Implement guardrails, fallbacks, red-teaming strategies, and human-in-the-loop (HITL) workflows.
  • Partner with legal and ethics teams to align with OneCall Responsible AI standards while working in a highly governed PHI space.
  • Build automated evaluation suites integrating user signals, continual learning cycles, and ongoing model updates.
  • Design and implement evaluation frameworks covering hallucination rate and factual consistency, relevance and precision or recall, latency and throughput plus system-level performance, and cost tracking and efficiency.
  • Partner with product, engineering, data science, security and ethics, and infrastructure teams to deliver robust and compliant AI systems.
  • Share technical insights and communicate findings clearly to cross-functional stakeholders and, where relevant, clients.
  • Share knowledge with peers and support technical learning across RAG, agents, prompt engineering, and AI safety.

Requirements

  • At least 5 years in software engineering, traditional ML engineering, or applied AI roles with hands-on building responsibilities.
  • Demonstrated experience shipping generative AI features or systems end-to-end, from prototyping through production.
  • Working knowledge of evaluation, design, experimentation frameworks, and data labeling strategies for LLM applications.
  • Experience with RAG architectures including vector-based retrieval; exposure to hybrid or graph-based approaches is a plus.
  • Experience with agentic patterns such as tool use, routing, memory management; multi-agent systems experience is a plus.
  • Experience with ReAct, RLAIF, and other HITL plus feedback loop approaches.
  • Experience with AI-specific tools and frameworks including orchestration frameworks, vector and graph databases, and Model + API ecosystems.
  • Strong background in system design, architecture, and production-grade deployment.
  • Familiarity with Azure cloud services, logging, monitoring, and observability patterns.
  • Healthcare, workers compensation, claims processing, or regulated industry experience is preferred.
  • Solid prompt engineering and context engineering skills, including familiarity with conversation design principles.
  • Clear communication skills, including the ability to explain technical concepts to non-technical collaborators and stakeholders.
  • Ability to collaborate across engineering, product, and data science teams; supporting or informally mentoring peers is a plus.
  • Familiarity with cost and latency tradeoffs for LLM workloads.
  • Comfort operating in high-ambiguity environments with cross-functional teams.
  • Eagerness to grow into a technical leadership role and support junior team members.
  • Client-facing consulting or enterprise transformation experience is a strong plus.
  • Ability to align with the company’s diversity, equity, inclusion, and belonging standards.
  • Ability to work independently and in a team environment.

Technologies

  • Azure AI Foundry Agent Service
  • Microsoft Agent Framework
  • LLM
  • RAG
  • CI/CD
  • APIs
  • Microservices
  • GenAIOps
  • HITL
  • ReAct
  • RLAIF
  • Azure cloud services
  • Vector databases
  • Graph databases
  • Model + API ecosystems
  • PHI

Work Location and Schedule

  • United States (remote)
  • Work is performed in the colleague’s home.
  • Ability to set up computer equipment within a home office is required.
  • Occasional in-person team meetings may be required depending on the position.

Work Environment and Physical/Mental Demands

  • Primarily sedentary work that may involve repetitive motions.
  • Ability to remain in a stationary position for extended periods, operate a computer and office equipment, and assess information and files stored electronically.
  • Ability to discern text on a monitor, input data using a keyboard, and adjust focus to distances of up to three feet.
  • Ability to learn new tasks, follow established processes, maintain focus, and complete tasks independently.
  • Ability to manage multiple tasks simultaneously and communicate professionally with colleagues and customers.
  • Ability to handle situations with speed or productivity requirements.
  • Ability to manage moderate to significant mental stress from a dynamic and heavy workload.

Compensation

USD 120,442 - 192,707 per year.

Similar Jobs