Evaluation & Testing Data Jobs

1 job found

Today

Applied AI Engineer

Microsoft Redmond, WA

Redmond, WA

Today

$102k - $219k/yr Full time Backend Developer Agentic Ai Ai Agent

Apply Now

Job Description

Microsoft is building production-ready, agentic AI services that help resolve Azure customer issues more quickly while reducing avoidable support demand. In this role, you will design and deliver AI service components that bring together LLM orchestration, multi-agent coordination, RAG, and evaluation, with secure deployments and full service lifecycle ownership from design through incident response. The work is based in Redmond, WA and supports onsite collaboration within ACES and partner engineering teams.

What you’ll do

Translate product specifications into AI service architectures by decomposing business intent into agent workflows, data source integrations, and scalable service designs.
Build agentic AI workflows using Azure AI Foundry Agent Service and the Microsoft Agent Framework, including multi-agent coordination, tool integrations, and agent lifecycle management.
Develop and iterate on LLM-based solutions with prompt engineering, model selection, and cost and token optimization, including RAG pipeline design.
Create evaluation systems using rubrics, golden datasets, and judge agents to validate agent correctness and safety before production deployment.
Write production-quality C# and Python with test-driven development, targeting reliability, performance, and security standards.
Deploy services via CI/CD pipelines, feature flags, and staged rollouts, paired with production observability using tracing, logging, and metrics.
Implement secure service patterns including RBAC, Managed Identities, and secrets management.
Integrate agents with Azure data sources and cloud-native services to ground responses in real-time signals.
Apply Responsible AI practices across agent development to support safe, fair, and compliant outputs.
Execute reliably within sprint and co-development commitments across ACES and partner engineering teams.
Own the end-to-end lifecycle of AI service components, covering design, development, testing, deployment, monitoring, and incident response.

Required qualifications

Bachelor’s Degree in Computer Science or a related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
Ability to meet Microsoft, customer, and/or government security screening requirements for this role.
Required to pass the Microsoft Cloud Background Check upon hire or transfer and every two years thereafter.

Technologies you may work with

Languages: C#, Python, C, C++, Java, JavaScript
Agent and AI: Azure AI Foundry Agent Service, Microsoft Agent Framework, LLM orchestration, multi-agent coordination, RAG, prompt engineering, cost and token optimization, evaluation frameworks, golden datasets, judge agents, vector retrieval systems, agent-to-agent communication protocols, Responsible AI
Delivery and operations: CI/CD pipelines, feature flags, tracing, logging, metrics
Security: RBAC, Managed Identities, secrets management
Azure services and architecture: Azure OpenAI, Azure AI Foundry, Azure Functions, Cosmos DB, Azure data sources, container orchestration, microservices architecture, and cloud-native deployment patterns

Compensation and work location

Location: Redmond, WA (onsite)
Estimated base salary: USD 102,100 - 219,200 per year
Software Engineering IC3 base pay note (U.S.): USD 102,100 - 202,200 per year
Software Engineering IC3 location variation: In the San Francisco Bay area and New York City metro areas, base pay is USD 133,800 - 219,200 per year

Education (minimum)

Bachelor's Degree in Computer Science or related technical field

Additional information

This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

View full job page →

Get Evaluation & Testing Data Jobs alerts

New Evaluation & Testing Data Jobs, daily.