Alchemy is hiring for a Lead AI Engineer role focused on agentic test automation powered by LLM-powered agents. This hybrid position is based in Tysons, VA (22182), and you’ll lead hands-on work that connects automated test generation, validation, and reporting with modern engineering workflows in GitHub.
You will design agent workflows that help teams produce and maintain reliable tests, ensure requirements are complete, and generate actionable summaries across pipelines and microservices. The work blends agent engineering (tool use, guardrails, structured outputs) with proven test automation frameworks including Karate and Playwright, integrated into CI/CD using GitHub Actions.
What you’ll do
- Build and operate tool-using, multi-step LLM-powered agents with guardrails.
- Design agent workflows for test generation and augmentation.
- Design agent workflows for requirements review and completeness validation.
- Design agent workflows for report generation and summarization.
- Implement and evolve CI/CD and test automation frameworks using GitHub Actions and supporting GitHub capabilities.
- Use GitHub Copilot in day-to-day development.
- Leverage GitHub platform features including PR checks, branch protections, CODEOWNERS, and templates.
- Automate workflows via GitHub APIs and webhooks as needed.
- Design and implement automation with Karate (API testing, contract-like checks, data-driven testing, mocks).
- Design and implement automation with Playwright (UI automation, selectors strategy, parallelization, and trace/video artifacts).
- Aggregate and normalize results across multiple microservices and multiple pipelines.
- Produce actionable automated reports including trend analysis, failure clustering, and service correlation.
What you bring
- Hands-on experience building LLM-powered agents (tool-using, multi-step reasoning, guardrails).
- Experience with prompting patterns, structured outputs using JSON schemas, evaluation, and reducing hallucinations.
- Strong experience designing agent workflows for test generation/augmentation, requirements review/completeness validation, and report generation/summarization.
- Strong proficiency with GitHub Copilot in daily development.
- Deep experience with GitHub Actions (CI/CD pipelines, reusable workflows, composite actions).
- Deep experience with GitHub capabilities including PR checks, branch protections, CODEOWNERS, and templates.
- Automation experience using GitHub APIs/webhooks as needed.
- Advanced automation experience with Karate and Playwright as described above.
- Strong understanding of test design and coverage across happy path, negative/validation, and edge/boundary scenarios.
- Experience with data setup/teardown strategies and test isolation.
- Proven ability to aggregate and normalize results from multiple microservices and multiple pipelines.
- Experience producing actionable automated reports including trend analysis, failure clustering, and service correlation.
Technologies
- LLM-powered agents
- JSON schemas
- GitHub Copilot
- GitHub Actions
- GitHub APIs
- GitHub webhooks
- Karate
- Playwright
Location and compensation
Work location: Hybrid remote in Tysons, VA 22182
Pay: From $70.00 per hour
Experience: 5 years (Generative AI and LLM-powered agents, required); 7 years (Test Automation Engineering, required)
