Senior AI Engineer, Testing
Senior
Agentic Ai
Agents
Ai Agent
Ai Agent Platform
Ai Evals
Ai Platform
Ai Security
Artificial Intelligence
Artificial Intelligence Engineer
Data Analysis
Data Analytics
Data Processing
Data Science
Database
Databases
Engineer
Generative AI
Generative Ai Engineering
Large Language Models
Machine Learning
Management
Programming
Qa Testing
Software Development
Software Engineering
SQL
Job Description
Build and own AI testing agents that plan, execute, and validate business process tests inside real SAP landscapes. At Glasslake, you will define what “good” looks like, ship end-to-end agent and LLM capabilities, and improve real production performance alongside SAP domain experts. The work is open-ended by design, with time to explore and test ideas before committing to solutions.
What you’ll do
- Frame the problem by converting observed agent behavior into a clear problem statement, supported by evidence and your own experiments.
- Define success with evaluation through metrics, benchmarks, and evals that measure agent performance over time.
- Ship end-to-end by designing and implementing agent and LLM features, including prompts, orchestration, and the infrastructure behind them.
- Own outcomes in production by monitoring real performance and iterating to improve results.
- Validate with domain experts by collaborating with SAP subject matter experts and presenting findings and decisions clearly.
- Raise the bar by reviewing code and designs and contributing to the team’s AI engineering practices.
What you bring
- 6+ years of engineering experience, including building AI agents or LLM features from scratch and shipping them to production with full ownership.
- Research-like rigor, using hypotheses, experiments, and verification before you build.
- Evaluation experience measuring agent performance via evals, benchmarks, or regression suites, and acting on results.
- Strong Python and SQL, plus hands-on experience with LLM APIs and agent frameworks.
- Infrastructure ownership, comfortable running workflow orchestration and cloud services behind your features.
- Production investigation skills, able to query databases and read logs and traces to determine what actually happened.
- Self-direction and clear communication, including with domain experts who are not engineers.
- AI-native habits, using AI tooling daily as a force multiplier.
Tools you’ll use
- Python, SQL
- LLM APIs, agent frameworks
- Workflow orchestration
Nice to have
- Experience with computer-use or browser-automation agents.
- Software testing background (test automation, QA for complex systems, or familiarity with SAP testing).
- Familiarity with enterprise ecosystems such as SAP, ERP, or CRM.
- Ability to work in TypeScript when a feature requires it.
- Early-stage startup experience where you owned a problem without a playbook.
- Experience mentoring engineers or serving as a technical reference.
Why Glasslake now
- Own open-ended agent problems end to end, from initial exploration to results in production.
- Time to explore a problem and test ideas before committing to a solution.
- Deploy agents inside multi-million dollar transformations at major SAP integrators, in a category being defined.
- Direct access to the founding team, customers, and SAP domain experts.
Location: San Francisco, CA (onsite)
Experience: 6+ years