Llm Evaluations Data Jobs

1 job found

Today

Manager, Global Private Equity Data and AI Engineer

The Carlyle Group Washington, DC

Washington, DC

Today

Full time Backend Developer Manager Artificial Intelligence

Apply Now

Job Description

Lead the engineering and evaluation of production AI for Global Private Equity, while helping investment teams access governed portfolio data through natural-language tools. This hands-on manager role combines building LLM-powered capabilities with data engineering, security controls, and reliability practices on a cloud platform built with Snowflake, dbt, and AWS.

The Carlyle Group’s platform connects portfolio company, deal, fund, valuation, and performance data from investment, portfolio monitoring, and enterprise systems. Your work will power reporting, applications, and AI-enabled tools that support investment workflows across valuations, fund finance, and investor relations.

What you will do

Design, build, and deploy LLM-powered tools, agents, and integrations so investment, valuation, and fund finance teams can work with governed portfolio data in natural language, including MCP servers and integrations with enterprise AI platforms and assistants. Own solutions end-to-end from discovery through production and adoption.
Define how AI quality is measured by creating evaluation cases, regression suites, and acceptance criteria. Track accuracy, cost, and latency, and gate releases based on results. Maintain clear boundaries between deterministic calculation/validation and model judgment so outputs trace back to source data.
Build controls into the architecture including OAuth/OIDC authentication, entitlement-based row-level security, SQL and tool guardrails, and safeguards for untrusted content and prompt injection. Ensure audit logging and collaborate with Information Security, Compliance, and Data Governance.
Lead and mentor a mixed team of contract developers, data engineers, and QA. Set technical direction, review designs and code, make build-versus-buy recommendations, and stay hands-on in the codebase while driving collaboration and accountability.
Maintain trusted governed data models and metadata including metric definitions, glossaries, and lineage, so reporting, applications, and AI consumers use a single source of truth.
Strengthen delivery reliability with code review and automated testing (unit tests, dbt tests, data reconciliation, and AI evaluations), plus CI/CD through GitHub Actions, infrastructure as code with Terraform on AWS, and disciplined promotion across development, QA, UAT, and production.
Own stability for production AI services and data pipelines, including incident response, root cause analysis, remediation, monitoring, alerting, and runbooks.
Support accurate, auditable outcomes for valuations and investor reporting metrics such as IRR, MOIC, DPI, and NAV, and participate in audit and control reviews.
Coordinate migration of legacy workloads by supporting the transition of remaining data and reporting systems to the cloud platform with vendor delivery teams.
Adopt AI coding assistants and agent workflows with human review, safety guardrails, and shared team knowledge to improve delivery speed and quality.
Work with business partners to deliver by translating ambiguous workflows into a prioritized backlog, running Agile delivery in Jira, and communicating progress, risks, dependencies, and technical decisions to senior stakeholders.

What you bring

Bachelor’s degree (required).
Concentration in computer science, engineering, finance, economics, or another quantitative discipline (preferred).
Cloud, data platform, or AI certification (for example AWS, Microsoft Azure, SnowPro, or dbt) (preferred).
7+ years of overall relevant experience in software engineering, data engineering, or technical analytics (required).
3+ years building and operating production software or data solutions used by business stakeholders (required).
2+ years leading engineering delivery, including owning a backlog, setting technical direction and coding standards, and reviewing developers or contractors (required).
Strong proficiency in Python and SQL, plus experience with C#/.NET or TypeScript (required).
Demonstrated experience shipping LLM-powered applications to production users, including tool use/agents, retrieval, structured outputs, and prompt/context design (required).
Experience evaluating AI systems by designing test cases and regression suites, diagnosing failures, tuning against measured results, and validating model output against source data (required).
Hands-on experience with a modern cloud data platform and transformation tooling (Snowflake, Databricks, or Azure Synapse with dbt or Spark). Snowflake and dbt strongly preferred (required).
Experience with AWS or Azure, CI/CD (for example GitHub Actions), and infrastructure as code with Terraform (required). AWS experience (ECS, Lambda, S3) is a plus.
Experience with Model Context Protocol (MCP), agent frameworks, or building tools and integrations for AI assistants (preferred).
Knowledge of data architecture including dimensional/semantic modeling, layered architectures, metadata/lineage, and data quality and reconciliation techniques (required).
Understanding of security for data and AI systems including OAuth/OIDC, role/entitlement-based access, and auditability (required).
Experience managing production support including incident management, root cause analysis, and remediation (preferred).
Familiarity with private equity or other financial services data such as fund structures, deals, valuations, portfolio monitoring, and performance metrics (strongly preferred).
Experience with Power BI or similar reporting tools (preferred).
Hands-on use of AI coding assistants (for example Claude Code, Cursor, GitHub Copilot) in a team setting, including code review and guardrails for AI-generated work (required).
Strong leadership, mentoring, and communication skills, including the ability to turn ambiguous business needs into working software and collaborate effectively with business teams and contractors (required).

Tools and platforms

Python, SQL, C#/.NET, TypeScript
LLM-powered applications, Model Context Protocol (MCP), MCP servers
Snowflake, dbt, Databricks, Azure Synapse
AWS (ECS, Lambda, S3), Azure
CI/CD (GitHub Actions), Terraform
OAuth/OIDC, row-level security, Jira, Power BI
AI evaluations, unit tests, data reconciliation
AI coding assistants: Claude Code, Cursor, GitHub Copilot

Location

Washington, DC (onsite)

Benefits

Retirement benefits
Health insurance
Life insurance and disability
Paid time off
Paid holidays
Family planning benefits
Various wellness programs
Annual discretionary incentive program (eligibility)

Compensation: The compensation range for this role is specific to Washington, D.C.

View full job page →

Get Llm Evaluations Data Jobs alerts

New Llm Evaluations Data Jobs, daily.