Progile Solutions LLC offers a remote Data Scientist role focused on AI evaluation analytics within an exclusive client engagement with a global consulting leader and a Fortune 100 enterprise AI Platform-as-a-Service provider. This position provides a competitive hourly rate of USD 50 to 75, the ability to influence measurement and interpretation of AI system performance, and collaboration with product, engineering, and business stakeholders from anywhere in a distributed workforce. You will develop metrics, evaluate AI capabilities, and clearly communicate insights to drive better outcomes for enterprise AI workflows.
Join a team that prioritizes rigorous measurement, data-driven decision making, and clear, actionable storytelling. For professionals who value impact, collaboration, and continuous learning, this is an opportunity to shape how enterprise AI features are tested and validated at scale.
Responsibilities
- Support the development and refinement of AI evaluation metrics, scoring approaches, and benchmark methodologies.
- Validate datasets, data-quality criteria, evaluation inputs, and analytical assumptions to improve reliability and reproducibility of results.
- Analyze evaluation data, performance trends, product telemetry, and measurement signals to generate actionable insights.
- Apply statistical analysis and experimentation techniques to assess AI system effectiveness and compare models, prompts, tools, or configurations.
- Identify trends, patterns, performance gaps, and opportunities to improve AI quality, workflow efficiency, user adoption, and customer outcomes.
- Communicate analytical findings, limitations, and recommendations clearly to product, engineering, and business stakeholders.
Requirements
- Minimum of five years of professional experience in data science, advanced analytics, applied statistics, or a related field.
- At least two years of experience focused on AI evaluation analytics, generative AI measurement, machine learning evaluation, or AI performance analysis.
- Experience designing metrics, analytical frameworks, experiments, benchmarks, or measurement approaches.
- Proficiency in Python, SQL, and commonly used data-analysis and statistical libraries.
- Experience working with large datasets, product telemetry, operational data, or customer-usage data.
- Familiarity with generative AI, large language models, AI agents, coding assistants, or enterprise AI applications.
- Familiarity with tools and platforms such as OpenAI models and APIs, Anthropic Claude, Claude Code, Claude Opus, GitHub Copilot, or comparable AI platforms.
- Understanding of statistical methods such as hypothesis testing, regression analysis, confidence intervals, sampling, variance analysis, and experiment design.
- Ability to determine whether evaluation results are valid, reliable, reproducible, and meaningful.
- Strong communication skills with the ability to explain analytical results clearly.
Technologies
- Python, SQL
- OpenAI models and APIs
- Anthropic Claude, Claude Code, Claude Opus
- GitHub Copilot
- OpenAI Evals, LangSmith, MLflow, Ragas, DeepEval
- Tableau, Power BI, Looker, Grafana, ServiceNow
Experience
- Designing metrics and analytical frameworks: 3 years (Required)
- Large datasets and extracting meaningful insights: 3 years (Required)
- Background in data science, analytics, statistics: 5 years (Required)
- Evaluating AI/ML systems and agentic AI applications: 2 years (Required)
- Experimentation, benchmarking, or performance analysis: 2 years (Required)
- OpenAI Evals, LangSmith, MLflow, Ragas, DeepEval: 3 years (Preferred)
Preferred Experience
- Experience evaluating AI/ML systems, generative AI applications, or agentic AI workflows.
- Experience with experimentation, benchmarking, model comparison, or performance analysis.
- Experience analyzing enterprise SaaS or PaaS applications, workflow platforms, ServiceNow applications, or similar enterprise technologies.
- Experience with software engineering, developer productivity, service-management, customer-support, or quality metrics.
- Familiarity with AI evaluation or observability tools such as OpenAI Evals, LangSmith, MLflow, Ragas, DeepEval, or similar tools.
- Experience with dashboards and visualization platforms such as Tableau, Power BI, Looker, or Grafana.
- Familiarity with predictive analytics, forecasting, segmentation, anomaly detection, or causal inference when relevant to improving product and customer outcomes.
About the engagement
Progile Solutions is partnering with a global consulting leader and a Fortune 100 enterprise AI Platform-as-a-Service provider to deliver data science and analytical expertise focused on measurement, reliability, and interpretation of AI system performance across enterprise PaaS, workflow, and ServiceNow related applications.
Pay
USD 50.00 - 75.00 per hour
Application questions
- LinkedIn profile and work experience validation is mandatory. Please provide your LinkedIn Profile ID
- This is a remote role; however, the client prefers candidates located on the West Coast and requires work during PDT hours. Can you work during PDT hours?
- Do you have hands on experience building or evaluating AI powered coding tools and software engineering agents such as Claude Code, Devin, OpenCode, or comparable platforms?
Language
English (Required)
Shift availability
Day Shift (Required)
Work location
Remote
