This hourly, remote contractor role supports Data Science AI training quality by overseeing quality assurance activities across projects, including content review, trainer feedback, and QA process scalability.
Role Overview
You will serve as a Data Scientist Quality Assurance Lead (QAL), monitoring consistency and quality across AI-generated data science training materials. The work focuses on reviewing trainer and QA outputs, validating results against established guidelines, and supporting scalable QA workflows through documentation and structured communication.
Key Responsibilities
- Quality monitoring: Perform spot-checks of data science items, identify quality issues, provide feedback through DMs, and escalate recurring or critical concerns.
- Technical review: Assess AI-generated data science explanations, including Python, R, and SQL snippets; modeling workflows; statistical interpretations; dashboards; experiment designs; and step-by-step reasoning.
- Trainer and QA communication: Share updates with trainers/QAs on Discord regarding guideline changes, workflow updates, and data science-specific quality expectations.
- Question handling: Address questions related to statistical assumptions, metrics, model selection, data leakage, validation, coding decisions, reproducibility, and rubric interpretation.
- Trainer/QA activation management: DM inactive contributors, encourage activation, track follow-ups, and flag availability issues.
- Documentation: Create and maintain data science style guides, trackers, FAQs, examples, honeypots, calibration tasks, and onboarding materials.
- Onboarding and training: Schedule and run onboarding and training calls to explain project expectations, workflows, rubrics, and data science review standards.
- Risk review: Flag misleading, overconfident, statistically invalid, or non-reproducible data science outputs.
- Process improvement: Identify recurring quality gaps and help develop scalable QA processes.
Required Qualifications
- Education: Bachelor’s, Master’s, or PhD degree in a closely related quantitative field, including Data Science, Statistics, Computer Science, Machine Learning, Mathematics, Economics, Engineering, or similar.
- English proficiency: Strong command of English to follow guidelines, communicate with teams, and deliver clear technical feedback.
- Experience: 3+ years of professional experience in data science, analytics, machine learning, statistical modeling, experimentation, data engineering, technical review, or data science education.
- Technical foundation: Strong understanding of statistics and probability; data cleaning and exploratory data analysis; feature engineering; supervised and unsupervised learning; model evaluation; experimentation; regression, classification, clustering; and validation methods.
- Rubric-based evaluation: Ability to assess content against detailed rubrics and identify issues including data leakage, flawed assumptions, incorrect metrics, weak methodology, non-reproducible code, hallucinated libraries/APIs, or misleading conclusions.
Preferred Skills
- Familiarity with tools such as Python, pandas, NumPy, scikit-learn, SQL, Jupyter, matplotlib, R, Spark, Git, MLflow, notebooks, dashboards, and cloud/data platforms.
- Experience leading or supporting remote teams of trainers, annotators, analysts, data scientists, engineers, educators, or QAs.
- Comfort using Discord, Google Sheets, Google Docs, trackers, dashboards, GitHub, and project management systems.
- High organization and the ability to maintain style guides, trackers, FAQs, onboarding materials, honeypots, calibration tasks, and quality documentation.
- Experience with AI training, data annotation, LLM evaluation, data science QA, or rubric-based technical review.
Technologies
Python, pandas, NumPy, scikit-learn, SQL, Jupyter, matplotlib, R, Spark, Git, MLflow, Discord, Google Sheets, Google Docs, GitHub
Contract Notes
This is an hourly, remote contractor position. There is no immediate project assigned; however, if you are qualified, you will be among the first experts contacted when relevant opportunities arise. The role also provides access to future projects through the expert network.
Selection Process
- AI interview
- Domain-specific task
- Interview with a recruiter
