DataJobs.io
← Back to all jobs

Job Description

Work with Microsoft’s Engineering Thrive Insights team to build measurement frameworks and AI data and experimentation systems that improve engineering outcomes at scale.

Responsibilities

  • Apply data science and business understanding to review technical projects and assess factors influencing final outcomes.
  • Evaluate project plans across resources, risks, contingencies, requirements, assumptions, and constraints.
  • Document key business objectives and translate goals into analytical and technical terms.
  • Communicate insights to stakeholders consistently.
  • Collaborate with immediate team members to define metrics and frameworks for evaluating AI/LLM systems and agentic product performance.
  • Experiment with models prior to selection (for example, LLM and prompting approaches) for product or solution tasks.
  • Diagnose basic issues and identify improvement opportunities across AI/LLM systems within the domain area.
  • Prototype and develop AI models in containerized environments (including AI Azure Foundry and LangChain) to operationalize, build, and validate scalable tools and applications.
  • Develop and deploy retrieval-augmented generation (RAG) workflows, including selection of data sources, retrieval systems, and generative models to support output quality.
  • Support senior team members in assessing and refining LLM systems and models.
  • Determine where to acquire data required to complete the project plan.
  • Use querying, visualization, and reporting to describe acquired data (format, quantity, identities, and other surface properties).
  • Explore data attributes and contribute to data quality reporting using AI tools (for example, Codex and GitHub Copilot), including task results, initial findings, and project impact.
  • Collaborate on data-science experiments using established methodologies, including statistics, optimization, and probability theory for general-purpose software and statistical packages.
  • Assess multiple tools and techniques and select appropriate options.
  • Partner with Solution Architects, Consultants, and Data Engineers on data preparation efforts.
  • Follow Microsoft privacy policy requirements for collecting data, preparing data, and using AI tools (including Codex and GitHub Copilot).
  • Identify data integrity problems.
  • Use knowledge of modeling and machine learning approaches (for example, state-of-the-art statistical analysis, causal modeling, reinforcement learning) to choose the best approach for objectives.
  • Select data preparation, model training and optimization, and evaluation methods based on validation, hypothesis testing, and proper use of data, including statistical and business significance.
  • Account for risks such as data leakage, bias-variance tradeoff, and methodological limitations.
  • Write required scripts using appropriate languages such as T-SQL, U-SQL, KQL, Python, and R.
  • Construct hypotheses, design controlled experiments, analyze results using statistical tests, and communicate findings to business stakeholders.
  • Communicate with diverse audiences on data-quality issues and initiatives.
  • Understand deployment operational considerations including performance, scalability, monitoring, maintenance, integration into engineering production systems, and stability.
  • Develop operational models that run at scale in partnership with data engineering teams.
  • Keep up to date on embedding models and vector databases, and understand current embedding/vector database practices.
  • Stay current on discipline trends, including tools, techniques, strategies, and processes that can improve efficiency and performance.
  • Attend internal research conferences and participate in on-hands training, when appropriate.
  • Contribute to thought leadership and IP best practices.

Requirements

  • Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field.
  • OR Master’s Degree in the same fields/areas AND 1+ year(s) of data-science experience (for example, managing structured and unstructured data, applying statistical techniques, and reporting results) or consulting experience.
  • OR Bachelor’s Degree in the same fields/areas AND 2+ years of data-science experience (for example, managing structured and unstructured data, applying statistical techniques, and reporting results).
  • OR equivalent experience.
  • Ability to meet Microsoft, customer and/or government security screening requirements.
  • Microsoft Cloud Background Check: pass upon hire/transfer and every two years thereafter.

Technologies

  • AI Azure Foundry, LangChain
  • T-SQL, U-SQL, KQL (Kusto Query Language), Python, R
  • Codex, GitHub Copilot
  • Retrieval-augmented generation, Spark-based big data platform technologies such as Databricks
  • Vector databases, embedding models
  • DORA, SPACE
  • AI coding tools, LLM, reinforcement learning
  • Regression modeling, hypothesis testing

Preferred Qualifications

  • Doctorate in the listed fields AND 1+ year(s) data-science experience OR Master’s Degree AND 3+ years data-science experience (as described in the application text).
  • Experience with observational causal inference methods, design of controlled experiments, regression modeling, and hypothesis testing.
  • Experience evaluating LLMs and/or AI agents, including designing offline/online evaluations, defining quality metrics, and measuring model behavior.
  • Experience designing surveys and analyzing response data.
  • Proficiency in Python.
  • Proficiency in analytical query languages such as SQL, KQL (Kusto Query Language), or similar.
  • Experience developing analytics on Spark-based big data platform technologies such as Databricks.
  • Data storytelling, including crafting and presenting data stories to business decision-makers.
  • Familiarity with developer productivity metrics and frameworks (for example DORA, SPACE, or similar).
  • Demonstrated effective use of AI coding assistants in the daily data science development workflow.

Location: Redmond, WA (onsite)
Salary: USD 102,100 - 219,200 per year
Minimum experience: 1 year

Similar Jobs