AI Engineer Consultant

Deloitte is hiring an AI Engineer Consultant for onsite AIOps/MLOps engineering in a Azure + Databricks environment under its Project Delivery Talent Model.

Responsibilities

• Monitor Databricks jobs and clusters by tracking job run status, cluster utilization, and auto-scaling behavior via Databricks Jobs UI and Azure Monitor, resolving failed or delayed pipeline runs.
• Build and maintain CI/CD pipelines in Azure DevOps using YAML, deploying notebooks, ML models, and Databricks workflows across dev/staging/prod with Databricks Repos and Git integration.
• Manage the MLflow model lifecycle: track experiments, register and version models in the MLflow Model Registry, and handle transitions for staging and production along with lineage.
• Maintain Delta Lake pipelines by enforcing data quality, schema standards, and ACID compliance across bronze/silver/gold layers for training and inference workloads.
• Monitor model performance and drift by setting up automated drift detection (data and concept drift) using Databricks native monitoring or custom Azure ML integration, triggering retraining when thresholds are breached.
• Optimize compute and cost by configuring Databricks clusters (job clusters vs. all-purpose), leveraging autoscaling and spot instances, and using Azure cost management dashboards to control spend.
• Implement observability using Azure Monitor and Log Analytics with end-to-end logging and alerting across Databricks, Azure ML, and downstream services via Application Insights and Log Analytics workspaces.
• Support security, access, and governance by configuring Unity Catalog for data and model governance, managing service principals, secrets via Azure Key Vault, and RBAC across workspaces.
• Collaborate on deployment using Azure ML endpoints, deploying models as real-time or batch endpoints via Azure ML Managed Endpoints or Databricks Model Serving, focused on scalability and low-latency inference.
• Provide on-call support and incident response: troubleshoot pipeline failures, cluster crashes, or endpoint downtime; perform root cause analysis and post-incident reviews to improve pipeline resilience.

Requirements

• 3-6+ years of experience in DevOps/MLOps/Data Engineering, including 1-2 years hands-on with Databricks and Azure.
• Strong proficiency in Python and/or Scala, plus SQL for data transformation and querying.
• Hands-on experience with Databricks components: Jobs, Workflows, Unity Catalog, Delta Lake, and Databricks Model Serving.
• Proficiency in the Azure ecosystem: Azure DevOps, Azure ML, Azure Monitor, Azure Key Vault, and Azure Data Factory.
• Experience with MLflow for experiment tracking and model registry management.
• Working knowledge of CI/CD practices and Infrastructure as Code using Terraform or ARM/Bicep templates.
• Understanding of ML lifecycle concepts: model training, validation, deployment, monitoring, and retraining.
• Education: Bachelor's degree, preferably in Computer Sciences, Information Technology, Computer Engineering, or a related IT discipline.

Technical tools

• Azure, Databricks, Azure Monitor, Databricks Jobs UI, Azure DevOps
• YAML, Databricks Repos, Git, MLflow, MLflow Model Registry
• Delta Lake, Databricks native monitoring, Azure ML, Databricks clusters, autoscaling, spot instances
• Application Insights, Log Analytics workspaces, Unity Catalog, service principals, Azure Key Vault, RBAC
• Azure ML Managed Endpoints, Databricks Model Serving, Python, Scala, SQL, CI/CD
• Infrastructure as Code, Terraform, ARM, Bicep, Azure Data Factory

Additional information

• Limited immigration sponsorship may be available.
• Travel: 10% on average, including potential overnight travel.

Preferred

• Familiarity with containerization (Docker) and orchestration (Kubernetes, if applicable).
• Analytical ability to manage multiple projects and prioritize tasks into manageable work products.
• Can operate independently or with minimum supervision.
• Excellent written and communication skills.
• Ability to deliver technical demonstrations.