Senior Data Engineer

Senior Data Engineer at Deloitte in Arlington Heights, IL, responsible for designing, building, and tuning ETL/ELT pipelines on Azure Databricks while collaborating with client and engagement teams and guiding data engineering standards within Deloitte's Project Delivery Model.

Responsibilities

• Maintain ongoing communication with Engagement Managers (Directors), project teams, and cross functional stakeholders, escalating items that require leadership attention
• Design, implement, and optimize ETL and ELT pipelines using Azure Data Factory and Databricks
• Author and tune PySpark and Spark SQL notebooks for large-scale data transformations
• Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog
• Lead design discussions with client architects and other counterparts
• Collaborate with multiple teams on data contracts and schema governance
• Direct the design and optimization of high-volume data pipelines
• Define and enforce data engineering standards including naming conventions, partitioning, cluster configurations, and Spark tuning
• Drive performance improvements through AQE tuning, effective clustering, broadcast joins, and shuffle partition management
• Design Databricks cluster policies, autoscaling configurations, and cost optimization approaches
• Perform root cause analysis on production incidents and implement durable fixes
• Mentor junior and mid-level engineers via code reviews and pair programming
• Evaluate new technologies and advocate for adoption (eg, DABs, DLT, Auto Loader, Serverless Compute, Event Hubs)

Requirements

• Proficiency in Python, PySpark, Spark SQL, and SQL Server
• Experience with Azure data services including Data Factory, Data Lake Storage Gen2, Key Vault, and Azure Monitor
• Hands-on with Databricks features such as Delta Lake, Unity Catalog, and Workflows
• Experience with Apache Airflow
• Version control and CI/CD tools using Git and Azure DevOps
• Deep understanding of Spark internals including DAG optimization, spill analysis, and data skew handling
• Advanced Delta Lake features such as time travel, deletion vectors, and predictive I/O
• Unity Catalog governance covering row and column security, external locations, and system tables
• Infrastructure as Code with Terraform and Azure ARM templates
• Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or related IT field, or equivalent experience
• Limited immigration sponsorship may be available
• Ability to travel about 10 percent, depending on client engagements

Technologies

• Python, PySpark, Spark SQL, SQL Server
• Azure Data Factory, Azure Data Lake Storage Gen2, Key Vault, Azure Monitor
• Databricks, Delta Lake, Unity Catalog, Workflows
• Apache Airflow
• Git, Azure DevOps
• Terraform, Azure ARM templates
• Auto Loader, Serverless Compute, DABs, DLT, Event Hubs

Additional Information

• Information for applicants with a need for accommodation: https://www2.deloitte.com/us/en/pages/careers/articles/join-deloitte-assistance-for-disabled-applicants.html