Data Jobs in Arlington Heights, IL

1 job found

Include remote jobs

19d ago

Senior Data Engineer

Deloitte Arlington Heights, IL

Arlington Heights, IL

19d ago

$95k - $150k/yr Full time Senior Azure Data Factory Azure Databricks

Apply Now

Job Description

Senior Data Engineer at Deloitte in Arlington Heights, IL, responsible for designing, building, and tuning ETL/ELT pipelines on Azure Databricks while collaborating with client and engagement teams and guiding data engineering standards within Deloitte's Project Delivery Model.

Responsibilities

Maintain ongoing communication with Engagement Managers (Directors), project teams, and cross functional stakeholders, escalating items that require leadership attention
Design, implement, and optimize ETL and ELT pipelines using Azure Data Factory and Databricks
Author and tune PySpark and Spark SQL notebooks for large-scale data transformations
Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog
Lead design discussions with client architects and other counterparts
Collaborate with multiple teams on data contracts and schema governance
Direct the design and optimization of high-volume data pipelines
Define and enforce data engineering standards including naming conventions, partitioning, cluster configurations, and Spark tuning
Drive performance improvements through AQE tuning, effective clustering, broadcast joins, and shuffle partition management
Design Databricks cluster policies, autoscaling configurations, and cost optimization approaches
Perform root cause analysis on production incidents and implement durable fixes
Mentor junior and mid-level engineers via code reviews and pair programming
Evaluate new technologies and advocate for adoption (eg, DABs, DLT, Auto Loader, Serverless Compute, Event Hubs)

Requirements

Proficiency in Python, PySpark, Spark SQL, and SQL Server
Experience with Azure data services including Data Factory, Data Lake Storage Gen2, Key Vault, and Azure Monitor
Hands-on with Databricks features such as Delta Lake, Unity Catalog, and Workflows
Experience with Apache Airflow
Version control and CI/CD tools using Git and Azure DevOps
Deep understanding of Spark internals including DAG optimization, spill analysis, and data skew handling
Advanced Delta Lake features such as time travel, deletion vectors, and predictive I/O
Unity Catalog governance covering row and column security, external locations, and system tables
Infrastructure as Code with Terraform and Azure ARM templates
Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or related IT field, or equivalent experience
Limited immigration sponsorship may be available
Ability to travel about 10 percent, depending on client engagements

Technologies

Python, PySpark, Spark SQL, SQL Server
Azure Data Factory, Azure Data Lake Storage Gen2, Key Vault, Azure Monitor
Databricks, Delta Lake, Unity Catalog, Workflows
Apache Airflow
Git, Azure DevOps
Terraform, Azure ARM templates
Auto Loader, Serverless Compute, DABs, DLT, Event Hubs

Additional Information

Information for applicants with a need for accommodation: https://www2.deloitte.com/us/en/pages/careers/articles/join-deloitte-assistance-for-disabled-applicants.html

View full job page →

Get Data Jobs in Arlington Heights, IL alerts

New Data Jobs in Arlington Heights, IL, daily.