Senior Data Engineer
Senior
Adls Gen2
Apache Airflow
Azure
Azure Arm Templates
Azure Data Factory
Azure Data Lake Storage
Azure Databricks
Azure Event Hubs
Azure Monitor
Big Data
Bigdata
Cloud
Cloud Platforms
Data
Data Architecture
Data Engineer
Data Governance
Data Integration
Data Lake
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Database
Databricks
DevOps
ETL
Infrastructure As Code
Key Vault
Microsoft Azure
Serverless Compute
Spark
SQL
Job Description
Deloitte in Morristown, New Jersey is seeking a Senior Data Engineer to design, build, and optimize end-to-end data solutions and ETL/ELT pipelines using Azure Data Factory, Databricks, and PySpark. This onsite role collaborates closely with engagement teams and client architects to deliver scalable data platforms, with a salary range of USD 95,000 to 150,000 per year and a bachelor’s degree required.
Responsibilities
- Maintain regular communications with engagement managers (directors), project teams, and stakeholders across functional and technical groups, escalating issues that require attention from engagement leadership.
- Design, build, and optimize ETL/ELT pipelines leveraging Azure Data Factory and Databricks.
- Develop and tune PySpark and Spark SQL notebooks for large-scale data transformations.
- Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog.
- Lead design discussions with client architects and other counterparts.
- Collaborate across teams to establish data contracts and agree on schemas.
- Lead the design and optimization of high-volume data pipelines.
- Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
- Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
- Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
- Conduct root cause analysis on production incidents and implement lasting fixes.
- Mentor junior and mid-level engineers through code reviews and pair programming.
- Evaluate new technologies and recommend adoption, such as DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs.
Requirements
- Proficiency in Python, PySpark, Spark SQL, and SQL Server.
- Experience with Azure services including Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
- Familiarity with Databricks components such as Delta Lake, Unity Catalog, and Workflows.
- Experience with Apache Airflow.
- Git and Azure DevOps tooling.
- Deep knowledge of Spark internals, including DAG optimization, spill analysis, and skew handling.
- Advanced Delta Lake features, including time travel, deletion vectors, and predictive I/O.
- Unity Catalog governance covering row/column security, external locations, and system tables.
- Infrastructure as Code experience with Terraform and Azure ARM templates.
- Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or related IT discipline, or equivalent experience.
- Limited immigration sponsorship may be available.
- Ability to travel up to 10% on average, depending on client and project needs.
Technologies
- Python
- PySpark
- Spark SQL
- SQL Server
- Azure Data Factory
- ADLS Gen2
- Key Vault
- Azure Monitor
- Databricks
- Delta Lake
- Unity Catalog
- Workflows
- Apache Airflow
- Git
- Azure DevOps
- DABs
- DLT
- Auto Loader
- Serverless Compute
- Azure Event Hubs
- Terraform
- Azure ARM templates
Additional Requirements
Information for applicants with a need for accommodation: Accommodation information for disabled applicants.
Similar Jobs