Serverless Compute Data Jobs

1 job found

22d ago

Senior Data Engineer

Deloitte Morristown, NJ

Morristown, NJ

22d ago

$95k - $150k/yr Full time Senior Adls Gen2 Apache Airflow

Apply Now

Job Description

Deloitte in Morristown, New Jersey is seeking a Senior Data Engineer to design, build, and optimize end-to-end data solutions and ETL/ELT pipelines using Azure Data Factory, Databricks, and PySpark. This onsite role collaborates closely with engagement teams and client architects to deliver scalable data platforms, with a salary range of USD 95,000 to 150,000 per year and a bachelor’s degree required.

Responsibilities

Maintain regular communications with engagement managers (directors), project teams, and stakeholders across functional and technical groups, escalating issues that require attention from engagement leadership.
Design, build, and optimize ETL/ELT pipelines leveraging Azure Data Factory and Databricks.
Develop and tune PySpark and Spark SQL notebooks for large-scale data transformations.
Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog.
Lead design discussions with client architects and other counterparts.
Collaborate across teams to establish data contracts and agree on schemas.
Lead the design and optimization of high-volume data pipelines.
Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
Conduct root cause analysis on production incidents and implement lasting fixes.
Mentor junior and mid-level engineers through code reviews and pair programming.
Evaluate new technologies and recommend adoption, such as DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs.

Requirements

Proficiency in Python, PySpark, Spark SQL, and SQL Server.
Experience with Azure services including Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
Familiarity with Databricks components such as Delta Lake, Unity Catalog, and Workflows.
Experience with Apache Airflow.
Git and Azure DevOps tooling.
Deep knowledge of Spark internals, including DAG optimization, spill analysis, and skew handling.
Advanced Delta Lake features, including time travel, deletion vectors, and predictive I/O.
Unity Catalog governance covering row/column security, external locations, and system tables.
Infrastructure as Code experience with Terraform and Azure ARM templates.
Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or related IT discipline, or equivalent experience.
Limited immigration sponsorship may be available.
Ability to travel up to 10% on average, depending on client and project needs.

Technologies

Python
PySpark
Spark SQL
SQL Server
Azure Data Factory
ADLS Gen2
Key Vault
Azure Monitor
Databricks
Delta Lake
Unity Catalog
Workflows
Apache Airflow
Git
Azure DevOps
DABs
DLT
Auto Loader
Serverless Compute
Azure Event Hubs
Terraform
Azure ARM templates

Additional Requirements

Information for applicants with a need for accommodation: Accommodation information for disabled applicants.

View full job page →

Get Serverless Compute Data Jobs alerts

New Serverless Compute Data Jobs, daily.