Senior Data Engineer
Senior
Azure
Azure Data Factory
Azure Data Lake Storage
Azure Databricks
Big Data
Bigdata
Cloud
Cloud Platforms
Data
Data Architecture
Data Engineer
Data Governance
Data Integration
Data Lake
Data Pipeline
Data Platform
Data Processing
Data Security
Database
Databricks
Engineer
ETL
Microsoft Azure
Spark
SQL
Job Description
Deloitte is seeking a Senior Data Engineer for an onsite appointment in Boston, MA. The role centers on designing, building, and optimizing end-to-end data pipelines and data solutions using Azure Data Factory, Databricks, and PySpark, with notable client-facing collaboration and mentorship responsibilities.
Responsibilities
- Maintain regular communication with Engagement Managers (Directors), project teams, and cross-functional technical partners, escalating issues to engagement leadership as needed.
- Create, implement, and optimize ETL and ELT pipelines using Azure Data Factory and Databricks.
- Develop and refine PySpark and Spark SQL notebooks for large-scale data transformations.
- Architect end-to-end data solutions spanning development, UAT, and production environments with Unity Catalog.
- Lead design discussions with client architects and other stakeholders.
- Collaborate across teams to establish data contracts and schema agreements.
- Lead the design and optimization of high-volume data pipelines.
- Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
- Drive performance improvements such as AQE tuning, clustering choices, broadcast joins, and shuffle partition management.
- Design Databricks cluster policies, autoscaling configurations, and cost optimization approaches.
- Investigate production incidents, perform root cause analysis, and implement durable fixes.
- Mentor junior and mid-level engineers through code reviews and pair programming.
- Evaluate new technologies and advise on adoption, including DABs, DLT, Auto Loader, Serverless Compute, and event hubs.
Requirements
- Proficiency in Python, PySpark, Spark SQL, and SQL Server.
- Experience with Azure components such as Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
- Databricks experience with Delta Lake, Unity Catalog, and Workflows.
- Apache Airflow.
- Git and Azure DevOps experience.
- Deep understanding of Spark internals including DAG optimization, spill analysis, and skew handling.
- Delta Lake advanced features such as time travel, deletion vectors, and predictive I/O.
- Unity Catalog governance including row and column security, external locations, and system tables.
- Infrastructure as Code using Terraform and Azure ARM templates.
- Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or related IT discipline, or equivalent experience.
- Limited immigration sponsorship may be available.
- Ability to travel up to 10 percent, depending on client engagements.
Technologies
- Python, PySpark, Spark SQL
- SQL Server
- Azure: Data Factory, Data Lake Storage Gen2, Key Vault, Monitor
- Databricks: Delta Lake, Unity Catalog, Workflows
- Apache Airflow
- Git, Azure DevOps
- Terraform, Azure Resource Manager (ARM) templates
- Delta Live Tables (DLT), Auto Loader
- Serverless Compute, event hubs
- DABs
The Team
The AI and Engineering group at Deloitte applies advanced engineering capabilities to build, deploy, and operate integrated sector solutions across software, data, AI, networking, and hybrid cloud infrastructure. These efforts empower clients to modernize technology and data platforms, transform mission-critical operations, and stay ahead by adopting the latest engineering innovations. Delivery models are tailored to meet each client engagement.
Accessibility and Accommodations
Applicants with accessibility needs can find information about available accommodations here: Deloitte assistance for disabled applicants.
Similar Jobs