Senior Data Engineer

Deloitte is seeking a Senior Data Engineer to help shape data pipelines and platform solutions that span development through production in Arlington, onsite. The role focuses on building, optimizing, and operating end-to-end data workflows using Azure Databricks, Azure Data Factory, and PySpark, across dev, UAT, and production environments. The position offers a salary range of USD 95,000 to 150,000 per year and requires a Bachelor's degree for consideration.

Responsibilities

• Communicate regularly with Engagement Managers (Directors), project team members, and representatives from diverse functional and technical teams, escalating issues that require management attention.
• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Write and tune PySpark and Spark SQL notebooks for large-scale data transformations.
• Architect end-to-end data solutions across development, UAT, and production using Unity Catalog.
• Lead design discussions with client architects and other counterparts.
• Collaborate with teams on data contracts and schema agreements.
• Lead the design and optimization of high-volume data pipelines.
• Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
• Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
• Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
• Conduct root cause analysis on production incidents and implement permanent fixes.
• Mentor junior and mid-level engineers via code reviews and pair programming.
• Evaluate new technologies and recommend adoption, such as Delta Live Tables, Auto Loader, serverless compute, and event hubs.

Requirements

• Proficiency with Python, PySpark, Spark SQL, and SQL Server.
• Experience with Azure services including ADF, ADLS Gen2, Key Vault, and Azure Monitor.
• Hands-on experience with Databricks features such as Delta Lake, Unity Catalog, and Workflows.
• Familiarity with Apache Airflow and source control using Git or Azure DevOps.
• Deep understanding of Spark internals, including DAG optimization, spill analysis, and skew handling.
• Knowledge of Delta Lake advanced features like time travel, deletion vectors, and predictive I/O.
• Experience with Unity Catalog governance, including row/column security, external locations, and system tables.
• Infrastructure as Code experience with Terraform and Azure ARM templates.
• Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or a related IT discipline, or equivalent experience.
• Limited immigration sponsorship may be available.
• Ability to travel approximately 10% on average, dependent on client engagements.

Technologies

• Python, PySpark, Spark SQL
• SQL Server
• Azure Data Factory, Azure Data Lake Storage Gen2, Key Vault, Azure Monitor
• Databricks, Delta Lake, Unity Catalog, Workflows
• Apache Airflow
• Git, Azure DevOps
• Terraform, Azure ARM templates
• DABs (Delta Live Tables), Auto Loader, Serverless Compute, Delta Live Tables, Azure Event Hubs

The Team

AI and Engineering at Deloitte harness cutting-edge capabilities to design, deploy, and operate integrated sector solutions across software, data, AI, networks, and hybrid cloud infrastructure. These efforts empower clients to transform mission-critical operations and modernize technology and data platforms, with delivery models tailored to meet each client’s needs.

Additional Requirements

Information for applicants with a need for accommodation: join Deloitte — accommodation for disabled applicants.