Deloitte is seeking a Senior Data Engineer to help shape data pipelines and platform solutions that span development through production in Arlington, onsite. The role focuses on building, optimizing, and operating end-to-end data workflows using Azure Databricks, Azure Data Factory, and PySpark, across dev, UAT, and production environments. The position offers a salary range of USD 95,000 to 150,000 per year and requires a Bachelor's degree for consideration.
Responsibilities
- Communicate regularly with Engagement Managers (Directors), project team members, and representatives from diverse functional and technical teams, escalating issues that require management attention.
- Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
- Write and tune PySpark and Spark SQL notebooks for large-scale data transformations.
- Architect end-to-end data solutions across development, UAT, and production using Unity Catalog.
- Lead design discussions with client architects and other counterparts.
- Collaborate with teams on data contracts and schema agreements.
- Lead the design and optimization of high-volume data pipelines.
- Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
- Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
- Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
- Conduct root cause analysis on production incidents and implement permanent fixes.
- Mentor junior and mid-level engineers via code reviews and pair programming.
- Evaluate new technologies and recommend adoption, such as Delta Live Tables, Auto Loader, serverless compute, and event hubs.
Requirements
- Proficiency with Python, PySpark, Spark SQL, and SQL Server.
- Experience with Azure services including ADF, ADLS Gen2, Key Vault, and Azure Monitor.
- Hands-on experience with Databricks features such as Delta Lake, Unity Catalog, and Workflows.
- Familiarity with Apache Airflow and source control using Git or Azure DevOps.
- Deep understanding of Spark internals, including DAG optimization, spill analysis, and skew handling.
- Knowledge of Delta Lake advanced features like time travel, deletion vectors, and predictive I/O.
- Experience with Unity Catalog governance, including row/column security, external locations, and system tables.
- Infrastructure as Code experience with Terraform and Azure ARM templates.
- Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or a related IT discipline, or equivalent experience.
- Limited immigration sponsorship may be available.
- Ability to travel approximately 10% on average, dependent on client engagements.
Technologies
- Python, PySpark, Spark SQL
- SQL Server
- Azure Data Factory, Azure Data Lake Storage Gen2, Key Vault, Azure Monitor
- Databricks, Delta Lake, Unity Catalog, Workflows
- Apache Airflow
- Git, Azure DevOps
- Terraform, Azure ARM templates
- DABs (Delta Live Tables), Auto Loader, Serverless Compute, Delta Live Tables, Azure Event Hubs
The Team
AI and Engineering at Deloitte harness cutting-edge capabilities to design, deploy, and operate integrated sector solutions across software, data, AI, networks, and hybrid cloud infrastructure. These efforts empower clients to transform mission-critical operations and modernize technology and data platforms, with delivery models tailored to meet each client’s needs.
Additional Requirements
Information for applicants with a need for accommodation: join Deloitte — accommodation for disabled applicants.
