Senior Data Engineer

Senior Data Engineer in Miami, FL (onsite) at Deloitte, with a salary range of USD 95,000 - 150,000 per year and a Bachelor's degree requirement, focusing on end-to-end data solutions and ETL/ELT pipelines using Azure Data Factory and Databricks, including governance, design, development, optimization, and mentoring within the Project Delivery Model.

Responsibilities

• Maintain regular communication with Engagement Managers, project teams, and cross-functional stakeholders, escalating issues that need engagement management input.
• Design, develop, and optimize ETL and ELT pipelines leveraging Azure Data Factory and Databricks.
• Author and tune PySpark and Spark SQL notebooks to handle large-scale data transformations.
• Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog.
• Steer design discussions with client architects and other counterparts to align on technical direction.
• Collaborate with multiple teams on data contracts and schema agreements to ensure interoperability.
• Lead the design and optimization of high-volume data pipelines.
• Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
• Drive performance improvements through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
• Design Databricks cluster policies, autoscaling settings, and cost-optimization strategies.
• Perform root cause analysis on production incidents and implement permanent fixes.
• Mentor junior and mid-level engineers via code reviews and pair programming.
• Evaluate new technologies and advise on adoption, such as Delta Apps (DABs), Delta Live Tables (DLT), Auto Loader, Serverless Compute, and Event Hubs.

Requirements

• Python, PySpark, Spark SQL, and SQL Server proficiency.
• Experience with Azure services including Data Factory, Data Lake Storage Gen2, Key Vault, and Azure Monitor.
• Hands-on work with Databricks components such as Delta Lake, Unity Catalog, and Workflows.
• Apache Airflow for workflow orchestration.
• Git and Azure DevOps for version control and CI/CD.
• Deep understanding of Spark internals, including DAG optimization, spill analysis, and skew handling.
• Advanced Delta Lake features such as time travel, deletion vectors, and predictive I/O.
• Unity Catalog governance covering row and column security, external locations, and system tables.
• Infrastructure as code experience with Terraform and Azure ARM templates.
• Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or related IT discipline, or equivalent experience.
• Limited immigration sponsorship may be available.
• Ability to travel about 10 percent on average based on client needs and engagements.

Technologies

• Python
• PySpark
• Spark SQL
• SQL Server
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2
• Key Vault
• Azure Monitor
• Databricks
• Delta Lake
• Unity Catalog
• Workflows
• Apache Airflow
• Git
• Azure DevOps
• Terraform
• Azure ARM templates
• DABs
• Delta Live Tables (DLT)
• Auto Loader
• Serverless Compute
• Event Hubs