Senior Azure Data Engineer
Senior
Azure
Azure Data Engineer
Azure Data Factory
Azure Data Lake
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Platform
Data Architecture
Data Engineer
Data Engineering
Data Factory
Data Factory Azure
Data Governance
Data Integration
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Security
Databricks
ETL
Microsoft Azure
Spark
SQL
Job Description
Own enterprise-grade lakehouse data engineering on Azure, building and optimizing scalable ETL/ELT pipelines with Azure Databricks.
Responsibilities
- Design and build scalable ETL/ELT pipelines using Azure Databricks with PySpark and Spark SQL
- Develop ingestion workflows from multiple source systems into Bronze/Silver/Gold layers (e.g., Event Hub, APIs, Azure SQL, Blob Storage)
- Apply best practices for Delta Lake and Unity Catalog, including schema evolution, versioning, and governance
- Optimize pipeline performance and cost using partitioning, caching, clustering, and Spark tuning
- Integrate Databricks with key Azure services including ADLS Gen2, Azure Data Factory (ADF), Azure Synapse, Event Hub, Key Vault, and Azure DevOps
- Implement secure access controls using managed identities, RBAC, and data protection practices such as masking
- Collaborate with analytics and business teams to deliver trusted datasets, metrics, and KPIs
- Provide technical leadership through mentoring, code reviews, and architectural guidance
- Work with enterprise architecture and security teams to align data solutions with data privacy/security policies
- Implement data quality checks, alerting, and lineage tracking using tools such as Great Expectations, Unity Catalog, and data observability platforms
- Build monitoring dashboards, alerts, and recovery mechanisms for critical pipelines
Requirements
- Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
- 5+ years of data engineering experience
- 3+ years with Azure Databricks / Apache Spark using PySpark
- Strong experience with Delta Lake, Unity Catalog, Databricks notebooks, and SQL-based warehousing patterns
- Hands-on experience with ADLS Gen2, Event Hub, ADF, Key Vault, and Synapse
- Strong coding skills in Python and SQL; Scala is a plus
- Understanding of data modeling for OLTP/OLAP, star/snowflake schemas, and performance tuning
- Experience implementing CI/CD for Databricks using Databricks CLI, Terraform, and/or Azure DevOps
- Databricks certification (e.g., Databricks Certified Data Engineer Professional)
- Experience with streaming/real-time pipelines using Structured Streaming and Event Hub (Kafka experience is referenced)
- ML platform exposure (e.g., MLflow or supporting ML pipelines)
- Reporting/consumption layer experience such as Power BI, Purview, or Microsoft Fabric
- Knowledge of governance/compliance standards including GDPR, HIPAA, and SOC2
- Must currently reside in the Dallas–Fort Worth (DFW) area and be able to work a hybrid schedule in Plano, TX
- Relocation is not available
- Candidates must be authorized to work in the United States without sponsorship now or in the future
Technologies
- Azure Databricks
- PySpark, Spark SQL
- Delta Lake, Unity Catalog
- Python, SQL, Scala
- ADLS Gen2
- Azure Data Factory (ADF)
- Azure Synapse
- Event Hub
- Key Vault, Azure DevOps
- Managed identities, RBAC, masking
- Great Expectations
- Data observability platforms
- Databricks notebooks, Databricks CLI
- Terraform
- Structured Streaming, Kafka
- MLflow
- Power BI, Purview, Microsoft Fabric
Benefits
- Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
- Health Savings Account (HSA) (for employees working 20+ hours per week)
- Life & Disability Insurance (for employees working 20+ hours per week)
- MetLife Voluntary Benefits
- Employee Assistance Program (EAP)
- 401K Retirement Savings Plan
- Direct Deposit & weekly epayroll
- Referral Bonus Programs
- Certification and training opportunities
Location: Plano, TX (onsite)
Compensation: USD 105,000 - 150,000 per year