DataJobs.io
← Back to all jobs

Job Description

Own enterprise-grade lakehouse data engineering on Azure, building and optimizing scalable ETL/ELT pipelines with Azure Databricks.

Responsibilities

  • Design and build scalable ETL/ELT pipelines using Azure Databricks with PySpark and Spark SQL
  • Develop ingestion workflows from multiple source systems into Bronze/Silver/Gold layers (e.g., Event Hub, APIs, Azure SQL, Blob Storage)
  • Apply best practices for Delta Lake and Unity Catalog, including schema evolution, versioning, and governance
  • Optimize pipeline performance and cost using partitioning, caching, clustering, and Spark tuning
  • Integrate Databricks with key Azure services including ADLS Gen2, Azure Data Factory (ADF), Azure Synapse, Event Hub, Key Vault, and Azure DevOps
  • Implement secure access controls using managed identities, RBAC, and data protection practices such as masking
  • Collaborate with analytics and business teams to deliver trusted datasets, metrics, and KPIs
  • Provide technical leadership through mentoring, code reviews, and architectural guidance
  • Work with enterprise architecture and security teams to align data solutions with data privacy/security policies
  • Implement data quality checks, alerting, and lineage tracking using tools such as Great Expectations, Unity Catalog, and data observability platforms
  • Build monitoring dashboards, alerts, and recovery mechanisms for critical pipelines

Requirements

  • Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
  • 5+ years of data engineering experience
  • 3+ years with Azure Databricks / Apache Spark using PySpark
  • Strong experience with Delta Lake, Unity Catalog, Databricks notebooks, and SQL-based warehousing patterns
  • Hands-on experience with ADLS Gen2, Event Hub, ADF, Key Vault, and Synapse
  • Strong coding skills in Python and SQL; Scala is a plus
  • Understanding of data modeling for OLTP/OLAP, star/snowflake schemas, and performance tuning
  • Experience implementing CI/CD for Databricks using Databricks CLI, Terraform, and/or Azure DevOps
  • Databricks certification (e.g., Databricks Certified Data Engineer Professional)
  • Experience with streaming/real-time pipelines using Structured Streaming and Event Hub (Kafka experience is referenced)
  • ML platform exposure (e.g., MLflow or supporting ML pipelines)
  • Reporting/consumption layer experience such as Power BI, Purview, or Microsoft Fabric
  • Knowledge of governance/compliance standards including GDPR, HIPAA, and SOC2
  • Must currently reside in the Dallas–Fort Worth (DFW) area and be able to work a hybrid schedule in Plano, TX
  • Relocation is not available
  • Candidates must be authorized to work in the United States without sponsorship now or in the future

Technologies

  • Azure Databricks
  • PySpark, Spark SQL
  • Delta Lake, Unity Catalog
  • Python, SQL, Scala
  • ADLS Gen2
  • Azure Data Factory (ADF)
  • Azure Synapse
  • Event Hub
  • Key Vault, Azure DevOps
  • Managed identities, RBAC, masking
  • Great Expectations
  • Data observability platforms
  • Databricks notebooks, Databricks CLI
  • Terraform
  • Structured Streaming, Kafka
  • MLflow
  • Power BI, Purview, Microsoft Fabric

Benefits

  • Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
  • Health Savings Account (HSA) (for employees working 20+ hours per week)
  • Life & Disability Insurance (for employees working 20+ hours per week)
  • MetLife Voluntary Benefits
  • Employee Assistance Program (EAP)
  • 401K Retirement Savings Plan
  • Direct Deposit & weekly epayroll
  • Referral Bonus Programs
  • Certification and training opportunities

Location: Plano, TX (onsite)
Compensation: USD 105,000 - 150,000 per year

Similar Jobs