Sr. Azure Data Engineer with Databricks

Build and optimize enterprise Azure and Databricks data platforms in a hybrid role based in Philadelphia.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for enterprise data and analytics initiatives
• Build data ingestion, transformation, and orchestration frameworks using Azure-native technologies
• Develop and optimize data processing solutions using Databricks and PySpark
• Implement and maintain Delta Lake and Lakehouse architectures
• Ensure data quality, accuracy, lineage, and reliability across data assets
• Perform root cause analysis and troubleshoot data-related issues
• Work on solutions leveraging:
  • Azure Databricks
  • Azure Data Lake Storage (ADLS Gen2)
  • Azure Data Factory (ADF)
  • Azure Synapse Analytics
  • Azure SQL Database
• Support data migration, modernization, and cloud transformation efforts
• Optimize performance, scalability, and cost efficiency for cloud-based data platforms
• Build and maintain logical and physical data models
• Create curated data layers for BI, reporting, and advanced analytics
• Partner with business users and data consumers to translate requirements into scalable technical solutions
• Implement CI/CD processes for data engineering solutions and support automated deployment and release management
• Develop monitoring, logging, and observability capabilities for data pipelines
• Follow enterprise data governance and security standards
• Participate in Agile/Scrum ceremonies including sprint planning, daily stand-ups, retrospectives, and backlog refinement
• Collaborate with Product Owners, Architects, QA teams, and business stakeholders
• Contribute to continuous improvement and engineering best practices

Requirements

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field
• 7+ years of experience in Data Engineering
• 5+ years hands-on experience with Azure data technologies
• 2+ years experience in Healthcare, Managed Care, Health Plans, or Payer organizations
• Strong expertise in:
  • Azure Databricks
  • Azure Data Lake Storage (ADLS)
  • Azure Data Factory (ADF)
  • PySpark
  • Python
  • SQL
  • Delta Lake and Data Modeling
• Extensive experience building enterprise-scale data pipelines and data integration solutions
• Strong understanding of Data Warehousing, Data Lake, and Lakehouse concepts
• Experience with large volumes of structured and unstructured data
• Experience with Git, CI/CD, and DevOps practices
• Strong analytical, troubleshooting, and problem-solving skills

Required Technical Skills

• Must Have: Azure Databricks
• Must Have: Azure Data Lake (ADLS Gen2)
• Must Have: Azure Data Factory (ADF)
• Must Have: PySpark
• Must Have: Python
• Must Have: SQL
• Must Have: Delta Lake
• Must Have: Data Modeling
• Must Have: ETL/ELT Development
• Must Have: Azure Cloud Platform
• Must Have: Git
• Must Have: CI/CD
• Must Have: Agile/Scrum
• Skills: Azure Synapse Analytics, Azure SQL Database

Preferred Qualifications

• Experience working with: Claims Data, Membership Data, Provider Data, Care Management Data
• Experience with Power BI or other reporting tools
• Familiarity with FHIR, HL7, or healthcare interoperability standards
• Experience with Azure DevOps and Infrastructure as Code (IaC)
• Exposure to AI/ML data pipelines and advanced analytics environments

Location & Work Model

• Philadelphia, PA (hybrid)
• Onsite presence required at least three days per week

Technology Stack

• Azure, Databricks, Azure Databricks, PySpark, Python, SQL
• Delta Lake and Lakehouse architectures, ETL/ELT
• Azure Data Lake Storage (ADLS Gen2), Azure Data Lake Storage (ADLS)
• Azure Data Factory (ADF), Azure Synapse Analytics, Azure SQL Database
• Git, CI/CD, Agile/Scrum, Data Modeling