Data Engineer

AmeriHealth Caritas is hiring an Azure Data Engineer to design, build, and optimize large-scale cloud data solutions on Microsoft Azure.

Responsibilities

• Design and implement scalable solutions using Azure Data Factory, Azure Databricks, Azure Event Hubs, Kafka, HVR, Snowflake, and Azure Data Lake Storage
• Build ETL/ELT pipelines for both batch and streaming workloads using PySpark and SQL, with fault tolerance, schema evolution, and idempotency
• Design and operate event-driven ingestion and processing with Event Hubs and Kafka; integrate CDC solutions using HVR
• Tune Databricks jobs and clusters to meet SLAs and cost targets (cluster sizing, autoscaling, caching, partitioning, shuffle optimization, and Delta Lake optimizations)
• Use Azure DevOps for backlog management, Git workflows, YAML pipelines, automated testing, and repeatable deployments across environments
• Conduct code reviews, enforce coding standards, and mentor junior developers to raise team capability
• Collaborate with product owners, data scientists, analysts, and architects to translate business needs into prioritized user stories and deliverables within Agile sprints
• Automate manual processes, optimize data delivery, and redesign infrastructure for greater scalability
• Identify and remediate issues in data management to improve data quality
• Apply data management best practices for master data, reference data, metadata, data quality, and data lineage

Requirements

• Bachelor’s degree in computer science, Information Systems, or Software Engineering
• 4+ years hands on experience with Azure Data Factory, Azure Databricks, Kafka, and Snowflake
• 2+ years hands on experience with HVR and Azure Event Hubs
• 8+ years experience with PySpark and SQL
• Expertise in dimensional modeling, SCD implementation, and data warehouse best practices
• Proven delivery of data projects in Agile teams (Scrum or Kanban)
• Practical experience with Azure DevOps for CI/CD, branching strategies, and pipeline automation
• Demonstrated ability to profile and optimize Databricks jobs and large-scale data processing workloads
• Track record of conducting code reviews and providing technical guidance
• Clear communication skills, able to present technical tradeoffs to technical and non-technical stakeholders
• Experience with Collibra data governance features
• Familiarity with MLOps patterns and integrating data platforms with ML pipelines
• Azure certifications such as DP 700, AZ 400, or Databricks Data Engineer certification
• Experience with cost optimization and observability tools for cloud data platforms
• Practical experience applying Gen AI to data engineering workflows, including LLMs, embeddings, and retrieval augmentation

Technologies

• Azure Data Factory, Azure Databricks, Azure Event Hubs, Kafka, HVR, Snowflake
• Azure Data Lake Storage, PySpark, SQL, Delta Lake
• Azure DevOps, Git, YAML
• CDC solutions, SCD, Dimensional modeling
• Scrum, Kanban
• Collibra, MLOps, ML pipelines
• Gen AI, LLMs, embeddings, retrieval augmentation
• DP 700, AZ 400, Databricks Data Engineer certification

Benefits

• Flexible work solutions including remote options and hybrid work schedules
• Competitive pay
• Paid time off including holidays and volunteer events
• Health insurance coverage for you and your dependents on Day 1
• 401(k)
• Tuition reimbursement
• more

Work Arrangement

• Hybrid, office based in Ellis Preserve office in Newtown Square, PA
• Core office days: Tuesday, Wednesday, Thursday

Remote Requirements (for roles that are 100% remote or hybrid)

• Minimum bandwidth requirement: 50 Mbps download and 5 Mbps upload
• Fully remote associates residing in states where service is required by contract, law, or regulation may submit for reimbursement

Minimum experience: 4 years