Senior Data Engineer

Optum is seeking a Senior Data Engineer to design, build, and operate scalable data architectures and governed pipelines that support healthcare data solutions. This role emphasizes ETL/ELT engineering, Azure and Databricks-based platforms, production operations, and HIPAA-aligned handling of sensitive data.

Key Responsibilities

• Lead the design and delivery of scalable, governed data engineering solutions, providing technical direction, implementation planning, risk management, and production readiness for data products used by business, analytics, and AI/ML stakeholders.
• Develop and maintain robust ETL/ELT pipelines to ingest, transform, and integrate data from heterogeneous sources, including relational databases, APIs, file shares, and end-user-computing assets.
• Build distributed data processing solutions using Python and PySpark, including reusable transformation logic and automation components.
• Optimize data platform performance and cloud usage using SQL tuning, Spark job optimization, storage format selection, partitioning strategies, cluster configuration, and workload scheduling.
• Develop and optimize workflows on Azure and Databricks, including medallion layering (bronze/silver/gold) and orchestrated pipelines to support enterprise data needs.
• Migrate business logic from legacy and end-user-computing systems into documented, testable transformation logic operating over governed tables.
• Define and implement data quality, validation, reconciliation, and parity testing practices to ensure accuracy, reliability, auditability, and consistency with replacement systems.
• Establish data lineage, ownership, and cataloguing, including documented rollup and allocation rules where source systems operate at different grains.
• Implement security and compliance controls for PHI/PII and confidential business data in accordance with HIPAA or similar regulations, including de-identification and masking for controlled sharing.
• Build and improve CI/CD, version control, and automated testing practices for data pipelines.
• Implement monitoring, alerting, logging, and operational runbooks to support reliability, timely issue detection, incident response, and continuous improvement of production platforms.
• Support AI/ML and advanced analytics workflows by provisioning trusted data and scalable, governed consumption patterns for data scientists, analysts, and business partners.
• Design and publish reusable data products, data services, APIs, and consumption layers to enable consistent downstream use for applications, analytics platforms, and AI/ML.
• Partner with product owners, business stakeholders, architects, data scientists, analysts, and engineering teams to translate requirements into scalable solutions and communicate technical tradeoffs.
• Design, develop, and deploy AI-enabled data solutions that address complex business challenges and support responsible AI use, including leveraging AI-powered development tools to improve engineering productivity and delivery quality.
• Mentor data engineers and analysts through design reviews, code reviews, best-practice guidance, and support to improve team delivery quality.

Required Qualifications

• 6+ years of overall experience in data engineering, data warehousing, distributed data processing, or enterprise data platform delivery.
• 6+ years of experience with SQL, data modeling, and performance tuning for large-scale systems.
• 3+ years of experience with ETL/ELT frameworks, pipeline orchestration, and integration across heterogeneous source systems.
• 3+ years of experience with Python for data processing, automation, and integration.
• 3+ years of experience with Databricks, Spark/PySpark, and Delta Lake.
• 3+ years of experience with public cloud platforms such as Azure or GCP.
• 1+ year of experience integrating data from heterogeneous source systems.
• 1+ year of experience with data quality, validation, reconciliation, observability, and production support practices for large-scale data pipelines.
• 1+ year of experience with Git, CI/CD platforms, automated testing frameworks, release management, and deployment practices for production data pipelines.
• 1+ year of experience with AI-powered development tools used responsibly to improve engineering productivity and delivery quality.

Technologies

• Python, PySpark, SQL, Databricks, Spark, Delta Lake
• Azure, GCP
• ETL, ELT
• Git, CI/CD
• HIPAA
• Azure Data Factory, Airflow, dbt
• Unity Catalog, Microsoft Purview
• APIs

Preferred Qualifications

• Experience with batch and near-real-time patterns, including event-driven ingestion, streaming pipelines, or comparable technologies.
• Experience with Azure Data Factory or comparable orchestration tooling such as Airflow, dbt, or similar.
• Experience migrating legacy or end-user-computing (spreadsheet- or macro-based) business logic into governed data pipelines.
• Experience with data governance, lineage, and catalog tooling (Unity Catalog, Microsoft Purview, or similar).
• Familiarity with AI/ML frameworks and supporting infrastructure for data science workflows.
• Experience working with PHI/PII and restricted client data in compliance with HIPAA or similar regulations, including de-identification and masking.
• Healthcare payer, actuarial, pricing, or financial-modelling domain experience.

Compensation

Salary for this role ranges from $91,700 to $163,700 annually based on full-time employment, with pay based on several factors.

Benefits

• Comprehensive benefits package
• Incentive and recognition programs
• Equity stock purchase
• 401k contribution

Work Arrangement

• Remote work from anywhere within the U.S.
• For hires in the Minneapolis or Washington, D.C. area, required to work in the office a minimum of four days per week.

Additional Information

• Application deadline: posted for a minimum of 2 business days or until a sufficient candidate pool has been collected; the job posting may come down early due to volume of applicants.
• Drug-free workplace: candidates are required to pass a drug test before beginning employment.

Location

Eden Prairie, MN (remote)

Experience Level

Minimum experience: 6 years