Data Engineer II

Essentia Health is seeking a Data Engineer II to help design and optimize scalable, complex data pipelines and workflows for large datasets. The role supports advanced data processing and scheduling, with emphasis on data quality, governance, security, and delivering batch and near real-time analytics that inform business-critical decisions.

Role focus

• Design and optimize scalable, robust data pipelines and workflows that meet functional and non-functional business requirements
• Implement advanced data processing and scheduling approaches for large-scale datasets
• Enable data-driven decision-making through batch and near real-time analytics
• Develop infrastructure to support real-time and batch processing workflows
• Deliver actionable data products and insights for key business tools

Key responsibilities

• Build and optimize pipelines and workflows to handle large datasets reliably
• Use frameworks such as Apache Spark, Azure Data Factory (ADF), and Apache Airflow to implement processing and scheduling
• Write and optimize advanced SQL queries for reporting and application needs
• Ensure data quality, governance, and security using automated testing and monitoring solutions
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements
• Provide technical guidance and mentorship to junior data engineers

Required qualifications

• Bachelor’s degree in computer science, data analytics, statistics, or a related field
• 2 to 5 years of experience in data engineering or a related role
• Proficiency in SQL and experience with database management systems
• Strong programming skills in Python, Java, or Scala
• Experience with pipelines using orchestration tools such as Apache Airflow or AWS Glue for ingestion and delivery
• Experience with modern data management technologies such as data lake, lakehouse, and data catalogs
• Experience supporting both batch and streaming data to enable near real-time analytics and reporting

Technology focus

• Apache Spark, ADF, Airflow, SQL
• Python, Java, Scala
• AWS Glue
• Data lake, lakehouse, data catalogs
• dbt, Microsoft Fabric
• AI and machine learning (for relevant data use cases)

Preferred qualifications

• Familiarity applying AI and machine learning techniques to data use cases
• Experience creating robust data transformation workflows, including metadata management, dependency tracking, and workload orchestration
• Experience with dbt
• Experience with Microsoft Fabric
• Excellent problem-solving skills and attention to detail
• Strong communication and collaboration skills

Compensation and schedule

• Compensation range: $89,523.20 - $134,284.80 per year
• Shift rotation: Day Rotation (United States of America)
• Shift time: 8:00 AM - 4:30 PM
• Holidays: No
• Call obligation: No

Location and logistics

• Location: Duluth, MN (Remote)
• Building location: Peerless Building
• Department: 1007670 ANALYTICS AND ARCHITECTURE - EH SS
• FTE: 1
• Possible remote option: Remote

Benefits

• Medical, dental, vision, life, and disability insurance
• 401(k) plan with employer contributions
• Training, tuition reimbursement, and educational programs
• Flexible scheduling
• Generous time off
• Wellness resources focused on physical, mental, and emotional health

Minimum experience: 2 years. Education: Bachelor’s degree in computer science, data analytics, statistics, or related field.