Mayo Clinic is recognized across more specialties than any other care provider by U.S. News & World Report. In Rochester, Minnesota, a Lead Data Engineer is being sought to advance the Enterprise Data & Analytics program with a blend of hands-on data engineering and technical leadership, while offering remote work options. This role emphasizes designing production-grade data pipelines, shaping platform capabilities, and guiding engineers through complex technical decisions.
Responsibilities
- Lead the design, prototyping, and development of data pipeline architectures.
- Drive internal process improvements by automating manual tasks, optimizing data delivery, and re-designing infrastructure for greater scalability.
- Perform cause analysis on internal and external data processes to identify opportunities for improvement.
- Apply advanced analytics to unstructured datasets and contribute to architectural discussions.
- Understand the overall architecture, participate in technical discussions, and communicate outcomes effectively as a senior individual contributor within the Data or Software Engineering teams.
- Serve on the Technical Review Board alongside the Manager and Principal Engineer and act as a liaison among management, software engineers, and principal engineers.
- Collaborate with software engineers to analyze, develop, and test functional requirements.
- Act as a hands-on technical leader who designs, develops, reviews, and optimizes production-grade data pipelines, data products, and platform capabilities.
- Maintain substantial contributions to production codebases while establishing engineering standards, mentoring teammates, and delivering scalable, resilient solutions.
- Mentor and coach engineers; explore design options, prototype new technologies, and evaluate technical feasibility.
- Operate within an Agile/Safe/Scrum environment to deliver high-quality software.
- Establish architectural principles and design patterns, and guide team members in their usage.
- Facilitate communication across front-end, back-end, data, and platform engineers.
- Assume a formal Engineering lead role within the area of expertise and stay current with industry trends and developments.
Requirements
- Bachelor’s degree in Computer Science/Engineering or related field with 6+ years of experience, or an Associate’s degree with 8+ years of experience.
- Knowledge of professional software engineering practices and the full software development lifecycle, including coding standards, code reviews, source control, build processes, testing, and operations.
- At least 5 years of experience in data engineering, data science, or analytical modeling, with foundational knowledge of related disciplines.
- Experience leading Data Engineering teams in Continuous Integration / Continuous Delivery models; building and leading highly resilient data products.
- Experience building test automation, unit testing coverage, data quality practices, monitoring, and observability.
- Minimum 5 years of experience with relational and NoSQL databases.
- Hands-on experience with cloud platforms (GCP, Azure, AWS) and cloud-based development and deployment.
- Proficiency with CI pipelines (Jenkins, GitHub Actions, or Azure Pipelines).
- Familiarity with tools such as Jira, GitHub, SharePoint, and Azure Boards.
- Experience using advanced data processing technologies such as Apache Spark, Hive, Airflow, Kafka, and GCP Dataflow.
- Knowledge of big data concepts, statistics, and data aspects of machine learning.
- Experience with Google BigQuery, FHIR APIs, and Vertex AI.
- Understanding of workflow scheduling solutions like Apache Airflow and Google Composer.
- Experience with Infrastructure as Code in a cloud environment (Kubernetes, Docker).
Technologies
- Python
- SQL
- Apache Spark
- Apache Flink
- Ray
- Hive
- Airflow
- Kafka
- Google Cloud Platform (GCP)
- BigQuery
- Vertex AI
- FHIR APIs
- Delta Lake
- Apache Iceberg
- Apache Hudi
- Parquet
- Avro
- ORC
- Jenkins
- GitHub Actions
- Azure Pipelines
- Jira
- SharePoint
- Azure Boards
- Kubernetes
- Docker
- Google Dataflow
- Google Composer
- AWS
Benefits
- Medical: Multiple plan options.
- Dental: Delta Dental or reimbursement account for flexible coverage.
- Vision: Affordable plan with national network.
- Pre-Tax Savings: HSA and FSAs for eligible expenses.
- Retirement: Competitive retirement package to secure your future.
Why Mayo Clinic
Mayo Clinic is consistently ranked highly across many specialties by U.S. News & World Report. The organization emphasizes patient care while investing in competitive compensation, comprehensive benefits, and opportunities for continuing education and career advancement to support a long, successful tenure.
Site Description
Locations extend beyond Minnesota to three major campuses: Phoenix/Scottsdale, AZ; Jacksonville, FL; and Rochester, MN, along with Mayo Clinic Health System campuses throughout the Midwest and international locations. Each Mayo Clinic site aims to provide a workplace that supports both professional growth and personal well-being, with details available about what each campus offers.
Equal Opportunity
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender identity, sexual orientation, national origin, protected veteran status or disability status. Mayo Clinic participates in E-Verify and may provide information from Form I-9 to confirm work authorization where necessary.
Recruiter
Recruiter: Laura Percival
