Senior Data Engineer

Vytwo is seeking a Senior Data Engineer to design and maintain scalable data pipelines and improve Databricks and Spark workloads so downstream analytics teams receive reliable, analytics-ready datasets. This role is based in Minneapolis, MN with onsite work.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark with PySpark and/or Scala.
• Build and optimize data workflows on Databricks, including Delta Lake, notebooks, and scheduled jobs.
• Ingest, transform, and curate large-scale structured and semi-structured datasets.
• Conduct performance tuning and cost optimization for Spark workloads and Databricks clusters.
• Implement data quality checks, monitoring, and error handling practices.
• Collaborate with analytics and business stakeholders to deliver well-modeled, analytics-ready data.
• Support batch processing and, where applicable, streaming data pipelines.
• Apply best practices for testing, documentation, security, and version control.

Required Qualifications

• 5+ years of experience in Data Engineering or a related role.
• Strong hands-on experience with Apache Spark (PySpark or Scala).
• Proven experience working in Databricks environments.
• Strong SQL skills, including experience with relational and analytical databases.
• Experience building and maintaining ETL/ELT pipelines at scale.
• Familiarity with modern data lake architectures; Delta Lake preferred.
• Experience using Git-based version control.
• Must be a US Citizen.

Technology Stack

• Apache Spark, PySpark, Scala
• Databricks, Delta Lake
• Git, SQL

Location and Eligibility

Minneapolis, MN (onsite). Must be currently located in Minnesota (MN), or willing to relocate. Must be a U.S. Citizen.

Benefits

• Flexible work from home options available.