Senior Data Engineer (Azure & Databricks)

Emergent Staffing is seeking a Senior Data Engineer with Azure Databricks expertise to design, build, and optimize a medallion based data platform and pipelines. This contract role is hybrid in Minneapolis, MN, and spans 6+ months. You will work with Delta Lake, Unity Catalog, and cloud data services to deliver analytics ready datasets in collaboration with analytics teams and business stakeholders.

This position emphasizes a collaborative culture and practical impact, with a focus on scalable architecture, governed data zones, and readiness for AI and machine learning initiatives. You'll automate deployments with Azure DevOps, integrate external data sources via REST APIs and SFTP, and support data factory pipelines, shaping data services used across the organization.

Responsibilities

• Design, develop, and optimize data pipelines in Azure Databricks using PySpark and SQL, applying Delta Lake and Unity Catalog best practices.
• Build modular, reusable libraries and utilities within Databricks to accelerate development and standardize workflows.
• Implement Medallion architecture (Bronze, Silver, Gold) for scalable, governed data zones.
• Ingest external data sources via REST APIs, SFTP file delivery, and SQL Server Managed Instance, with validation, logging, and schema enforcement.
• Utilize parameter driven jobs and manage compute using Spark clusters and Databricks serverless.
• Collaborate with data analytics teams and business stakeholders to understand requirements and deliver analytics ready datasets.
• Monitor and troubleshoot Azure Data Factory pipelines (jobs, triggers, activities, data flows) to identify and resolve failures and data issues.
• Automate deployments and manage code using Azure DevOps for CI/CD, version control, and environment management.
• Contribute to documentation, architectural design, and continuous improvement of data engineering best practices.
• Support the design and readiness of the data platform for AI and machine learning initiatives.

Requirements

• Strong expertise with Azure Databricks, including PySpark, Delta Lake, Unity Catalog, and the ability to build reusable libraries, utility notebooks, and parameterized jobs.
• Advanced SQL skills with experience working in Azure SQL Database and/or SQL Server Managed Instance.
• Experience designing, troubleshooting, and supporting data pipelines using Azure Data Factory.
• Proven ability to integrate external data sources, including REST APIs and SFTP.
• Working knowledge of Azure DevOps for CI/CD, version control, and parameterized deployments.
• Demonstrated experience partnering closely with data analytics teams and business stakeholders, supported by strong communication, problem solving, and collaboration skills.
• Interest or experience in preparing data platforms to support AI and machine learning initiatives.

Technologies

• Azure Databricks
• PySpark
• Delta Lake
• Unity Catalog
• SQL
• Azure SQL Database
• SQL Server Managed Instance
• Azure Data Factory
• Spark
• Azure DevOps
• REST APIs
• SFTP
• Power BI

Our Vetting Process

1. Application (5 minutes)
2. Online Assessment (40 minutes)
3. Initial Phone Interview (30-45 minutes)
4. Virtual Interview with Hiring team
5. Onsite Interview
6. Job Offer