Emergent Staffing seeks a Senior Data Engineer with a strong focus on Azure Databricks to design and scale a medallion-based data platform. The role centers on PySpark, Delta Lake, and SQL within Databricks, building scalable data pipelines and governance layers while collaborating closely with analytics teams and business stakeholders. This is a 6+ month contract in Bloomington, MN with a hybrid in-office schedule (Tue/Thu) and US work authorization required.
Responsibilities
- Design, develop, and optimize data pipelines in Azure Databricks using PySpark and SQL, applying Delta Lake and Unity Catalog best practices.
- Build modular, reusable libraries and utilities within Databricks to accelerate development and standardize workflows.
- Implement Medallion architecture (Bronze, Silver, Gold layers) for scalable, governed data zones.
- Integrate external data sources via REST APIs, SFTP file delivery, and SQL Server Managed Instance, implementing validation, logging, and schema enforcement.
- Utilize parameter-driven jobs and manage compute using Spark clusters and Databricks serverless.
- Collaborate with data analytics teams and business stakeholders to understand requirements and deliver analytics-ready datasets.
- Monitor and troubleshoot Azure Data Factory (ADF) pipelines (jobs, triggers, activities, data flows) to identify and resolve job failures and data issues.
- Automate deployments and manage code using Azure DevOps for CI/CD, version control, and environment management.
- Contribute to documentation, architectural design, and continuous improvement of data engineering best practices.
- Support the design and readiness of the data platform for AI and machine learning initiatives.
Requirements
- Strong expertise with Azure Databricks, including PySpark, Delta Lake, Unity Catalog, and the ability to build reusable libraries, utility notebooks, and parameterized jobs.
- Advanced SQL skills with experience working in Azure SQL Database and/or SQL Server Managed Instance.
- Experience designing, troubleshooting, and supporting data pipelines using Azure Data Factory.
- Proven ability to integrate external data sources, including REST APIs and SFTP.
- Working knowledge of Azure DevOps for CI/CD, version control, and parameterized deployments.
- Demonstrated experience partnering closely with data analytics teams and business stakeholders, supported by strong communication, problem-solving, and collaboration skills.
- Interest or experience in preparing data platforms to support AI and machine learning initiatives.
Technologies
Azure Databricks, PySpark, Delta Lake, Unity Catalog, SQL, SQL Server Managed Instance, Azure SQL Database, REST APIs, SFTP, Spark, Databricks serverless, Azure Data Factory, Azure DevOps, Power BI
Vetting process
- Application (5 minutes)
- Online Assessment (40 minutes)
- Initial Phone Interview (30-45 minutes)
- Virtual Interview with Hiring team
- Onsite Interview
- Job Offer!
Location
Bloomington, MN with a hybrid schedule (in-office Tue/Thu). This is a contract role requiring US work authorization.
Compensation
Hourly rate: USD 60 - 85 per hour
