Structured Streaming Data Jobs

1 job found

1d ago

Sr Databricks Data Engineer

Deloitte Pittsburgh, PA

Pittsburgh, PA

1d ago

$116k - $229k/yr Full time Apache Airflow Automation Big Data

Apply Now

Job Description

Senior Databricks Data Engineer within Deloitte AI and Data practice, accountable for designing, building, and optimizing cloud-based data engineering solutions on Databricks to modernize data platforms, enable analytics and AI, and drive measurable business outcomes. This role is onsite in Pittsburgh, PA, with a salary range of USD 116,200 to 229,100 per year.

Responsibilities

Establish and advocate leading practices for data architecture, integration, and modeling, documenting standards and promoting adherence across teams.
Own the end-to-end design, development, and maintenance of robust data pipelines and architectures to support enterprise-scale data needs.
Lead initiatives to enhance data quality, streamline operations, and scale data processes.
Assess, pilot, and integrate new big data and analytics technologies to keep the organization at the cutting edge; provide leadership and mentorship to data engineers and architects to support growth and project success.
Advise on and implement governance, security, and compliance strategies tailored to cloud-based data ecosystems.
Translate technical concepts and business value for stakeholders across the organization, including executives, business leads, and technology teams.
Guide the adoption of CI/CD practices using tools such as Azure DevOps, AWS CodePipeline, Jenkins, TFS, and PowerShell to streamline deployments and operations.

Requirements

Bachelor's degree in Computer Science, Engineering, or a related field
More than five years of hands-on data engineering experience focusing on Databricks across AWS, Azure, or Google Cloud Platform (GCP)
Proficiency with Lakehouse architectures, Apache Spark, Delta Lake, cloud-native databases and storage solutions, and distributed compute platforms
Experience with data warehousing and 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
At least one year leading complex, cross-functional data projects and technical teams, including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources
Ability to travel approximately 50 percent, depending on client engagements
Limited immigration sponsorship may be available
Master's degree in Computer Science, Engineering, or a related field
Experience with one or more cloud ecosystems (AWS, Azure, GCP) and associated big data services
Experience tuning and optimizing performance in Databricks and Apache Spark environments
Experience with Databricks Lakeflow
Experience with artificial intelligence and machine learning solutions

Technologies

Databricks
Amazon Web Services (AWS)
Microsoft Azure
Google Cloud Platform (GCP)
Apache Spark
Delta Lake
Unity Catalog
Delta Live Tables
Autoloader
Structured Streaming
Databricks Workflows
Apache Airflow
Azure DevOps
AWS CodePipeline
Jenkins
TFS
PowerShell
PySpark
Databricks Lakeflow

Benefits

Discretionary annual incentive program
Benefits package aligned with Core Talent Model

Qualifications Required

Bachelor's degree in Computer Science, Engineering, or a related field
Five or more years of hands-on data engineering experience with a focus on Databricks on AWS, Azure, or GCP
Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
Experience with data warehousing, 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
At least one year leading complex, cross-functional data projects and technical teams, including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources
Ability to travel approximately 50 percent
Limited immigration sponsorship may be available

Preferred

Master's degree in Computer Science, Engineering, or a related field
Experience with one or more cloud ecosystems (AWS, Azure, GCP) and their big data services
Experience tuning and optimizing performance in Databricks and Apache Spark environments
Experience with Databricks Lakeflow
Experience with artificial intelligence and machine learning solutions

View full job page →

Get Structured Streaming Data Jobs alerts

New Structured Streaming Data Jobs, daily.