Databricks Workflows

Databricks Workflows Jobs

31 jobs found

1d ago

Data/Analytics Engineer

Public Health Foundation Enterprises, In Los Angeles, CA

Los Angeles, CA

1d ago

$10,214 - $13,376/mo Full time Manager Analytics Automation

Apply Now

Job Description

Public Health Foundation Enterprises, In is seeking a senior Data/Analytics Engineer to lead the Community Programs Databricks environment from its Los Angeles, CA campus. The role centers on end-to-end data architecture and governance, delivering a Medallion-based platform that unifies housing, justice, and clinical operations while creating scalable data products for program tracking, executive reporting, and predictive modeling. The position is onsite in Los Angeles with a monthly salary range of USD 10,214 to 13,376.

Responsibilities

Develop and scale a governed Databricks platform that serves as the authoritative data source across housing, justice, and clinical systems.
Architect and deploy automated, production-grade data pipelines and CI/CD workflows using Databricks, Terraform, and GitHub.
Translate complex program logic into scalable data models and reusable data assets.
Create scalable semantic models that map a client’s journey across multiple programs, standardizing touchpoints over time.
Institute secure-by-design data systems handling sensitive PHI and CJI data with Unity Catalog for centralized governance, audit logging, and access control.
Enable advanced analytics by delivering curated datasets, APIs, and environments for machine learning and NLP applications.
Collaborate with leadership and cross-functional teams to shape data usage for policy, operations, and outcomes.
Provide technical leadership by mentoring staff and establishing engineering standards for long-term scalability and maintainability.
Programmatically map complex, multi-source program utilization (CHAMP, DD, HMIS) to funding streams for strict fiscal and grant tracking compliance.
Design, implement, and maintain a scalable Databricks lakehouse using Medallion principles; build ETL/ELT pipelines that integrate structured and unstructured data across systems, and ensure interoperability with standards such as FHIR and HL7v2.
Automate data lifecycles with GitHub Actions and Terraform, applying software engineering practices like version control, automated testing, and continuous integration before production deployment.
Design and maintain the Unified Data Model for Community Programs, leading architectural efforts to stitch cross-program client journeys and develop multi-dimensional structures mapping utilization to funding streams.
Establish enterprise data governance, including RBAC/ABAC, data lineage, audit logs, and data loss prevention to safeguard regulated data.
Develop scalable data products, curated datasets, APIs, and analytical layers that support reporting, dashboards, and advanced analytics; optimize performance with SQL and BI tools.
Provide technical leadership and delivery oversight for data engineers, analysts, and scientists, promoting best practices in system design.
Build and maintain environments required for advanced analytics, partnering with data scientists and DHS Security teams to ensure models and data products are scalable and compliant.

Requirements

Significant experience in Enterprise Data Architecture.
Strong background in Cloud Infrastructure.
Extensive experience in Analytics Engineering.
A relevant bachelor’s degree.
Proven track record managing complex data lifecycles within large-scale Databricks environments.
Solid background in Cloud Data Engineering.
Experience with Infrastructure Automation.
Option I: Two years in a lead capacity delivering complex data infrastructure and architecture projects, including automated ETL/ELT pipelines, Lakehouse management (Databricks), and enterprise data security at a level equivalent to the Los Angeles County Principal Information Systems Analyst.
Option II: Bachelor’s degree in Information Technology, Computer Science, Data Engineering, or Data Science with six years applied experience in data engineering, infrastructure automation (CI/CD), and enterprise data management; two years in a lead role. A Master’s or Doctoral degree may substitute up to two years of general experience.
DHS Live Scan is required.
Expert proficiency in Python focusing on data manipulation with Pandas and PySpark, and automation.
Expert proficiency in SQL including query optimization, CTEs, and window functions.
Advanced data visualization skills for executive insights in Tableau or Power BI.
Cloud automation experience with Terraform and GitHub Actions.

Technologies

Databricks
Terraform
GitHub
GitHub Actions
Unity Catalog
SQL
Tableau
Power BI
Python
Pandas
PySpark

Physical Demands

Stand: Occasionally
Walk: Occasionally
Sit: Frequently
Handling / Fingering: Frequently
Reach Outward: Occasionally
Reach Above Shoulder: Occasionally
Climb, Crawl, Kneel, Bend: Occasionally
Lift / Carry: Occasionally up to 25 lbs
Push / Pull: Occasionally up to 25 lbs
See: Constantly
Taste / Smell: Not Applicable

Work Environment

General office setting with indoor climate control.

Selection Process

Live technical interview: shortlisted candidates will complete a live technical assessment to test SQL and Python proficiency, architectural problem solving within the Databricks ecosystem, and the ability to translate programmatic logic into scalable data models.

View full job page →

5d ago

Senior Data Engineer

Deloitte Austin, TX

Austin, TX

5d ago

$95k - $150k/yr Full time Senior Apache Airflow Azure Data Factory

Apply Now

Job Description

In Deloitte's Austin hub, the Senior Data Engineer collaborates on client engagements to design and optimize data pipelines and analytics solutions. The role centers on building ETL and ELT workflows, developing PySpark and Spark SQL notebooks, and leading data engineering efforts within the Project Delivery Model to enable scalable, data-driven outcomes.

Position details

Location: Austin, TX (onsite)
Salary: USD 95,000 - 150,000 per year
Education: Bachelor's degree

Responsibilities

Maintain regular communication with Engagement Managers (Directors), project teams, and representatives from multiple functional or technical groups, escalating issues as needed for management review
Design, build, and optimize ETL and ELT pipelines using Azure Data Factory and Databricks
Develop and refine PySpark and Spark SQL notebooks for large-scale data transformations
Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog
Lead design discussions with client architects and other counterparts
Collaborate with teams on data contracts and schema agreements
Lead the design and optimization of high‑volume data pipelines
Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning
Drive performance improvements through AQE tuning, clustering strategies, broadcast joins, and shuffle partition management
Design Databricks cluster policies, autoscaling configurations, and cost optimization approaches
Conduct root cause analyses on production incidents and implement durable fixes
Mentor junior and mid-level engineers through code reviews and pair programming
Evaluate new technologies and recommend adoption (for example, DABs, DLT, Auto Loader, Serverless Compute, and event hubs)

Requirements

Proficiency in Python, PySpark, Spark SQL, and SQL Server
Experience with Azure components such as Data Factory, ADLS Gen2, Key Vault, and Azure Monitor
Familiarity with Databricks technologies including Delta Lake, Unity Catalog, and Workflows
Knowledge of Apache Airflow
Git or Azure DevOps experience
Deep understanding of Spark internals, including DAG optimization, spill analysis, and skew handling
Delta Lake advanced features such as time travel, deletion vectors, and predictive I/O
Unity Catalog governance aspects including row/column security, external locations, and system tables
Infrastructure as code with Terraform and Azure ARM templates
Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or a related IT discipline, or equivalent experience
Limited immigration sponsorship may be available
Ability to travel approximately 10% on average, based on client and project needs

Technologies

Python, PySpark, Spark SQL, SQL Server
Azure: Data Factory (ADF), ADLS Gen2, Key Vault, Azure Monitor
Databricks: Delta Lake, Unity Catalog, Workflows
Apache Airflow
Git, Azure DevOps
Delta technologies and optimization: DABs, DLT, Auto Loader, Serverless Compute, event hubs
Infrastructure as code: Terraform, Azure ARM templates

View full job page →

5d ago

Data Engineer

Proteam Solutions Dublin, OH

Dublin, OH

5d ago

Full time Azure Databricks Big Data Bigdata

Apply Now

Job Description

Proteam Solutions is seeking a Senior Data Engineer to join onsite in Dublin, OH. This role focuses on designing, developing, and optimizing enterprise-scale data pipelines with Databricks, supporting analytics, reporting, and downstream applications. You will play a key part in building scalable Lakehouse architectures and ensuring reliable, high-quality data across critical data workflows. The position calls for deep Databricks expertise, strong data engineering fundamentals, and a collaborative approach with cross-functional teams.

Responsibilities

Design and build scalable ETL and ELT pipelines using Databricks
Develop PySpark and Spark SQL transformations
Construct Bronze, Silver, and Gold data layers following the Medallion Architecture
Implement Delta Live Tables, Workflows, and Unity Catalog
Optimize Spark jobs for scalability and performance
Design dimensional data models for analytical workloads
Implement data quality validations and monitoring
Collaborate with analytics, engineering, and architecture teams
Build reusable frameworks and automation
Support CI/CD deployment pipelines and Git-based development

Requirements

7–10+ years of Data Engineering experience
Extensive hands-on Databricks experience
Strong PySpark programming
Advanced Spark SQL skills
Experience with Delta Lake
Experience implementing Medallion Architecture
Hands-on Delta Live Tables
Unity Catalog experience
Strong SQL optimization and performance tuning
Experience with AWS, Azure, or GCP
CI/CD and Git experience
Strong understanding of distributed data systems
Experience designing enterprise data models

Technologies

Databricks
PySpark
Spark SQL
Delta Lake
Medallion Architecture
Delta Live Tables
Unity Catalog
AWS
Azure
GCP
Git
CI/CD

Preferred Qualifications

Databricks certification
Experience with streaming data
Experience building reusable data frameworks
Experience with enterprise governance
Exposure to AI-assisted development tools including Databricks AI Genie

View full job page →

8d ago

Data Engineer – Healthcare Analytics Platform

Guidehouse Washington, DC

Washington, DC

8d ago

$77k - $129k/yr Full time Analytics Business Intelligence Claims Data Pipelines

Apply Now

Job Description

Guidehouse is seeking a Data Engineer to design and implement an enterprise Contract Performance Analytics platform for a large healthcare system. This onsite role in Washington, DC centers on data architecture, ELT/ETL pipeline development, and the integration of clinical, claims, and operational data to power analytics across the organization.

Responsibilities

Design, develop, and maintain robust ETL/ELT pipelines to ingest, transform, and load healthcare data from diverse structured and unstructured sources
Create pipelines to process data from CMS and payer files (CCLF, paid claims, PUG) as well as Epic Caboodle and Clarity data models and extracts
Build and optimize data models to support analytics, reporting, and operational use cases, including BI and downstream analytics consumption
Transform raw data into standardized, analytics-ready canonical data models and curated data marts
Develop lakehouse or medallion architecture, data ingestion patterns, and orchestration frameworks
Implement and maintain CI/CD pipelines for data engineering workflows, including pipelines and scheduled jobs, using version control and automation tools
Collaborate with database administrators, analysts, and application teams to integrate data sources, design schemas, and support downstream data consumers
Ensure data quality, integrity, and accuracy through validation, monitoring, logging, and alerting
Support data migration, integration, and modernization initiatives, including legacy upgrades, optimization of large-scale ETL pipelines, query performance, and cloud adoption efforts
Troubleshoot and resolve issues in development and production environments to maintain stable and reliable data pipelines
Document data flows, pipelines, test cases, and technical solutions to support knowledge sharing and compliance requirements
Stay current with emerging tools, technologies, and best practices in data engineering and cloud platforms

Requirements

US Citizenship or a Green Card is required
Bachelor’s degree in Computer Science, Data Analytics, Software Engineering, Information Systems, or related fields
Minimum of five (5) years of experience in data engineering, ETL/ELT development, or data platform engineering in a healthcare setting
Experience working with healthcare data, including claims, clinical, payer, or population health datasets
Experience with healthcare data interoperability standards (e.g., FHIR, HL7)
Proficiency in Python and SQL for data engineering and transformation workloads
Hands-on experience designing and building ETL/ELT pipelines and data ingestion frameworks
Experience with modern cloud data platforms or ETL/ELT tools (e.g., Databricks, Azure Data Factory, AWS Glue)
Experience working with lakehouse or medallion-style architectures for analytics platforms
Strong knowledge of relational database design, data warehouses, and/or data lakes (e.g., star/snowflake schemas)
Experience with relational and/or distributed data systems, including data modeling
Experience working in a cloud environment (AWS or Azure) supporting data solutions
Experience with CI/CD practices and version control tools (e.g., Git)
Experience using monitoring and logging tools to support data pipeline reliability
Experience working with PHI and healthcare data privacy/security requirements
Ability to work effectively in an Agile development environment
Strong analytical and troubleshooting skills, with the ability to convey technical concepts clearly to clients, engineers, and business stakeholders
Ability to work independently in a fast-paced, client-facing environment

Technologies

Python
SQL
Databricks
Azure Data Factory
AWS Glue
Git
Tableau
Power BI
AWS
Azure

Benefits

Medical, Rx, Dental & Vision Insurance
Personal and Family Sick Time & Company Paid Holidays
Position may be eligible for a discretionary variable incentive bonus
Parental Leave and Adoption Assistance
401(k) Retirement Plan
Basic Life & Supplemental Life
Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
Short-Term & Long-Term Disability
Student Loan PayDown
Tuition Reimbursement, Personal Development & Learning Opportunities
Skills Development & Certifications
Employee Referral Program
Corporate Sponsored Events & Community Outreach
Emergency Back-Up Childcare Program
Mobility Stipend

Nice to have

Previous experience working with Epic and/or Athena in a healthcare setting for data engineering
Previous experience with Exasol or similar analytics platform
AWS, Azure, Databricks, Snowflake or other data engineering–related certifications
Experience with data visualization or analytics tools (e.g., Tableau, Power BI)
Exposure to microservices-based architectures or AI/ML-enabled data pipelines
Prior consulting experience

View full job page →

9d ago

Data Engineer with Financial Services, Databricks and MDM Experience

Yoh, A Day & Zimmermann Company Jersey City, NJ

Jersey City, NJ

9d ago

$65 - $80/hr Full time Cloud Cloud Platform Data Engineer

Apply Now

Job Description

Yoh, A Day & Zimmermann Company invites you to join a data engineering team in Jersey City, NJ on a hybrid schedule. This role offers a competitive hourly rate of USD 65 to 80 and a benefits-rich environment for professionals working on financial services data. You will design and operate scalable data pipelines, support master data management, and build cloud-native solutions on platforms like Databricks and Snowflake to meet operational, analytical, and regulatory needs. A Bachelor’s degree in Computer Science, Information Systems, or Engineering is required.

Responsibilities

Design, develop, and support scalable data pipelines and enterprise data integration solutions.
Build and maintain batch and real-time data ingestion, transformation, and processing frameworks.
Develop cloud-native data engineering solutions supporting enterprise data lake, warehouse, and lakehouse platforms.
Implement ETL and ELT processes for structured, semi-structured, and unstructured data sources.
Support Master Data Management initiatives across security, account, client, and reference data domains.
Collaborate with data architects, business analysts, governance teams, and application teams to support enterprise data initiatives.
Implement data quality validation, monitoring, metadata management, and lineage processes.
Support cloud migration and modernization efforts involving legacy and enterprise data platforms.
Optimize data processing, storage, and pipeline performance for scalability and operational efficiency.
Ensure compliance with enterprise security, governance, and regulatory standards within financial services environments.
Support reporting, analytics, and downstream consumption platforms through reliable and trusted data delivery.

Requirements

Strong hands-on experience in Data Engineering and enterprise-scale data integration.
Proven experience developing scalable ETL/ELT pipelines and distributed data processing solutions.
Experience working with modern cloud-based data platforms and data ecosystems.
Strong SQL expertise along with programming or scripting experience in Python, PySpark, or Snowpark.
dbt for data transformation and modeling
dbt for ELT pipeline development within Snowflake/Databricks
dbt for modular, reusable SQL-based data workflows
dbt for data testing, documentation, and version control integration
Experience with cloud platforms such as Azure, AWS, or GCP, including integration with Snowflake and Databricks.
Solid understanding of data lake, data warehouse, and lakehouse architectures, and their implementation across platforms.
Experience with orchestration and workflow tools (e.g., Airflow, Databricks Workflows, Snowflake Tasks) for pipeline scheduling and automation.
Experience supporting Master Data Management and enterprise data governance initiatives.
Familiarity with metadata management, data lineage, data cataloging, and data quality processes.
Experience integrating APIs and microservices.
Experience integrating file-based ingestion (batch).
Real-time/streaming data experience (e.g., Kafka, Spark Streaming).

Must Haves

Financial services experience
Data pipeline development
Databricks and Delta Lake
Master Data Management integration (Golden Source)
Strong SQL and data engineering skills
Oracle to Cloud migration experience
Data quality and reconciliation

Preferred Qualifications

Financial services or banking industry experience
Experience supporting regulatory, risk, compliance, or operational reporting data environments
Exposure to real-time data processing and streaming technologies
Familiarity with CI/CD processes and infrastructure automation
Strong analytical, troubleshooting, and problem-solving skills
Excellent communication and collaboration skills

Education

Bachelor’s degree in Computer Science, Information Systems, or Engineering

Tech Stack

Databricks, Delta Lake, Snowflake, Snowpark
Python, PySpark, SQL
dbt
Azure, AWS, GCP
Airflow, Databricks Workflows, Snowflake Tasks
Kafka, Spark Streaming

Compensation

Estimated rate: USD 65.00 - 80.00 per hour

Benefits

Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
Health Savings Account (HSA) (for employees working 20+ hours per week)
Life & Disability Insurance (for employees working 20+ hours per week)
MetLife Voluntary Benefits
Employee Assistance Program (EAP)
401K Retirement Savings Plan
Direct Deposit & weekly epayroll
Referral Bonus Programs
Certification and training opportunities

View full job page →

10d ago

Databricks Data Engineer

Deloitte Seattle, WA

Seattle, WA

10d ago

$87k - $171k/yr Full time Azure Databricks Big Data Data Analysis

Apply Now

Job Description

Deloitte is seeking a Databricks Data Engineer to join its Seattle on-site team. The role centers on designing, building, and optimizing cloud-based data engineering solutions with Databricks and Spark to enable large-scale data transformation and data-driven decision making across client engagements. You will collaborate with business and technology colleagues to modernize data environments and support analytics and AI use cases, with a salary range of USD 86,700 to 170,900 per year.

Responsibilities

Design, build, and maintain data pipelines and engineering solutions using Databricks, Apache Spark, Python, and SQL
Support modernization of enterprise data platforms by implementing cloud-based ingestion, transformation, and storage patterns
Develop and maintain batch and streaming data workflows that underpin reporting, analytics, and artificial intelligence use cases
Apply data architecture, integration, and data modeling practices to improve data quality, scalability, and platform performance
Collaborate with client and Deloitte teams to troubleshoot issues, implement enhancements, and support large-scale data transformation programs

Requirements

Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, Mathematics, or Statistics
3+ years of experience in data engineering, ETL development, or data platform engineering
2+ years of experience building data pipelines in Databricks using Apache Spark, Python, and SQL
Experience implementing data models, data integration patterns, and cloud-based data processing workflows
Experience working with at least one cloud platform: AWS, Azure, or GCP
Ability to travel about 50% on average, depending on client assignments
Limited immigration sponsorship may be available

Technologies

Databricks, Apache Spark, Python, SQL
Delta Lake, Databricks Workflows, Unity Catalog
Git, CI/CD pipelines
AWS, Microsoft Azure, Google Cloud Platform (GCP)

The Team

Deloitte's Core AI & Data practice helps organizations modernize data platforms, strengthen enterprise data foundations, and scale analytics and artificial intelligence capabilities across the business
The team architects, engineering, and deploy cloud-based data solutions to improve decision-making, enable innovation, and support large-scale transformation
Practitioners collaborate across business and technology functions to tackle challenges in data modernization, governance, platform engineering, and insight delivery

Preferred

Experience with Delta Lake, Databricks Workflows, and Unity Catalog
Experience using Git-based version control and CI/CD pipelines
Experience supporting batch and streaming data workloads
Experience with data warehousing, lakehouse architecture, and performance optimization
Experience supporting analytics or machine learning data pipelines
Experience working in consulting or client service delivery environments

Information for applicants with a need for accommodation

Information is available at the following Deloitte page: Join Deloitte assistance for disabled applicants

View full job page →

11d ago

Databricks Data Engineer

Deloitte Atlanta, GA

Atlanta, GA

11d ago

$87k - $171k/yr Full time Artificial Intelligence Azure Databricks Big Data

Apply Now

Job Description

Deloitte seeks an onsite Databricks Data Engineer in Atlanta, GA, offering a salary range of USD 86,700 - 170,900 per year and requiring a Bachelor's degree plus at least 3 years of data engineering experience.

Responsibilities

Create and sustain data pipelines and engineering solutions using Databricks, Apache Spark, Python, and SQL
Support modernization of enterprise data platforms by deploying cloud-based ingestion, transformation, and storage patterns
Design and operate batch and streaming data workflows to enable reporting, analytics, and AI use cases
Apply data architecture, integration, and modeling practices to improve data quality, scalability, and performance
Collaborate with client and Deloitte teams to troubleshoot issues, implement enhancements, and support large-scale data transformation programs

Requirements

Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, Mathematics, or Statistics
Minimum of three years in data engineering, ETL development, or data platform engineering
At least two years building data pipelines in Databricks using Spark, Python, and SQL
Experience implementing data models, data integration patterns, and cloud-based processing workflows
Experience with one or more cloud platforms (AWS, Azure, or Google Cloud Platform)
Willingness to travel approximately 50% depending on client engagements
Limited immigration sponsorship may be available

Technologies

Databricks
Apache Spark
Python
SQL
Delta Lake
Unity Catalog
Databricks Workflows
Git and Git-based version control
AWS
Microsoft Azure
Google Cloud Platform (GCP)
CI/CD

The Team

Deloitte's Core AI & Data practice helps organizations modernize data platforms, strengthen enterprise data foundations, and scale analytics and artificial intelligence capabilities across the business. The team collaborates with clients to architect, engineer, and deploy cloud-based data solutions that improve decision-making, enable innovation, and support large-scale transformation. Practitioners work across business and technology functions to address data modernization challenges.

Preferred

Experience with Delta Lake, Databricks Workflows, and Unity Catalog
Proficiency with Git-based version control and continuous integration/continuous deployment pipelines
Experience supporting batch and streaming data workloads
Background in data warehousing, lakehouse architecture, and performance optimization
Experience supporting analytics or machine learning data pipelines
Experience working in consulting or client service delivery environments