DataJobs.io
← Back to all jobs

Job Description

Build and operate the data foundation for manufacturing intelligence and fleet operations using modern lakehouse and streaming technologies.

Responsibilities

  • Design, build, and maintain batch and streaming ingestion pipelines into Delta Lake using Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, or Structured Streaming.
  • Implement change data capture (CDC) feeds sourced from AWS DMS and SAP SLT.
  • Set up and maintain bronze, silver, and gold data layers following MOPS naming conventions and data dictionary standards.
  • Develop incremental dbt models on Databricks, using MERGE and upsert patterns.
  • Apply Slowly Changing Dimension (SCD) Type 2 patterns in dbt, including support for late or out-of-order records.
  • Model manufacturing datasets to support analytics and reporting, including unit-level traceability (SFC), work orders, nonconformances, station results, and first-pass yield.
  • Process large-scale and semi-structured data using PySpark and Databricks SQL.
  • Optimize performance and cost using clustering, partitioning, OPTIMIZE, VACUUM, file sizing, and compute configuration.
  • Improve reliability and data quality with dbt tests, Lakeflow expectations, and Lakehouse Monitoring.
  • Set and manage freshness SLAs and implement automated alerts.
  • Orchestrate pipelines using Databricks Workflows or Airflow.
  • Support Redshift-to-Databricks migration via SQL conversion, parallel validation, reconciliation, and cutover.
  • Collaborate with analysts and manufacturing teams to define data requirements, document lineage, and publish data contracts and table definitions in Unity Catalog.
  • Maintain CI/CD using Databricks Asset Bundles and GitHub Actions across development, staging, and production.
  • Manage MOPS-specific Databricks configuration, including Unity Catalog permissions, compute resources, SQL warehouses, service principals, secrets, and OAuth authentication under central platform governance.
  • Configure and troubleshoot AWS ingestion components including DMS, Kinesis, S3, and IAM, coordinating Terraform-based provisioning with infrastructure teams.
  • Monitor pipeline health, performance, and Databricks Unit (DBU) costs; maintain operational documentation and support incident response and root cause analysis.

Requirements

  • Bachelor’s degree in computer science, information systems, engineering, or a related discipline (or equivalent practical experience).
  • 6+ years of data engineering experience.
  • Strong proficiency in SQL, Python, and PySpark.
  • Hands-on production experience with Databricks, including Delta Lake, Unity Catalog permissions, Workflows, and Spark performance tuning.
  • Production experience with dbt, including incremental modeling and testing.
  • Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading.
  • Experience deploying data code through Git-based workflows and CI/CD tools such as GitHub Actions or Databricks Asset Bundles.
  • Working knowledge of AWS data services including S3, IAM, DMS, or Kinesis, and infrastructure-as-code tools such as Terraform.

Technologies

  • Databricks, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, Structured Streaming
  • AWS DMS, SAP SLT, dbt, MERGE, upsert, Slowly Changing Dimension (SCD) Type 2
  • PySpark, Databricks SQL, OPTIMIZE, VACUUM, Lakehouse Monitoring
  • Databricks Workflows, Airflow, Redshift, Unity Catalog
  • Databricks Asset Bundles, GitHub Actions, Unity Catalog permissions, SQL warehouses
  • service principals, OAuth authentication, AWS Kinesis, AWS S3, AWS IAM, Terraform, Databricks Unit (DBU)
  • Git-based workflows, SAP MES, SAP HANA, SFC

Benefits

  • Pre-tax commuter benefits
  • Employer (HireArt) subsidized healthcare benefits (eligibility begins on the first of the month following 60 days of service)
  • Flexible Spending Account for healthcare-related costs
  • HireArt covers all costs for short- and long-term disability and life insurance
  • 401k package

Preferred Qualifications

  • Databricks Certified Data Engineer Associate or Professional certification; AWS certification such as Solutions Architect Associate is a plus
  • Experience preparing data for Databricks AI/BI dashboards, Genie, machine learning, or large language model applications
  • Experience with manufacturing or industrial data, including quality, yield, traceability, or manufacturing execution systems (MES)
  • Understanding of data governance and compliance requirements related to data pipelines

Job Details

  • Location: Foster City, CA (onsite)
  • Work type: contract
  • Salary: USD 68 per hour
  • Commitment: full-time, 12-month contract position staffed via HireArt
  • Availability: onsite role for candidates local to the Foster City, CA area

Similar Jobs