Data Engineer - Fleet Operations
Apache Spark
Aws Dms
Big Data
Bigdata
Change Data Capture
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Analytics Tools
Data Architecture
Data Build Tool
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lake
Data Lakehouse
Data Management
Data Modeling
Data Operations
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Transforms
Data Warehouse
Database
Databases
Databricks
Databricks Lakehouse
Databricks Lakehouse Platform
Dbt
Delta Lake
ETL
Informatica
Integration
Lakehouse
Spark
SQL
Streaming Data
Structured Streaming
Job Description
Build and operate the data foundation for manufacturing intelligence and fleet operations using modern lakehouse and streaming technologies.
Responsibilities
- Design, build, and maintain batch and streaming ingestion pipelines into Delta Lake using Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, or Structured Streaming.
- Implement change data capture (CDC) feeds sourced from AWS DMS and SAP SLT.
- Set up and maintain bronze, silver, and gold data layers following MOPS naming conventions and data dictionary standards.
- Develop incremental dbt models on Databricks, using MERGE and upsert patterns.
- Apply Slowly Changing Dimension (SCD) Type 2 patterns in dbt, including support for late or out-of-order records.
- Model manufacturing datasets to support analytics and reporting, including unit-level traceability (SFC), work orders, nonconformances, station results, and first-pass yield.
- Process large-scale and semi-structured data using PySpark and Databricks SQL.
- Optimize performance and cost using clustering, partitioning, OPTIMIZE, VACUUM, file sizing, and compute configuration.
- Improve reliability and data quality with dbt tests, Lakeflow expectations, and Lakehouse Monitoring.
- Set and manage freshness SLAs and implement automated alerts.
- Orchestrate pipelines using Databricks Workflows or Airflow.
- Support Redshift-to-Databricks migration via SQL conversion, parallel validation, reconciliation, and cutover.
- Collaborate with analysts and manufacturing teams to define data requirements, document lineage, and publish data contracts and table definitions in Unity Catalog.
- Maintain CI/CD using Databricks Asset Bundles and GitHub Actions across development, staging, and production.
- Manage MOPS-specific Databricks configuration, including Unity Catalog permissions, compute resources, SQL warehouses, service principals, secrets, and OAuth authentication under central platform governance.
- Configure and troubleshoot AWS ingestion components including DMS, Kinesis, S3, and IAM, coordinating Terraform-based provisioning with infrastructure teams.
- Monitor pipeline health, performance, and Databricks Unit (DBU) costs; maintain operational documentation and support incident response and root cause analysis.
Requirements
- Bachelor’s degree in computer science, information systems, engineering, or a related discipline (or equivalent practical experience).
- 6+ years of data engineering experience.
- Strong proficiency in SQL, Python, and PySpark.
- Hands-on production experience with Databricks, including Delta Lake, Unity Catalog permissions, Workflows, and Spark performance tuning.
- Production experience with dbt, including incremental modeling and testing.
- Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading.
- Experience deploying data code through Git-based workflows and CI/CD tools such as GitHub Actions or Databricks Asset Bundles.
- Working knowledge of AWS data services including S3, IAM, DMS, or Kinesis, and infrastructure-as-code tools such as Terraform.
Technologies
- Databricks, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, Structured Streaming
- AWS DMS, SAP SLT, dbt, MERGE, upsert, Slowly Changing Dimension (SCD) Type 2
- PySpark, Databricks SQL, OPTIMIZE, VACUUM, Lakehouse Monitoring
- Databricks Workflows, Airflow, Redshift, Unity Catalog
- Databricks Asset Bundles, GitHub Actions, Unity Catalog permissions, SQL warehouses
- service principals, OAuth authentication, AWS Kinesis, AWS S3, AWS IAM, Terraform, Databricks Unit (DBU)
- Git-based workflows, SAP MES, SAP HANA, SFC
Benefits
- Pre-tax commuter benefits
- Employer (HireArt) subsidized healthcare benefits (eligibility begins on the first of the month following 60 days of service)
- Flexible Spending Account for healthcare-related costs
- HireArt covers all costs for short- and long-term disability and life insurance
- 401k package
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional certification; AWS certification such as Solutions Architect Associate is a plus
- Experience preparing data for Databricks AI/BI dashboards, Genie, machine learning, or large language model applications
- Experience with manufacturing or industrial data, including quality, yield, traceability, or manufacturing execution systems (MES)
- Understanding of data governance and compliance requirements related to data pipelines
Job Details
- Location: Foster City, CA (onsite)
- Work type: contract
- Salary: USD 68 per hour
- Commitment: full-time, 12-month contract position staffed via HireArt
- Availability: onsite role for candidates local to the Foster City, CA area
Similar Jobs
G