Staff Data Engineer (Autonomous Vehicle Data)
Big Data
Bigdata
Data
Data Architecture
Data Engineer
Data Engineering
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Databricks
Databricks Lakehouse
Databricks Pyspark
Databricks Spark
ETL
Information Technology (IT)
Lakehouse Governance
Reporting and Analytics
Software Development
Spark
SQL
Telemetry Data
Job Description
Own production data products for ADAS and autonomous vehicle analytics, and build Databricks-based pipelines that deliver governed, analytics-ready datasets.
Responsibilities
- Own production data products end-to-end, from ingestion through curated, analytics-ready datasets, including silver-to-gold pipelines for feature-performance reporting, safety workflows, and self-service analysis.
- Design and operate reliable Databricks pipelines for large, raw vehicle telemetry: decoding, deduplication, ordering, completeness, and handling late or missing data.
- Run the self-service analytics platform from the engineering side, including workspace onboarding, compute, and governed access so partner teams can analyze approved ADAS data without one-off pipelines.
- Raise production standards by defining data quality checks, monitoring, and alerting, and driving fixes before downstream scorecards and investigations are wrong.
- Ship changes through a standard path using GitHub, Databricks Asset Bundles, automated tests, and a dev-to-prod promotion flow with rollback.
- Increase reuse by modeling datasets so scientists, analysts, and product partners can consume them without rediscovering lineage, grain, or known quality limits.
- Apply enterprise data governance including classification, access control, lineage, and privacy constraints for customer and vehicle data.
- Lead cross-team execution by partnering with upstream data engineers, scientists, product managers, and safety stakeholders on contracts, feasibility, and production support.
- Mentor senior and mid-level engineers through design reviews, code reviews, and incident response, improving platform operability.
- Enable agentic analytics in production by building the data layer agents rely on: an entity spine joining vehicle, event, software, and road context; gold datasets for investigations; and expert procedures encoded for agent execution.
- Keep agent outputs trustworthy by grounding results in approved data, returning evidence, evaluating workflows against known cases, and enforcing access, privacy, and AI-review requirements.
- Connect production analytics to agent workflows through APIs and tool interfaces, including MCP, so investigations, fleet scans, and KPI explanations reuse the same datasets in alignment with the enterprise agent platform.
Requirements
- 7+ years of experience building and operating production data platforms.
- Bachelor’s degree in computer science, Software Engineering, or a closely related technical field, or equivalent demonstrable professional software engineering experience.
- Deep hands-on experience with Python, SQL, and Spark on a lakehouse (e.g., Databricks).
- Experience designing batch and incremental pipelines that remain correct with late, duplicate, and incomplete source data.
- Production operations experience: monitoring, alerting, incident response, and cost-aware performance tuning.
- Experience releasing data code via CI/CD, including infrastructure-as-code or bundle-based deployment.
- Experience putting agent or LLM workflows on production data: multi-source retrieval, tool use, and checks ensuring answers are grounded in data.
- Sound judgment on data governance, access control, and sensitive vehicle or customer data.
- Ability to set technical direction for a domain and align engineers and non-engineers.
- Clear writing and speaking skills for design docs and production runbooks.
- Automotive, ADAS, or vehicle-telemetry experience.
- Ability to work across more than one cloud, especially Azure and GCP.
- Experience building self-service analytics for analysts and scientists not on your team.
- Familiarity with Unity Catalog, data contracts, and medallion (bronze, silver, gold) architecture.
- Experience supporting safety or regulatory reporting where incorrect rows have real consequences.
- Experience with agent frameworks, MCP or other tool-calling interfaces, and evaluation of agent answers against known cases.
- Familiarity with diagnostic or investigative workflows requiring root-cause explanations.
Technologies
- Databricks
- Databricks Asset Bundles
- GitHub
- Python
- SQL
- Spark
- CI/CD
- Infrastructure-as-code
- Unity Catalog
- Medallion (bronze, silver, gold) architecture
- APIs
- MCP
Benefits
- Expected base compensation: $159,200 - $220,500 per year.
- Bonus potential via an incentive pay program based on company performance, job level, and individual performance.
- Medical
- Dental
- Vision
- Health Savings Account
- Flexible Spending Accounts
- Retirement savings plan
- Sickness and accident benefits
- Life insurance
- Paid vacation & holidays
- Tuition assistance programs
- Employee assistance program
- GM vehicle discounts
Job Details
- Location: Warren, MI (hybrid)
- Hybrid expectations: report to Warren Global Technical Center or Austin Technical Center three times per week, at minimum (or other frequency dictated by the business if more than 3 days).
Accommodations: General Motors provides opportunities to all job seekers, including individuals with disabilities. For reasonable accommodations during the job search or application process, email or call 1-800-865-7580 and include the specific accommodation requested, the job title, and the requisition number.