DataJobs.io
← Back to all jobs

Job Description

This Senior Data Engineering role supports scalable batch and real-time data platforms on the Databricks lakehouse, with responsibilities spanning ETL/ELT development, governance and security, and support for machine learning and generative AI operationalization.

Location

  • New York, NY (onsite)
  • Troy, MI (onsite minimum four days per week)

Salary

USD 89,400 - 166,398 per year

Experience Level

5+ years of experience

Key Responsibilities

  • Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
  • Build robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
  • Implement data quality checks, monitoring, and pipeline optimization to improve reliability and performance.
  • Design and manage modern Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers).
  • Develop and maintain Delta Lake tables while ensuring optimal query performance.
  • Perform performance tuning using Databricks best practices, including Photon Engine, Liquid Clustering, and Partitioning.
  • Configure and administer Unity Catalog to manage data governance across the enterprise.
  • Implement security controls including catalog, schema, table, row-level, and column-level security.
  • Enable data lineage, auditing, and compliance requirements across the data platform.
  • Develop and maintain pipelines using Databricks Workflows.
  • Manage deployments using Databricks Asset Bundles (DABs).
  • Implement CI/CD pipelines using GitHub, GitLab, and DevOps best practices.
  • Provision and manage infrastructure using Terraform and Infrastructure-as-Code methodologies.
  • Prepare and curate datasets for machine learning and generative AI applications.
  • Support feature engineering and feature store implementations.
  • Enable vector search and retrieval capabilities for LLM solutions and AI-driven use cases.
  • Collaborate with Data Science and AI teams to operationalize ML workloads.

Required Technologies and Skills

  • PySpark and SQL
  • Databricks Lakehouse platform experience
  • Delta Lake
  • Unity Catalog
  • Databricks Workflows
  • Structured Streaming
  • Auto Loader
  • Medallion Architecture (Bronze, Silver, Gold)
  • Photon Engine, Liquid Clustering, and Partitioning
  • Databricks Asset Bundles (DABs)
  • Git, GitHub/GitLab, DevOps, and CI/CD
  • Terraform and Infrastructure-as-Code
  • pytest
  • Cloud experience with at least one of: Azure, GCP, AWS
  • LLM, vector search, and feature store

Benefits

  • Paid time off based on employee grade (A-F), including Vacation (12-25 days), Company paid holidays, Personal Days, and Sick Leave
  • Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
  • Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
  • Life and disability insurance
  • Employee assistance programs
  • Other benefits as provided by local policy and eligibility

Onsite Work Requirement

Candidates must be able to work onsite at the customer location in Troy, MI a minimum of four (4) days per week.

Similar Jobs