Senior Data Engineer - Databricks
Apache Spark
Azure Data Engineer
Azure Databricks
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Infrastructure
Cloud Platform
Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Engineering Lead
Data Governance
Data Integration
Data Integration Tools
Data Lake
Data Lakehouse
Data Management
Data Modeling
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Dlt
Databricks Lakehouse
Databricks Lakehouse Platform
Databricks Pyspark
Databricks Workflows
Delta Lake
DevOps
DevSecOps
ETL
Facilities Management
Feature Store
Gcp Data Engineer
Informatica
Information Technology (IT)
Infrastructure As Code
Lead Data Engineering
Security Automation
Security Devops
Spark
SQL
Vector Databases
Job Description
This Senior Data Engineering role supports scalable batch and real-time data platforms on the Databricks lakehouse, with responsibilities spanning ETL/ELT development, governance and security, and support for machine learning and generative AI operationalization.
Location
- New York, NY (onsite)
- Troy, MI (onsite minimum four days per week)
Salary
USD 89,400 - 166,398 per year
Experience Level
5+ years of experience
Key Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
- Build robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
- Implement data quality checks, monitoring, and pipeline optimization to improve reliability and performance.
- Design and manage modern Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers).
- Develop and maintain Delta Lake tables while ensuring optimal query performance.
- Perform performance tuning using Databricks best practices, including Photon Engine, Liquid Clustering, and Partitioning.
- Configure and administer Unity Catalog to manage data governance across the enterprise.
- Implement security controls including catalog, schema, table, row-level, and column-level security.
- Enable data lineage, auditing, and compliance requirements across the data platform.
- Develop and maintain pipelines using Databricks Workflows.
- Manage deployments using Databricks Asset Bundles (DABs).
- Implement CI/CD pipelines using GitHub, GitLab, and DevOps best practices.
- Provision and manage infrastructure using Terraform and Infrastructure-as-Code methodologies.
- Prepare and curate datasets for machine learning and generative AI applications.
- Support feature engineering and feature store implementations.
- Enable vector search and retrieval capabilities for LLM solutions and AI-driven use cases.
- Collaborate with Data Science and AI teams to operationalize ML workloads.
Required Technologies and Skills
- PySpark and SQL
- Databricks Lakehouse platform experience
- Delta Lake
- Unity Catalog
- Databricks Workflows
- Structured Streaming
- Auto Loader
- Medallion Architecture (Bronze, Silver, Gold)
- Photon Engine, Liquid Clustering, and Partitioning
- Databricks Asset Bundles (DABs)
- Git, GitHub/GitLab, DevOps, and CI/CD
- Terraform and Infrastructure-as-Code
- pytest
- Cloud experience with at least one of: Azure, GCP, AWS
- LLM, vector search, and feature store
Benefits
- Paid time off based on employee grade (A-F), including Vacation (12-25 days), Company paid holidays, Personal Days, and Sick Leave
- Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
- Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
- Life and disability insurance
- Employee assistance programs
- Other benefits as provided by local policy and eligibility
Onsite Work Requirement
Candidates must be able to work onsite at the customer location in Troy, MI a minimum of four (4) days per week.