DataJobs.io
← Back to all jobs

Job Description

EXL Service is hiring a Lead Databricks Data Engineer for an onsite role in New York, NY. This position focuses on building and modernizing enterprise data platforms using Databricks Lakehouse architecture, with technical leadership across governance, delivery, performance, and operational excellence. If you’ve led large-scale data engineering efforts for analytics, AI/ML, and reporting, this role puts you at the center of platform decisions.

Compensation: USD 15 - 18 per hourly. Experience: 10-12 years. Employment type: Full-Time.

What you’ll do

  • Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
  • Architect and develop scalable ETL/ELT pipelines for both batch and real-time processing.
  • Define enterprise data models, ingestion frameworks, and transformation strategies.
  • Lead implementation of Delta Lake and Unity Catalog, including data governance frameworks.
  • Build and optimize highly scalable Spark and PySpark solutions for large-volume datasets.
  • Drive cloud data platform modernization and migration initiatives.
  • Serve as technical lead for Databricks and Data Engineering programs.
  • Set engineering best practices, coding standards, and architecture guidelines.
  • Run architecture reviews, code reviews, and solution assessments.
  • Mentor Data Engineers, Technical Leads, and junior team members, while collaborating across disciplines including Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.
  • Design secure, scalable, and resilient cloud-native data solutions on Azure, AWS, or GCP.
  • Optimize cost, performance, reliability, and availability.
  • Lead CI/CD and Infrastructure-as-Code for data workloads.
  • Define monitoring, observability, and operational excellence practices.
  • Establish data governance, lineage, cataloging, and security controls; ensure alignment with organizational and regulatory requirements.
  • Implement data quality frameworks and monitoring solutions; partner with security and compliance teams to enforce data protection standards.
  • Translate business requirements into scalable technical solutions; lead delivery planning, risk management, and resource coordination.
  • Present architecture decisions and technical recommendations to leadership teams.

What you bring

  • Strong Databricks Lakehouse Platform and Databricks Workflows experience.
  • Hands-on experience with Delta Lake, Unity Catalog, Structured Streaming, Databricks SQL, Delta Live Tables (DLT), and Databricks Asset Bundles.
  • Proficiency with Apache Spark, PySpark, Spark SQL, and real-time streaming architectures.
  • Experience with Kafka and supporting structured, semi-structured, and unstructured processing.
  • Cloud data platform knowledge across Azure Databricks and related services (including Azure Data Factory (ADF), ADLS Gen2, Azure Synapse Analytics, and Azure Key Vault), plus exposure to AWS (including S3, EMR, and Glue) and GCP (including BigQuery and Pub/Sub).
  • Experience implementing secure, resilient data architectures using IAM and cloud storage patterns.
  • Demonstrated expertise in data warehousing approaches such as Dimensional Modeling, Star & Snowflake Schema, Data Vault, and Master Data Management, along with Data Governance frameworks.
  • Proven leadership driving architectural decisions and managing technical teams.
  • Overall IT experience: 12-15 years. Databricks experience: 6+ years. Spark/PySpark data engineering: 7+ years.
  • Degree: Bachelor’s or Master’s in Computer Science, Engineering, Information Technology, or a related field.
  • Certifications (preferred/required as listed): Databricks Certified Data Engineer Professional, Azure Data Engineer Associate, AWS Data Analytics Specialty, and Google Professional Data Engineer; TOGAF or Cloud Architecture certifications are a plus.
  • Experience or exposure to MLOps and AI Engineering, plus Snowflake and Databricks integration and Microsoft Fabric.
  • Exposure to Data Mesh and data governance tools such as Collibra, Alation, or Informatica.
  • Expertise in data quality frameworks such as Great Expectations or Deequ.
  • Familiarity with supporting engineering tooling and platforms, including Python, SQL, Git, Azure DevOps, GitHub Actions, Jenkins, Terraform, Docker, and Kubernetes.

Technology Tower: Data & AI. Practice Area: Data Engineering. Primary Skill: Databricks Lakehouse Engineering.

Similar Jobs