Sr Databricks Data Engineer

Responsibilities

• Develop and promote best-in-class approaches for data architecture, integration, and modelling; document standards and drive adoption across teams.
• Lead the design, development, and maintenance of robust data pipelines and architectures that support large-scale, enterprise data needs.
• Drive excellence by leading initiatives to improve data quality, operational efficiency, and process scalability.
• Evaluate, pilot, and integrate new big data and analytics technologies, ensuring the organization remains at the cutting edge; lead, coach, and develop teams of data engineers and architects to ensure effective project delivery.
• Advise on, design, and implement governance, security, and compliance strategies tailored to modern cloud data ecosystems.
• Explain technical concepts and business value to a range of stakeholders, including executives, business leads, and technology teams.
• Oversee the implementation of CI/CD practices with tools such as Azure DevOps, AWS Code Pipeline, Jenkins, TFS, or PowerShell for streamlined deployments and operations.
• Provide clear guidance to colleagues and stakeholders to align efforts and ensure successful outcomes.

Requirements

• Ability to work independently and collaboratively within a team environment.
• Strong written and verbal communication skills.
• Meticulous attention to detail and a commitment to high-quality work products.
• Proven ability to build and sustain professional relationships.
• Experience leading projects or workstreams.
• Ability to manage and prioritize multiple tasks in a fast-paced, dynamic setting.
• Strong interpersonal skills and a professional demeanor.
• Ability to meet deadlines.

Technologies

• Databricks
• AWS
• Azure
• GCP
• Delta Lake
• Apache Spark
• PySpark
• Unity Catalog
• Delta Live Tables
• Autoloader
• Structured Streaming
• Databricks Workflows
• Apache Airflow
• Azure DevOps
• AWS Code Pipeline
• Jenkins
• TFS
• PowerShell
• Databricks Lakeflow

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 5+ years of hands-on data engineering experience with a focus on Databricks on AWS, Azure, or GCP.
• Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.
• Experience with data warehousing, 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.
• 1+ year leading complex, cross-functional data projects and technical teams, including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data engineering pipelines, code, and compute resources.
• Ability to travel 50% on average based on client engagement and project needs.
• Limited immigration sponsorship may be available.

Preferred Qualifications

• Master's degree in Computer Science, Engineering, or a related field.
• Experience across one or more of AWS, Azure, and GCP cloud ecosystems and associated big data services.
• Experience tuning and optimizing performance in Databricks and Apache Spark environments.
• Experience with Databricks Lakeflow.
• Experience with artificial intelligence and machine learning solutions.