Databricks Data Engineer, Lead

The Lead Databricks Data Engineer role at Booz Allen Hamilton is a hands-on position focused on designing, building, and maintaining data pipelines in Databricks. You will provide technical leadership, mentorship, and oversight of scalable data platform work across Databricks and AWS, with emphasis on ETL execution, data modeling, security, and performance.

Key Responsibilities

• Design, build, and maintain robust data pipelines within Databricks, contributing independently while providing technical leadership and mentorship to the team
• Create and optimize end-to-end data flows and manage extract, transform, load (ETL) processes
• Design data models and ensure alignment with data architecture needs
• Ensure data security and compliance across data platforms and pipelines
• Collaborate with cross-functional teams to support delivery and platform outcomes
• Continuously improve performance and scalability of data processing workloads
• Mentor a multi-disciplinary team of analysts, data engineers, developers, and data consumers in a fast-paced, agile environment
• Oversee the assessment, design, building, and maintenance of scalable platforms for clients

Required Qualifications

• 8+ years of experience designing, developing, deploying, and managing ETL processes and data pipelines, including comprehensive logging and robust error handling to troubleshoot and resolve issues
• 3+ years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Databricks tools such as Delta Lake, Databricks Delta, and Unity Catalog
• 3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations
• Experience with data modeling, including when to use dimensional versus relation data models within a medallion architecture (Bronze, Silver, or Gold)
• Experience using AWS services including AWS Glue, Amazon S3, and Amazon RDS
• Knowledge of performance optimization techniques for ETL jobs such as partitioning, bucketing, caching, and troubleshooting to improve query performance
• Ability to design and implement data models from defining schemas through optimizing data structures that support efficient queries and analytics
• Public Trust
• Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology

Technology Stack

• Databricks
• AWS
• Databricks notebooks
• Clusters
• Job orchestration tools
• Delta Lake
• Databricks Delta
• Unity Catalog
• Python
• PySpark
• AWS Glue
• Amazon S3
• Amazon RDS

Nice If You Have

• Experience with AWS cloud-native data services such as AWS Lambda or CloudWatch that support serverless architectures for data processing
• Experience with tools such as AWS Glue Data Catalog or Collibra for managing metadata, data lineage, and cataloging data assets
• Knowledge of data governance practices, data lineage, and compliance requirements, including GDPR or CCPA, supporting data quality, privacy, and security
• Ability to explain complex technical concepts to a non-technical audience, including tradeoffs in architectural design

Vetting and Compliance

• Selected applicants will be subject to a government investigation and may need to meet eligibility requirements of the U.S. government client
• Public Trust determination is required

Compensation

The projected compensation range for this position is $77,600.00 to $176,000.00 (annualized USD).

Benefits

• Health, life, disability, financial, and retirement benefits
• Paid leave
• Professional development
• Tuition assistance
• Work-life programs
• Dependent care
• Recognition awards program acknowledging employees for exceptional performance and superior demonstration of Booz Allen values
• Full-time and part-time employees working at least 20 hours per week on a regular basis are eligible to participate in Booz Allen’s benefit programs

Work Location and Model

Location: Bethesda, MD (onsite). Work will primarily be performed full-time at a customer facility, with collaboration required as part of the role.

Identity Verification and Interview Policies

• Complete an identity verification process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity fraud
• You are expected to be on camera during interviews and assessments
• The company reserves the right to take your picture to verify identity and prevent fraud
• The use of AI or other tools to assist with responses during interviews is prohibited unless permission is explicitly provided