Senior Data Engineer

The Senior Data Engineer / Databricks Specialist will support a federal project by building and optimizing high-performance Spark applications and AWS Databricks enterprise data infrastructure, with an emphasis on secure and reliable ETL/ELT for mission-critical environments.

Key Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
• Design, build, and maintain ETL/ELT pipelines following Medallion Architecture (Bronze, Silver, and Gold), ensuring data integrity, completeness, and lineage.
• Administer Databricks resources including workspaces, clusters, scheduled jobs, access controls, Unity Catalog, and notebooks.
• Manage AWS services such as S3, Lambda, IAM, Aurora, and Glue, and provision infrastructure using Terraform.
• Optimize queries and Spark jobs, and troubleshoot large-scale data processing solutions.
• Integrate structured, semi-structured, and unstructured data from APIs and databases.
• Implement security and compliance controls in DevSecOps CI/CD pipelines using Jenkins, GitLab CI, or equivalent tooling to support FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
• Add data quality checks, anomaly detection, validation rules, and automated lineage tracking.
• Work with cross-functional teams in Agile/Scrum delivery for multi-contractor federal environments, providing technical leadership and production on-call support.

Required Qualifications

• 3+ years of dedicated, hands-on Databricks development.
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
• Strong proficiency in Python and SQL, plus working knowledge of REST API based data ingestion.
• Hands-on production experience with AWS S3, IAM, Lambda, Glue, and Aurora.
• Experience with Terraform infrastructure as code and CI/CD tools such as Jenkins or GitLab CI, or equivalent.
• Experience delivering in Agile/Scrum environments.
• Bachelor’s degree in Computer Science, Information Technology, Data Science, or a related engineering discipline.
• Active or previous High-Risk Public Trust (Tier 4) clearance.

Preferred Qualifications

• Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and MLflow or Model Registry for AI/ML pipelines.
• Experience supporting federal government contracts and FISMA/FedRAMP environments.

Preferred Certifications

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Databricks Certified Machine Learning Associate
• AWS Certified Solutions Architect – Associate or Professional
• AWS Certified Data Analytics – Specialty

Technologies and Tools

• AWS Databricks, Spark, Python, PySpark, SQL, Delta Lake
• Medallion Architecture, Unity Catalog
• Databricks workspaces, Databricks clusters, Unity Catalog notebooks, Databricks Scheduled Jobs
• S3, Lambda, IAM, Aurora, Glue, Terraform
• Jenkins, GitLab CI, REST API
• Apache Spark, Delta Live Tables (DLT), MLflow, Model Registry
• DevSecOps CI/CD, ETL/ELT, Agile/Scrum

Location and Salary

Location: Washington, DC (onsite). Salary: USD 110,000 - 115,000 per year.

Benefits

• 401(k) matching
• Dental insurance
• Employee discount
• Flexible spending account
• Health insurance
• Health savings account
• Life insurance
• Paid time off
• Professional development assistance
• Referral program
• Vision insurance

Security Clearance

Confidential (Required)

Application Screening Question

Do you have at least 3 years of hands-on experience with AWS Databricks, including PySpark, Spark SQL, and Delta Lake?