Senior Data Engineer focusing on Databricks to design and build enterprise-grade data platforms, services, and pipelines for Steampunk clients, with an emphasis on strong communication and problem solving. Location: McLean, VA onsite.
Responsibilities
- Lead and architect migration of data environments with performance and reliability in mind.
- Assess and understand the ETL jobs, workflows, BI tools, and reporting landscape.
- Answer technical inquiries related to customization, integration, enterprise architecture, and general feature functionality of data products.
- Design and implement database and data warehouse solutions on Databricks.
- Key skills required: Databricks, Python, SQL.
- Support an Agile software development lifecycle.
- Contribute to the growth of the Data Exploitation Practice.
Requirements
- Ability to hold a position of public trust with the US government.
- Experience: 12+ years with a Bachelor's degree or 8+ years with a Master's degree.
- 8-12 years industry experience in data engineering with a passion for solving complex problems.
- 8-12 years direct data engineering experience, including:
- Designing and building data pipelines and data products on modern lakehouse platforms (e.g., Databricks, Apache Spark), including Delta Lake, distributed processing, and performance optimization.
- Supporting or integrating GenAI/ML workflows (feature engineering, vector storage, RAW pipelines, or model lifecycle using tools like MLFlow).
- Operating in enterprise or regulated cloud environments with platform constraints, security controls, and data governance frameworks.
- Building batch and streaming data pipelines using distributed processing frameworks (e.g., Apache Spark Structured Streaming) within a lakehouse architecture.
- Experience with search and retrieval patterns for analytics or AI use cases (e.g., indexing, semantic search, vector-based retrieval).
- Strong proficiency in SQL and Python for data engineering, including distributed data processing frameworks (e.g., Apache Spark).
- Advanced working SQL knowledge with relational databases, query authoring and optimization, and familiarity with various databases.
- Ability to lead technical discussions with technical and non-technical stakeholders, translating business needs into actionable data solutions and guiding tradeoffs.
- Demonstrated ability to evaluate feasibility, identify risks, and provide pragmatic recommendations in complex environments.
- Experience navigating cross-team environments and driving alignment through structured problem-solving and clear communication.
- Experience constructing complex queries to analyze results in a data processing environment.
- Strong judgment balancing technical constraints, business needs, and long-term scalability.
- Experience architecting transactional and data warehouse systems.
- Experience aggregating results and compiling information for reporting from multiple datasets.
- Experience working in an Agile environment.
- Experience supporting project teams of developers and data scientists who build web-based interfaces, dashboards, reports, and analytics or ML models.
Technologies
- Databricks
- Python
- SQL
- Apache Spark
- Delta Lake
- MLFlow
- Apache Spark Structured Streaming
Overview
We seek a seasoned Senior Data Engineer to collaborate with our team and clients on enterprise-grade data platforms, services, and pipelines. The role emphasizes strong communication and customer-facing skills, alongside a passion for data and problem solving.
Contributions
- Lead and architect migration of data environments with performance and reliability.
- Assess and understand the ETL jobs, workflows, BI tools, and reports.
- Address technical inquiries concerning customization, integration, enterprise architecture, and general feature functionality of data products.
- Design and implement database and data warehouse solutions on Databricks.
- Key skills: Databricks, Python, SQL.
- Support an Agile software development lifecycle.
- Contribute to the growth of our Data Exploitation Practice.
About Steampunk
Steampunk considers multiple factors when determining compensation, including geography, requirements, education, knowledge, skills, competencies, and experience. The projected salary range for this position is $125,000 to $165,000 per year. This estimate represents typical annual pay for the role, and total compensation may include additional benefits. Learn more about Steampunk benefits as part of the package.
Identity Statement
As part of the application workflow, candidates are expected to be on camera during interviews and assessments. We reserve the right to capture a photo to verify identity and prevent fraud.
