Senior Data Engineer (Top Secret Cleared)

The Senior Data Engineer will design, implement, and optimize data pipelines and governance for government-facing data solutions, collaborating with stakeholders and IT teams to deliver scalable data architectures within a Top Secret/DHS EOD context.

Responsibilities

• Identify, design, and implement internal process improvements to increase scalability, optimize data delivery, and automate manual processes.
• Develop and design data pipelines to support end-to-end solutions.
• Create and maintain artifacts such as schemas, data dictionaries, and ETL transforms.
• Integrate data pipelines with AWS cloud services to extract actionable insights.
• Engage with stakeholders to support data infrastructure needs and resolve data-related technical issues.
• Design robust dataflows to support both raw data and expected data.
• Provide Tier 3 technical support for deployed applications and dataflows.
• Define and communicate a clear product vision aligned with user needs and business objectives.
• Create and manage product roadmaps reflecting innovation and growth strategies.
• Collaborate with a government product owner and a product team of 7-8 FTEs.
• Work with the data engineering team to design and launch new features.
• Coordinate and document dataflows and capabilities.
• Occasionally support off-hours deployments, including evenings or weekends, as needed.

Requirements

• Bachelor’s degree with 4 years of related professional experience.
• Expertise in distributed computing frameworks for large-scale data processing.
• Familiarity with Amazon Web Services or other cloud environments.
• Experience with datastores such as PostgreSQL, S3, Redshift, MongoDB/DynamoDB, Redis, Elasticsearch/OpenSearch, and SQL.
• Proficient in Python with libraries like pandas and PySark, NiFi, Airflow, AWS Lambda or similar technologies.
• Working knowledge of Docker, Kubernetes, JMS/SQS, SNS and Kafka.
• Familiar with Linux/Unix server environments.
• Experience with Agile development methodology.
• Publishing and/or presenting design reports.
• Coordinating with other team members to reach project milestones and deadlines.
• Working knowledge of collaboration tools such as Jira and Confluence.

Technologies

• Python
• pandas
• PySark
• NiFi
• Airflow
• Amazon Web Services (AWS)
• AWS Lambda
• Docker
• Kubernetes
• JMS/SQS
• SNS
• Kafka
• Linux/Unix
• Jira
• Confluence
• PostgreSQL
• S3
• Redshift
• MongoDB
• DynamoDB
• Redis
• Elasticsearch/OpenSearch
• SQL

Benefits

• 401(k) matching
• Dental insurance
• Health insurance
• Life insurance
• Vision insurance

Preferred Qualifications

• Master’s degree or equivalent experience in a related field.
• Familiarity and experience with the Intelligence Community (IC) and the intel workflow.
• Familiarity and experience with the Department of Homeland Security (DHS).
• Direct experience with DHS and IC component data architectures and environments, with IC-GovCloud experience preferred.
• Experience with cloud message APIs and push notifications.
• Strong interest in applying new tools and technologies to solve national security problems.
• Working knowledge of public keys and digital certificates.
• Experience in DevOps environments.
• Expertise with a range of COTS, GOTS, and open source tools for data integration and visualization.
• Specialization in object oriented programming languages, scripting, and databases.

Pay

Salary range: $110,000 - $170,000 per year. Compensation is determined by factors including location, experience, skills, education, certifications, seniority, and business needs. This range may be modified in the future.

Clearance

Applicants selected will be subject to a security investigation and may need to meet eligibility requirements for access to classified information.

Work Location

Hybrid remote in Washington, DC 20590