Cleared Data Engineer
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Database
Databases
Databricks
ETL
Informatica
Reporting and Analytics
Software Development
Spark
SQL
Job Description
Guidehouse is seeking a Cleared Data Engineer to design and maintain scalable batch and streaming data pipelines in Databricks, supporting cybersecurity metrics, risk analysis, and reporting. This position requires an Active Secret clearance and is based in Arlington, VA (onsite).
Key Responsibilities
- Design, develop, test, deploy, and maintain scalable batch and streaming data pipelines in Databricks using Python, PySpark, SQL, and reusable engineering patterns.
- Convert raw source data into curated, analysis-ready datasets and dimensional models for cybersecurity metrics, risk analysis, dashboards, and executive reporting.
- Create automated data-quality controls for ETL jobs, including checks for schema, completeness, validity, consistency, uniqueness, timeliness, referential integrity, and source-to-target reconciliation.
- Define handling procedures for failed quality checks, such as quarantining invalid records, stopping critical loads, generating alerts, documenting exceptions, and coordinating remediation with data owners.
- Schedule and orchestrate production workflows; monitor job status, freshness, throughput, latency, resource utilization, and failures; troubleshoot root causes; and restore processing within agreed service levels.
- Improve reliability, performance, and cost efficiency through transformation optimization, workload configuration, incremental processing, partitioning, and compute resource management.
- Implement and maintain data governance and quality assurance standards, including access controls, metadata, data classification, ownership, lineage, retention, auditability, and protection of sensitive cybersecurity information.
- Maintain source-to-target mappings, data dictionaries, transformation rules, validation evidence, and operational documentation such as standard operating procedures and runbooks.
- Work with cybersecurity analysts, data owners, system administrators, dashboard developers, and business stakeholders to define data requirements, metrics, thresholds, and acceptance criteria.
- Maintain client relationships by anticipating stakeholder needs, communicating delivery risks and dependencies, and resolving issues in a timely and professional manner.
- Organize and lead client meetings by coordinating schedules, preparing and distributing agendas and pre-read materials, facilitating discussions, recording minutes, maintaining documentation, and tracking action items through closure.
- Participate in peer reviews, testing, release activities, and continuous-improvement efforts to strengthen engineering practices and reduce recurring defects.
Required Qualifications
- ACTIVE and CURRENT SECRET federal security clearance
- Bachelor’s degree
- SIX (6) years total experience in a data or programming discipline, including THREE (3) years in data engineering
- Proficiency in data engineering programming languages such as Python, PySpark, and/or SQL
- Experience with scheduling, monitoring, troubleshooting, and optimizing production ETL/ELT pipelines
- Experience with data engineering principles and best practices
- Excellent problem-solving skills and attention to detail
- Selected Candidate must be able to commute to client and Guidehouse offices as needed
Technologies
- Databricks
- Python
- PySpark
- SQL
Benefits
- Medical, Rx, Dental & Vision Insurance
- Personal and Family Sick Time & Company Paid Holidays
- Position may be eligible for a discretionary variable incentive bonus
- Parental Leave and Adoption Assistance
- 401(k) Retirement Plan
- Basic Life & Supplemental Life
- Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
- Short-Term & Long-Term Disability
- Student Loan PayDown
- Tuition Reimbursement, Personal Development & Learning Opportunities
- Skills Development & Certifications
- Employee Referral Program
- Corporate Sponsored Events & Community Outreach
- Emergency Back-Up Childcare Program
- Mobility Stipend
Travel and Clearance
- Travel Required: Up to 50%
- Clearance Required: Active Secret
Nice to Have
- Master’s degree in computer science or related field
- Experience working with system and application logs
- Familiarity with the Databricks platform
- Strong communication and collaboration skills, with the ability to work effectively in a team environment
Compensation
USD 85,000 - 141,000 per yearly