Data Engineer - Managing Consultant

Guidehouse is seeking a Data Engineer with Managing Consultant responsibilities for an on-site role in McLean, Virginia. The position centers on designing and delivering scalable, governed data pipelines and analytics solutions using Databricks and AWS, with a focus on security, observability, and enterprise data availability. This role blends hands-on engineering with governance and delivery leadership to sustain trusted data across the organization.

Responsibilities

• Design, build, and optimize scalable, production-grade data ingestion, transformation, and analytics-ready pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services to enable trusted data access across enterprise use cases.
• Engineer standardized pipelines supporting batch and near real-time processing, integrating legacy data sources with modern cloud-native services to improve enterprise data availability and reduce data access gaps.
• Execute full delivery lifecycle by supporting intake, discovery, source profiling, and technical design, while maintaining requirements traceability and alignment with Architecture Review Board and governance expectations.
• Implement and maintain governed data pipelines with embedded metadata, lineage, and data quality controls to meet technical, security, and documentation requirements before production deployment.
• Develop and operationalize data engineering frameworks that incorporate observability, monitoring, alerting, and resilient error handling to maintain platform stability and support 99.9% availability targets.
• Partner with platform engineering and cloud operations teams to integrate pipelines with AWS services (S3, Glue, Kafka/Kinesis, APIs) for secure, scalable data movement and cross-platform interoperability.
• Enable governed analytics and self-service data consumption through curated datasets, semantic layers, and SQL warehouse integration to support enterprise reporting, dashboards, and advanced analytics use cases.
• Apply security and compliance controls aligned to IRS cybersecurity policies, including RBAC/ABAC, data masking, encryption, and audit logging to protect sensitive data and maintain regulatory compliance.

Requirements

• Bachelor’s degree required
• Minimum eight (8) years of data engineering experience in cloud-based environments
• Minimum four (4) years of hands-on Databricks experience designing scalable data pipelines
• Advanced proficiency in Python, PySpark, and SQL for large-scale data processing and pipeline development in lakehouse architectures
• Demonstrated experience building and optimizing data pipelines leveraging Delta Lake, medallion architecture, and modern data ingestion frameworks (batch and streaming)
• Strong experience with AWS data platforms and services (S3, IAM, VPC, Glue, streaming frameworks) and integration with enterprise data ecosystems
• Experience delivering production-ready data solutions incorporating metadata management, lineage, data quality, and observability frameworks
• Familiarity with DevSecOps and CI/CD pipeline implementation, including automation, testing, and deployment within cloud data environments
• Knowledge of data governance, security, and compliance requirements within regulated environments, including FISMA and FedRAMP High

Technologies

• Databricks, Delta Lake, Delta Live Tables, Auto Loader
• AWS: S3, Glue, IAM, VPC, Kafka, Kinesis, APIs
• Python, PySpark, SQL
• Delta Sharing, Unity Catalog, Genie, Clean Rooms
• Informatica EDC/Axon, EventBridge
• Terraform, GitHub Actions, Databricks Asset Bundles
• Photon, Z-ordering, liquid clustering

Benefits

• Medical, Rx, Dental & Vision Insurance
• Personal and Family Sick Time & Company Paid Holidays
• Discretionary variable incentive bonus may apply
• Parental Leave and Adoption Assistance
• 401(k) Retirement Plan
• Basic Life & Supplemental Life
• Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
• Short-Term & Long-Term Disability
• Student Loan PayDown
• Tuition Reimbursement, Personal Development & Learning Opportunities
• Skills Development & Certifications
• Employee Referral Program
• Corporate Sponsored Events & Community Outreach
• Emergency Back-Up Childcare Program
• Mobility Stipend

Job Family

Data Engineering & Architecture Consulting

Travel

Up to 10%

Clearance

Ability to Obtain Public Trust

Nice to Have

• Experience supporting federal data platforms or large-scale enterprise data modernization efforts within IRS, Treasury, or similar regulated environments
• Hands-on experience with Databricks Unity Catalog, Delta Sharing, Genie, and Clean Rooms for governed data access and collaboration
• Experience implementing streaming and near real-time data pipelines using Kafka, Kinesis, EventBridge, or similar technologies
• Familiarity with Informatica EDC/Axon or enterprise metadata/catalog tooling
• Experience with performance optimization techniques (Photon, Z-ordering, liquid clustering) for large-scale data workloads
• Exposure to CI/CD automation using Terraform, GitHub Actions, and Databricks Asset Bundles for infrastructure and pipeline deployment
• Advanced cloud or Databricks certifications (AWS, Databricks) in good standing