Senior Data Engineer

Handshake is building reliable infrastructure for a career marketplace and AI products, and the Senior Data Platform Engineer role is a central part of that mission. You will help shape and operate the systems that move data with confidence, support analytics and streaming use cases, and enable dependable foundations for AI workloads. The position is based onsite in San Francisco, CA.

Compensation: USD 166,000 - 207,000 per year. Handshake uses standard cash compensation ranges for U.S.-based roles based on function, level, and geographic location, benchmarked against similar stage growth companies. Final offers are determined by multiple factors, including geographic location and candidate experience and expertise.

What you’ll be doing

• Set the technical direction and roadmap for the Airflow and Astronomer data orchestration platform while owning production operations, including deployment, DAG packaging, scheduler and worker capacity, upgrades, observability, and incident response.
• Develop “paved paths” for authoring, testing, deploying, and operating workflows, including reusable operators, CI checks, local and staging environments, and runbooks.
• Design and operate streaming and change data capture (CDC) pipelines using tools such as Pub/Sub, Dataflow or Beam, and Datastream to serve analytics and product use cases.
• Make data delivery resilient to retries, duplicates, schema changes, late events, backfills, and replay, including defining latency, freshness, and reliability targets.
• Strengthen the cloud foundation for data workloads using Kubernetes, Terraform, IAM, secrets, CI/CD, and cost-aware capacity management.
• Lead cross-team decisions on data contracts, interfaces, and operational ownership, mentor engineers, and coordinate with application, analytics, ML, and cloud partners on scalable platform approaches.
• Participate in on-call support, troubleshoot production failures across systems, and turn incidents into durable improvements.
• Build dependable data and orchestration foundations for AI workloads, including batch inference, evaluation datasets, and agent-facing data.

What you’ll bring

• Strong software engineering skills in Python and experience building and operating production data or distributed systems.
• Deep hands-on Airflow experience beyond writing DAGs, including scheduling and execution behavior, deployment, scaling, upgrades, monitoring, and debugging failures.
• Experience with event-driven or streaming infrastructure, including a message broker or managed event bus and a stream processing system.
• Solid understanding of CDC, delivery guarantees, idempotency, ordering, schema evolution, and production recovery or replay.
• Cloud infrastructure and infrastructure-as-code experience, comfortable with containers, Kubernetes, CI/CD, access controls, and production observability.
• A track record leading ambiguous infrastructure initiatives, setting technical direction, making pragmatic architecture tradeoffs, and driving adoption across teams.
• Clear communication and a demonstrated ability to partner across teams while owning systems through production support.

Tools you may work with

• Python, Airflow, Astronomer, Pub/Sub, Dataflow, Beam, Datastream, Datadog
• BigQuery, Kubernetes, Terraform, IAM, CI/CD, GKE, Cloud Storage, Spacelift
• dbt, Spark, Dataproc

Benefits

• Equity in a fast-growing company
• 401(k) match, competitive compensation, and financial coaching
• Paid parental leave, fertility benefits, and parental coaching
• Medical, dental, & vision plus mental health support, and a $500 wellness stipend
• $2,000 learning stipend and ongoing development
• Internet, commuting, and free lunch/gym in the SF office
• Flexible PTO, 15 holidays + 2 flex days
• Team outings and referral bonuses

Extra credit

• GCP experience across Pub/Sub, Dataflow, Datastream, BigQuery, GKE, and Cloud Storage
• Experience with Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc
• Supporting both batch and low-latency consumers, including product-facing data services or ML features
• Experience supporting ML or AI workloads, such as inference pipelines, reproducible evaluations, or governed data access