Datastream Jobs

1 job found

Today

Senior Data Engineer

Handshake San Francisco, CA

San Francisco, CA

Today

$166k - $207k/yr Full time Senior Apache Airflow Application Security

Apply Now

Job Description

Handshake is building reliable infrastructure for a career marketplace and AI products, and the Senior Data Platform Engineer role is a central part of that mission. You will help shape and operate the systems that move data with confidence, support analytics and streaming use cases, and enable dependable foundations for AI workloads. The position is based onsite in San Francisco, CA.

Compensation: USD 166,000 - 207,000 per year. Handshake uses standard cash compensation ranges for U.S.-based roles based on function, level, and geographic location, benchmarked against similar stage growth companies. Final offers are determined by multiple factors, including geographic location and candidate experience and expertise.

What you’ll be doing

Set the technical direction and roadmap for the Airflow and Astronomer data orchestration platform while owning production operations, including deployment, DAG packaging, scheduler and worker capacity, upgrades, observability, and incident response.
Develop “paved paths” for authoring, testing, deploying, and operating workflows, including reusable operators, CI checks, local and staging environments, and runbooks.
Design and operate streaming and change data capture (CDC) pipelines using tools such as Pub/Sub, Dataflow or Beam, and Datastream to serve analytics and product use cases.
Make data delivery resilient to retries, duplicates, schema changes, late events, backfills, and replay, including defining latency, freshness, and reliability targets.
Strengthen the cloud foundation for data workloads using Kubernetes, Terraform, IAM, secrets, CI/CD, and cost-aware capacity management.
Lead cross-team decisions on data contracts, interfaces, and operational ownership, mentor engineers, and coordinate with application, analytics, ML, and cloud partners on scalable platform approaches.
Participate in on-call support, troubleshoot production failures across systems, and turn incidents into durable improvements.
Build dependable data and orchestration foundations for AI workloads, including batch inference, evaluation datasets, and agent-facing data.

What you’ll bring

Strong software engineering skills in Python and experience building and operating production data or distributed systems.
Deep hands-on Airflow experience beyond writing DAGs, including scheduling and execution behavior, deployment, scaling, upgrades, monitoring, and debugging failures.
Experience with event-driven or streaming infrastructure, including a message broker or managed event bus and a stream processing system.
Solid understanding of CDC, delivery guarantees, idempotency, ordering, schema evolution, and production recovery or replay.
Cloud infrastructure and infrastructure-as-code experience, comfortable with containers, Kubernetes, CI/CD, access controls, and production observability.
A track record leading ambiguous infrastructure initiatives, setting technical direction, making pragmatic architecture tradeoffs, and driving adoption across teams.
Clear communication and a demonstrated ability to partner across teams while owning systems through production support.

Tools you may work with

Python, Airflow, Astronomer, Pub/Sub, Dataflow, Beam, Datastream, Datadog
BigQuery, Kubernetes, Terraform, IAM, CI/CD, GKE, Cloud Storage, Spacelift
dbt, Spark, Dataproc

Benefits

Equity in a fast-growing company
401(k) match, competitive compensation, and financial coaching
Paid parental leave, fertility benefits, and parental coaching
Medical, dental, & vision plus mental health support, and a $500 wellness stipend
$2,000 learning stipend and ongoing development
Internet, commuting, and free lunch/gym in the SF office
Flexible PTO, 15 holidays + 2 flex days
Team outings and referral bonuses

Extra credit

GCP experience across Pub/Sub, Dataflow, Datastream, BigQuery, GKE, and Cloud Storage
Experience with Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc
Supporting both batch and low-latency consumers, including product-facing data services or ML features
Experience supporting ML or AI workloads, such as inference pipelines, reproducible evaluations, or governed data access

View full job page →

Get Datastream Jobs alerts

New Datastream Jobs, daily.