Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst)
Job Description
Photon seeks a Senior Data Engineer in New York, NY (onsite) to lead data federation and lakehouse initiatives.
Responsibilities
- Data Federation Architecture: Design and implement federated query layers using Starburst Enterprise or Trino to enable high-speed analytics across distributed data sources with minimal data movement.
- ETL/ELT Pipeline Development: Build scalable, distributed data processing pipelines with Python and Apache Spark (PySpark).
- Lakehouse Implementation: Manage and optimize Delta Lake, Apache Iceberg, or Apache Hudi to bring ACID transactions to the data lake.
- Performance Tuning: Optimize Spark jobs and SQL queries across the federation layer to minimize latency and control compute costs.
- Governance & Security: Implement fine-grained access control and data masking within the federation engine to safeguard privacy across connected platforms.
Requirements
- Python & Spark: 5+ years of experience with Python and deep expertise in Apache Spark tuning (partitioning, shuffling, caching).
- Data Federation Tools: Hands-on experience with Starburst Enterprise, Trino (Presto), or Dremio.
- Lakehouse Ecosystem: Proven track record working with Delta Lake or Iceberg architectures.
- Cloud Platforms: Extensive experience with AWS (EMR, S3, Glue), Azure (Databricks, ADLS), or GCP.
- SQL Mastery: Expert-level SQL skills for complex analytical queries and query plan analysis.
- Data Modeling: Proficiency in designing Star/Snowflake schemas and understanding Medallion Architecture (Bronze, Silver, Gold layers).
Technologies
- Python
- Apache Spark
- PySpark
- Starburst Enterprise
- Trino (Presto)
- Dremio
- Delta Lake
- Iceberg
- Hudi
- AWS
- EMR
- S3
- Glue
- Azure
- Databricks
- ADLS
- GCP
- SQL
- dbt
- Terraform
- Pulumi
- Kubernetes (K8s)
Benefits
- Medical, vision, and dental benefits
- 401k retirement plan
- Variable pay/incentives
- Paid time off
- Paid holidays
Compensation, Benefits and Duration
- Minimum Compensation: USD 40,000
- Maximum Compensation: USD 140,000
- Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and good faith estimate for the role.
- Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full-time employees.
- This position is not available for independent contractors
- No applications will be considered if received more than 120 days after the date of this post
Similar Jobs
N