Lead Data Engineer
CI/CD
Data
Data Analysis
Data Architecture
Data Engineer
Data Governance
Data Integration
Data Integration Tools
Data Management
Data Pipeline
Data Platform
Data Processing
Data Security
Database
DevOps
Devops Tools
Engineer
ETL
Informatica
Software Development
Software Engineering
Workflow Orchestration
Job Description
Lead Data Engineer role on Prove’s Data Science team, focused on building high-throughput pipelines and DAG-based workflows that support verified digital trust.
Responsibilities
- Architect, build, and maintain resilient data pipelines powering real-time and batch inference, transforming raw intelligence into verifiable scientific statements
- Translate complex, multidisciplinary domain logic into efficient, maintainable, modular DAGs and execution workflows
- Implement and advocate for data governance frameworks, including metadata management and semantic data ontologies
- Partner with data science, informatics, and product engineering teams to deliver high-integrity data extracts for revenue-generating products
- Continuously evaluate, benchmark, and adopt data engineering tools, frameworks, and storage paradigms
- Optimize distributed data processing for high availability, computational efficiency, and cost governance
- Drive engineering excellence through code reviews, architectural discussions, automated testing, and CI/CD practices
- Promote and maintain Prove’s cultural values: humility, passion, inclusion, and leadership
Requirements
- 7+ years of professional experience in data engineering or developing complex, data-driven software products
- Deep, first-principles understanding of data systems, distributed compute engines, data modeling, and modern pipeline architecture
- Proven ability to collaborate across technical and scientific functions, converting cross-disciplinary logic into concrete data systems
- High intellectual curiosity and proactive experimentation with new approaches and technologies
- Hands-on experience with modern data orchestration, e.g. Airflow, Dagster, or Prefect, and large-scale distributed frameworks
- Strong passion for learning about products and markets via in-house and external training
Technologies
- Airflow
- Dagster
- Prefect
- Linux/Unix
- Shell scripting
- CI/CD
Preferred Qualifications
- Advanced degree (Master’s or Ph.D.) in Computer Science, Mathematics, Statistics, Physics, or a related quantitative field
- Experience architecting solutions for identity resolution, graph data structures, or fraud prevention systems
- Fluent, terminal-native experience in Linux/Unix, command-line utilities, and shell scripting
- Experience in high-growth or pre-IPO Technology companies
Benefits
- Competitive salary plus Bonus Plan (for eligible roles) and an Equity Plan
- Modern Health for financial, mental, and physical wellness
- 401(k) Retirement Plan & Match (US Offices) and Local Country Pension (International Offices)
- Unlimited Vacation and flexible hours
- Comprehensive medical benefits for you and your family
- Emotional & Physical Wellness: access to wellness services (EAP & Prove Well-Being Reimbursement)
- Bottomless snacks & beverages for certain office locations
- Daily GrubHub stipend for lunch if coming into the office (US Offices)
Location
- Hybrid - New York, NY
Additional Information
- Department: Data Science
- Reports to: Director of Informatics
- FLSA status: Exempt
- Anticipated salary pay range (Metro 2 areas): $175,000 - $200,000 plus company bonus
- Metro 2 definition: NYC metro area, Seattle metro area, Los Angeles metro area, and the Miami metro area
- Metro 3 definition: all other cities across the domestic United States, excluding the San Francisco Bay Area
Similar Jobs
J