Principal Data Engineer
Artificial Intelligence
Aws Data Engineer
Aws Data Engineering
Big Data
Cloud Data Engineering
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Engineering Lead
Data Integration
Data Integration Tools
Data Pipeline
Data Platform
Data Processing
Data Science
Database
Databases
Engineering
ETL
Informatica
Integration
Lead Data Engineering
Software Development
SQL
Technical Lead
Job Description
Lead the evolution of a production AWS data platform supporting human genetics, multi-omics, and high-content imaging workflows.
- Technical ownership and architecture of the AWS environment, including account structure, IAM, networking, compute, and storage.
- Cost governance for the platform: budgets, alerting, tagging enforcement, rightsizing, and storage lifecycle and tiering.
- Developer operations for the data platform: infrastructure as code, CI/CD, container builds, and secrets and environment management.
- Platform modernization and scaling: upgrade the data lake, managed orchestration, metadata, and data ingestion workflows to support future data production and strengthen lab-to-cloud data transfer.
- Cross-functional partnership with ML and computational biology leaders to define the interface between the platform and modelling.
Requirements
- 10+ years building and operating production data infrastructure, with experience architecting and owning platforms end to end.
- Deep AWS expertise across S3, IAM, EC2, Batch, Lambda, and Spark (EMR or equivalent).
- Experience as technical owner of production cloud environments, including multi-account architecture and cross-account access.
- Strong Python and SQL with a background in production-grade pipeline development.
- Production experience with workflow orchestration such as Nextflow, Airflow, Temporal, Step Functions, Redun, or similar, plus judgement to evaluate and select tools.
- Infrastructure as code using CloudFormation or CDK, plus CI/CD, containers, and observability; DevOps is a core competency.
- Data modelling and warehouse or lakehouse design, including metadata management, lineage, and data quality practices.
- Cost governance at scale including budgets, tagging, rightsizing, and storage lifecycle across hundreds of terabytes.
- Fluency with scientific data semantics (studies, samples, donors, cell lines, assays) and the ability to translate scientist questions into data models.
Technologies
- AWS, S3, IAM, EC2, Batch, Lambda, Spark, EMR
- Python, SQL
- Nextflow, Airflow, Temporal, Step Functions, Redun
- CloudFormation, CDK, CI/CD, containers, observability
- Databricks, OpenMetadata, Snowflake, SageMaker, MLflow, Weights & Biases
- Feature stores
- LLM, MCP servers, tool APIs, Infrastructure as code
Benefits
- Healthcare coverage
- Annual incentive program
- Retirement benefits
- A broad range of other benefits
Preferred Qualifications
- Familiarity with genetics and genomics data.
- Microscopy or high-content screening data experience, including large binary object stores, transfer optimization, and image streaming for model training.
- Experience with Databricks, OpenMetadata, or a comparable catalog in production.
- Experience with Snowflake, especially governed access patterns over cloud object storage.
- ML infrastructure experience: SageMaker or Databricks ML; MLflow or Weights & Biases; feature stores.
- Experience serving data to LLM or agent clients (MCP servers, tool APIs).
- Experience leading data platform architecture as a senior or principal individual contributor, ideally in biotech, pharma R&D, or a research institute.
Location: Cambridge, MA (onsite)
Salary: USD 179,000 - 236,000 per year
Similar Jobs
