Data Engineer Lead

Build production-ready data platforms at AIG, partnering closely with product and stakeholders while keeping engineering standards high. This hands-on Data Engineer Lead role combines strong pipeline delivery with leadership through reviews, mentorship, and documentation. You will also leverage GenAI tooling, including Claude Code and Snowflake Cortex, as part of day-to-day development to improve code quality and productivity.

Based in Jersey City, NJ (onsite), this role includes a $125,000 to $135,000 base salary range and eligibility for a bonus under the terms of the applicable incentive plan.

What you’ll do

• Design, build, and operate production-grade PySpark and Python pipelines on AWS EMR, Glue, and S3, with integrated data quality checks and observability.
• Own end-to-end delivery of data products, from requirements through production readiness, working in an Agile/Scrum workflow that includes sprint planning, CI/CD, release management, and production readiness.
• Turn business and product requirements into concrete technical designs, including pipeline structure, data models, and defined SLAs.
• Apply architecture and design patterns with clear rationale, evaluating trade-offs and aligning decisions with peers and stakeholders.
• Set and enforce team engineering standards, including coding conventions, data quality frameworks, reusable pipeline patterns, and observability hooks.
• Drive technical quality through thorough code reviews and pairing on difficult problems.
• Use GenAI tooling across the development lifecycle, including Claude Code CLI and Snowflake Cortex, to improve solution quality and delivery speed.
• Collaborate with product owners to refine requirements, surface trade-offs, and push back constructively when scope is not technically workable.
• Communicate design decisions and delivery updates to both technical and non-technical stakeholders.
• Mentor junior engineers through reviews, pairing, and knowledge sharing while building team capability.
• Produce clear technical documentation as a standard deliverable, including Architecture Decision Records (ADRs), runbooks, data dictionaries, and pipeline lineage notes.
• Evaluate and drive adoption of productivity tooling and best practices, setting team norms for safe and effective use.

What you bring

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related technical field.
• 7+ years of hands-on data engineering experience delivering production data pipelines end-to-end.
• Strong command of PySpark, Python, and SQL, including Spark optimization, partition management, and query performance tuning in Snowflake.
• Hands-on experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM, and CloudWatch.
• Working knowledge of architecture and design principles such as lakehouse patterns and fault-tolerant ingestion, applied in practice.
• Strong analytical and problem-solving skills, including decomposing complex requirements and reasoning through trade-offs.
• Clear communication skills to explain technical decisions and design rationale to engineering peers and business stakeholders.
• Self-motivated learner, interested in staying current with evolving tools, frameworks, and engineering best practices.

Technologies you’ll work with

PySpark, Python, AWS EMR, AWS Glue, Amazon S3, AWS Aurora, AWS Lambda, IAM, CloudWatch, SQL, Snowflake, Snowflake Cortex, Claude Code, Claude Code CLI, GenAI, Agile/Scrum, CI/CD, Architecture Decision Record (ADR).

Benefits and growth

• Eligible bonus in accordance with the terms of the applicable incentive plan.
• Comprehensive benefits package including health, wellbeing, financial security, and professional development.
• Hybrid flexibility.
• Competitive compensation, with a base salary range of $125,000 to $135,000.
• Created a Total Rewards Program, extending benefits beyond time spent at work.

Culture fit

At AIG, the Data Office is a group of builders, thinkers, and pragmatic problem-solvers. Great data engineering is treated as a team sport, with careful documentation, collaborative design reviews, mentorship across every level, and close partnership with business stakeholders. GenAI is encouraged as a practical force multiplier, not a novelty.

Nice to have

• AWS certification such as Solutions Architect Associate or equivalent.
• Familiarity with insurance or financial services data (claims, policy, risk, or actuarial structures).
• Experience writing ADRs, design notes, or technical documentation in a collaborative engineering team.
• Demonstrated use of GenAI tooling (for example Claude Code CLI or Snowflake Cortex) to improve productivity or solution quality.

In-person collaboration: AIG values in-person collaboration and asks team members to be primarily in the office.