Lead Software Engineer - Data Engineer

JPMorganChase's CIB Regulatory Reporting Team in Jersey City, NJ is seeking a hands-on Lead Software Engineer - Data Engineer to architect and deliver Spark-based ETL/ELT data solutions. This onsite role combines production Python and PySpark development with Spark SQL in an agile setting, guided by AI-assisted development practices to boost security, stability, and scalability. The position offers a competitive USD 152,000 - 215,000 per year and requires at least five years of relevant experience.

Responsibilities

• Develops innovative software solutions and conducts technical troubleshooting with an emphasis on data engineering and Spark-based ETL/ELT, going beyond routine approaches.
• Produces secure, production-grade Python/PySpark code and Spark SQL logic, and reviews others’ Spark jobs, SQL logic, and data issues end-to-end.
• Leads the adoption of enterprise-authorized AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes, establishing validation standards and promoting reuse of effective patterns across the team.
• Utilizes the Software Development Life Cycle toolchain, including AI-assisted development and automation capabilities, to maximize the value of automation efforts.
• Identifies opportunities to automate remediation of recurring issues to enhance the operational stability of software applications and data pipelines, including lakehouse maintenance automation.
• Leads evaluation sessions with external vendors, startups, and internal teams to assess architectural designs and applicability within existing systems and information architecture, including EMR/Databricks, lakehouse formats, and governance patterns.
• Leverages and grows communities of practice across Software Engineering to promote new technologies, with a focus on Spark performance, Iceberg best practices, and data platform operations.
• Contributes to a culture that values diversity, opportunity, inclusion, and respect.

Requirements

• Formal training or certification in software engineering accompanied by at least five years of applied experience.
• 5+ years building production data engineering or software engineering solutions across design, development, testing, and operations.
• Hands-on experience delivering system design, application development, testing, and operational stability for large-scale data pipelines.
• Advanced proficiency in Python with strong PySpark experience; solid practice in Spark SQL and SQL fundamentals (data modeling, optimization, execution plans).
• Proven track record leading use of approved AI-assisted development tools for coding, code review, and test acceleration, with the ability to set team expectations for validating AI outputs.
• Strong understanding of responsible AI use in engineering, including data sensitivity, secure handling of inputs/outputs, and adherence to resiliency and security requirements; experience coaching peers on safe adoption.
• Experience with AWS data management patterns (S3, AWS Glue Data Catalog) and familiarity with other cloud data platforms.
• Platform experience delivering and operating Spark workloads on EMR and/or Databricks, including tuning, troubleshooting, monitoring, cost, and performance optimization.
• Lakehouse expertise with Apache Iceberg in production, covering table design, partitioning, schema evolution, compaction, small-file mitigation, snapshot retention, backfills, and reprocessing patterns.
• Proficiency in automation and continuous delivery methods (CI/CD, automated testing, repeatable deployments for data pipelines).

Technologies

• Python
• PySpark
• Spark SQL
• Apache Spark
• Databricks
• EMR
• AWS S3
• AWS Glue Data Catalog
• Apache Iceberg
• Delta Lake
• Kafka
• Java
• GitHub Copilot
• Claude

Benefits

• Comprehensive health care coverage
• On-site health and wellness centers
• Retirement savings plan
• Backup childcare
• Tuition reimbursement
• Mental health support
• Financial coaching
• Commission-based pay
• Discretionary incentive compensation (cash and/or forfeitable equity)

About Us

JPMorganChase is one of the oldest financial institutions, delivering innovative financial solutions to millions of consumers, small businesses, and a broad range of corporate, institutional, and government clients under the J.P. Morgan and Chase brands. With a history spanning over two centuries, the firm is a leader across investment banking, consumer and small business banking, commercial banking, financial transaction processing, and asset management.

About the Team

J.P. Morgan's Commercial & Investment Bank operates as a global leader across banking, markets, securities services, and payments. Corporations, governments, and institutions worldwide rely on the team for strategic advisory, capital raising, risk management, and liquidity support in more than 100 countries.