Data Engineer 5

Capital One seeks a Data Engineer 5 (Senior Manager, IC) to lead a major data transformation within Enterprise Data Tech (EDT). The role focuses on designing and delivering scalable, resilient, cloud-first data pipelines and platforms, using technologies commonly applied in the AI space.

Responsibilities

• Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions using full-stack development tools and technologies
• Influence a team of developers, data analysts, and data scientists with expertise across machine learning, distributed microservices, lakehouse architecture, and full-stack systems
• Use programming languages including Python and Spark, along with open-source relational and NoSQL databases and cloud data warehousing platforms such as Databricks and Snowflake
• Stay current with data technology trends through experimentation, learning new technologies, participating in internal and external technology communities, and mentoring members of the data community
• Work with product managers and software engineers to deliver robust cloud-first data solutions that support experiences for millions of Americans seeking financial empowerment
• Independently design, build, and deliver cloud data solutions and applications with little or no support from supervisors or managers
• Architect and apply common data engineering design patterns to support code quality, maintainability, and reusability across platforms and pipelines
• Design and build data pipelines and platforms emphasizing scalability, resilience, and operational efficiency, with performance under increasing data volume and business demand
• Act as a force multiplier through a blend of hands-on technical contribution, innovation, mentoring, and elevation of peers and junior engineers
• Lead and execute large-scale, transformative data initiatives end to end, including critical architectural decisions and platform evaluations such as Snowflake versus Databricks based on technical and business requirements
• Serve as an ambassador for data engineering by communicating technical concepts and data outcomes clearly to internal and external stakeholders to drive alignment

Requirements

• Bachelor's degree or higher in Computer Science or a related quantitative field (Statistics, Economics, Operations Research, Analytics, Mathematics, Engineering)
• At least 6 years of experience in application development (internship experience does not apply)
• At least 4 years of experience in distributed data
• At least 4 years of experience with SQL
• At least 4 years programming with Python, Java, or Scala
• At least 4 years designing and developing data pipelines
• At least 2 years of experience in data modeling and designing end-to-end data solutions using both relational and non-relational database systems

Technologies

• Python, Spark
• Databricks, Snowflake
• SQL, NoSQL, relational databases
• AWS, EMR, Glue
• Lakehouse architecture, microservices
• Redshift
• MongoDB, Cassandra, DynamoDB
• Airflow, Dagster
• Monte Carlo, Splunk
• Real-time, streaming data pipelines

Location

New York, NY (onsite)

Compensation

USD 250,800 - 286,200 per year

Benefits

• Health, financial, and other benefits supporting total well-being
• Performance-based incentive compensation, including cash bonus(es) and/or long term incentives (LTI)

Preferred Qualifications

• Master’s degree in Computer Science or a related field
• 8+ years of experience in data engineering
• 5+ years of hands-on experience designing, deploying, and operating data workloads in AWS
• 5+ years of experience building or supporting distributed data or compute workloads using EMR, Spark, Glue, or Databricks
• 5+ years of experience designing, implementing, and operating real-time or streaming data pipelines
• 5+ years of experience working with unstructured or semistructured data using NoSQL databases (MongoDB, Cassandra, DynamoDB)
• 5+ years of experience designing and supporting data warehousing solutions (Snowflake, Redshift)
• 4+ years of data modeling experience
• 3+ years of experience working in an Agile development environment
• 3+ years of experience developing user-centric reusable data products
• 3+ years of experience working on data observability (Monte Carlo, Splunk) or data orchestration tools (Airflow, Dagster)