Senior Lead Data Engineer

Capital One is seeking a Senior Lead Data Engineer to lead enterprise-grade, streaming-focused data platform development and distributed pipeline engineering in a low-latency environment.

Responsibilities

• Lead the design and build of an enterprise scalable, low-latency, fault-tolerant streaming data platform to deliver meaningful, timely insights
• Build next-generation distributed streaming data pipelines and analytics data stores using streaming frameworks such as Flink and Spark Streaming
• Develop streaming and analytics solutions using Java, Scala, and Python
• Lead a group of engineers building data pipelines on medium to large scale datasets using technologies including Spark, Flink, Kafka, Snowflake, AWS Big Data Services, and Redshift
• Shape best practices for data pipeline design, data architecture, and processing for structured and unstructured data
• Partner with agile teams, with a focus on CI/CD and application resiliency standards, and coordinate with Cyber & Security
• Collaborate across agile teams to design, develop, test, implement, and support solutions using full-stack development tools and technologies
• Lead developers with experience in machine learning, distributed microservices, and full stack systems
• Use cloud-based data warehousing services such as Redshift and Snowflake, along with open source RDBMS and NoSQL databases
• Perform unit tests and conduct code reviews to ensure rigorous design, clean implementation, and performance tuning
• Collaborate with digital product managers to deliver robust cloud-based solutions that support financial empowerment for millions of Americans
• Stay current with technology trends by experimenting with and learning new technologies; participate in internal and external technology communities and mentor others

Requirements

• Bachelor’s Degree
• 6+ years of experience in application development (internship experience does not apply)
• 2+ years of experience with big data technologies
• 1+ year experience with cloud computing: AWS, Microsoft Azure, or Google Cloud

Preferred Qualifications

• Master’s Degree
• 9+ years of application development experience, including Python, SQL, Scala, or Java
• 4+ years of experience with a public cloud (AWS, Microsoft Azure, or Google Cloud)
• 5+ years experience with distributed data/computing tools such as MapReduce, Hadoop, Hive, EMR, Kafka, Spark, Gurobi, or MySQL
• 4+ years working on real-time data and streaming applications
• 4+ years experience implementing NoSQL systems (Mongo, Cassandra)
• 4+ years of data warehousing experience (Redshift or Snowflake)
• 4+ years of experience with UNIX/Linux, including basic commands and shell scripting
• 2+ years of experience with Agile engineering practices
• Experience leveraging interactive AI tooling to accelerate productivity, using capabilities beyond basic code completion

Technologies

• Flink, Spark Streaming, Java, Scala, Python, Spark, Kafka
• Snowflake, AWS Big Data Services, Redshift
• Open Source RDBMS, NoSQL
• AWS, Microsoft Azure, Google Cloud
• CI/CD

Benefits

• Eligible for performance-based incentive compensation, which may include cash bonus(es) and/or long term incentives (LTI)
• Comprehensive, competitive, and inclusive health, financial, and other benefits supporting total well-being

Compensation and Location

• Location: Plano, TX (onsite)
• Salary: $209,000 - $238,500 per year
• Performance-based incentive compensation eligibility (cash bonus(es) and/or LTI)

Additional Notes

• Capital One may sponsor authorization for a new qualified applicant for this position
• Applications expected to be accepted for a minimum of 5 business days
• No agencies please; Equal Opportunity Employer (EOE, including disability/vet); non-discrimination per applicable federal, state, and local laws
• Drug-free workplace
• Accommodation contact: Capital One Recruiting at 1-800-304-9102 or [email protected]
• Recruiting process technical support: [email protected]
• Third-party products/services/tools referenced through the site are not provided, endorsed, guaranteed, or liable by Capital One