Senior Data Engineer (Customer Data Products)

NBME is modernizing an AWS-based data platform to support analytics, AI/ML, and business decision-making, and this Senior Data Engineer role is central to that effort. You will build and optimize data lakes and scalable pipelines, partner with technical and business stakeholders, and help bring emerging data and AI technologies into production-grade capabilities.

What you’ll work on

• Develop, test, deploy, orchestrate, monitor, document, and troubleshoot cloud-based data engineering processes across the full development lifecycle, including feature stores and vector databases, aligned with best practices and security standards.
• Collaborate with data scientists, AI researchers, data and enterprise architects, and business stakeholders to identify, extract, clean, and format structured and unstructured data for AI/ML training, fine-tuning, and feature extraction.
• Lead evaluation, research, and experimentation using batch and streaming technologies, LLM data preparation frameworks, and MLOps tools.
• Act as a technical lead for emerging AI and data technologies, enabling broader adoption of modern data techniques across the organization.
• Contribute to defining and refining data engineering processes and procedures.
• Support ETL developer education and development related to cloud-based data engineering initiatives to enable the transition toward data engineering practice.
• Ensure corporate data integrity, accuracy, and security, with attention to data quality for Data Services, Analytics, and Master Data Management.
• Own technical design and delivery by helping coordinate solutions, including development, testing, and release to production.
• Build automated, scalable, test-driven data pipelines.
• Use software development practices such as Git, CI/CD, and release management to enhance AWS CI/CD pipelines.
• Work with Data Engineers and DevOps engineers to improve DataOps tools and frameworks.

Key requirements

• Bachelor’s Degree
• 7+ years of experience in application development (internship experience does not apply)
• 4+ years with big data technologies
• 4+ years with AWS cloud computing
• 4+ years in application development including Python, SQL, Scala, or Java
• 4+ years with distributed data or computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, MySQL, etc.)
• 4+ years building real-time data and streaming applications
• 4+ years implementing NoSQL (Mongo, Cassandra)
• 4+ years with data warehousing (Redshift)
• 6+ years with UNIX/Linux, including basic commands and shell scripting
• 7+ years with Agile engineering practices
• 7+ years of SQL optimization
• 4+ years with PySpark
• 3+ years with process orchestration including Airflow, KubeFlow, AWS Step Functions, or Luigi

Technologies you’ll use

AWS, Python, SQL, Scala, Java, MapReduce, Hadoop, Hive, EMR, Kafka, Spark, MySQL, Mongo, Cassandra, Redshift, UNIX/Linux, shell scripting, Git, CI/CD, Airflow, KubeFlow, AWS Step Functions, Luigi, PySpark, feature stores, vector databases (Pinecone, Milvus, pgvector), LLM data preparation frameworks, MLOps tools, DataOps tools, APIs, Data-as-a-Service (DaaS), Amazon Redshift.

Preferred qualifications

• Proven experience implementing Generative AI, LLM data preparation pipelines, and vector databases (Pinecone, Milvus, pgvector)
• Strong experience building and maintaining feature stores for machine learning models
• Experience building highly scalable, secure, production-ready APIs and Data-as-a-Service (DaaS) platforms
• AWS Certified Data Engineer or AWS Certified Machine Learning – Specialty certifications
• 3+ years of experience with Machine Learning
• Experience building a Data-as-a-service platform
• Experience building APIs

Benefits

• Healthcare, Dental, Prescription, and Vision plans
• 401(k) with match
• Tuition Reimbursement Plan
• Commuter Benefit: Public Transit or Parking options
• Remote-friendly workplace

Hybrid position details

• This role is designated as hybrid, with occasional onsite work for team collaboration or meetings.
• NBME is prioritizing applicants within a 50-mile radius from the Philadelphia office.
• Onsite interviews and onboarding at the Philadelphia office may be required; advance notice will be provided if applicable.

Location: Philadelphia, PA (hybrid). Salary range: USD 127,337 - 160,000 per year.