Data Pipeline

Data Pipeline Jobs

593 jobs found

22d ago

Senior Machine Learning Engineer (AI Foundations)

Capital One McLean, VA

McLean, VA

22d ago

$162k - $185k/yr Full time Ai Ml Artificial Intelligence Big Data

Apply Now

Job Description

The Senior Machine Learning Engineer, AI Foundations, contributes to Capital One's production ML initiatives within an Agile team. The role centers on scalable ML architecture, code review for models and applications, and ensuring the reliability and performance of ML systems, with opportunities to apply contemporary ML engineering practices.

Compensation

Salary: USD 161,800 - 184,600 per year.

Responsibilities

Design, build, and deliver ML models and components that address real business needs, collaborating with Product and Data Science teams.
Guide ML infrastructure decisions based on modeling techniques and issues, including model choice, data and feature selection, training, hyperparameter tuning, dimensionality, bias/variance, and validation.
Address complex problems by writing and testing application code, developing and validating ML models, and automating tests and deployment.
Collaborate within a cross-functional Agile team to create and enhance software enabling state-of-the-art big data and ML applications.
Retrain, maintain, and monitor models in production.
Leverage or build cloud-based architectures, technologies, and platforms to deliver optimized ML models at scale.
Construct optimized data pipelines to feed ML models.
Apply continuous integration and continuous deployment best practices, including test automation and monitoring, to ensure successful deployment of ML models and application code.
Ensure all code is well-managed to reduce vulnerabilities, models are well-governed from a risk perspective, and the ML practices align with Responsible and Explainable AI.
Use programming languages such as Python, Scala, or Java.

Requirements

Bachelor’s Degree.
At least 4 years of experience programming with Python, Scala, or Java (Internship experience does not apply).
At least 3 years of experience designing and building data-intensive solutions using distributed computing.
At least 2 years of on-the-job experience with industry-recognized ML frameworks (scikit-learn, PyTorch, Dask, Spark, or TensorFlow).
At least 1 year of experience productionizing, monitoring, and maintaining models.

Technologies

Python
Scala
Java
scikit-learn
PyTorch
Dask
Spark
TensorFlow

Benefits

Health benefits
Financial benefits
Performance-based incentive compensation (cash bonuses and/or long-term incentives)

View full job page →

22d ago

Senior Machine Learning Engineer (AI Foundations)

Capital One New York, NY

New York, NY

22d ago

$177k - $201k/yr Full time Ai Ml Artificial Intelligence Big Data

Apply Now

Job Description

The Senior Machine Learning Engineer will join Capital One's AI Foundations team to productionize machine learning applications at scale, design robust ML architectures, and ensure the high availability and performance of ML systems. This role encompasses end-to-end ML engineering work, including data pipelines, cloud infrastructure, CI/CD, and responsible AI practices.

Responsibilities

Design, build, and deploy ML models and components to address real world business needs, in collaboration with Product and Data Science teams.
Guide ML infrastructure choices using knowledge of modeling techniques, including model selection, data and feature decisions, training, hyperparameters, dimensionality, bias/variance, and validation.
Tackle complex problems by writing and testing application code, developing and validating ML models, and automating tests and deployment.
Collaborate within a cross functional Agile team to create software that enables state-of-the-art big data and ML applications.
Retrain, maintain, and monitor models in production.
Leverage or build cloud-based architectures, technologies, and platforms to deliver optimized ML models at scale.
Construct optimized data pipelines to feed ML models.
Apply continuous integration and continuous deployment best practices, including test automation and monitoring, to ensure successful deployment of ML models and application code.
Ensure code quality and governance, manage risk, and follow responsible and explainable AI practices.
Utilize programming languages such as Python, Scala, or Java.

Requirements

Bachelor’s Degree
At least 4 years of experience programming with Python, Scala, or Java (internship experience does not apply)
At least 3 years designing and building data‑intensive solutions using distributed computing
At least 2 years of on‑the‑job experience with ML frameworks (scikit-learn, PyTorch, Dask, Spark, or TensorFlow)
At least 1 year of experience productionizing, monitoring, and maintaining models

Technologies

Python, Scala, Java
scikit-learn, PyTorch, Dask, Spark, TensorFlow
AWS, Azure, Google Cloud Platform

Benefits

Health benefits
Financial benefits
Performance-based incentive compensation, including cash bonuses and long-term incentives

What you’ll do

Design, build, and deliver ML models and components that solve real world business problems, in collaboration with Product and Data Science teams.
Inform ML infrastructure choices with expertise in modeling techniques, including model selection, data and feature choices, training, hyperparameters, dimensionality, bias/variance, and validation.
Address complex problems by authoring and testing application code, developing and validating ML models, and automating tests and deployment.
Work within a cross-functional Agile team to create software that enables cutting‑edge big data and ML capabilities.
Retrain, maintain, and monitor models in production.
Utilize or build cloud-based architectures and platforms to deliver optimized ML models at scale.
Develop optimized data pipelines to feed ML models.
Apply CI/CD practices with test automation and monitoring to support successful deployments of ML models and code.
Ensure code quality and governance, mitigate vulnerabilities, and adhere to responsible and explainable AI standards.
Work with Python, Scala, or Java as primary programming languages.

Basic Qualifications

Bachelor’s Degree
At least 4 years programming with Python, Scala, or Java (internship experience does not apply)
At least 3 years designing and building data‑intensive solutions using distributed computing
At least 2 years on-the-job experience with ML frameworks (scikit-learn, PyTorch, Dask, Spark, or TensorFlow)
At least 1 year of experience productionizing, monitoring, and maintaining models

Preferred Qualifications

1+ years building, scaling, and optimizing ML systems
1+ years gathering and preparing data for ML models
2+ years developing performant, resilient, and maintainable code
Experience deploying ML solutions in a public cloud such as AWS, Azure, or Google Cloud Platform
Master’s or doctoral degree in computer science, electrical engineering, mathematics, or a related field
3+ years working with distributed file systems or multi-node database paradigms
Contributed to open source ML software
Authored or co-authored a paper on a ML technique, model, or concept
3+ years building production-ready data pipelines that feed ML models
Experience designing, implementing, and scaling complex data pipelines for ML models and evaluating their performance
Experience leveraging interactive AI tooling to accelerate productivity beyond basic code completion

View full job page →

22d ago

Senior Lead Machine Learning Engineer (Intelligent Foundations and Experiences)

Capital One New York, NY

New York, NY

22d ago

$251k - $286k/yr Full time Senior AI Enablement Ai Ml

Apply Now

Job Description

Capital One is hiring a Senior Lead Machine Learning Engineer in New York, NY onsite to productionize ML applications, lead cross-functional teams, and build scalable AI-driven products, with a salary range of USD 250,800 - 286,200 per year.

Responsibilities

Lead cross functional pods of software, data, and ML engineers to develop AI and ML capabilities for Credit and Financial Risk Management products, acting as a technical mentor on core technologies
Design and deliver AI powered products and components that address real business needs, applying model experimentation, LLM inference, similarity search, and agentic AI within a collaborative Product and Data Science setting
Partner with engineers, data scientists, and designers to develop scalable AI powered products that improve associate performance and deliver standout customer value
Guide ML infrastructure choices based on modeling techniques and challenges, including model selection, data and feature considerations, training, hyperparameters, dimensionality, bias/variance, and validation
Develop and validate ML models, write application code, and automate tests and deployment to address complex problems
Retrain, maintain, and monitor deployed models in production
Utilize or build cloud based architectures and platforms to deploy scalable ML models
Create efficient data pipelines to feed ML models
Apply CI/CD best practices, with test automation and monitoring, to ensure reliable deployment of ML models and code
Ensure code is well managed to minimize vulnerabilities, maintain model governance, and follow Responsible and Explainable AI practices
Utilize a broad mix of open source and SaaS AI technologies, with proficiency in Python, Scala, or Java

Requirements

Bachelor’s Degree required
8+ years designing and building data‑intensive solutions on distributed computing platforms; internship experience not eligible
4+ years programming in Python, Scala, or Java
3+ years building, scaling, and optimizing ML systems
2+ years leading teams delivering ML solutions

Technologies

Python
Scala
Java
AWS Bedrock
Google Cloud
Azure
Retrieval Augmented Generation (RAG)

Benefits

Health benefits
Financial benefits
Performance-based incentive compensation

View full job page →

22d ago

Data Scientist for Analytics (Engineer/Scientist Support)

CACI Norco, CA

Norco, CA

22d ago

$70k - $143k/yr Full time Analyst Analytics Data Analysis

Apply Now

Job Description

A Data Scientist for Analytics to support DoD led assessments of Defense and Integrated Combat Systems through data analysis, automation, and technical reporting, on site in Norco, CA.

Responsibilities

Analyze data from Defense and Integrated Combat Systems training, testing, and evaluation events to identify trends in system performance, collaborating with senior analysts on interpretation.
Develop and maintain automated scripts in MATLAB or Python to streamline data extraction, cleaning, and transformation; training and mentorship are provided.
Prepare draft technical inputs, summaries, and visual presentations to support government led reports and briefings.
Effectively communicate analysis inputs to government analysts and government leads.
Collaborate with multidisciplinary teams, sharing knowledge with both government and contractor members as directed.
Support the refinement and documentation of data analysis workflows and reporting processes under government direction.
Proactively expand knowledge of Defense and Integrated Combat Systems, analytical tools, and related technologies.

Requirements

Bachelor’s degree in a related field.
Ability to obtain and maintain a DoD SECRET clearance.
Willingness and ability to travel occasionally.
Flexibility to adjust schedule during test events and surge periods.
Capacity to sit or stand for extended periods.
U.S. citizen; must be willing to work on site in Norco, California.
Working knowledge of MATLAB or Python; coursework or personal projects are advantageous.
Familiarity with tool verification and validation practices.
Strong analytical and critical thinking abilities.
Familiarity with core data analysis techniques and visualization tools.
Quick to learn new systems, tools, and technical domains.
Excellent written and verbal communication skills, with strong interpersonal abilities and a team-oriented mindset.

Benefits

Healthcare
Wellness benefits
Financial benefits
Retirement benefits
Family support
Continuing education
Time off benefits

What you can expect

A culture of integrity
An environment of trust
A focus on continuous growth

Pay Range

The proposed salary range for this position is $69,600 to $142,700 USD per year. Final compensation may vary based on location, contract terms, experience, education, and certifications.

View full job page →

22d ago

TAGM Data Engineer

KBR Redstone Arsenal, AL

Redstone Arsenal, AL

22d ago

Full time Analytics Business Intelligence Data Analytics

Apply Now

Job Description

KBR seeks a hybrid Data Engineer and Analytics Steward based at Redstone Arsenal, AL to own end-to-end data pipelines, governance, and analytics delivery.

Responsibilities

Design, build, validate, and maintain scalable data pipelines and architectures to support data integration, storage, and analytics across the organization.
Extract, transform, and load data from diverse sources into centralized data warehouses or data lakes to enable consistent reporting and governance.
Automate manual data processes and optimize the data delivery infrastructure to improve efficiency and reliability.
Establish and enforce data governance policies, data quality standards, and compliance protocols to safeguard data assets.
Serve as the primary point of contact for data-related inquiries, ensuring data accuracy, consistency, and security across the organization.
Maintain data catalogs and metadata repositories to support data discovery, lineage, and stewardship.
Develop, publish, and maintain interactive dashboards and reports using Power BI for business insight delivery.
Translate complex business requirements into intuitive data visualizations; apply DAX and Power Query to model data for end users.
Architect, query, and manage relational and non-relational databases; monitor performance, perform routine maintenance, and troubleshoot data infrastructure issues.
Write complex, highly optimized SQL queries to support both engineering pipelines and reporting tools.

Requirements

Three to five years of professional experience in data engineering, database administration, business intelligence, or data governance.
Bachelor’s degree from an accredited institution.
Ability to obtain and/or maintain a DoD secret clearance.

Technologies

Python
Scala
Power BI
DAX
Power Query
SQL
MS SQL Server
PostgreSQL
Oracle
Snowflake
BigQuery
Master Data Management (MDM)

View full job page →

22d ago

Data Engineer

Booz Allen Hamilton Fort Meade, MD

Fort Meade, MD

22d ago

$78k - $176k/yr Full time Ai Ml Big Data Bigdata

Apply Now

Job Description

Booz Allen Hamilton is seeking a Data Engineer to join the Fort Meade based team on onsite work. The role centers on building and governing scalable data pipelines and platforms that enable AI and ML workloads, with collaboration across AI Model Engineers, Network/Cloud Architects, and other technical leaders on mission-driven initiatives. This position emphasizes turning diverse data into trusted, well-governed streams that support decision making and advanced analytics.

Responsibilities

Lead data engineering activities on mission-driven projects within the industry.
Oversee the development and deployment of pipelines and platforms that organize disparate data and make it meaningful.
Collaborate with AI Model Engineers, Network and Cloud Architects, the ISSE, and other technical leaders to guide scalable, mission-aligned data pipelines.
Apply expertise in data engineering, exploratory analysis, and feature development to shape how datasets are assessed, designed, transformed, and maintained for AI/ML workloads.
Ensure high-quality, well-governed data flows that support reliable model performance across operational, research, and enterprise systems.
Monitor and improve pipeline performance in coordination with operations teams.
Develop dashboards and metrics for data and pipeline monitoring.
Build automated ETL workflows and CI/CD pipelines for data processes.
Optimize data workflows across on-prem, cloud, or hybrid environments.

Requirements

6+ years of experience as a data engineer.
Experience designing, implementing, and maintaining scalable data pipelines for structured and unstructured data.
Experience preparing validated, normalized, and secure datasets for AI/ML model development.
Experience implementing data governance practices, including access control, lineage tracking, and dataset documentation.
Experience developing dashboards and metrics for data and pipeline monitoring.
Experience building automated ETL workflows and CI/CD pipelines for data processes.
Experience optimizing data workflows across on-prem, cloud, or hybrid environments.
Ability to monitor and improve pipeline performance in coordination with operations teams.
TS/SCI clearance with a polygraph.
HS diploma or GED with 6+ years of experience with data normalization and data pipelines, or Bachelor’s degree with 4+ years of experience in data normalization and data pipelines.

Technologies

Spark
Databricks
Hadoop
Hive
Kafka
AWS
Azure

Benefits

Health benefits
Life insurance
Disability benefits
Financial benefits
Retirement benefits
Paid leave
Professional development
Tuition assistance
Work-life programs
Dependent care

Clearance

TS/SCI clearance with polygraph is required.

Identity Statement

As part of the hiring process, you will complete an identity verification process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity fraud. You are expected to be on camera during interviews and assessments. Booz Allen reserves the right to take your picture to verify identity and prevent fraud.

Candidate AI Usage Policy

AI is a part of Booz Allen's daily work, and the company supports responsible and ethical use of AI tools. The candidate process remains based on your skills and knowledge, and the use of artificial intelligence or other tools to assist with responses during interviews is prohibited unless explicit permission is provided.

Work Model

Remote: There may be occasions when in-person work at a Booz Allen or customer facility is required.
Hybrid: Regular in-person work at a Booz Allen facility is expected, with potential visits to customer facilities as needed.
Onsite: Work primarily at a Booz Allen office or customer facility, collaborating directly with colleagues and customers as required.

Compensation

The projected compensation range for this position is USD 77,600.00 to USD 176,000.00 per year (annualized).

Commitment to Non-Discrimination

All qualified applicants will receive consideration for employment without regard to disability, veteran status, or any other status protected by applicable law.

View full job page →

22d ago

Data Engineer

Amazon.com Services LLC New York, NY

New York, NY

22d ago

$145k - $197k/yr Full time Amazon Quicksight Analytics AWS

Apply Now

Job Description

Onsite in New York, NY, this data engineering role focuses on building a greenfield finance data platform for FAIM within Amazon Ads. You will own the end-to-end data warehouse, ETL pipelines, and reporting, working closely with Finance Managers, PM-Ts, Scientists, and Engineering to shape the data foundation of next generation ad products.

Responsibilities

Define the technical direction for the FAIM data warehouse, ETL pipelines, and the reporting layer from end to end.
Design and operate Datanet/ETLM jobs, Cradle profiles, Andes datasets, and dashboards that finance partners rely on as the source of truth.
Ingest telemetry from across Amazon’s data ecosystem (Andes subscriptions, EDX, S3, internal services) into a clean, query-ready data layer.
Deliver on OP1 and OP2 cycles, support MBR/QBR rhythms, and respond to ad-hoc executive requests with a bias for action.
Turn messy, multi-source data into well-documented dimensional models that scale with the organization.
Lead code and design reviews while establishing data quality and pipeline reliability standards.

Requirements

3+ years of data engineering experience.
1+ year developing and operating large-scale data structures for business intelligence analytics using ETL/ELT processes.
1+ year developing and operating large-scale data structures for business intelligence analytics using data modeling.
1+ year developing and operating large-scale data structures for business intelligence analytics using SQL.
Experience with data modeling, warehousing, and building ETL pipelines.

Technologies

SQL
Redshift
Andes
Datanet/ETLM
Cradle
Andes 3.0
Redshift Spectrum
EDX
QuickSight
SPICE
Python
Kiro
Claude Code
AWS Redshift
S3
AWS Glue
EMR
Kinesis
FireHose
Lambda
IAM

Benefits

Health insurance
401(k) matching
Paid time off
Parental leave

Description

This position represents a ground-up, greenfield initiative within Amazon Ads Finance, focusing on the FAIM team’s ambition to advance agentic AI advertising products. There are no legacy pipelines or inherited dashboards to contend with, allowing the candidate to shape data infrastructure from the ground up in a fast-moving environment.

What we are building includes a finance data platform that powers FAIM, the Full-Funnel Agentic Intelligence & Models team, responsible for the next generation of agentic AI advertising products. You will contribute to pipelines and models that convert raw data into actionable decisions for new products, and support self-service reporting that scales across Engineering, Science, PM-T, and Design for multiple AI-native advertising initiatives. This is a startup-like team within Amazon Ads Finance with a clear, ambitious vision and the runway to execute it well.

We seek a senior Data Engineer who brings strong SQL proficiency and at least 3 years of experience architecting and operating production ETL on Redshift, Andes, or comparable systems at scale. Hands-on expertise with the Amazon data stack, including Datanet/ETLM, Cradle, Andes 3.0, Redshift Spectrum, EDX, and QuickSight (SPICE), is essential. A solid foundation in dimensional data modeling, including fact/dim design, slowly changing dimensions, and informed denormalization and partitioning decisions, is required, along with Python for orchestration and data quality tooling. The ideal candidate will set data quality, lineage, SLA, and reliability standards, navigate ambiguity to deliver durable data products, and mentor junior engineers as the team grows. AI-native experience with automation and defect/opportunity detection using tools such as Kiro or Claude Code is highly valued.