Data Engineering Lead

Data Engineering Lead Jobs

8 jobs found

Today

Senior Data Engineer (Must have Microsoft Fabric!)

MMD Services, Inc Downers Grove, IL

Downers Grove, IL

Today

$65 - $85/hr Full time Backend Developer Senior Azure

Apply Now

Job Description

Senior Data Engineer role focused on hands-on data architecture and prototyping across the Microsoft Azure stack, working closely with executive stakeholders and contributing to AI initiatives.

Design and architect data solutions across the Microsoft Azure stack, with direct hands-on build responsibility
Build prototypes and proofs of concept to validate architecture decisions before full-scale implementation
Translate C-suite requirements and KPIs into clear, actionable technical user stories for the engineering team
Provide technical direction by guiding and mentoring a team of mid-level and junior engineers supplied by an external vendor
Act as primary technical liaison between developers and business/executive stakeholders
Own architecture decisions for Synapse and Data Factory pipelines, focusing on scalability and performance
Contribute to the AI roadmap through AI POCs, Microsoft Copilot rollout, and early agentic AI initiatives
Present technical strategy and progress updates to non-technical executive audiences

Requirements

5+ years of hands-on experience in data architecture or lead data engineering roles
Hands-on architecture and lead engineering experience (design systems and build prototypes, not only advise)
Strong Microsoft Azure stack experience, including Azure Synapse and Azure Data Factory
Microsoft Fabric experience (must have)
Strong Python and SQL skills ( PySpark is a plus)
Exceptional communication skills to bridge technical and non-technical audiences
Experience guiding and mentoring junior and mid-level engineers

Technologies

Microsoft Azure
Azure Synapse
Data Factory
Microsoft Fabric
Python
SQL
PySpark
Databricks
Microsoft Copilot

Benefits

Flexible schedule

Nice to Have

Databricks experience (considered a bridge if you do not yet have Fabric hands-on time)
Interest or experience in AI, especially agentic AI agents, Copilot, or similar tools

Location: Downers Grove, IL (hybrid)

Pay: USD 65 - 85 per hourly

Experience: 5 years minimum

View full job page →

1d ago

Senior Data Engineer

Hearst Media Services Troy, MI

Troy, MI

1d ago

Full time Backend Developer Senior 3d Design Tools

Apply Now

Job Description

The Senior Data Engineer role at Hearst Media Services (MOTOR Information Systems) focuses on architecting, designing, building, and evolving an AWS-based data platform. The position requires technical ownership across scalability, reliability, performance, and cost while delivering data pipelines and data products.

Responsibilities

Analyze large and complex data sets to support pipeline and platform decisions.
Design, develop, and coordinate the generation of data pipelines.
Collaborate with the team to plan and solve complex technical problems.
Design for, or optimize existing pipelines, with emphasis on performance and cost efficiency.

Requirements

Expert Python and SQL development.
Strong AWS experience, including: AWS Lambda, AWS Glue, AWS State Machines, AWS Fargate, AWS CloudFormation, and Amazon RedShift.
Strong experience with semi-structured data such as XML, CSV, JSON, and Parquet.
Strong automation experience.
Experience with Databricks, including Delta Lake, notebooks, and SQL analytics.
Experience with Kubernetes, including cluster orchestration, container networking, and pod management.
Experience with DevOps and CI/CD, including deployment orchestration with Octopus Deploy.
Experience with agentic AI, including LangGraph, MCP servers, and agent frameworks.
Experience with data handling, including Pandas/NumPy, Apache Spark, and Ray.

Technologies

Python, SQL
AWS Lambda, AWS Glue, AWS Step Functions, AWS Fargate, AWS CloudFormation, Amazon RedShift
XML, CSV, JSON, Parquet
Databricks, Delta Lake, Databricks Notebooks, SQL Analytics
Kubernetes
Octopus Deploy
LangGraph, MCP Servers
Pandas, NumPy, Apache Spark, Ray
Agent frameworks

Location & Schedule

Location: Troy, MI (hybrid)
Hybrid schedule: 3 days onsite in Troy, MI

Education

Bachelor's Degree.

Benefits

No visa sponsorship available at this time.

Helpful Experience (Nice to Have)

Experience with LLMs, machine learning & deep learning libraries, AI application frameworks, and vector databases.
Experience with automotive data or the automotive industry.
Experience with Agile or Scrum.

About MOTOR Information Systems (Hearst)

MOTOR is the industry leader in automotive data and insights. The organization powers the aftermarket ecosystem with comprehensive datasets, and is investing in modernizing its data platform to support the next generation of products and AI-driven capabilities.

View full job page →

4d ago

Principal Engineer I - Senior Data Engineer

Software Resources, Inc. Phoenix, AZ

Phoenix, AZ

4d ago

Full time Azure Data Factory Azure Data Lake Azure DevOps

Apply Now

Job Description

Join a hybrid team to build and modernize an enterprise data platform supporting the Bank Regulatory Reporting Program.

Design, build, and implement critical components of an enterprise data platform aligned to strategic data and analytics needs for the Bank Regulatory Reporting Program
Develop end-to-end data engineering solutions across the full data lifecycle, including efficient technical hygiene and implementation using Azure DevOps for the Bank Regulatory Reporting and LFI strategy/roadmap
Build ETL and data pipeline capabilities covering creation, transformation, storage, archiving, analysis, and sharing across the Bank and partner systems
Write dynamic, parameterized, and reusable PySpark code
Create shared frameworks including helper functions and companion notebook structures
Perform performance tuning and optimization for Spark workloads
Implement Azure DevOps CI/CD pipelines for data solutions in accordance with Bank technical standards
Apply Test Driven Development (TDD) to data solutions designed, built, and implemented
Lead implementation of outcomes, recommendations, and designs from Data Governance and Enterprise Architecture
Conduct data exploration and profiling using tools such as SSAS
Participate in data architecture decisions
Provide technical mentorship to team members and other data professionals
Collaborate with product owners and business stakeholders to understand business requirements and operational processes
Implement quality and compliance by design in all data solutions
Include support for unstructured data and big data methods

Key skills & requirements

5+ years of experience (minimum)
Expert in PySpark (required)
8+ years data engineering experience in ETL concepts and processes, including enterprise data warehouse capabilities and database principles; preferably strong to expert in Azure Data Factory, Azure Synapse Analytics, and/or Databricks
5+ years designing, implementing, and supporting cloud data solutions; Azure Data Lake, Azure Data Factory, Azure Data Services, Azure Synapse, Azure Logic Apps, and Azure DevOps experience strongly preferred
Bachelor’s degree in engineering or related field
Expert RDBMS and query language experience with at least one of: T-SQL, PL/SQL, Spark SQL
Ability to design and build reusable, dynamic PySpark components including companion notebook frameworks, shared helper functions/libraries, and parameter-driven ingestion and transformation patterns
Support and influence migration from Azure Synapse to Microsoft Fabric, including modernization of patterns and frameworks
Expert conceptual, logical, and physical data design
Azure certifications such as Azure Fundamentals, Azure Fabric Data Engineer, Azure Data Scientist, and/or Azure DevOps Engineer
Experience using design tools for conceptual architecture diagrams and data flow diagrams such as Visio, Archimate, Lucidchart
Excellent communication skills verbal and written

Tools & technologies

PySpark, Azure DevOps, ETL
Azure Data Factory, Azure Synapse Analytics, Databricks
Azure Data Lake, Azure Data Services, Azure Logic Apps, Microsoft Fabric
SSAS
T-SQL, PL/SQL, Spark SQL
Visio, Archimate, Lucidchart
SAS, Tableau, PowerBI

Location & work schedule

Phoenix, AZ (hybrid)
4 days on site and Fridays remote in Phoenix, AZ or Dallas, TX

Benefits

Medical, dental, and vision coverage
401(k) with company match
Short-term disability
Life insurance with AD&D

Preferred experience

Experience in Agile, SAFe, and/or Scrum
Experience integrating with data quality, data catalog, and data lineage tools
Additional cloud data certifications with major providers (Azure, AWS, GCP)
Banking/financial services or other highly regulated industry experience
Regulatory Reporting experience, including Large Financial Institution requirements

View full job page →

7d ago

Lead Specialist, Federal Data Engineer

KPMG Washington, DC

Washington, DC

7d ago

Full time Manager Big Data Data Analysis

Apply Now

Job Description

KPMG is seeking a Lead Specialist, Federal Data Engineer to support its Federal Advisory practice in Washington, DC. In this onsite role, you will help clients design, modernize, and optimize data platforms using modern big data tools. You will also lead cross-functional teams and contribute to innovation efforts and technical thought leadership.

What you’ll do

Apply modern big data tools and approaches to solve complex business objectives.
Develop and update big data platform requirements to evaluate client platforms, identify gaps in technologies and processes, and provide recommendations for platform modernization, cloud migration, and integration.
Design tools and services for cloud DevOps environments, including performance monitoring, security monitoring, deployment and configuration, continuous integration and build servers, and cloud resource creation scripts.
Lead multi-disciplinary and cross-functional teams, workshops, and innovation sessions with clients to uncover business opportunities and drive optimization.
Support efforts to pursue innovations, define target solutions, and build extendable data platforms; participate in developing and presenting thought leadership.

What you bring

Minimum five (5) years of technical data engineering experience; U.S. Federal government consulting experience preferred.
Bachelor’s degree from an accredited college or university.
Required: Python and SQL; preferred: Advana, Databricks, Spark.
Experience with development tools and methodologies such as Agile, GIT, test driven development, and CI/CD release management.
Experience with data architecture and integration, plus coding and testing patterns using existing open-source software platforms.
Experience leading project teams and engagements.
Ability to travel as required to support firm engagements.
Must possess a U.S. Government Secret clearance.

Benefits and time off

Medical and dental plans
Vision coverage
Disability and life insurance
401(k) plans
Personal well-being benefits to support mental health
Personal Time Off per fiscal year
Calendar of holidays to be observed during the year
Two breaks each year where employees will not be required to use Personal Time Off

Additional notes

Material job duties are listed above. KPMG also notes that criminal history may have a direct, adverse, and negative relationship with some material job duties. Applicants will be considered with arrest and conviction records in accordance with applicable fair chance ordinances and acts.

Salary ranges by city outside of California can be reviewed here: https://kpmg.com/us/en/how-we-work/pay-transparency.html/?id=M101ADV_3_26.

View full job page →

9d ago

Lead Generative AI Engineer, VP

Citi Jacksonville, FL

Jacksonville, FL

9d ago

$126k - $189k/yr Full time Backend Developer Manager Agentic Ai

Apply Now

Job Description

Citi is hiring a Lead Generative AI Engineer (VP) to design, develop, and deploy generative AI solutions for intelligent operations and automation.

Responsibilities

Lead hands-on development and delivery of intelligent operations capabilities, including advanced chatbots, agentic workflow automation, and AI-assisted decision platforms
Fine-tune, prompt-engineer, and deploy open-source LLMs (including Llama, Mistral, Gemma, etc.) and proprietary models using Google Vertex AI
Implement RAG (Retrieval-Augmented Generation) pipelines
Build and operationalize AI safety frameworks using guardrails (for example, NeMo Guardrails and Llama Guard) plus deterministic controls to reduce hallucinations, support data privacy, and maintain regulatory compliance
Design and deploy scalable AI pipelines using Python and cloud-native architectures with containerization (Docker, Kubernetes)
Apply financial services technology expertise to align AI model usage with risk management, auditability, and financial data security requirements

Requirements

Deep understanding of LLM architectures, vector databases (Pinecone, Milvus, Chroma), embedding models, and AI orchestration frameworks (Google ADK, LangChain, etc.)
Hands-on experience with Google Vertex AI (or equivalent cloud AI suites) and deploying open-source models such as Llama, Mistral, and Gemma
Proven production experience implementing deterministic controls, semantic routing, guardrails, and bias and hallucination detection
Expert-level Python skills, including AI/ML libraries: PyTorch, Hugging Face, Pandas, NumPy; plus API development with FastAPI and Flask
Strong containerization and delivery experience with OpenShift, Docker, Kubernetes, and CI/CD pipelines
Experience with financial services technology, including financial products, compliance requirements, and the competitive AI landscape in fintech
Demonstrated ability to lead AI and automation efforts from concept through production, establishing credible senior-level technical direction

Technologies

Google Vertex AI
Python, Docker, Kubernetes, OpenShift
NeMo Guardrails, Llama Guard
LLMs: Llama, Mistral, Gemma
RAG (Retrieval-Augmented Generation)
Pinecone, Milvus, Chroma
Google ADK, LangChain
PyTorch, Hugging Face, Pandas, NumPy
FastAPI, Flask
CI/CD pipelines, containerization

Beneficial Skills & Qualifications

Experience deploying and tuning models on Google Vertex AI or equivalent enterprise-grade cloud AI platforms
Familiarity with additional AI orchestration tools and emerging open-source LLM frameworks beyond core requirements
Bachelor’s degree or equivalent experience; Master’s degree in Computer Science, Artificial Intelligence, or a related technical discipline preferred

Benefits

Hybrid working model: 3 days in the office and 2 days working remotely
Continuous learning and development opportunities
Competitive financial wellbeing and benefits package
Medical, dental & vision coverage
401(k)
Life, accident, and disability insurance
Wellness programs
Paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays
Discretionary and formulaic incentive and retention awards (for eligible employees)

Job Details

Location: Jacksonville, FL (hybrid)
Employment type: Full time
Job family group: Technology
Job family: Applications Development
Salary range: USD 125,760 - 188,640 per year
Primary location (listed): Irving Texas United States
Anticipated posting close date: Aug 15, 2026

Automated Processing

We use automated processing, including artificial intelligence, for legitimate business interests to identify and align candidate skills and abilities to the job opening. This automated processing does not involve relying on automatic or autonomous decision-making.

Additional Notes

Other relevant skills: Artificial Intelligence (AI), Containerization, Generative AI, Large Language Models (LLMs), Python (Programming Language)

View full job page →

9d ago

Lead AI Engineer

Vytwo Dallas, TX

Dallas, TX

9d ago

Full time Manager Artificial Intelligence Azure

Apply Now

Job Description

Vytwo is building production-ready AI solutions and is hiring a Lead AI Engineer to help design and deploy end-to-end data and AI pipelines on Azure Databricks and other Azure cloud services. This onsite role in Dallas, TX focuses on moving AI from experimentation to reliable production systems, including deployment, monitoring, and optimization across ML and Generative AI use cases.

What you will do

Design and build end-to-end data and AI pipelines using Azure Databricks.
Develop robust ETL/ELT workflows with Python (including PySpark) and SQL.
Implement CI/CD pipelines for Databricks deployments, including jobs, notebooks, and workflows.
Integrate Databricks with Azure services such as Data Lake, Blob Storage, Key Vault, Azure OpenAI, and Azure Functions.
Optimize Databricks jobs for performance, cost, and reliability.
Create reusable, modular code to support maintainable production systems.
Collaborate with data scientists and platform teams to move models from experimentation to production deployment.
Implement logging, monitoring, and error handling for production pipelines.
Develop and deploy ML and Generative AI models, including LLMs, embeddings, and RAG pipelines for NLP, computer vision, and predictive analytics.
Fine-tune LLMs using LoRA or QLoRA, and integrate with Azure OpenAI or Hugging Face models.
Build vector search and retrieval pipelines using FAISS or Azure Cognitive Search.
Support responsible AI practices, including bias detection and model governance.

What you bring

Hands-on experience with Azure Databricks, including jobs, workflows, and clusters (with Unity Catalog preferred).
Python skills with strong emphasis on PySpark rather than only pandas.
SQL experience, including complex joins, window functions, and analytical queries.
Working knowledge of Azure cloud components such as ADLS Gen2, ADF, Key Vault, and IAM concepts.
Experience with pipeline orchestration and deployment, including CI/CD and environment promotion.
Experience with Azure DevOps.
Strong understanding of the ML lifecycle and MLOps best practices.
Experience deploying models using MLflow or similar frameworks.

Technologies you will work with

Azure Databricks; Databricks workflows, jobs, and notebooks
Azure cloud infrastructure including Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions
Python, PySpark, SQL; ETL/ELT
CI/CD pipelines; Logging and Monitoring
ML and Generative AI; MLflow
LoRA, QLoRA, Hugging Face models
FAISS and Azure Cognitive Search; vector search and RAG pipelines
Unity Catalog; ADLS Gen2; ADF; IAM concepts; Azure DevOps
NLP, computer vision, predictive analytics

Strong advantage

Experience with ML and Generative AI workloads on Databricks.
Experience with RAG, embeddings, or inference pipelines.
Terraform / ARM / Bicep experience for infrastructure.
Databricks Asset Bundles experience.
Airflow or ADF orchestration experience.
Production monitoring and cost optimization experience.
Knowledge of LangChain or similar frameworks for AI application development.
Experience with Azure AI services such as Azure Machine Learning or Azure Cognitive Services.

View full job page →

9d ago

Principal Data Engineer, People Data

Okta Washington, DC

Washington, DC

9d ago

$197k - $271k/yr Full time Big Data Data Analysis Data Architecture

Apply Now

Job Description

Okta is hiring a Principal Data Engineer, People Data to architect and deliver scalable data platforms that support candidate and employee lifecycle insights.

Responsibilities

Design, build, and evolve high-scale data solutions that act as reference architecture and make data access easier across the organization
Create data quality frameworks including monitoring, anomaly detection, and alerting, with governance, lineage tracking, and change management aligned to externally reported numbers
Drive consistent data modeling patterns, naming conventions, documentation standards, and metric governance across the data organization
Lead cross-functional technical initiatives across data verticals to speed delivery and harden data systems
Operate effectively in ambiguity and make technical decisions that balance short-term delivery with long-term infrastructure investment
Champion data contracts by implementing data contracts and schema evolution practices to improve reliability across global product teams
Mentor and scale teams by raising the technical bar, mentoring senior engineers, leading cross-functional task forces, and fostering engineering excellence and data-driven decision making
Stay current with the data landscape and adopt emerging technologies and practices to guide long-term team direction

Requirements

10+ years in data engineering, with proven senior or lead experience
Federal government compliance and security experience is highly desirable
Expert-level experience with SQL; ETL/ELT tools including Airflow and dbt
MPP database experience including Snowflake and Redshift
Extensive hands-on AWS experience: S3, Lambda, EMR, ECR, EKS
People Data expertise in HR data covering candidate lifecycle, employee records, compensation, and organizational data
Experience designing data models for recruiting analytics, engagement signals, and workforce planning metrics
Experience building and maintaining batch and realtime pipelines using Spark and Kafka
Deep experience with lakehouse and warehouse architectures such as Databricks and Snowflake
Modern file formats experience including Iceberg and Delta
Demonstrated ability to integrate AI tools to optimize or redesign workflows with measurable impact such as efficiency or quality improvements
Proven hands-on collaboration with Data Science, Business, and Product teams
Familiarity with Workday data structures is a bonus

Technologies

SQL, Airflow, dbt
Snowflake, Redshift
AWS: S3, Lambda, EMR, ECR, EKS
Spark, Kafka
Databricks
Iceberg, Delta

Location and Salary

Location: Washington, DC (hybrid)
Salary: USD 197,000 - 270,600 per year

The Okta Experience

Supporting Your Well-Being
Driving Social Impact
Developing Talent and Fostering Connection + Community

View full job page →

9d ago

Principal Data Engineer, People Data

Okta San Francisco, CA

San Francisco, CA

9d ago

$197k - $271k/yr Full time Apache Airflow Big Data Bigdata

Data Engineering Lead Jobs

Get Data Engineering Lead Jobs alerts