Senior Data Engineer

The Senior Data Engineer role at Delta Airlines focuses on delivering secure, scalable, and AI-ready data solutions that support IT Service Assurance. You will lead modernization of AWS data platforms and help enable operational intelligence and trusted data products across ITSM, CMDB, observability, telemetry, and enterprise platforms.

Responsibilities

• Lead independently across complex data initiatives by identifying improvement opportunities, shaping solutions, and driving delivery from concept through production.
• Design and modernize secure, scalable AWS data platforms, pipelines, lakehouse patterns, and reusable data products aligned to enterprise architecture and governance standards.
• Provide senior technical leadership across ingestion, transformation, modeling, metadata, data quality, lineage, performance optimization, and production reliability.
• Build and optimize batch, incremental, event-driven, and near-real-time integrations across APIs, databases, files, ServiceNow, CMDB, observability, telemetry, and enterprise platforms.
• Develop efficient, maintainable Python and SQL for data pipeline development, orchestration, transformations, analytics layers, APIs, automation, and reusable engineering frameworks.
• Create trusted data products using automated controls for completeness, accuracy, freshness, reconciliation, monitoring, alerting, and exception handling.
• Advance AI-ready architectures that prepare structured and unstructured data for Amazon SageMaker, Amazon Bedrock, LLMs, RAG, vector databases, semantic search, agents, and MCP-enabled solutions.
• Design retrieval, embedding, indexing, enrichment, and context-management patterns to improve usability, discoverability, and responsible AI outcomes.
• Drive operational excellence using CI/CD, Infrastructure as Code, automated testing, source control, code reviews, runbooks, production monitoring, and resilient deployment practices.
• Apply security, privacy, identity, access control, encryption, retention, data classification, and responsible AI requirements throughout the data lifecycle.
• Influence stakeholders, mentor engineers, promote reusable standards, evaluate emerging technologies, and guide modernization efforts that improve scalability, automation, cost, and business value.

Requirements

• At least 18 years of age and authorization to work in the United States.
• High school diploma, GED, or high school equivalency.
• Embraces diverse people, thinking, and styles.
• Consistently prioritizes safety and security of self, others, and personal data.
• Strong analytical and interpersonal skills with the ability to develop effective relationships and influence without direct authority.
• Superior written and verbal communication skills, including executive presence to represent the business across the organization.
• 7+ years of IT experience, including 4+ years hands-on experience designing, developing, and supporting enterprise data pipelines, data platforms, lakehouses, or analytical data solutions.
• Demonstrated experience designing and implementing cloud-based data lake, data warehouse, or lakehouse architectures using AWS services.
• Hands-on experience with Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, AWS Lambda, Amazon EventBridge, AWS Lake Formation, AWS IAM, and related data services.
• Advanced proficiency in Python and SQL for ingestion, transformation, automation, analysis, data services, and performance optimization.
• Experience designing scalable ETL and ELT pipelines integrating APIs, databases, structured files, semi-structured content, and cloud services.
• Strong experience with dimensional and relational data modeling, including normalized, denormalized, and analytics-oriented structures, such as layered raw, curated, published, and analytics architectures.
• Strong understanding of incremental ingestion, change-data capture, watermarking, deduplication, record versioning, historical tracking, late-arriving data, schema evolution, and reconciliation.
• Experience implementing metadata, lineage, cataloging, source-to-target mapping, data-quality testing, freshness monitoring, logging, alerting, and exception handling.
• Working knowledge of Amazon SageMaker, Amazon Bedrock, foundation models, LLM application architectures, RAG, embeddings, vector search, semantic retrieval, and AI-ready data preparation.
• Experience preparing structured and unstructured data through parsing, chunking, enrichment, embedding generation, indexing, access filtering, and context management.
• Strong understanding of data governance, data classification, privacy, encryption, retention, identity, authorization, and secure handling of personal and sensitive information.
• Experience with Amazon Redshift development concepts, including SQL optimization, workload performance, distribution and sort strategies, materialized views, stored procedures, access controls, and Parquet.
• Experience with Git, automated testing, CI/CD, and Infrastructure as Code such as AWS CDK or CloudFormation.
• Ability to communicate complex technical concepts, data limitations, architecture decisions, AI data requirements, and implementation plans to technical and business audiences.
• Demonstrated ability to lead technical design, mentor engineers, conduct code and architecture reviews, establish reusable standards, and manage work across concurrent Agile initiatives.
• Consistent commitment to safety, security, responsible AI, operational reliability, and protection of personal and sensitive data.

Technologies

• AWS, Amazon SageMaker, Amazon Bedrock, LLMs, RAG, vector databases
• Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, AWS Lambda, AWS EventBridge, AWS Lake Formation, AWS IAM
• Python, SQL, ServiceNow, CMDB, Amazon CloudWatch, Dynatrace, PagerDuty, Sumo Logic
• Amazon OpenSearch Service, PostgreSQL, pgvector
• OpenID Connect, OAuth 2.0, CI/CD, Infrastructure as Code, AWS CDK, CloudFormation
• Git, AWS Glue Data Quality, Great Expectations, dbt
• Amazon Redshift SQL optimization, Parquet, ETL, ELT, Lakehouse, Lake Formation

Preferred Qualifications

• AWS Certified Data Engineer, AWS Certified Solutions Architect, AWS Certified Machine Learning Engineer, or another relevant AWS certification.
• Hands-on experience with Amazon SageMaker, Amazon Bedrock, Bedrock Knowledge Bases, Bedrock Agents or AgentCore, Amazon OpenSearch Service, foundation models, embedding models, and production generative AI architectures.
• Experience implementing enterprise RAG solutions, semantic search, hybrid retrieval, reranking, retrieval evaluation, grounding, citations, and AI-powered knowledge repositories.
• Experience designing or integrating vector databases and vector-capable platforms such as Amazon OpenSearch Service, PostgreSQL with pgvector, or comparable technologies.
• Experience with prompt engineering, model evaluation, guardrails, hallucination reduction, responsible AI controls, observability, and secure AI deployment patterns.
• Deep knowledge of ServiceNow data, APIs, table structures, reference fields, ITSM processes, configuration management data, and CMDB relationships.
• Experience integrating observability and operational platforms such as Dynatrace, PagerDuty, Amazon CloudWatch, ServiceNow, Sumo Logic, or similar enterprise services.
• Experience implementing OAuth 2.0, OpenID Connect, identity federation, role-based access controls, and document- or row-level authorization for data and AI retrieval.
• Experience with automated data testing and quality frameworks such as AWS Glue Data Quality, Great Expectations, dbt tests, or comparable technologies.
• Demonstrated history of leading data modernization, cloud migration, operational intelligence, platform engineering, generative AI, or AI-enablement initiatives.

Benefits

• Competitive salary, industry-leading profit sharing program, and performance incentives
• 401(k) with generous company contributions up to 9%
• Paid time off including vacation, holidays, paid personal time, maternity and parental leave
• Comprehensive health benefits including medical, dental, vision, short/long term disability and life benefits
• Family care assistance through fertility support, surrogacy and adoption assistance, lactation support, subsidized back-up care, and programs that help with loved ones in all stages
• Holistic Wellbeing programs including access to an employee assistance program, free financial coaching, and extensive resources supporting mental health
• Domestic and International space-available flight privileges for employees and eligible family members
• Career development programs to achieve your long-term career goals
• World-wide partnerships to engage in community service and innovative goals created to focus on sustainability and reducing their carbon footprint
• Business Resource Groups created to connect employees with common interests to promote inclusion, provide perspective and help implement strategies
• Recognition rewards and awards through the platform Unstoppable Together
• Access to over 500 discounts, specialty savings and voluntary benefits through Deltaperks such as car and hotel rentals and auto, home, and pet insurance, legal services, and childcare

Location

Atlanta, GA (onsite)