Lead Data Engineer

Vertage is hiring a Lead Data Engineer for a hybrid role in the United States (hybrid to Charlotte is preferred). In this position, you will design and deliver enterprise data engineering solutions across the AWS ecosystem, with a strong focus on scalable data pipelines, streaming architectures, lake and warehouse platforms, and operational excellence. The role spans end-to-end ownership, from architecture and governance through production support and modernization.

Compensation: USD 80 - 85 per hour. Start and end date: 12 Oct 2026 – 24 Oct 2027. Respond by: 10 Oct 2026.

Responsibilities

• Lead the design, architecture, and implementation of enterprise data engineering solutions across AWS
• Collaborate with Lead Developers, Data Scientists, Architects, Product Owners, and business stakeholders to define technical strategy and scalable solutions
• Provide technical leadership and mentorship to Data Engineers and development teams, promoting engineering excellence and best practices
• Drive architectural decisions with Data Architects and Solution Architects to ensure scalability, security, reliability, and maintainability
• Design and oversee data warehouse and data lake solutions balancing usability, performance, and long-term sustainability
• Establish engineering standards for data modeling, ETL frameworks, pipeline reliability, monitoring, and operational excellence
• Lead end-to-end solution delivery, aligning with business requirements, enterprise architecture standards, and regulatory requirements
• Oversee production support and operational management for AWS-based data platforms, including root-cause analysis and performance optimization
• Champion data quality, governance, observability, and data stewardship across platforms and teams
• Identify opportunities to modernize data architecture and improve operational efficiency using automation and cloud-native technologies

Requirements

• 8+ years of experience in Data Engineering, including 5+ years working extensively within AWS ecosystems
• Expert-level experience with AWS services including S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, and Step Functions
• Experience creating AI applications with AWS Bedrock
• Strong experience building enterprise-scale data lake and data warehouse solutions using Lake Formation, Amazon Redshift, and Amazon Athena
• Extensive experience with Kafka-based streaming architectures, preferably Confluent Kafka
• Advanced SQL and data modeling skills, including dimensional modeling and data vault
• Deep expertise designing, developing, and optimizing scalable, resilient data pipelines in AWS
• Solid understanding of distributed processing frameworks, especially PySpark and EMR
• Advanced Python development with hands-on PySpark experience
• Expertise in Infrastructure as Code using Terraform
• Experience implementing CI/CD frameworks using GitHub and GitHub Actions
• Deep knowledge of AWS IAM, including roles, policies, governance, and security best practices
• Strong workflow orchestration experience using AWS Step Functions, Apache Airflow, or equivalent platforms
• Experience leading cloud migration and enterprise data platform modernization initiatives
• Understanding of data governance, metadata management, data quality frameworks, and observability
• Ability to lead hands-on development while providing direction, code reviews, and engineering oversight across multiple initiatives
• Experience building development environments, infrastructure standards, security controls, and migration strategies across multiple AWS accounts and environments
• Ability to architect and govern enterprise-scale pipelines, ETL, ingestion frameworks, and orchestration workflows
• Ability to identify data gaps, define remediation plans, and implement automation that improves analytical capabilities
• Ability to design highly reliable pipelines with focus on data quality, observability, resiliency, and operational supportability
• Experience building and optimizing large-scale data warehousing solutions that support business users and system performance
• Strong leadership and communication skills to influence architecture and explain complex technical concepts to technical and non-technical stakeholders
• Proven track record mentoring engineers and building high-performing teams
• Experience leading cross-functional initiatives across data engineering, analytics, architecture, platform engineering, and business stakeholders
• Experience building AI-ready data pipelines and ML workflows, including feature engineering and MLOps
• Proficiency in Python, SQL, Spark, and Generative AI technologies
• Hands-on experience with AWS, Azure, or Databricks and cloud-based AI/data platforms
• Knowledge of LLMs, RAG, and vector databases for AI-powered applications and intelligent search
• Understanding of responsible AI, governance, quality, and security practices
• English: Professional working proficiency

Preferred Qualifications

• AWS Certified Data Engineer, AWS Solutions Architect, or equivalent cloud certifications
• Experience implementing enterprise data governance and metadata management platforms
• Experience with real-time analytics, event-driven architectures, and streaming data platforms
• Knowledge of modern data architecture patterns including Data Mesh, Lakehouse, and domain-oriented design
• Experience leading large-scale cloud transformation or enterprise data modernization programs
• Skills — Security Controls, Machine learning, Architecture, Enterprise Data Management, Python programming, Data, Confluent, Amazon, Feature engineering, Data streaming, Data Quality, Code reviews, EcoSYSTEMS, Identity and access management, Extract/transform/load, System Migration, Software applications, and more (all listed as Intermediate)