Lead Data Engineer

Circle K seeks a Lead Data Engineer to steer the design, development, and deployment of data solutions that yield actionable insights. This onsite role in Clemmons, NC leads a team of data engineers and collaborates with cross functional teams to strengthen data infrastructure, CI/CD pipelines, and analytics capabilities. A Bachelor’s or Master’s degree is required, with a minimum of eight years of experience in data engineering.

Responsibilities

• Apply advanced data engineering principles to design and implement data loading and aggregation frameworks across diverse areas of the organization.
• Collect and process raw data in structured, semi-structured, and unstructured forms using batch and real-time processing frameworks.
• Build and optimize data solutions within enterprise data warehouses and big data repositories, with a emphasis on cloud migration.
• Advance capabilities for Enterprise Data Platform partners to meet the needs of product, engineering, and business teams.
• Leverage enterprise system experience with Databricks, Snowflake, and cloud platforms such as Azure, AWS, and GCP.
• Utilize Python, Spark, and SQL to construct robust pipelines for efficient data processing and analysis.
• Develop CI/CD pipelines to automate the build, test, and deployment processes for data solutions.
• Apply data modeling techniques to design and optimize data schemas, ensuring data integrity and performance.
• Lead continuous improvement efforts to enhance the performance, reliability, and scalability of the data infrastructure.
• Collaborate with data scientists, analysts, and other stakeholders to translate business requirements into technical solutions.
• Implement best practices for data governance, security, and compliance to safeguard data assets.

Requirements

• Bachelor’s or Master’s degree in computer science, engineering, or a related field.
• Eight or more years of experience in a data engineering role, with proven success designing and building data pipelines, ETL processes, and data warehouses.
• Strong proficiency in SQL, Python, and Spark.
• Extensive experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Hands-on work with big data technologies including Hadoop, Spark, Kafka, and distributed computing frameworks.
• Knowledge of data lake and data warehouse tools such as Databricks, Snowflake, Amazon Redshift, Google BigQuery, Azure Data Factory, and Airflow.
• Experience implementing CI/CD pipelines for automated build, test, and deployment of data solutions.
• Solid understanding of data modeling, data warehousing architectures, and data management best practices.
• Excellent communication and leadership abilities, with a track record of collaborating across cross-functional teams and guiding technical decisions.
• Relevant certifications (for example Azure, Databricks, Snowflake) are a plus.

Technologies

• Python
• Spark
• SQL
• Databricks
• Snowflake
• AWS
• Azure
• GCP
• Hadoop
• Kafka
• Airflow
• Google BigQuery
• Amazon Redshift
• Azure Data Factory