Senior Data Engineer
Job Description
Drive enterprise data infrastructure for The Home Depot by building pipelines, validation processes, and in-house solutions that improve reliability and data integrity.
Responsibilities
- Apply deep business understanding to solution approach; build trust and collaboration with internal customers and cross-functional teams
- Educate technical and non-technical partners on advanced analytics and translate needs into actionable data work
- Understand IT requirements to successfully address business problems and pursue new data science opportunities as a competitive advantage
- Partner with project teams and business stakeholders to define project goals
- Set direction on prioritization and help ensure quality of delivery
- Provide mentoring and coaching to more junior roles to strengthen technical competencies
- Collaborate with managers and team members on workload and resource distribution; support recruiting and hiring efforts
- Execute analytical work with high efficiency and quality, selecting and interpreting advanced analytical methodologies
- Communicate insights, recommendations, and impacts clearly to technical and non-technical leaders and business partners
- Prepare progress updates and presentations related to project and solution development
- Lead requirements gathering, design, and development for complex applications and programs
- Stay current on developments in advanced analytics, technical skill sets, and additional data sources
- Contribute to continuous improvement of predictive and prescriptive analytics by developing replicable solutions
- Support online Data Engineer work that captures customer data and converts business requirements into infrastructure, datasets, and transformations
- Build, test, and maintain database pipeline architectures; develop data validation methods and data analysis tools
- Create and maintain APIs for data retrieval and support enterprise in-house solutions with monitoring, continuous testing, and validation
- Use big-data techniques to cleanse, organize, and transform data, while defending and updating data structures and integrity via automated processes
- Establish design standards and assurance processes to ensure compatibility and operability of data connections, flows, and storage requirements
- Review internal and external business and product requirements for data operations and recommend system and storage changes or upgrades
Requirements
- Minimum 4 years of experience
- Must be 18 years of age or older
- Must be legally permitted to work in the United States
- Education: bachelor's degree program or equivalent degree in a field related to the job
Preferred Qualifications
- Bachelors or Masters in a quantitative field (Analytics, Computer Science, Math, Physics, Statistics, etc.) or relevant work experience
- 5+ years of relevant work experience
- Experience in predictive modeling, data mining, and data analysis
- Experience developing and testing ETL jobs/pipelines, configuring orchestration, automated CI/CD, writing automation scripts, and supporting pipelines in production
- Experience in Python and high-level programming languages
- Experience defining and capturing metadata and rules for ETL processes
- Experience building Batch and Streaming pipelines
- Experience writing analytical SQL queries and performance-tuning queries
- Ability to stitch and maintain data from multiple sources
- Ability to use JavaScript, React, and QA apps (Retina, KPI Shield, Alert Goose)
- Ability to produce tags for site data
- Ability to code in Python and Google BigQuery to stitch and enrich raw data
- Proven ability to use PySpark, AirFlow, and DataProc to engineer and automate data flow pipelines
- Ability to optimize pipeline run time and lower cost on slots/storage consumption
- Ability to prioritize requests and manage a product roadmap
- Coaching junior engineers to improve code, best practices, and understanding of data engineering principles
- Strong verbal and written communication skills at all levels
- Leadership experience: none required
Technology
- Python, JavaScript, React
- ETL, ETL jobs/pipelines, BigQuery, PySpark, AirFlow, DataProc
- CI/CD, SQL
- Batch pipelines, Streaming pipelines
Location and Work Setup
- Atlanta, GA
- Onsite
Travel
- Overnight travel: 5% to 20% of the time