Senior Data Engineer

The Senior Data Engineer will design, develop, and maintain scalable data pipelines and architectures that support advanced analytics and business intelligence, leveraging big data ecosystems, cloud platforms, and robust data modeling.

Responsibilities

• Design, build, and optimize large-scale ETL pipelines using Informatica, Talend, Apache Spark, and Apache Hive.
• Create and sustain data architectures on cloud environments including AWS, Azure Data Lake, and other public clouds.
• Partner with cross-functional teams to capture requirements and translate them into scalable data solutions.
• Develop and implement data models employing dimensional modeling techniques for data warehousing and analytics.
• Oversee data integration from linked data, RESTful APIs, and additional external sources.
• Ensure data quality, security, and compliance throughout the data lifecycle.
• Support machine learning model deployment through efficient data pipeline integration.
• Apply SQL, Python, Java, Bash, and shell scripting to automate processes and improve efficiency.
• Contribute to Agile practices by participating in sprint planning, reviews, and continuous improvement efforts.
• Document data workflows and architecture diagrams to promote transparency and knowledge sharing.

Requirements

• Proven experience as a Data Engineer or similar role focusing on big data systems and cloud-based architectures.
• Extensive knowledge of AWS services (S3, Redshift, Glue), Azure Data Lake, Hadoop components, and other public cloud offerings.
• Strong proficiency with SQL databases including Microsoft SQL Server, Oracle, and cloud-based options.
• Hands-on ETL/ELT development experience using Informatica or Talend.
• Demonstrated expertise in data modeling techniques, including dimensional modeling for enterprise data warehouses.
• Proficiency in programming languages such as Python and Java, with automation scripting in VBA, Bash, and shell scripting.
• Familiarity with business intelligence tools like Looker or similar platforms for reporting and analysis.
• Understanding of RESTful API integration for linked data access.
• Knowledge of analytics approaches related to database design, query management, model training, and analytics methodologies.
• Experience delivering value in fast-paced environments using Agile methodologies.

Technologies

• Informatica
• Talend
• Apache Spark
• Apache Hive
• AWS
• Azure Data Lake
• Hadoop
• S3
• Redshift
• Glue
• Microsoft SQL Server
• Oracle
• SQL
• Python
• Java
• VBA
• Bash scripting
• Shell Scripting
• Looker
• RESTful API

Pay

$65.00 - $85.00 per hour

Location

New York, NY (remote)