Lead Data Engineer

Kaizen Dynamics is seeking a Lead Data Engineer to support the OCTO Enterprise Data team and the Cloud Data Exchange. In this onsite contract role, you will contribute to the design, development, implementation, and ongoing support of complex enterprise data pipelines and governance-aligned workflows using native Azure tooling.

You will work across cloud and on-prem connectivity, ETL and transformation activities, and pipeline automation for loading data from multiple sources into landing layers and SQL targets, while coordinating with stakeholders and other agencies to resolve requirements, constraints, and feasibility across the current and future data systems.

Responsibilities

• Analyze, design, develop, implement, replicate, and support complex enterprise data projects.
• Interface with other agencies and consult with user departments on system requirements; advise on environment constraints and operating difficulties for the current state; resolve problems using cloud solutions and help develop and replicate future enhancements to District data systems.
• Apply strong Extraction, Transformation and Loading (ETL) expertise using frameworks such as Azure Data Factory, Synapse, Databricks, and Informatica by gathering requirements from stakeholders or analyzing existing code and performing enhancements or new development.
• Establish cloud and on-prem connectivity across systems including ADLS, ADF, and Synapse, as well as Databricks.
• Provide hands-on work with Azure services including Azure Data Factory or Azure Synapse, MSSQL DB, Azure SQL DB, Azure Data Lake Storage Gen2, Blob Storage, and Python.
• Create end-to-end pipelines that read from multiple sources and load into a landing layer or SQL tables.
• Use data integration and pipeline tools such as Informatica, Synapse, Apache NiFi, and Apache Airflow.
• Work with multiple data formats, including database formats (Oracle, SQL Server, DB2, Quickbase), text formats (CSV, XML), and binary formats (Parquet, AVRO).
• Develop, standardize, and optimize data workflow and pipelines while adhering to best practices.
• Support enterprise data governance by ensuring data accuracy, consistency, security, and reliability.
• Automate, monitor, manage, and trigger alerts for data pipelines and workflows.
• Analyze and evaluate system changes for feasibility, and provide alternative solutions as well as backup and rollback procedures.
• Develop new systems, perform upgrades, and deliver enhancements to existing systems while ensuring approved standards and consistency after changes.
• Develop complex programs and reports using database query language.
• Handle and secure sensitive data in line with its level of sensitivity.
• Convey technical and functional concepts for a specific technical specialty, and identify improvements to project standards to support high-quality services and products.
• Coordinate programming activities with the application development team when required, and collaborate with business and technical personnel to resolve intent, inaccuracy, or feasibility issues related to processing and project design.
• Interact with executive-level business users or technical experts and may serve as a niche technical SME (Subject Matter Expert).

Requirements

• Bachelor’s degree in Information Technology or a related field, or equivalent experience.
• SQL experience: 10 years (Required).
• ETL processes experience: 10 years (Required).

Technologies

• Azure Data Factory, Azure Synapse, Databricks
• Informatica
• ADLS, Azure Data Lake Storage Gen2, Blob Storage
• MSSQL Db, Azure SQL DB
• Python
• Apache NiFi, Apache Airflow
• Oracle, SQL Server, DB2, Quickbase
• CSV, XML, Parquet, AVRO

Location and Engagement

• Work location: In person (Washington, DC)
• Job type: Contract (also listed as full-time in job types)

Compensation: USD 70 - 80 per hour.