Data Engineer

Data Engineer position with the NYC Department of City Planning, onsite in Manhattan, NY.

Location

• Manhattan, NY (onsite)

Compensation

• USD 100,000 to 112,000 per year

Responsibilities

• Design, deploy, and scale the existing data infrastructure to sustain operations, improve processes, and extend the team’s capabilities.
• Assess the technologies used by the team and advocate for new tools; design and implement enhancements to products and workflows to ensure the best tool fit.
• Develop data products and pipelines, including:
  • operating data ingestion pipelines and managing data storage
  • transforming input data with standardization, geocoding, merging, aggregating, and basic spatial operations
  • reviewing outputs by writing custom scripts and/or using QAQC tools to identify errors, inconsistencies, and edge cases
• From concept to completion, design new data products, define specifications with product owners, craft technical designs with teammates, support sprint planning, contribute to documentation, and deliver a finished product.
• Collaborate with other data engineers in sprint planning, design reviews, code reviews, and pair programming.
• Learn and modernize complex legacy data systems, bringing infrastructure into a contemporary development stack.
• Share achievements and lessons learned with NYC Planning and other city agencies, as well as public audiences through talks, blogs, or other formats.

Requirements

• A bachelor’s degree in computer science, engineering, or a related field from an accredited college plus four years of full-time experience in IT automation engineering, monitoring engineering, or infrastructure management; or
• Eight years of full-time experience in IT automation engineering, monitoring engineering, or infrastructure management; or
• Education and/or experience equivalent to either of the above.

Technologies

• SQL variants (PostgreSQL, DuckDB, MySQL, SQL Server)
• Python
• Data Warehousing and Data Lakes
• dbt
• AWS, GCP, Azure
• Containerization, CI/CD, Git
• PostGIS, GeoPandas
• BigQuery, Snowflake, Redshift
• Apache Spark, Airflow, Prefect, Dagster

Benefits

• Public Service Loan Forgiveness

Residency

• New York City residency is not required for this position