DataJobs.io
← Back to all jobs

Job Description

This Data Engineer position is based in Quincy, MA and is onsite. The assignment is for 12 months with the possibility of extension. The role focuses on Databricks-centered data engineering, including ETL/ELT development, Spark performance work, and orchestration of data processing workflows.

Role Overview

You will work on building and improving data pipelines and Databricks-based solutions. The scope includes ETL/ELT implementation, development of Spark jobs, data transformations, and ongoing optimization of data processing performance and workflows.

Responsibilities

  • Design and implement Databricks solutions using Scala and Spark, including work with data frames and notebooks/sql.
  • Develop and optimize Spark jobs, data transformations, and end-to-end data processing workflows on Databricks.
  • Perform performance tuning and enhancement for pipelines using Scala/PySpark and SQL.
  • Work with Databricks storage and governance components, including Delta Lake storage and Unity Catalog.
  • Apply Autoloader patterns for ingestion workflows and support Lakehouse and Medallion architecture implementations.
  • Support Databricks operational setup, including clusters configuration and sizing.
  • Contribute to job coordination using Databricks job orchestration and streaming workflows via Spark structured streaming.
  • Utilize Databricks serverless options as relevant to solution design.

Required Experience

  • 8+ years of extensive data engineering experience in Databricks and ETL/ELT using data pipelines.
  • SQL and processor-related experience.
  • 8 years of hands-on development implementing Databricks solutions with Scala and Spark, including data frames and notebooks/sql.
  • 8+ years of hands-on development with performance tuning and enhancement using Scala/PySpark and SQL.
  • Strong working knowledge of Databricks concepts including Delta Lake storage, Unity Catalog, Autoloader, Lakehouse architecture, and Medallion architecture.
  • Experience with Dataset and data frame work in Databricks environments.
  • Familiarity with Databricks clusters configuration and sizing and Databricks job orchestration.
  • Experience with Spark structured streaming.

Technical Skills

  • Databricks
  • ETL/ELT
  • SQL
  • Scala, Spark, PySpark
  • Data frames, Dataset
  • Notebooks/sql
  • Delta Lake storage
  • Unity Catalog, Autoloader
  • Lakehouse architecture, Medallion architecture
  • Databricks serverless options
  • Databricks clusters configuration and sizing
  • Databricks job orchestration
  • Spark structured streaming

Location and Engagement

  • Location: Quincy, MA (onsite)
  • Duration: 12 Months + Extension

Compensation

USD 86 per hour.

Similar Jobs