Spark Structured Streaming Data Jobs

1 job found

Today

Databricks Data Engineer

i4DM Remote 🌍 Remote

Remote

Today

Full time Artificial Intelligence Big Data Bigdata

Apply Now

Job Description

i4DM is hiring a hands-on Databricks Data Engineer to build and run scalable data and analytics solutions on the Databricks Lakehouse for federal mission needs.

Responsibilities

Design, develop, and maintain scalable batch and streaming data pipelines using Databricks, Apache Spark, PySpark, and Spark SQL
Build and manage Delta Lake tables using the medallion (bronze/silver/gold) architecture
Develop real-time and near-real-time ingestion using Spark Structured Streaming and messaging platforms such as Kafka
Configure and manage Databricks clusters, jobs, and workflows in production environments
Implement data governance, access controls, and security using Unity Catalog
Integrate data from multiple source systems and destinations, supporting ETL/ELT and pipeline orchestration
Optimize existing data workflows and Spark jobs for performance, reliability, and cost efficiency
Integrate Databricks development with CI/CD and enterprise SDLC tooling, including Git-based version control
Collaborate with data scientists and analysts on data models and support machine learning and AI use cases, including model lifecycle management with MLflow
Support advanced analytics such as anomaly detection, risk scoring, and fraud analytics
Monitor and troubleshoot processing jobs; implement data quality checks and observability for high availability
Document data processes, frameworks, pipelines, and data mappings for technical and non-technical audiences
Work with scrum teams, product owners, and client stakeholders to deliver end-to-end data solutions
Stay current on Databricks platform capabilities and industry trends to recommend best-fit tools and technologies

Requirements

Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent experience)
4+ years of experience in data engineering, analytics engineering, or big data development
2+ years hands-on experience with the Databricks platform
Proficiency in Apache Spark, PySpark, and Spark SQL
Production experience with Databricks clusters, jobs/workflows, Delta Lake, and Unity Catalog
Experience with medallion architecture and Spark Structured Streaming
Strong Python and SQL for data engineering and analysis
Experience with ETL/ELT and data pipeline orchestration
Familiarity with cloud platforms such as AWS, Azure, or Google Cloud and their native data services
Experience integrating data solutions with CI/CD pipelines and Git-based version control
Understanding of data governance, security, and access control best practices
Experience working in Agile development environments
Ability to obtain and maintain a Public Trust determination
Excellent analytical, problem-solving, and communication skills with ability to work with technical and non-technical stakeholders

Technologies

Databricks
Apache Spark, PySpark, Spark SQL
Delta Lake
Unity Catalog
Medallion architecture
Spark Structured Streaming
Kafka
ETL/ELT
CI/CD
Git-based version control
MLflow
Python, SQL
AWS, Azure, Google Cloud
ML/AI use cases

Preferred Qualifications

Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional) or cloud platform certification
Experience implementing ML or AI solutions in Databricks, including MLflow-based model lifecycle management
Knowledge of machine learning, AI, or NLP techniques, including text mining
Experience supporting fraud analytics, risk scoring, or anomaly detection
Experience with distributed data and streaming tools such as Kafka, Hadoop, Hive, or Amazon EMR
Experience with data quality frameworks and observability/monitoring tooling
Experience with NoSQL databases
Experience with visualization packages such as Plotly, Seaborn, or ggplot2
Experience supporting federal government or regulated-industry programs, especially the Department of Veterans Affairs

View full job page →

Get Spark Structured Streaming Data Jobs alerts

New Spark Structured Streaming Data Jobs, daily.