Streaming Data

Streaming Data Jobs

4 jobs found

1d ago

Sr. Data Engineer

Sunbelt Rentals, Inc. Fort Mill, SC

Fort Mill, SC

1d ago

$130k - $179k/yr Full time Senior Big Data Bigdata

Apply Now

Job Description

Sunbelt Rentals, Inc. is hiring a Sr. Data Engineer to design, build, and maintain end-to-end IoT data pipelines into Databricks. The position focuses on streaming ingestion, data quality, Delta Lake operations, and supporting production stability across the data platform.

Onsite Location

Fort Mill, SC (onsite)

Compensation

USD 130,315 - 179,183 per yearly

Base pay range: $130,315.00 - 179,183.40

Experience Level

Minimum experience: 5 years

Core Responsibilities

Design, build, and maintain end-to-end IoT data pipelines from edge devices into Databricks using the medallion architecture (Bronze/Silver/Gold).
Lead development and optimization of streaming ingestion pipelines using Apache Kafka and Databricks Structured Streaming.
Define and enforce data engineering best practices, including pipeline design patterns, data quality frameworks, and schema governance.
Architect and manage Delta Lake tables, including partitioning strategies, Z-ordering, and vacuuming to support performance and cost efficiency.
Collaborate with infrastructure and cloud teams to configure and manage Databricks clusters, Unity Catalog, and Delta Live Tables.
Mentor junior and mid-level data engineers through code reviews, pair programming, and technical guidance.
Partner with data scientists and analysts to support downstream analytical and ML requirements.
Troubleshoot production pipeline failures and drive root cause analysis and resolution.
Evaluate and introduce new tools and technologies to continuously improve the data platform.

Required Qualifications

5+ years of data engineering experience, including at least 2 years working extensively with Databricks.
Strong proficiency in Python and SQL, with experience writing production-grade Spark code.
Hands-on experience building and managing Kafka-based streaming pipelines.
Deep familiarity with Delta Lake, Delta Live Tables, and the Databricks ecosystem, including Unity Catalog, Workflows, and MLflow.
Experience with IoT data patterns such as high-frequency telemetry ingestion, device state management, and time-series data.
Solid understanding of cloud infrastructure ( Azure preferred) and infrastructure-as-code concepts.
Strong communication skills, including the ability to lead technical discussions and translate requirements into architecture.

Technologies

Databricks
Apache Kafka
Databricks Structured Streaming
Delta Lake
Delta Live Tables
Unity Catalog
Workflows
MLflow
Python
SQL
Spark
Azure
Infrastructure-as-code
Bronze/Silver/Gold (medallion architecture)
Z-ordering
vacuuming
IoT
ML

Preferred Qualifications

Experience with Geotab or similar fleet telematics platforms.
Databricks Certified Data Engineer Associate or Professional certification.
Familiarity with Azure Key Vault, Event Hubs, or Azure Data Factory.
Experience with dbt for data transformation.
Bachelor’s degree in computer science, Information Technology, Data Engineering, or equivalent experience.

Benefits

Health, Dental and Vision plans
401(k) Match
Volunteer time off
Short-term and long-term disability
Accident, Life and Travel insurance, as well as flexible spending
Tuition Reimbursement Options
Employee Assistance Program (EAP)
Length of Service Awards

Benefits Eligibility

You will become eligible for benefits on the first of the month following 30 days from your start date.

Paid Time Off

12-25 vacation days depending on years of service
5 sick days
6 holidays
2 half day holidays
2 floating holidays
1 inclusion day
1 volunteer day

Physical Demands and Safety Requirements

Must be able to bend, squat, crouch and/or reach and lift up to 25 pounds or more, as required by the job.
Some Sunbelt jobs may require driving for long periods of time, loading and unloading heavy equipment, performing work in extreme weather conditions including rain, wind, or excessive temperatures and/or night and weekend work.
All duties must be performed according to Sunbelt’s safety policies and guidelines.
Reasonable accommodations may be made to comply with ADA/ADAAA.

View full job page →

2d ago

Senior Data Engineer

Software Resources, Inc. Glendale, CA

Glendale, CA

2d ago

$90 - $93/hr Full time Senior Big Data Data Architecture

Apply Now

Job Description

The Senior Data Engineer will support enterprise data engineering initiatives by designing, building, and optimizing scalable data solutions. The role includes owning and operating the Core Data platform, delivering both batch and streaming Spark pipelines, and applying governance across AWS, Kubernetes, and Airflow.

Role Overview

This position focuses on Core Data platform delivery and ongoing operations. The engineer will manage Databricks platform governance, develop pipelines with PySpark and related tooling, and ensure platform reliability through monitoring, cost awareness, automation, and troubleshooting across deployed services.

Key Responsibilities

Manage Databricks platform governance, including Unity Catalog, ACLs, lineage, and data discovery and privacy tooling
Design, write, test, and deploy batch and streaming data pipelines using PySpark, Scala, SQL, Python, and Kotlin
Meet with stakeholders to gather requirements and translate them into scalable data platform solutions
Use knowledge of Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrations
Explain Spark architecture and pipeline behavior to stakeholders to identify root causes and recommend solutions
Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross-platform integrations
Build and maintain Kubernetes containers and containerized utilities that support deployed data platform services
Apply networking knowledge to troubleshoot connectivity and integration errors across platform components
Perform platform administration such as provisioning and removing access, assessing resource utilization, monitoring platform health and cost, and evaluating stakeholder requests
Collaborate with engineers, architects, and product managers to drive Core Data platform success and participate in agile or scrum ceremonies
Maintain documentation for platform changes, standards, and pipeline configurations to support data quality and governance
Develop and maintain scalable data engineering solutions, including building and enhancing data pipelines and workflows
Support cloud-based data platform initiatives and collaborate with engineering teams on system architecture and design
Implement infrastructure automation and deployment standards and troubleshoot and optimize data processing environments

Required Qualifications

Strong experience with Databricks
Advanced Python development skills
Experience with Terraform and Infrastructure as Code
5+ years of relevant data engineering experience
Ability to participate in architecture and system design discussions
Strong problem-solving and data platform implementation experience

Additional Qualifications

5+ years of data engineering experience developing and operating large-scale data pipelines
Deep hands-on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or Scala
Strong understanding of Spark architecture (executors, stages, partitioning, shuffle) with ability to explain performance tuning tradeoffs to technical and non-technical stakeholders
Proficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshooting
Proficient in SQL with advanced performance tuning capabilities
Hands-on production experience with Airflow (MWAA) for orchestrating data pipelines
Experience managing Databricks governance, including ACLs, Unity Catalog, lineage, and access provisioning
Proficiency in Python and at least one additional language (Scala, Kotlin, or SQL-driven pipeline tooling)
Experience designing and optimizing scalable ETL/ELT pipelines integrating diverse structured and unstructured data sources
AWS-primary experience (compute, storage, networking, IAM); experience with other cloud providers is transferable
Proficiency with Docker and Kubernetes for building and maintaining containerized data platform services
Working knowledge of networking concepts to diagnose cross-platform integration and connectivity issues
Familiarity with Snowflake and comparable tooling relative to the Databricks ecosystem
Experience implementing CI/CD and DevOps practices using Git-based workflows
Experience implementing data quality checks, monitoring, and logging for pipeline reliability
Self-starting problem solver with strong analytical and communication skills and willingness to learn new tooling and trends
Familiar with Scrum and Agile methodologies
Bachelor’s Degree in Computer Science, Information Systems, or related field, or equivalent work experience; Master’s Degree is a plus

Preferred Qualifications

Experience with AWS
Cloud-based data platform experience
Data pipeline optimization and automation expertise

Technology Stack

Databricks, Apache Spark
Python, PySpark, Scala, SQL, Kotlin
Terraform, Infrastructure as Code
AWS, Kubernetes, Docker
Airflow (MWAA)
Unity Catalog
Git-based workflows, CI/CD

Location and Schedule

Glendale, CA (hybrid)
Duration: 12+ months

Compensation

Pay rate: USD 90 - 93 per hourly (DOE)

Benefits

Medical, dental, and vision coverage
401(k) with company match
Short-term disability
Life insurance with AD&D

View full job page →

13d ago

Data Engineer

JANSON Fairfax, VA

Fairfax, VA

13d ago

$135k - $135k/yr Full time Big Data Bigdata Data

Apply Now

Job Description

JANSON is seeking a Data Engineer (Engineer IV) to support a U.S. Department of Defense (DoD) initiative focused on data pipelines, data warehouses/lakes, integration, validation, metadata, and compliance/security.

Responsibilities

Extract, transform, and load (ETL) data from sources including DoD Advana and Army Vantage into data warehouses or data lakes
Clean, normalize, and enrich data during transformation
Configure automated workflows to move data between systems using Apache Airflow, AWS Glue, or Azure Data Factory
Optimize pipelines for scalability and reliability
Implement real-time data processing via streaming pipelines
Build streaming pipelines using Apache Kafka, Apache Flink, or Spark Streaming
Process and analyze data in real-time for applications such as fraud detection or IoT analytics
Design and implement relational databases (e.g., MySQL, PostgreSQL) or NoSQL databases (e.g., MongoDB, Cassandra) based on project requirements
Ensure efficient schema design and indexing for performance
Build and maintain data warehouses using Snowflake, Amazon Redshift, or Google BigQuery
Optimize storage and query performance for analytical workloads
Support management of data lakes and integration within existing organizational policies
Interpret policies to support long-term sustainability when integrating products into existing data lakes
Integrate data from APIs, databases, flat files, and third-party systems
Handle diverse data formats including JSON, XML, CSV, and Parquet
Migrate data between systems during upgrades or transitions (e.g., on-premises to cloud)
Configure APIs to expose data for consumption by other systems or applications
Manage data quality and governance at the 21st TSC level per USAEUR-AF policies and guidance
Implement checks to ensure accuracy, completeness, and consistency
Identify and resolve data anomalies or errors
Document data lineage, definitions, and transformations for traceability
Implement and manage compliance and security measures for data systems
Support regulatory compliance including GDPR, HIPAA, or CCPA
Apply security controls such as encryption, access control, and auditing
Partner with internal teams to help rectify or mitigate data quality issues in analyses and tool configuration
Work with PEOs and PMs of Systems of Record to understand and correct data issues feeding multiple data platforms

Requirements

18 years of recent specialized experience
OR: Major IT Certification plus 14 years of recent specialized experience
OR: Associate’s degree in computer science, information management, or related discipline plus 14 years of recent specialized experience
OR: Bachelor’s degree in computer science, information management, or related discipline plus 10 years of recent specialized experience
OR: Master’s degree in computer science, information management, or related discipline plus 7 years of recent specialized experience
IA Technical Level II certification or higher
CCNA-Security
CompTIA CySA+
Global Industrial Cyber Security Professional (GICSP)
GIAC Security Essentials (GSEC)
CompTIA Security+ CE
EC-Council Certified Network Defense (CND)
(ISC)² Certified System Security Practitioner (SSCP)
Bachelor’s degree (required)
Recent specialized experience: 10 years (required)
IAT Level II or higher (required)

Technologies

Apache Airflow, AWS Glue, Azure Data Factory
Apache Kafka, Apache Flink, Spark Streaming
MySQL, PostgreSQL, MongoDB, Cassandra
Snowflake, Amazon Redshift, Google BigQuery
APIs, JSON, XML, CSV, Parquet
GDPR, HIPAA, CCPA
Encryption, access control, auditing

Location and Work Details

Work location: Kaiserslautern, Germany
Clearance: Secret
Job type: Full-time
Work location: In person
Position: Data Engineer (Engineer IV)

Compensation

USD 135,000 per year

Application Questions

Do you have a issue living in Kaiserslautern, Germany for 1-2 years.
What is your salary range?

View full job page →

14d ago

GCP BigQuery Data Engineer

Virtues Irving, TX

Irving, TX

14d ago

$100k - $110k/yr Full time Analytics Big Data Bigdata

Apply Now

Job Description

Virtues is seeking a GCP BigQuery Data Engineer to design cloud-native, scalable data platforms and build real-time ingestion pipelines on Google Cloud. This onsite role in Irving, TX focuses on event-driven streaming architectures that support near real-time analytics and enterprise reporting.

Responsibilities

Design, develop, and optimize enterprise data warehouse solutions using Google BigQuery.
Design, develop, and maintain real-time data ingestion pipelines using Apache NiFi to extract, transform, and route streaming data into Apache Kafka topics.
Build scalable event-driven streaming architectures using Apache NiFi, Apache Kafka, and Google BigQuery to enable high-throughput, fault-tolerant, low-latency processing.
Configure and optimize Apache NiFi processors for extraction, transformation, routing, filtering, schema validation, error handling, retry mechanisms, and reliable delivery to Kafka.
Develop streaming ingestion solutions that allow Google BigQuery to consume Kafka event streams and transform near real-time events into analytical tables and reporting datasets.
Design data models and implement ETL/ELT processes to move data from raw to curated and published layers.
Design, create, and manage large-scale BigQuery datasets, including temporary/permanent and internal/external tables.
Optimize BigQuery workloads using query tuning, partitioning, clustering, and cost optimization techniques to improve performance and reduce cloud costs.
Monitor, troubleshoot, and optimize streaming workloads by tuning NiFi flows, Kafka topics/partitions, and BigQuery streaming ingestion to support high availability, data integrity, minimal latency, and cost-efficient processing.
Build scalable data lake frameworks and ingestion pipelines using cloud-native GCP technologies.
Develop reporting and visualization solutions using Looker, Looker Studio (Data Studio), Connected Sheets, and other BigQuery reporting tools.
Collaborate with data architects, modelers, developers, DevOps engineers, project managers, and business stakeholders to deliver scalable enterprise analytics solutions and continuous improvements to the data platform.

Requirements

7+ years of experience in Data Engineering, Data Warehousing, or Big Data platforms.
Must-have skills: Apache NiFi and Apache Kafka.
Strong hands-on experience with Google Cloud Platform (GCP), including BigQuery, Cloud Dataflow, Pub/Sub, and Google Cloud Storage (GCS).
Hands-on experience designing and supporting real-time streaming pipelines using Apache NiFi, Apache Kafka, and Google BigQuery.
Experience using BigQuery Console/Query Editor for data management, performance tuning, and SQL development.
Strong experience designing scalable data models, ETL/ELT pipelines, data lake architectures, and enterprise data warehouse solutions.
Experience with batch and streaming data ingestion using GCP services.
Thorough understanding of BigQuery cost structure, including storage, ingestion, and query costs, with experience implementing query optimization, partitioning, clustering, and other cost optimization techniques.
Experience managing large-scale datasets, including temporary/permanent and internal/external BigQuery tables.
Experience developing reporting and visualization solutions using Looker, Looker Studio (Data Studio), and Connected Sheets.
Excellent verbal and written communication skills, with the ability to collaborate with technical teams, business stakeholders, senior management, and executive leadership.
Experience working in Agile environments and collaborating with cross-functional teams to deliver enterprise data engineering solutions.

Preferred Qualifications

Google Cloud certifications.
Experience delivering enterprise-scale analytics and cloud data solutions.

Technologies

Google BigQuery
Apache NiFi
Apache Kafka
Google Cloud Dataflow
Google Pub/Sub
SQL
ETL/ELT
Google Cloud Storage (GCS)
Looker
Looker Studio (Data Studio)
Connected Sheets
CI/CD
DevOps
Agile/Scrum
Python
Spark
Data Modeling
Data Warehousing
Data Lakes
Batch & Streaming Pipelines

Compensation and Location

Salary: USD 100,000 - 110,000 per year.

Work location: In person (Irving, TX).

View full job page →

Streaming Data Jobs

Sr. Data Engineer

Senior Data Engineer

Data Engineer

GCP BigQuery Data Engineer

Get Streaming Data Jobs alerts