Azure Event Hubs

Azure Event Hubs Data Jobs

2 jobs found

8d ago

Senior Data Engineer

firstPROPhiladelphia Philadelphia, PA

Philadelphia, PA

8d ago

$60 - $70/hr Contract Senior Analytics Automation

Apply Now

Job Description

FirstPROPhiladelphia is seeking a Senior Data Engineer to design and operate real-time streaming pipelines for connected medical devices in a hybrid contract-to-hire engagement in Philadelphia. The role centers on building scalable data flows with Azure Databricks and related Azure services, enabling ML and analytics workflows within a Medallion architecture. The successful candidate will collaborate with data scientists, software engineers, and business stakeholders to translate telemetry into reliable, actionable insights.

Engagement and compensation

Location: Philadelphia, PA (hybrid)
Job type: Contract (contract-to-hire)
Hourly rate: USD 60 - 70
Education and experience requirements: 6+ years of data engineering/analytics/warehousing with a Bachelor's degree in Computer Science, Mathematics, Engineering, or a related field; OR 3+ years with a Master's degree in a related technical field

Responsibilities

Design and implement batch and streaming data pipelines using Azure and Databricks, focusing on real-time IoT telemetry within a Medallion architecture
Develop ETL/ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data
Build and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams
Implement real-time streaming solutions via Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns with cost-effective throughput, partitioning, and downstream delivery to Databricks
Optimize production Databricks pipelines using PySpark, Spark SQL, and Delta Lake, including Spark tuning for performance, reliability, and cost
Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on‑premises systems with root-cause analysis and corrective actions
Collaborate with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities
Apply data quality, validation, and privacy-first practices, delivering reliable pipelines through engineering standards, documentation, testing, and CI/CD

Requirements

Bachelor's degree in Computer Science, Mathematics, Engineering, or a related field and 6+ years of professional experience in data engineering, analytics, or warehousing; OR Master's degree in a related technical field and 3+ years of professional experience in data engineering, analytics, or warehousing
5+ years designing, building, and operating big data and real-time streaming pipelines across cloud and on‑premises environments
5+ years applying DevOps and CI/CD practices to data and analytics workloads
Production experience delivering data services, APIs, or microservices for downstream data consumption
Strong, hands-on experience designing and building production-grade data pipelines on Databricks
Demonstrated Spark optimization and tuning in Databricks, including performance analysis, partitioning strategies, caching, shuffle optimization, and cost‑aware pipeline design
Strong experience with Azure cloud services for data engineering and streaming workloads
Experience delivering data services, APIs, or microservices for data consumption
Data quality, validation, and privacy‑aware handling for regulated or sensitive data

Technologies

Databricks
Spark, PySpark, Spark SQL
Delta Lake
Azure, including Azure Event Hubs and Azure Stream Analytics
Azure Databricks
Medallion architecture

Outcomes

Onboard to the Azure Databricks environment and contribute to troubleshooting, stabilization, and optimization of existing batch and streaming pipelines
Stand up Azure streaming ingestion for telemetry data and deliver production-ready pipelines integrated with Databricks to support API, ML, and downstream analytics within the Medallion framework
Design and deliver data services or consumption patterns that enable business and ML teams to access near‑real‑time telemetry data reliably, securely, and at scale

Work hours and travel

Willingness to assist in troubleshooting and analysis during off-hour production problems as needed
The hybrid setup requires a minimum of two days per week in the downtown Philadelphia office

View full job page →

21d ago

Senior Data Engineer

Deloitte Morristown, NJ

Morristown, NJ

21d ago

$95k - $150k/yr Full time Senior Adls Gen2 Apache Airflow

Apply Now

Job Description

Deloitte in Morristown, New Jersey is seeking a Senior Data Engineer to design, build, and optimize end-to-end data solutions and ETL/ELT pipelines using Azure Data Factory, Databricks, and PySpark. This onsite role collaborates closely with engagement teams and client architects to deliver scalable data platforms, with a salary range of USD 95,000 to 150,000 per year and a bachelor’s degree required.

Responsibilities

Maintain regular communications with engagement managers (directors), project teams, and stakeholders across functional and technical groups, escalating issues that require attention from engagement leadership.
Design, build, and optimize ETL/ELT pipelines leveraging Azure Data Factory and Databricks.
Develop and tune PySpark and Spark SQL notebooks for large-scale data transformations.
Architect end-to-end data solutions across development, UAT, and production environments using Unity Catalog.
Lead design discussions with client architects and other counterparts.
Collaborate across teams to establish data contracts and agree on schemas.
Lead the design and optimization of high-volume data pipelines.
Define and enforce data engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning.
Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management.
Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies.
Conduct root cause analysis on production incidents and implement lasting fixes.
Mentor junior and mid-level engineers through code reviews and pair programming.
Evaluate new technologies and recommend adoption, such as DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs.

Requirements

Proficiency in Python, PySpark, Spark SQL, and SQL Server.
Experience with Azure services including Azure Data Factory, ADLS Gen2, Key Vault, and Azure Monitor.
Familiarity with Databricks components such as Delta Lake, Unity Catalog, and Workflows.
Experience with Apache Airflow.
Git and Azure DevOps tooling.
Deep knowledge of Spark internals, including DAG optimization, spill analysis, and skew handling.
Advanced Delta Lake features, including time travel, deletion vectors, and predictive I/O.
Unity Catalog governance covering row/column security, external locations, and system tables.
Infrastructure as Code experience with Terraform and Azure ARM templates.
Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or related IT discipline, or equivalent experience.
Limited immigration sponsorship may be available.
Ability to travel up to 10% on average, depending on client and project needs.

Technologies

Python
PySpark
Spark SQL
SQL Server
Azure Data Factory
ADLS Gen2
Key Vault
Azure Monitor
Databricks
Delta Lake
Unity Catalog
Workflows
Apache Airflow
Git
Azure DevOps
DABs
DLT
Auto Loader
Serverless Compute
Azure Event Hubs
Terraform
Azure ARM templates

Additional Requirements

Information for applicants with a need for accommodation: Accommodation information for disabled applicants.

View full job page →

Azure Event Hubs Data Jobs

Senior Data Engineer

Senior Data Engineer

Get Azure Event Hubs Data Jobs alerts