Azure Data Lakehouse

Azure Data Lakehouse Jobs

4 jobs found

Today

Data Engineer

Pyx Health Inc Remote 🇺🇸 Remote

Remote

Today

Full time Python Apache Airflow Azure

Apply Now

Job Description

Pyx Health Inc is hiring a Healthcare Data Engineer to help build and maintain Azure-based data infrastructure and dependable data pipelines. In this remote role, you will work with Databricks, Airflow, Python, and SQL to support healthcare data workflows, with an emphasis on reliability, governance, and data quality for downstream reporting.

What you’ll do

Build and maintain data pipelines on Azure to ingest, transform, and clean healthcare data using Databricks, PySpark, and Delta Lake
Implement pipeline logic based on defined specifications, applying architectural guidance from senior engineers
Create reusable testing frameworks that help ensure pipeline reliability
Monitor pipelines for failures and performance issues, escalating complex problems when needed
Develop and maintain Airflow DAGs with appropriate error handling and retry logic
Support deployment and configuration of pipelines using Astronomer on Azure via Astro CLI
Reduce manual intervention by improving pipeline reliability over time
Implement data models in Delta Lake for efficient storage and retrieval, including merge/upsert patterns
Design and optimize Delta Lake tables for analytic workloads
Use Unity Catalog to support data governance, organization, and security
Apply Change Data Capture (CDC) patterns under the direction of senior team members
Write and optimize T-SQL and Databricks SQL scripts, stored procedures, and ETL support
Work across the Azure ecosystem including ADLS Gen2, Key Vaults, Logic Apps, and Azure DevOps
Follow established security and scalability standards when building data infrastructure
Ensure scripts and datasets are well-documented to support enterprise data governance
Design and continuously improve automated data quality monitoring, reconciliation, and alerting to support reliable downstream reporting
Troubleshoot and resolve pipeline failures by documenting root causes and resolutions
Collaborate with data scientists, analysts, and business stakeholders to understand reporting requirements and troubleshoot data issues
Participate in code reviews and incorporate feedback to improve code quality
Document pipelines, processes, and implementation decisions clearly and consistently
Stay current with data engineering technologies, especially in the healthcare space

What you’ll bring

2–4 years of experience as a Data Engineer or closely related data role
Hands-on experience with Azure cloud services such as ADLS, Databricks, or similar
Working knowledge of SQL for scripting and data modeling, including T-SQL, Spark, and Databricks SQL
Ability to contribute to technical projects with moderate oversight
Strong communication skills, including comfort asking questions and sharing updates with cross-functional partners
Familiarity with CI/CD, Git, and version control workflows
Solid problem-solving skills with a systematic approach to debugging
Healthcare data standards and regulations knowledge (such as HIPAA and HL7) is a plus
Bachelor’s degree in Computer Science, Information Technology, Engineering, or related field (or equivalent work experience)

Core must-haves

Databricks SQL (Mid): Delta Lake fundamentals, basic merge/upsert patterns, familiarity with CDC concepts
Python (Mid): Pipeline logic, data transformation, SQL scripting; some PySpark/Spark DataFrame experience
Databricks Spark Notebooks (Mid): Notebook-based development, basic cluster usage, Delta table operations
Airflow Python Development (Mid): Ability to write and maintain DAGs, including error handling and retry patterns
Airflow Astro Configuration (Foundational): Familiarity with Astronomer or willingness to learn; basic Astro CLI usage
Azure Ecosystem (Foundational): Working knowledge of ADLS Gen2, Key Vaults, and Azure DevOps
T-SQL (Foundational–Mid): Basic stored procedures, SQL Server querying, and data manipulation

Nice to have

Azure Data Factory (Foundational): Basic familiarity with pipeline authoring and triggers
Git + Azure DevOps CI/CD (Foundational): Experience with branching, pull requests, peer review processes, version control best practices, and deploying production data pipelines using CI/CD workflows

Location: Remote (remote).

View full job page →

Today

Data Engineer - Business Analytics

Primoris Services Corporation Texas 📍 United States Remote

Texas

Today

$135k - $155k/yr Full time Analytics Azure Data Lakehouse Big Data

Apply Now

Job Description

Primoris Services Corporation is hiring a Data Engineer focused on Business Analytics, supporting an enterprise analytics platform built around Azure and Databricks. In this role, you will design and optimize analytics-ready datasets using a modern Lakehouse approach, delivering curated Silver and Gold assets with the semantic layer implemented in Databricks.

What you’ll do

Design, build, and maintain scalable data pipelines and curated datasets within the Databricks Lakehouse (Silver and Gold layers).
Develop and manage analytics-ready data models and semantic structures in Databricks based on enterprise reporting needs.
Transform raw data into business-consumable datasets using standardized definitions, KPIs, and metrics.
Apply dimensional modeling and other data modeling techniques within a distributed data environment.
Own and deliver the semantic layer in Databricks to support consistent definitions across reporting and analytics use cases.
Enable downstream consumption by visualization and analytics tools such as Power BI, without relying on embedded semantic modeling.
Partner with business analysts and reporting teams to improve dataset usability, performance, and scalability.
Optimize datasets and models for performance, including query efficiency and scalability for enterprise reporting workloads.
Ensure data quality, integrity, and consistency across the data lifecycle.
Implement and enforce data governance standards, including security, masking, access controls, and cost controls.
Translate business requirements into scalable data solutions in collaboration with business leaders and stakeholders.
Align data structures to enterprise data dictionary and KPI definitions.
Communicate dataset and data model designs clearly to technical and non-technical stakeholders.
Support platform evolution within Azure and Databricks.
Contribute to best practices for CI/CD, deployment automation, and data engineering standards.
Support the expansion of data capabilities, including assistance with AI/ML use cases where applicable.

What you bring

Bachelor’s degree in Information Technology, Computer Science, Business Analytics, or a related field.
6+ years of experience in data engineering, data modeling, or analytics engineering roles.
Strong expertise in Databricks Unity Catalog, SQL, and Python for transformations and pipeline development.
Proven experience designing and implementing data models within modern data platforms such as Lakehouse architecture.
Experience with medallion architecture (Bronze/Silver/Gold layers).
Knowledge of BI tools such as Power BI, with emphasis on dataset consumption rather than ownership of the semantic layer.
Experience with CI/CD pipelines, version control, and deployment processes.
Familiarity with Agile/Scrum delivery methodologies.

Tools and technologies

Databricks, Databricks Lakehouse, Databricks Unity Catalog
SQL, Python
Azure
Power BI
CI/CD pipelines
Agile/Scrum

Pay and benefits

Salary: USD 135,000 - 155,000 per year
Pay equity compensation range: $135,000 - $155,000
401k with employer match
Health/Dental/Vision insurance plans
Paid time off
10 paid holidays
Stock purchase plan

Location and work conditions

Location: Texas (remote)
Occasional travel may be required
Flexible work hours may be necessary to meet project deadlines

Salary note: Pay equity compensation range is USD $135,000 - $155,000.

View full job page →

2d ago

Lead AI Engineer

Vytwo Dallas, TX

Dallas, TX

2d ago

Full time Manager Artificial Intelligence Azure

Apply Now

Job Description

Vytwo is building production-ready AI solutions and is hiring a Lead AI Engineer to help design and deploy end-to-end data and AI pipelines on Azure Databricks and other Azure cloud services. This onsite role in Dallas, TX focuses on moving AI from experimentation to reliable production systems, including deployment, monitoring, and optimization across ML and Generative AI use cases.

What you will do

Design and build end-to-end data and AI pipelines using Azure Databricks.
Develop robust ETL/ELT workflows with Python (including PySpark) and SQL.
Implement CI/CD pipelines for Databricks deployments, including jobs, notebooks, and workflows.
Integrate Databricks with Azure services such as Data Lake, Blob Storage, Key Vault, Azure OpenAI, and Azure Functions.
Optimize Databricks jobs for performance, cost, and reliability.
Create reusable, modular code to support maintainable production systems.
Collaborate with data scientists and platform teams to move models from experimentation to production deployment.
Implement logging, monitoring, and error handling for production pipelines.
Develop and deploy ML and Generative AI models, including LLMs, embeddings, and RAG pipelines for NLP, computer vision, and predictive analytics.
Fine-tune LLMs using LoRA or QLoRA, and integrate with Azure OpenAI or Hugging Face models.
Build vector search and retrieval pipelines using FAISS or Azure Cognitive Search.
Support responsible AI practices, including bias detection and model governance.

What you bring

Hands-on experience with Azure Databricks, including jobs, workflows, and clusters (with Unity Catalog preferred).
Python skills with strong emphasis on PySpark rather than only pandas.
SQL experience, including complex joins, window functions, and analytical queries.
Working knowledge of Azure cloud components such as ADLS Gen2, ADF, Key Vault, and IAM concepts.
Experience with pipeline orchestration and deployment, including CI/CD and environment promotion.
Experience with Azure DevOps.
Strong understanding of the ML lifecycle and MLOps best practices.
Experience deploying models using MLflow or similar frameworks.

Technologies you will work with

Azure Databricks; Databricks workflows, jobs, and notebooks
Azure cloud infrastructure including Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions
Python, PySpark, SQL; ETL/ELT
CI/CD pipelines; Logging and Monitoring
ML and Generative AI; MLflow
LoRA, QLoRA, Hugging Face models
FAISS and Azure Cognitive Search; vector search and RAG pipelines
Unity Catalog; ADLS Gen2; ADF; IAM concepts; Azure DevOps
NLP, computer vision, predictive analytics

Strong advantage

Experience with ML and Generative AI workloads on Databricks.
Experience with RAG, embeddings, or inference pipelines.
Terraform / ARM / Bicep experience for infrastructure.
Databricks Asset Bundles experience.
Airflow or ADF orchestration experience.
Production monitoring and cost optimization experience.
Knowledge of LangChain or similar frameworks for AI application development.
Experience with Azure AI services such as Azure Machine Learning or Azure Cognitive Services.

View full job page →

4d ago

Sr Data Engineer, Data Analytics & Intelligence, NA

Vantage Data Centers Management Company LLC Denver, CO

Denver, CO

4d ago

$130k - $155k/yr Full time Senior Artificial Intelligence Azure Data Factory

Apply Now

Job Description

Build and scale governed Azure data foundations for enterprise reporting, operational intelligence, and AI-enabled consumption for Operations in North America.

Responsibilities

Design, build, and maintain reliable, scalable data pipelines using Python and PySpark on the Microsoft Azure data platform
Develop and operate batch and incremental pipelines using Azure Data Factory for orchestration and Azure Data Lake Storage Gen2 as the primary data store
Create and maintain curated lakehouse / gold-layer datasets and semantic-model inputs for governed operational insights and AI-enabled consumption
Independently implement SQL- and Spark-based transformations to produce curated datasets for enterprise reporting, analytics, and downstream use
Own assigned pipelines and datasets, including monitoring, troubleshooting, performance optimization, documentation, and production support
Work with Azure Synapse and Microsoft Fabric / Lakehouse patterns where applicable, plus related Azure analytics services
Prepare structured operational data for AI-enabled use cases by documenting business rules, source lineage, data reliability constraints, known quality limitations, and data dictionary definitions
Support source visibility and confidence context, including Data Reliability & Trust Indicator integration where applicable
Contribute to ontology, taxonomy, semantic model, and data dictionary alignment across operational domains (KPIs, incidents, work orders, and more)
Collaborate with business analysts, operations SMEs, data stewards, IT Global, and cross-functional stakeholders to translate requirements into working data solutions
Apply data governance, security, access-control, data classification, and engineering standards for compliant, maintainable, scalable solutions
Identify, document, and route data-quality issues to accountable owners to improve source correction rather than masking defects downstream
Participate in code reviews, technical discussions, sprint planning, and platform improvement initiatives
Proactively identify data quality issues, pipeline risks, platform dependencies, and improvement opportunities, and communicate them clearly
Develop and maintain PySpark notebooks and jobs to ingest, transform, validate, and curate data within the enterprise data platform
Build and modify Azure Data Factory pipelines for batch and incremental ingestion
Implement Spark transformations to write curated datasets to Azure Data Lake Storage Gen2 and/or Fabric Lakehouse patterns using established folder structures, naming conventions, and governance standards
Create and maintain SQL views, tables, lakehouse objects, and semantic-model inputs to support analytics, operational intelligence, and AI-enabled consumption patterns
Prepare datasets for Fabric Data Agent / AI agent use cases with documentation including business rules, joins, grain, quality limitations, source lineage, and operational definitions
Respond to pipeline failures, data validation issues, operational alerts, and data-quality escalations with root-cause analysis and remediation steps
Perform performance tuning for Spark jobs and SQL workloads (partitioning, filtering, incremental logic, query optimization, and resource-aware design)
Validate outputs with business partners, operations SMEs, and data stewards; address defects or discrepancies through documented correction paths
Support observability, logging, and auditability practices for pipelines and AI-consumable datasets where applicable
Commit code using Git, follow branching standards, support pull request reviews, and participate in CI/CD using GitHub, Azure DevOps, or similar tools
Update documentation for pipelines, datasets, data contracts, data dictionaries, business rules, and operational runbooks as changes are made
Execute assigned backlog items within sprint timelines; raise risks, dependencies, or blockers early
Additional duties as assigned by management

Requirements

Bachelor’s degree in Engineering, Computer Science, Data Analytics, or a related field, or equivalent experience
Minimum 5–8 years of experience in data engineering, analytics engineering, or a closely related technical data role
Proficiency in Python for data pipelines, automation, and data processing workflows, including PySpark-based transformations
Proficiency in SQL for querying, transformation, analytical processing, model validation, and data quality checks
Solid understanding of ETL/ELT pipelines, data transformation patterns, data integration concepts, incremental processing, and production support practices
Experience analyzing enterprise data sources to identify relationships, transformations, business rules, grain, ownership, and quality constraints
Experience building solutions on the Microsoft Azure platform, including exposure to Azure Data Factory, Azure Synapse, Azure Data Lake Storage Gen2, Microsoft Fabric / Lakehouse patterns, and related analytics services
Working knowledge of data modeling fundamentals, including fact/dimension tables and semantic models
Experience supporting governed data products, including metadata, lineage, issue documentation, access-control awareness, data-quality validation, and operational runbooks
Experience with source control and CI/CD workflows using GitHub or Azure DevOps
Strong communication and collaboration skills across IT Global, business SMEs, data governance partners, platform teams, and operations stakeholders
Experience in Agile environments and using collaboration or tracking tools such as Jira or similar tools
Travel required is expected to be up to 10% but may increase over time

Technologies

Python, PySpark, Spark
Microsoft Azure, Azure Data Factory, Azure Data Lake Storage Gen2, Azure Synapse
Microsoft Fabric, Fabric Lakehouse, Microsoft Fabric / Lakehouse patterns
SQL
Git, GitHub, Azure DevOps
Jira, CI/CD

Benefits

Medical, dental, and vision coverage
Life and AD&D
Short and long-term disability coverage
Paid time off
Employee assistance
401k participation with company match
Above market total compensation package
Comprehensive suite of health and welfare, retirement, and paid leave benefits

Salary Range

USD 130,000 - 155,000 per year
Range is based on Colorado market data and may vary in other locations
Compensation may depend on qualifications, skills, competencies, and experience and may fall outside the range shown

Additional Details

On-site role in Denver, CO (3 days on site required, 2 days flexible)
Eligible for company benefits including medical, dental, and vision coverage, life and AD&D, short and long-term disability coverage, paid time off, employee assistance, and 401k with company match

Physical Demands and Special Requirements

Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions
Occasionally required to stand, walk, sit; use hands to handle or feel objects; reach with hands and arms; climb stairs; balance; stoop or kneel; talk and hear
Occasionally required to lift and/or move up to 25 pounds

Desired Qualifications

Experience with distributed data processing frameworks, including Apache Spark
Experience preparing governed data products for AI-enabled use cases, including Microsoft Fabric Lakehouse, semantic models, Fabric/Data Agent patterns, ontology or taxonomy alignment, and explainable AI outputs
Familiarity with data observability, metadata management, lineage, data contracts, reliability indicators, and operational best practices in production environments
Familiarity with additional Azure services such as Azure Functions or Logic Apps in support of data workflows
Experience supporting data platform enhancement, refactoring, modernization, or reusable architecture initiatives
Experience working with structured and unstructured operational sources, such as enterprise applications, operational workflows, documents, dashboards, and knowledge assets
Experience working in a scaling or fast-paced organization where priorities evolve quickly and practical delivery discipline is required

View full job page →

Azure Data Lakehouse Jobs

Data Engineer

Data Engineer - Business Analytics

Lead AI Engineer

Sr Data Engineer, Data Analytics & Intelligence, NA

Get Azure Data Lakehouse Jobs alerts