Databricks Lakeflow

Databricks Lakeflow Jobs

4 jobs found

Today

Data Engineer

Pyx Health Inc Remote 🇺🇸 Remote

Remote

Today

Full time Python Apache Airflow Azure

Apply Now

Job Description

Pyx Health Inc is hiring a Healthcare Data Engineer to help build and maintain Azure-based data infrastructure and dependable data pipelines. In this remote role, you will work with Databricks, Airflow, Python, and SQL to support healthcare data workflows, with an emphasis on reliability, governance, and data quality for downstream reporting.

What you’ll do

Build and maintain data pipelines on Azure to ingest, transform, and clean healthcare data using Databricks, PySpark, and Delta Lake
Implement pipeline logic based on defined specifications, applying architectural guidance from senior engineers
Create reusable testing frameworks that help ensure pipeline reliability
Monitor pipelines for failures and performance issues, escalating complex problems when needed
Develop and maintain Airflow DAGs with appropriate error handling and retry logic
Support deployment and configuration of pipelines using Astronomer on Azure via Astro CLI
Reduce manual intervention by improving pipeline reliability over time
Implement data models in Delta Lake for efficient storage and retrieval, including merge/upsert patterns
Design and optimize Delta Lake tables for analytic workloads
Use Unity Catalog to support data governance, organization, and security
Apply Change Data Capture (CDC) patterns under the direction of senior team members
Write and optimize T-SQL and Databricks SQL scripts, stored procedures, and ETL support
Work across the Azure ecosystem including ADLS Gen2, Key Vaults, Logic Apps, and Azure DevOps
Follow established security and scalability standards when building data infrastructure
Ensure scripts and datasets are well-documented to support enterprise data governance
Design and continuously improve automated data quality monitoring, reconciliation, and alerting to support reliable downstream reporting
Troubleshoot and resolve pipeline failures by documenting root causes and resolutions
Collaborate with data scientists, analysts, and business stakeholders to understand reporting requirements and troubleshoot data issues
Participate in code reviews and incorporate feedback to improve code quality
Document pipelines, processes, and implementation decisions clearly and consistently
Stay current with data engineering technologies, especially in the healthcare space

What you’ll bring

2–4 years of experience as a Data Engineer or closely related data role
Hands-on experience with Azure cloud services such as ADLS, Databricks, or similar
Working knowledge of SQL for scripting and data modeling, including T-SQL, Spark, and Databricks SQL
Ability to contribute to technical projects with moderate oversight
Strong communication skills, including comfort asking questions and sharing updates with cross-functional partners
Familiarity with CI/CD, Git, and version control workflows
Solid problem-solving skills with a systematic approach to debugging
Healthcare data standards and regulations knowledge (such as HIPAA and HL7) is a plus
Bachelor’s degree in Computer Science, Information Technology, Engineering, or related field (or equivalent work experience)

Core must-haves

Databricks SQL (Mid): Delta Lake fundamentals, basic merge/upsert patterns, familiarity with CDC concepts
Python (Mid): Pipeline logic, data transformation, SQL scripting; some PySpark/Spark DataFrame experience
Databricks Spark Notebooks (Mid): Notebook-based development, basic cluster usage, Delta table operations
Airflow Python Development (Mid): Ability to write and maintain DAGs, including error handling and retry patterns
Airflow Astro Configuration (Foundational): Familiarity with Astronomer or willingness to learn; basic Astro CLI usage
Azure Ecosystem (Foundational): Working knowledge of ADLS Gen2, Key Vaults, and Azure DevOps
T-SQL (Foundational–Mid): Basic stored procedures, SQL Server querying, and data manipulation

Nice to have

Azure Data Factory (Foundational): Basic familiarity with pipeline authoring and triggers
Git + Azure DevOps CI/CD (Foundational): Experience with branching, pull requests, peer review processes, version control best practices, and deploying production data pipelines using CI/CD workflows

Location: Remote (remote).

View full job page →

13d ago

Machine Learning Engineer

Interwell Health Remote 🌍 Remote

Remote

13d ago

Full time Ai Ml Ai Workflows Analytics

Apply Now

Job Description

What Interwell Health offers

This remote role focuses on meaningful, real world impact in healthcare. You will own the full lifecycle of machine learning initiatives, from data preparation and model building to deployment, monitoring, and ongoing optimization. Work alongside engineers, product managers, and clinicians to develop new ML products and enhance existing systems, with opportunities to tackle traditional ML and large language model based capabilities in a collaborative, cross functional environment.

Responsibilities

Develop and deliver end to end machine learning solutions, defining technical requirements, architecting scalable systems, and implementing monitoring, logging, and maintenance workflows.
Collaborate closely with engineers, product managers, clinicians, and cross functional partners to build new ML products and enhance existing systems.
Lead the design and implementation of MLOps frameworks, including pipeline development, CI/CD integration, drift detection, retraining workflows, and rollback strategies.
Monitor model performance in production, identify issues, propose remediation steps, and ensure strong test coverage and system reliability.
Apply contemporary software engineering practices to deliver scalable, secure, and maintainable AI/ML systems.
Develop and tailor API integrations to enable seamless connectivity between cloud based systems and ML services.
Participate in architectural discussions to ensure ML platforms meet compliance, performance, and scalability standards.

Requirements

Bachelor's degree in Computer Science, Data Analytics, Software/Computer Engineering, Computational Statistics, Mathematics, or a related discipline.
Minimum 2 years of professional ML experience, with 3+ years of end-to-end ML development in production (data prep, feature engineering, modeling, calibration, deployment, monitoring, maintenance).
3+ years of MLOps experience building production pipelines (CI/CD, model registry, feature store), implementing monitoring and drift detection, and automating retraining.
3+ years of Python for production ML (testing, packaging, type hints, linting) and SQL for analytical and production workloads; Scala is a plus.
2+ years working with distributed compute and cloud ML environments (for example Spark/Databricks on Azure, AWS, or GCP) and modern data ecosystems (data lakes, DBMS).
Strong debugging and optimization skills across data and ML workflows.
Track record of ownership and problem solving—driving measurable impact and quality under ambiguity and evolving requirements.
Ability to communicate technical decisions clearly and contribute to documentation and design discussions.
Demonstrated system design and architecture skills for scalable, high performance ML services and batch plus streaming workflows; familiarity with API design and service integration patterns.
Proven understanding of tradeoffs in latency, cost, performance, and compliance.

Technologies

Python
SQL
Scala
Spark
Databricks
Azure
AWS
GCP

Preferred

1+ years of Databricks experience plus some experience in infrastructure or networking
1+ years implementing LLM based solutions in production, including prompt/response design, evaluation frameworks, guardrails and safety, latency and cost optimization
1+ years designing compliant ML platforms (HIPAA, SOC 2) and working with PHI/PII governance, access controls, and auditability

View full job page →

13d ago

Data and Analytics Engineer

Digital Strategy LLC Remote 🇺🇸 Remote

Remote

13d ago

$100k - $130k/yr Full time Analytics Big Data Bigdata

Apply Now

Job Description

Data and Analytics Engineer at Digital Strategy LLC supports a U.S. Department of Energy client by designing, delivering, and operating end-to-end data products on a Databricks based cloud platform. The role blends data engineering, analytics-product development, governance, and AI-assisted delivery in a fully remote setting. Compensation ranges from USD 100,000 – 130,000 per year.

Responsibilities

Collaborate with architects, analysts, BI developers, team leads, DOE staff, and Digital Strategy leadership to ensure each data product aligns with mission priorities and the broader platform architecture.
Convert ambiguous business questions and operational needs into well scoped technical designs, delivery plans, and product increments.
Lead assigned workstreams from intake through delivery by clarifying outcomes, sequencing activities, tracking progress, and managing dependencies, risks, and stakeholder expectations.
Assess incoming requests and surface trade-offs among urgency, scope, quality, architecture, and team capacity for client and leadership visibility.
Communicate recommendations, constraints, and technical trade-offs clearly to stakeholders with diverse technical backgrounds while maintaining professionalism when priorities shift.
Apply and continually improve the team’s architecture and delivery standards, highlighting exceptions and proposing refinements.
Migrate legacy SQL Server and SSIS workloads to Databricks native pipelines and transformation patterns.
Design and operate ingestion pipelines for structured, semi-structured, and unstructured sources, including relational data, JSON, XML, spreadsheets, and documents.
Model Bronze, Silver, and Gold data layers with clear contracts, quality gates, lineage, and promotion logic between tiers.
Translate transactional source data into BI ready facts, dimensions, semantic structures, and curated data products.
Write and optimize production grade SQL and Python for reusable transformations, utilities, automation, and operational tasks.
Design consumption-layer products appropriate to the use case, such as Power BI or Databricks AI/BI dashboards, lightweight Databricks Apps, APIs, Genie Spaces, and governed NLQ experiences.
Embed data quality checks, validation rules, automated tests, documentation, and operational telemetry into every data product.
Leverage AI and emerging platform capabilities selectively to speed delivery, improve usefulness, supportability, or stakeholder access to insights.
Participate as a hands-on technical contributor for assigned products, making implementation decisions within architectural guardrails and escalating material trade-offs as needed.
Mentor analysts, BI developers, engineers, and citizen developers in data-as-code, Git-based collaboration, modular development, automated testing, code review, and CI/CD.
Conduct constructive design and code reviews that explain reasoning and help teammates develop independent judgment.
Create reusable templates, reference implementations, utilities, and practical documentation to enable less-experienced contributors to deliver safely and consistently.
Promote disciplined engineering without over-engineering urgent work; choose controls and patterns appropriate to risk, lifespan, and audience.
Identify patterns across products that can become reusable Digital Strategy capabilities, accelerators, demonstrations, or proposal assets.
Manage metadata, data lineage, ownership, and catalog organization in Unity Catalog.
Implement least-privilege access controls at the row, column, and object level for built and maintained products.
Coordinate with the DOE Databricks platform team on environment configuration, CI/CD, and promotion across development, test, and production workspaces.
Instrument data products with logging, alerts, event information, system-table reporting, and data quality monitoring to surface issues proactively.
Apply naming conventions, object taxonomy, documentation standards, and release criteria consistently across products and environments.
Protect sensitive information and comply with applicable federal, client, and platform security requirements.
Guide data products through design, development, testing, release, and post-production operation, ensuring required evidence and documentation are complete before promotion.
Maintain delivered products for reliability, performance, usability, and cost-effectiveness by tracing issues to root causes rather than patching symptoms.
Proactively surface gaps, risks, and improvement opportunities and collaborate with teammates and client staff to drive resolution.
Improve standard operating procedures and delivery practices when experience reveals gaps or friction.
Balance immediate client value with maintainability to avoid creating fragile production dependencies.

Requirements

7+ years of professional experience in data engineering, analytics engineering, or related fields, including at least 3 years owning production data products on a cloud analytics platform.
Bachelor's degree in Data Science, Data Analytics, Computer Science, Engineering, or a related technical field, or equivalent professional experience.
Proven track record of owning data products from design through release, support, and enhancement.
Experience leading technical workstreams, articulating trade-offs, reviewing peers, and mentoring without formal management authority.
Advanced production-grade SQL across analytical workloads, including complex transformations and dimensional modeling.
Production Python experience for data engineering, automation, utilities, APIs, or lightweight data applications.
Experience building and operating production-grade ingestion, transformation, and orchestration workflows on an enterprise cloud data platform.
Demonstrated implementation of analytical models and translation of transactional schemas into BI-ready facts, dimensions, and curated datasets.
Hands-on experience with source control, automated testing, code review, CI/CD, and promoting data-platform assets across environments.
Practical familiarity with data quality controls, observability, metadata, lineage, and security at row, column, or object levels.
Ability to build or support analytics consumption products such as dashboards, semantic models, APIs, interactive tools, or natural-language data experiences.
Capability to quickly learn a client’s mission and business context to inform technical decisions beyond literal requirements.
Experience translating complex concepts for diverse client stakeholders and co-developing solutions rather than awaiting detailed instructions.
Strong presentation of recommendations and trade-offs with clear reasoning, while staying open to challenge and adjusting direction when warranted by evidence.
Judgment in stakeholder interactions and professional composure when priorities shift, deadlines tighten, or feedback is challenging.
Clear written and verbal communication, including concise status updates, technical documentation, and decision records.

Technologies

Databricks, Unity Catalog, Lakeflow Spark Declarative Pipelines, Lakeflow Jobs
Delta Lake, Spark SQL, PySpark, SQL Server, SSIS
Power BI, Genie Spaces, Databricks Apps, Data Quality Monitoring
Declarative Automation Bundles, AWS, IAM, S3, VPC
Microsoft Fabric, Snowflake, Git, Python, SQL, dbt
Databricks AI/BI dashboards

Benefits

401(k)
401(k) matching
Dental insurance
Health insurance
Life insurance
Paid time off
Vision insurance

What success looks like during the first year

Within the initial months, achieve productive work within the DOE domain, the Databricks environment, and Digital Strategy delivery processes.
Take ownership of at least one data product from intake and design through production release, operation, and enhancement.
Modernize or significantly improve a legacy pipeline, data model, reporting process, or analytics product.
Introduce or reinforce automated quality checks, testing, observability, security, and documentation in assigned products.
Elevate team capability by guiding team members and citizen developers to adopt source control, modular development, code review, automated testing, or CI/CD practices.
Turn a recurring data request or manual process into a governed, reusable solution when value is clear.
Contribute reusable patterns, utilities, templates, accelerators, or lessons that benefit delivery beyond a single product or workstream.

About Digital Strategy LLC

Digital Strategy LLC is a federal IT and management consulting firm focused on data modernization, analytics, automation, and AI enabled delivery for U.S. government clients. The company emphasizes combining DOE and federal domain expertise with strong technical execution and seeks engineers who improve products and teams while delivering reliable, reusable capabilities.

Application Question(s)

In one to two sentences, identify a production data product you personally owned and your contribution.
In one to two sentences, describe how you balanced an urgent data request with longer-term engineering work.
In one to two sentences, identify how you helped a teammate adopt data-as-code, testing, or CI/CD practices.
In one to two sentences, identify a production data issue you traced to its root cause and prevented from recurring.
In one to two sentences, identify your role in building a medallion or comparable layered data solution and how data progressed from raw to curated layers.
Optionally, in one to two sentences, identify one way you have used AI to accelerate data or analytics engineering while maintaining validation and quality controls.
Can you consistently maintain availability from 9:00 a.m. to 5:00 p.m. Eastern Time? Briefly describe your expected working hours in Eastern Time.

Experience

Databricks production data product development and ownership: 1 year (Preferred)
Data engineering or analytics engineering: 7 years (Required)
Cloud analytics platform production data product ownership: 3 years (Required)

Language

English (Required)

View full job page →

25d ago

Sr Databricks Data Engineer

Deloitte Pittsburgh, PA

Pittsburgh, PA

25d ago

$116k - $229k/yr Full time Apache Airflow Automation Big Data

Apply Now

Job Description

Senior Databricks Data Engineer within Deloitte AI and Data practice, accountable for designing, building, and optimizing cloud-based data engineering solutions on Databricks to modernize data platforms, enable analytics and AI, and drive measurable business outcomes. This role is onsite in Pittsburgh, PA, with a salary range of USD 116,200 to 229,100 per year.

Responsibilities

Establish and advocate leading practices for data architecture, integration, and modeling, documenting standards and promoting adherence across teams.
Own the end-to-end design, development, and maintenance of robust data pipelines and architectures to support enterprise-scale data needs.
Lead initiatives to enhance data quality, streamline operations, and scale data processes.
Assess, pilot, and integrate new big data and analytics technologies to keep the organization at the cutting edge; provide leadership and mentorship to data engineers and architects to support growth and project success.
Advise on and implement governance, security, and compliance strategies tailored to cloud-based data ecosystems.
Translate technical concepts and business value for stakeholders across the organization, including executives, business leads, and technology teams.
Guide the adoption of CI/CD practices using tools such as Azure DevOps, AWS CodePipeline, Jenkins, TFS, and PowerShell to streamline deployments and operations.

Requirements

Bachelor's degree in Computer Science, Engineering, or a related field
More than five years of hands-on data engineering experience focusing on Databricks across AWS, Azure, or Google Cloud Platform (GCP)
Proficiency with Lakehouse architectures, Apache Spark, Delta Lake, cloud-native databases and storage solutions, and distributed compute platforms
Experience with data warehousing and 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
At least one year leading complex, cross-functional data projects and technical teams, including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources
Ability to travel approximately 50 percent, depending on client engagements
Limited immigration sponsorship may be available
Master's degree in Computer Science, Engineering, or a related field
Experience with one or more cloud ecosystems (AWS, Azure, GCP) and associated big data services
Experience tuning and optimizing performance in Databricks and Apache Spark environments
Experience with Databricks Lakeflow
Experience with artificial intelligence and machine learning solutions

Technologies

Databricks
Amazon Web Services (AWS)
Microsoft Azure
Google Cloud Platform (GCP)
Apache Spark
Delta Lake
Unity Catalog
Delta Live Tables
Autoloader
Structured Streaming
Databricks Workflows
Apache Airflow
Azure DevOps
AWS CodePipeline
Jenkins
TFS
PowerShell
PySpark
Databricks Lakeflow

Benefits

Discretionary annual incentive program
Benefits package aligned with Core Talent Model

Qualifications Required

Bachelor's degree in Computer Science, Engineering, or a related field
Five or more years of hands-on data engineering experience with a focus on Databricks on AWS, Azure, or GCP
Experience with Lakehouse architecture, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
Experience with data warehousing, 3NF, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
At least one year leading complex, cross-functional data projects and technical teams, including Delta Live Tables, Autoloader, Structured Streaming, Databricks Workflows, Apache Airflow, Unity Catalog, automated CI/CD pipelines, and performance optimization of data pipelines, code, and compute resources
Ability to travel approximately 50 percent
Limited immigration sponsorship may be available

Preferred

Master's degree in Computer Science, Engineering, or a related field
Experience with one or more cloud ecosystems (AWS, Azure, GCP) and their big data services
Experience tuning and optimizing performance in Databricks and Apache Spark environments
Experience with Databricks Lakeflow
Experience with artificial intelligence and machine learning solutions

View full job page →

Databricks Lakeflow Jobs

Data Engineer

Machine Learning Engineer

Data and Analytics Engineer

Sr Databricks Data Engineer

Get Databricks Lakeflow Jobs alerts