Databricks Delta

Databricks Delta Jobs

4 jobs found

Today

Data Engineer, Finance

SHIELD AI San Diego, CA

San Diego, CA

Today

$97k - $140k/yr Full time Ai/ml Data Engineering Artificial Intelligence Big Data

Apply Now

Job Description

Shield AI is seeking a Data Engineer, Finance in San Diego, CA (onsite) to build a reliable, governed data foundation that supports Finance planning, decision support, analytics, and AI-enabled Finance workflows. You will help create reusable Databricks integrations, strengthen semantic-layer infrastructure, and operate production-grade data services that keep teams moving with confidence.

This role blends production data engineering with platform-minded optimization. You will design for observability and resilient operations, implement data-quality controls, and work on the services and integrations behind Databricks Genie and other AI-enabled Finance workflows. Shield AI also supports faster delivery using AI coding assistants, with a strong expectation to validate correctness, security, and performance.

What you’ll do

Build and operate scalable pipelines and integrations connecting enterprise data to Finance products.
Develop semantic-layer infrastructure and governed data access for analytics, planning, and AI applications.
Implement end-to-end pipeline observability, data-quality controls, and resilient operations.
Engineer services and integrations that support Databricks Genie and other AI-enabled Finance workflows.
Use AI coding assistants to accelerate development, while independently validating correctness, security, and performance.
Apply FinOps practices to manage platform spend and optimize cost without compromising service quality.
Partner with analytics engineers and enterprise teams on architecture, data contracts, and sustainable ownership.

What you bring

Hands-on production data engineering experience with SQL and Python or PySpark.
Experience with cloud data platforms, orchestration, data modeling, and governed serving patterns.
Ability to design systems with observability, recoverability, security, and maintainability.
Proficiency with AI coding assistants and independent validation of generated work.
Practical experience enabling AI capabilities on enterprise data, ideally in Databricks; adjacent experience with Snowflake Cortex or comparable platforms is welcome.
Strong cross-functional collaboration and clear production ownership.

Technologies you’ll work with

Databricks
SQL
Python
PySpark
Snowflake Cortex
Databricks Genie
Delta Lake

Compensation and benefits

$97,000 - $140,000 USD per year plus bonus, benefits, and equity.

Pay within the listed range plus bonus, benefits, and equity.
Pay within the listed range above plus a temporary benefits package (applicable after 60 days of employment).

Preferred qualifications

Experience with Databricks, Delta Lake, cloud infrastructure, and deployment automation.
Experience developing AI application backends, model integrations, or LLM-based workflows.
Experience with FinOps, financial data, or enterprise-system integrations.

View full job page →

1d ago

Data Engineering Lead

Builders Atlanta, GA

Atlanta, GA

1d ago

Full time Manager Azure Azure Data Engineer

Apply Now

Job Description

Builders is committed to reliability, fairness, and customer-first service, earning an A rating by AM Best and recognition through Top Workplaces “Culture Excellence” honors. This role supports an insurance business built on strong partnerships and mutual respect, with an onsite opportunity in Atlanta, GA to lead a platform that helps teams modernize and govern data for property and casualty insurance.

As the technical owner of the Azure Databricks lakehouse platform, you will drive architecture, modernization, governance alignment, and engineering leadership to deliver a scalable, secure, reliable, high-performing foundation for property and casualty data.

What you’ll do

Design the Databricks lakehouse architecture using a bronze/silver/gold approach on Delta Lake for policy, rating, claims, billing, reinsurance, and customer data.
Plan and lead phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into Azure Databricks, including historical loads, incremental/CDC loads, reconciliation, and decommissioning support.
Model core property and casualty insurance entities: accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries.
Provide data for Actuarial and Finance use cases including written/earned/unearned premium, loss triangles, IBNR (incurred but not reported) inputs, loss ratios, and Schedule P and NAIC statutory reporting.
Integrate third-party sources such as geospatial and weather data.
Define and operationalize Unity Catalog design: metastore, catalogs, schemas, development/test/production environments, naming conventions, permissions, row/column-level security, and lineage.
Manage pipeline orchestration and operations in Databricks, including Databricks Workflows (Lakeflow Jobs), scheduling, dependencies, retries, alerting and SLA monitoring, and production failure handling with runbooks.
Own performance and cost work: Spark tuning, partitioning, liquid clustering, Photon, and cluster/SQL warehouse sizing.
Partner with IT and a Platform Engineer on infrastructure for storage, networking, security, identity, IaC, and CI/CD.
Set standards for pipelines, testing, code review, and documentation, and lead mentoring and review for Data Engineers and junior contractors.
Collaborate with the data governance lead to implement access controls, PII masking, lineage, and data quality rules.
Perform other duties as assigned.

What you bring

Associate degree or an equivalent combination of education and experience.
10+ years of data engineering experience, including 6+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake), plus experience leading or mentoring engineering teams.
Databricks Data Engineer Professional certification; Azure data or solutions architect certification is preferred.
Strong knowledge of Unity Catalog: governance model, permissions, row/column-level security, lineage, and system tables.
Experience leading large data migrations from core insurance and legacy systems into a lakehouse or data warehouse, with property and casualty domain knowledge and familiarity with the policy and claims lifecycle and core P&C KPIs.
Knowledge of data modeling including dimensional modeling, data vault, SCD patterns, and insurance data models.
Expert technical skills across Spark internals and performance tuning, streaming (structured streaming, Auto Loader), CDC patterns, MERGE, and Lakeflow/Delta Live Tables.
Expert-level SQL and Python, with software engineering practices such as testing, code review, and modular design.
Knowledge of data infrastructure: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID, VNets, and Private Link.
Experience scheduling and monitoring production pipelines with Databricks Workflows and/or Azure Data Factory, including dependencies, parameterization, and monitoring with Databricks system tables.
Experience with Git-based version control (GitHub or Bitbucket), including branching strategies, pull request reviews, and Databricks Git folders.
Infrastructure-as-code and CI/CD for data using Terraform, Databricks Asset Bundles, Azure DevOps, GitHub Actions, or Bitbucket Pipelines.
Experience with Guidewire data (Guidewire DataHub/InfoCenter, Cloud Data Access) is preferred, including supporting actuarial reserving and pricing; API and integration tools such as MuleSoft are preferred.
Knowledge of insurance data standards such as ACORD is preferred.

Benefits at Builders

Competitive Salary
Bonus Structure
Profit Sharing
Medical, Dental, Vision Insurance
Employer Paid Short Term Disability
Employer Paid Long Term Disability
Employer Paid Life Insurance
Voluntary Life Insurance
401K with Company Match
PTO

Equal Opportunity

Builders Insurance Group is an equal opportunity employer. Applicants are welcomed from all walks of life, and the company does not discriminate based on any protected status. This is part of building a diverse and inclusive workplace.

View full job page →

1d ago

Data Engineer

Builders Atlanta, GA

Atlanta, GA

1d ago

Full time Azure Data Engineer Azure Data Lake Big Data

Apply Now

Job Description

Build, govern, and optimize Azure Databricks lakehouse pipelines supporting property and casualty insurance data migrations.

Responsibilities

Design and build pipelines for the phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into the Azure Databricks lakehouse
Implement historical loads, incremental/CDC (Change Data Capture) loads, and reconciliation
Create bronze and silver layer transformations in PySpark and Spark SQL following the medallion architecture on Delta Lake
Support data modeling for property and casualty insurance, including policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries
Integrate third-party data sources such as geospatial and weather data
Schedule and run production pipelines using Databricks Workflows (Lakeflow Jobs) or Azure Data Factory, including dependencies, parameters, retries, and alerting
Investigate and resolve pipeline failures
Apply Unity Catalog controls defined by the Data Engineering Lead and Data Governance lead, including permissions, PII (Personally Identifiable Information) masking, tags, and lineage
Implement data quality checks, error handling, and monitoring
Tune pipeline performance and cost using partitioning, liquid clustering, and right-sized job compute
Use Git version control (GitHub or Bitbucket) and CI/CD with Databricks Asset Bundles to version, review, and deploy code across development, test, and production
Write documentation and runbooks, participate in code reviews, and gradually take ownership of contractor-built pipelines
Perform other duties as assigned

Requirements

Associate degree or equivalent combination of education and experience
Databricks engineering certification
3+ years in data engineering, including 1+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake)
Experience with property and casualty insurance data, ideally Guidewire data, and experience migrating from legacy source systems
Knowledge of Python/PySpark and SQL (Structured Query Language)
Knowledge of Unity Catalog: permissions, row/column-level security, and lineage
Experience with incremental loading patterns such as CDC, MERGE, SCD (Slowly Changing Dimension), and Auto Loader
Knowledge of data modeling, including dimensional modeling
Experience scheduling and monitoring pipelines with Databricks Workflows and/or Azure Data Factory
Experience with Git-based version control (GitHub or Bitbucket), including branching, pull request reviews, and Databricks Git folders
Experience with Azure data services including ADLS (Azure Data Lake Storage) Gen2, Azure Data Factory, and Key Vault
Knowledge of CI/CD basics (Databricks Asset Bundles, Azure DevOps, GitHub Actions, or Bitbucket Pipelines)
Experience with streaming (Structured Streaming, Event Hubs)
Experience with API and integration tools such as MuleSoft (added advantage)

Technologies

Azure Databricks
Delta Lake
Unity Catalog
Lakeflow
Databricks SQL
PySpark
Spark SQL
Databricks Workflows
Azure Data Factory
Git, GitHub, Bitbucket
Databricks Asset Bundles
Python
SQL (Structured Query Language)
ADLS (Azure Data Lake Storage) Gen2
Azure DevOps
GitHub Actions
Bitbucket Pipelines
Key Vault
Structured Streaming
Event Hubs
Auto Loader
MERGE
SCD (Slowly Changing Dimension)
CDC (Change Data Capture)
MuleSoft
API (Application Programming Interface)

Benefits

Competitive Salary
Bonus Structure
Profit Sharing
Medical, Dental, Vision Insurance
Employer Paid Short Term Disability
Employer Paid Long Term Disability
Employer Paid Life Insurance
Voluntary Life Insurance
401K with Company Match
PTO

Location and Experience

Atlanta, GA (onsite)
Minimum experience: 3 years
Education: Associate degree

Equal Opportunity Employer

Builders welcomes applicants from all walks of life and does not discriminate based on protected status.

View full job page →

3d ago

Databricks Data Engineer, Lead

Booz Allen Hamilton Bethesda, MD

Bethesda, MD

3d ago

$78k - $176k/yr Full time Manager Aws Data Engineer Big Data

Apply Now

Job Description

The Lead Databricks Data Engineer role at Booz Allen Hamilton is a hands-on position focused on designing, building, and maintaining data pipelines in Databricks. You will provide technical leadership, mentorship, and oversight of scalable data platform work across Databricks and AWS, with emphasis on ETL execution, data modeling, security, and performance.

Key Responsibilities

Design, build, and maintain robust data pipelines within Databricks, contributing independently while providing technical leadership and mentorship to the team
Create and optimize end-to-end data flows and manage extract, transform, load (ETL) processes
Design data models and ensure alignment with data architecture needs
Ensure data security and compliance across data platforms and pipelines
Collaborate with cross-functional teams to support delivery and platform outcomes
Continuously improve performance and scalability of data processing workloads
Mentor a multi-disciplinary team of analysts, data engineers, developers, and data consumers in a fast-paced, agile environment
Oversee the assessment, design, building, and maintenance of scalable platforms for clients

Required Qualifications

8+ years of experience designing, developing, deploying, and managing ETL processes and data pipelines, including comprehensive logging and robust error handling to troubleshoot and resolve issues
3+ years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Databricks tools such as Delta Lake, Databricks Delta, and Unity Catalog
3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations
Experience with data modeling, including when to use dimensional versus relation data models within a medallion architecture (Bronze, Silver, or Gold)
Experience using AWS services including AWS Glue, Amazon S3, and Amazon RDS
Knowledge of performance optimization techniques for ETL jobs such as partitioning, bucketing, caching, and troubleshooting to improve query performance
Ability to design and implement data models from defining schemas through optimizing data structures that support efficient queries and analytics
Public Trust
Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology

Technology Stack

Databricks
AWS
Databricks notebooks
Clusters
Job orchestration tools
Delta Lake
Databricks Delta
Unity Catalog
Python
PySpark
AWS Glue
Amazon S3
Amazon RDS

Nice If You Have

Experience with AWS cloud-native data services such as AWS Lambda or CloudWatch that support serverless architectures for data processing
Experience with tools such as AWS Glue Data Catalog or Collibra for managing metadata, data lineage, and cataloging data assets
Knowledge of data governance practices, data lineage, and compliance requirements, including GDPR or CCPA, supporting data quality, privacy, and security
Ability to explain complex technical concepts to a non-technical audience, including tradeoffs in architectural design

Vetting and Compliance

Selected applicants will be subject to a government investigation and may need to meet eligibility requirements of the U.S. government client
Public Trust determination is required

Compensation

The projected compensation range for this position is $77,600.00 to $176,000.00 (annualized USD).

Benefits

Health, life, disability, financial, and retirement benefits
Paid leave
Professional development
Tuition assistance
Work-life programs
Dependent care
Recognition awards program acknowledging employees for exceptional performance and superior demonstration of Booz Allen values
Full-time and part-time employees working at least 20 hours per week on a regular basis are eligible to participate in Booz Allen’s benefit programs

Work Location and Model

Location: Bethesda, MD (onsite). Work will primarily be performed full-time at a customer facility, with collaboration required as part of the role.

Identity Verification and Interview Policies

Complete an identity verification process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity fraud
You are expected to be on camera during interviews and assessments
The company reserves the right to take your picture to verify identity and prevent fraud
The use of AI or other tools to assist with responses during interviews is prohibited unless permission is explicitly provided

View full job page →

Databricks Delta Jobs

Data Engineer, Finance

Data Engineering Lead

Data Engineer

Databricks Data Engineer, Lead

Get Databricks Delta Jobs alerts