Cloud Data Migration Jobs

1 job found

Today

Data Engineer

Builders Atlanta, GA

Atlanta, GA

Today

Full time Azure Data Engineer Azure Data Lake Big Data

Apply Now

Job Description

Build, govern, and optimize Azure Databricks lakehouse pipelines supporting property and casualty insurance data migrations.

Responsibilities

Design and build pipelines for the phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into the Azure Databricks lakehouse
Implement historical loads, incremental/CDC (Change Data Capture) loads, and reconciliation
Create bronze and silver layer transformations in PySpark and Spark SQL following the medallion architecture on Delta Lake
Support data modeling for property and casualty insurance, including policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries
Integrate third-party data sources such as geospatial and weather data
Schedule and run production pipelines using Databricks Workflows (Lakeflow Jobs) or Azure Data Factory, including dependencies, parameters, retries, and alerting
Investigate and resolve pipeline failures
Apply Unity Catalog controls defined by the Data Engineering Lead and Data Governance lead, including permissions, PII (Personally Identifiable Information) masking, tags, and lineage
Implement data quality checks, error handling, and monitoring
Tune pipeline performance and cost using partitioning, liquid clustering, and right-sized job compute
Use Git version control (GitHub or Bitbucket) and CI/CD with Databricks Asset Bundles to version, review, and deploy code across development, test, and production
Write documentation and runbooks, participate in code reviews, and gradually take ownership of contractor-built pipelines
Perform other duties as assigned

Requirements

Associate degree or equivalent combination of education and experience
Databricks engineering certification
3+ years in data engineering, including 1+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake)
Experience with property and casualty insurance data, ideally Guidewire data, and experience migrating from legacy source systems
Knowledge of Python/PySpark and SQL (Structured Query Language)
Knowledge of Unity Catalog: permissions, row/column-level security, and lineage
Experience with incremental loading patterns such as CDC, MERGE, SCD (Slowly Changing Dimension), and Auto Loader
Knowledge of data modeling, including dimensional modeling
Experience scheduling and monitoring pipelines with Databricks Workflows and/or Azure Data Factory
Experience with Git-based version control (GitHub or Bitbucket), including branching, pull request reviews, and Databricks Git folders
Experience with Azure data services including ADLS (Azure Data Lake Storage) Gen2, Azure Data Factory, and Key Vault
Knowledge of CI/CD basics (Databricks Asset Bundles, Azure DevOps, GitHub Actions, or Bitbucket Pipelines)
Experience with streaming (Structured Streaming, Event Hubs)
Experience with API and integration tools such as MuleSoft (added advantage)

Technologies

Azure Databricks
Delta Lake
Unity Catalog
Lakeflow
Databricks SQL
PySpark
Spark SQL
Databricks Workflows
Azure Data Factory
Git, GitHub, Bitbucket
Databricks Asset Bundles
Python
SQL (Structured Query Language)
ADLS (Azure Data Lake Storage) Gen2
Azure DevOps
GitHub Actions
Bitbucket Pipelines
Key Vault
Structured Streaming
Event Hubs
Auto Loader
MERGE
SCD (Slowly Changing Dimension)
CDC (Change Data Capture)
MuleSoft
API (Application Programming Interface)

Benefits

Competitive Salary
Bonus Structure
Profit Sharing
Medical, Dental, Vision Insurance
Employer Paid Short Term Disability
Employer Paid Long Term Disability
Employer Paid Life Insurance
Voluntary Life Insurance
401K with Company Match
PTO

Location and Experience

Atlanta, GA (onsite)
Minimum experience: 3 years
Education: Associate degree

Equal Opportunity Employer

Builders welcomes applicants from all walks of life and does not discriminate based on protected status.

View full job page →

Get Cloud Data Migration Jobs alerts

New Cloud Data Migration Jobs, daily.