Data Engineer

Build, govern, and optimize Azure Databricks lakehouse pipelines supporting property and casualty insurance data migrations.

Responsibilities

• Design and build pipelines for the phased migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into the Azure Databricks lakehouse
• Implement historical loads, incremental/CDC (Change Data Capture) loads, and reconciliation
• Create bronze and silver layer transformations in PySpark and Spark SQL following the medallion architecture on Delta Lake
• Support data modeling for property and casualty insurance, including policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries
• Integrate third-party data sources such as geospatial and weather data
• Schedule and run production pipelines using Databricks Workflows (Lakeflow Jobs) or Azure Data Factory, including dependencies, parameters, retries, and alerting
• Investigate and resolve pipeline failures
• Apply Unity Catalog controls defined by the Data Engineering Lead and Data Governance lead, including permissions, PII (Personally Identifiable Information) masking, tags, and lineage
• Implement data quality checks, error handling, and monitoring
• Tune pipeline performance and cost using partitioning, liquid clustering, and right-sized job compute
• Use Git version control (GitHub or Bitbucket) and CI/CD with Databricks Asset Bundles to version, review, and deploy code across development, test, and production
• Write documentation and runbooks, participate in code reviews, and gradually take ownership of contractor-built pipelines
• Perform other duties as assigned

Requirements

• Associate degree or equivalent combination of education and experience
• Databricks engineering certification
• 3+ years in data engineering, including 1+ years hands-on with Databricks (PySpark, Spark SQL, Delta Lake)
• Experience with property and casualty insurance data, ideally Guidewire data, and experience migrating from legacy source systems
• Knowledge of Python/PySpark and SQL (Structured Query Language)
• Knowledge of Unity Catalog: permissions, row/column-level security, and lineage
• Experience with incremental loading patterns such as CDC, MERGE, SCD (Slowly Changing Dimension), and Auto Loader
• Knowledge of data modeling, including dimensional modeling
• Experience scheduling and monitoring pipelines with Databricks Workflows and/or Azure Data Factory
• Experience with Git-based version control (GitHub or Bitbucket), including branching, pull request reviews, and Databricks Git folders
• Experience with Azure data services including ADLS (Azure Data Lake Storage) Gen2, Azure Data Factory, and Key Vault
• Knowledge of CI/CD basics (Databricks Asset Bundles, Azure DevOps, GitHub Actions, or Bitbucket Pipelines)
• Experience with streaming (Structured Streaming, Event Hubs)
• Experience with API and integration tools such as MuleSoft (added advantage)

Technologies

• Azure Databricks
• Delta Lake
• Unity Catalog
• Lakeflow
• Databricks SQL
• PySpark
• Spark SQL
• Databricks Workflows
• Azure Data Factory
• Git, GitHub, Bitbucket
• Databricks Asset Bundles
• Python
• SQL (Structured Query Language)
• ADLS (Azure Data Lake Storage) Gen2
• Azure DevOps
• GitHub Actions
• Bitbucket Pipelines
• Key Vault
• Structured Streaming
• Event Hubs
• Auto Loader
• MERGE
• SCD (Slowly Changing Dimension)
• CDC (Change Data Capture)
• MuleSoft
• API (Application Programming Interface)

Benefits

• Competitive Salary
• Bonus Structure
• Profit Sharing
• Medical, Dental, Vision Insurance
• Employer Paid Short Term Disability
• Employer Paid Long Term Disability
• Employer Paid Life Insurance
• Voluntary Life Insurance
• 401K with Company Match
• PTO

Location and Experience

• Atlanta, GA (onsite)
• Minimum experience: 3 years
• Education: Associate degree

Equal Opportunity Employer

• Builders welcomes applicants from all walks of life and does not discriminate based on protected status.