DataJobs.io
← Back to all jobs

Job Description

Better Debt Solutions is building an enterprise data platform on Microsoft Fabric, and you will help design, scale, and optimize it for operational reporting and AI-driven insights. This fully in-office role in Irvine, CA offers clear visibility with direct reporting to executive leadership (CEO & COO) and the autonomy to architect and grow the company’s data footprint.

You will work on a Fabric-based lakehouse/warehouse environment, focusing on reliable ingestion, performant transformations, and data reliability with monitoring and observability. If you care about building systems that stay fast and trustworthy as they scale, this role is a strong match.

What you’ll do

  • Design and implement scalable data warehouse and lakehouse architectures using Microsoft Fabric and Azure.
  • Organize data structures in OneLake, Fabric Warehouse, and lakehouse layers using a medallion approach (bronze, silver, gold).
  • Build and maintain ingestion and transformation pipelines with Data Factory in Fabric (pipelines, Copy jobs) and Spark/Python notebooks.
  • Ingest data from REST APIs and relational databases including MySQL/MariaDB, PostgreSQL/Redshift, and SQL Server, including sources behind on-premises data gateways.
  • Create dimensional models and star schemas with SCD Type 2 history and conformed dimensions to support cross-functional reporting.
  • Centralize business logic in curated gold-layer tables to keep semantic models consistent and power near-real-time operational dashboards.
  • Profile and tune Spark and SQL workloads using partitioning, file sizing, join strategies, and caching to reduce run times.
  • Convert legacy full reload processes into incremental and CDC pipelines to minimize compute overhead.
  • Monitor and manage Fabric capacity (CU consumption) using the Fabric Capacity Metrics app to avoid throttling during peak workloads.
  • Maintain Delta tables with OPTIMIZE, VACUUM, and V-Order to improve read performance in Direct Lake and warehouse queries.
  • Audit pipelines, notebooks, and dataflows to remove redundancy and consolidate duplicated logic.
  • Implement automated monitoring for freshness, volume, and completeness with proactive alerting across production pipelines.
  • Reconcile loaded data with source systems to identify ingestion gaps and anomalies at the root cause.
  • Perform safe historical corrections and backfills using dry runs and validation before production deployment.
  • Collaborate with vendor partners, business users, BI, and analytics teams to deliver cost-efficient, scalable data solutions.

What you’ll bring

  • Bachelor’s degree in Computer Science, Data Science, Information Systems, or related field (or equivalent professional experience).
  • 7+ years of hands-on experience in Data Engineering or Data Platform Development, including direct production experience with Microsoft Fabric and Azure.
  • Strong proficiency in Python, PySpark, and advanced SQL.
  • Proven ability to identify, decompose, and solve complex data challenges independently and with a team.
  • Track record of reducing run times and compute costs through workload optimization, incremental loading, and dimensional modeling (star schema, SCD Type 2).
  • Practical experience with Git-based development and deployment promotion workflows using Azure DevOps or GitHub.
  • Excellent communication skills, including bridging conversations across executive leadership (CEO, COO), vendor partners, and non-technical stakeholders.

Bonus points

  • Microsoft Fabric Data Engineer Associate (DP-700) certification.
  • Familiarity with Azure AI Foundry or Azure Machine Learning.
  • Experience leveraging AI-assisted coding tools to accelerate pipeline development.
  • Familiarity with secondary cloud ecosystems such as Google Cloud Platform (GCP).
  • Prior DBA experience managing data security, redundancy, and high availability.

Compensation and benefits

  • $140,000 - $155,000 per year
  • Full-Time (Exempt/Salary)
  • Comprehensive medical, dental, and vision insurance coverage options
  • Paid vacation and paid company holidays throughout the year
  • Lunch is on us every Friday!
  • 401(k) retirement savings plan options
  • Vibrant team culture with regular quarterly social events, team outings, and annual holiday parties

Technologies you’ll work with

Microsoft Fabric, Azure, OneLake, Fabric Warehouse, Data Factory in Fabric, Spark, Python, PySpark, REST APIs, MySQL, MariaDB, PostgreSQL, Redshift, SQL Server, on-premises data gateways, SCD Type 2, star schema, Delta tables, OPTIMIZE, VACUUM, V-Order, Direct Lake, Fabric Capacity Metrics app, Fabric capacity (CU), Azure DevOps, GitHub, and SQL.

Similar Jobs