Medallion Framework Data Jobs

1 job found

Today

Senior Data Engineer

Hillwood Development Company LLC Dallas, TX

Dallas, TX

Today

Full time Senior Ai/ml Data Engineering Change Data Capture

Apply Now

Job Description

Hillwood’s Corporate IT team in Dallas is hiring a Senior Data Engineer to build and evolve an enterprise Data Lakehouse on the Databricks platform.

Responsibilities

Design, develop, and maintain scalable Databricks data pipelines to integrate data from enterprise systems, including JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius, and other applications
Develop and maintain data models across Bronze, Silver, and Gold layers using Hillwood’s enterprise data architecture and medallion framework
Implement incremental processing and change data capture (CDC) to keep data current, accurate, and cost-efficient
Administer and maintain Unity Catalog, including schema management, governance, permissions, and lineage tracking
Design and support production-ready pipelines with automated scheduling, monitoring, alerting, and recovery
Optimize workflows, jobs, and compute resources for performance, scalability, and cost efficiency
Establish engineering best practices including version control, code reviews, automated testing, and CI/CD
Create and maintain technical documentation, standards, and operational procedures for long-term sustainability and knowledge transfer
Develop and implement data quality frameworks (validation rules, reconciliation processes, monitoring) to ensure accuracy and reliability
Investigate and resolve data quality issues by identifying and addressing root causes
Partner with business stakeholders and data owners to define data standards, ownership, business definitions, and quality expectations
Ensure certified and business-critical datasets align with the data governance framework and quality standards
Leverage AI-assisted development tools (coding assistants and platform-native AI capabilities) to accelerate development and improve code quality
Support standards for responsible use of AI-assisted software development
Continuously evaluate emerging AI technologies to improve engineering processes, platform operations, and delivery
Prepare and optimize lakehouse data assets for AI, machine learning, and natural language access use cases
Build data foundations for advanced analytics and AI, including feature stores, vector search, retrieval architectures, and document-based data pipelines
Work with AI engineers, developers, and stakeholders to integrate structured and unstructured data within the Databricks ecosystem
Support AI workloads and platform capabilities, including model serving, agent frameworks, and emerging GenAI
Identify and onboard new structured, unstructured, and third-party data sources to expand enterprise platform value
Collaborate with finance and business unit stakeholders to understand processes, data requirements, and reporting needs
Validate outputs and business metrics with data owners prior to dataset certification for business use
Communicate risks, trade-offs, and technical concepts to both technical and non-technical audiences
Partner closely with stakeholders to align data capabilities with operational objectives and business priorities
Partner with external engineering resources, review solution quality, and assume long-term ownership of delivered pipelines and platform components
Serve as an internal subject matter expert on architecture, operations, and ongoing evolution of Hillwood’s Enterprise Data Lakehouse

Requirements

SQL, Python, PySpark, and modern data engineering development practices
Strong understanding of lakehouse architectures, data modeling principles, and enterprise data management
Ability to design scalable, reliable, maintainable solutions for reporting, analytics, and AI use cases
Knowledge of data governance, data quality, metadata management, and data lineage
Familiarity with source control, testing methodologies, code reviews, and CI/CD
Strong analytical and problem-solving skills to identify root causes and implement solutions
Ability to balance performance, scalability, maintainability, and cost
Effective written and verbal communication skills, including explaining technical concepts to non-technical audiences
Strong organizational skills and attention to detail; ability to manage multiple priorities in a fast-paced environment
Ability to work independently and collaborate with business stakeholders, technical teams, and external partners
Curiosity and adaptability for emerging technologies, AI capabilities, and evolving data engineering practices
Customer-focused mindset with commitment to high-quality, business-driven solutions
Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related field (required)
Minimum five years of data engineering experience, including at least two years on Databricks production solutions using PySpark, Spark SQL, Delta Lake, and Databricks Workflows
Experience designing, building, and maintaining enterprise data pipelines, integrations, and lakehouse architectures in a cloud environment
Hands-on experience administering and utilizing Unity Catalog (metadata management, lineage, security, access controls)
Experience operating production data platforms, including monitoring, alerting, incident response, troubleshooting, and performance optimization
Experience implementing data quality controls, reconciliation, and governance standards
Experience integrating data from ERP, financial, operational, or other enterprise systems
Experience with Microsoft Azure and Git-based tools, including Azure DevOps and/or GitHub
Experience leveraging AI-assisted development tools in a professional software engineering workflow
Experience supporting AI/ML initiatives through data engineering, including feature engineering, vector search, retrieval-augmented generation (RAG), and model-serving pipelines or related technologies
Demonstrated success partnering with business stakeholders to understand requirements, validate data, and deliver business-focused solutions

Technologies

Databricks
SQL, Python, PySpark, Spark SQL
Delta Lake
Databricks Workflows
Unity Catalog
Bronze, Silver, Gold, medallion framework
Change data capture (CDC)
CI/CD, Git
Azure DevOps, GitHub
Microsoft Azure
JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius
Feature stores, vector search, retrieval architectures, document-based data pipelines
Model serving, agent frameworks, GenAI, retrieval-augmented generation (RAG)

Benefits

Medical Insurance
Dental Insurance
Vision Insurance
Flexible Spending Accounts (FSA)
Health Savings Accounts (HSA)
Vacation time
Sick time
Holidays
Paid family caregiver leave
Mental health and wellness resources
Life Insurance
Disability Insurance
401(k) with company match
Tuition assistance
Additional programs to support Associates and their families

Preferred Experience

Databricks Data Engineer certification or equivalent Databricks credentials
Experience supporting JD Edwards, OneStream, Yardi, Entrata, UKG, Medius, or similar enterprise platforms
Experience working with data in real estate, construction, energy, financial services, or related industries
Experience transitioning, stabilizing, and optimizing data solutions developed by third-party consultants or implementation partners
Experience supporting large language model (LLM), generative AI, or agent-based solution development
Experience designing solutions that combine structured and unstructured data for AI-enabled business applications

Location: Dallas, TX (onsite)

View full job page →

Get Medallion Framework Data Jobs alerts

New Medallion Framework Data Jobs, daily.