Senior Data Engineer

Hillwood’s Corporate IT team in Dallas is hiring a Senior Data Engineer to build and evolve an enterprise Data Lakehouse on the Databricks platform.

Responsibilities

• Design, develop, and maintain scalable Databricks data pipelines to integrate data from enterprise systems, including JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius, and other applications
• Develop and maintain data models across Bronze, Silver, and Gold layers using Hillwood’s enterprise data architecture and medallion framework
• Implement incremental processing and change data capture (CDC) to keep data current, accurate, and cost-efficient
• Administer and maintain Unity Catalog, including schema management, governance, permissions, and lineage tracking
• Design and support production-ready pipelines with automated scheduling, monitoring, alerting, and recovery
• Optimize workflows, jobs, and compute resources for performance, scalability, and cost efficiency
• Establish engineering best practices including version control, code reviews, automated testing, and CI/CD
• Create and maintain technical documentation, standards, and operational procedures for long-term sustainability and knowledge transfer
• Develop and implement data quality frameworks (validation rules, reconciliation processes, monitoring) to ensure accuracy and reliability
• Investigate and resolve data quality issues by identifying and addressing root causes
• Partner with business stakeholders and data owners to define data standards, ownership, business definitions, and quality expectations
• Ensure certified and business-critical datasets align with the data governance framework and quality standards
• Leverage AI-assisted development tools (coding assistants and platform-native AI capabilities) to accelerate development and improve code quality
• Support standards for responsible use of AI-assisted software development
• Continuously evaluate emerging AI technologies to improve engineering processes, platform operations, and delivery
• Prepare and optimize lakehouse data assets for AI, machine learning, and natural language access use cases
• Build data foundations for advanced analytics and AI, including feature stores, vector search, retrieval architectures, and document-based data pipelines
• Work with AI engineers, developers, and stakeholders to integrate structured and unstructured data within the Databricks ecosystem
• Support AI workloads and platform capabilities, including model serving, agent frameworks, and emerging GenAI
• Identify and onboard new structured, unstructured, and third-party data sources to expand enterprise platform value
• Collaborate with finance and business unit stakeholders to understand processes, data requirements, and reporting needs
• Validate outputs and business metrics with data owners prior to dataset certification for business use
• Communicate risks, trade-offs, and technical concepts to both technical and non-technical audiences
• Partner closely with stakeholders to align data capabilities with operational objectives and business priorities
• Partner with external engineering resources, review solution quality, and assume long-term ownership of delivered pipelines and platform components
• Serve as an internal subject matter expert on architecture, operations, and ongoing evolution of Hillwood’s Enterprise Data Lakehouse

Requirements

• SQL, Python, PySpark, and modern data engineering development practices
• Strong understanding of lakehouse architectures, data modeling principles, and enterprise data management
• Ability to design scalable, reliable, maintainable solutions for reporting, analytics, and AI use cases
• Knowledge of data governance, data quality, metadata management, and data lineage
• Familiarity with source control, testing methodologies, code reviews, and CI/CD
• Strong analytical and problem-solving skills to identify root causes and implement solutions
• Ability to balance performance, scalability, maintainability, and cost
• Effective written and verbal communication skills, including explaining technical concepts to non-technical audiences
• Strong organizational skills and attention to detail; ability to manage multiple priorities in a fast-paced environment
• Ability to work independently and collaborate with business stakeholders, technical teams, and external partners
• Curiosity and adaptability for emerging technologies, AI capabilities, and evolving data engineering practices
• Customer-focused mindset with commitment to high-quality, business-driven solutions
• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related field (required)
• Minimum five years of data engineering experience, including at least two years on Databricks production solutions using PySpark, Spark SQL, Delta Lake, and Databricks Workflows
• Experience designing, building, and maintaining enterprise data pipelines, integrations, and lakehouse architectures in a cloud environment
• Hands-on experience administering and utilizing Unity Catalog (metadata management, lineage, security, access controls)
• Experience operating production data platforms, including monitoring, alerting, incident response, troubleshooting, and performance optimization
• Experience implementing data quality controls, reconciliation, and governance standards
• Experience integrating data from ERP, financial, operational, or other enterprise systems
• Experience with Microsoft Azure and Git-based tools, including Azure DevOps and/or GitHub
• Experience leveraging AI-assisted development tools in a professional software engineering workflow
• Experience supporting AI/ML initiatives through data engineering, including feature engineering, vector search, retrieval-augmented generation (RAG), and model-serving pipelines or related technologies
• Demonstrated success partnering with business stakeholders to understand requirements, validate data, and deliver business-focused solutions

Technologies

• Databricks
• SQL, Python, PySpark, Spark SQL
• Delta Lake
• Databricks Workflows
• Unity Catalog
• Bronze, Silver, Gold, medallion framework
• Change data capture (CDC)
• CI/CD, Git
• Azure DevOps, GitHub
• Microsoft Azure
• JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius
• Feature stores, vector search, retrieval architectures, document-based data pipelines
• Model serving, agent frameworks, GenAI, retrieval-augmented generation (RAG)

Benefits

• Medical Insurance
• Dental Insurance
• Vision Insurance
• Flexible Spending Accounts (FSA)
• Health Savings Accounts (HSA)
• Vacation time
• Sick time
• Holidays
• Paid family caregiver leave
• Mental health and wellness resources
• Life Insurance
• Disability Insurance
• 401(k) with company match
• Tuition assistance
• Additional programs to support Associates and their families

Preferred Experience

• Databricks Data Engineer certification or equivalent Databricks credentials
• Experience supporting JD Edwards, OneStream, Yardi, Entrata, UKG, Medius, or similar enterprise platforms
• Experience working with data in real estate, construction, energy, financial services, or related industries
• Experience transitioning, stabilizing, and optimizing data solutions developed by third-party consultants or implementation partners
• Experience supporting large language model (LLM), generative AI, or agent-based solution development
• Experience designing solutions that combine structured and unstructured data for AI-enabled business applications

Location: Dallas, TX (onsite)