Lakehouse

Lakehouse Data Jobs

14 jobs found

1d ago

Lead Data Scientist

Hyatt Corporate Office Chicago, IL 🇺🇸 Remote

Chicago, IL

1d ago

$160k - $170k/yr Full time Ai Ml Artificial Intelligence Data

Apply Now

Job Description

Lead Data Scientist, individual contributor on Hyatt's AIML team focused on Search, Personalization, and Agents.

Responsibilities

Own the design, development, evaluation, and optimization of AI and ML solutions that support Hyatt’s guest, colleague, and operational experiences.
Provide technical leadership, mentor peers, influence architecture and product direction, and raise the overall technical bar for applied AI at Hyatt.
Serve as a hands-on technical lead for high impact AI and machine learning initiatives.
Lead solution design, modeling decisions, experimentation strategy, and technical tradeoff discussions.
Partner with ML engineering and data engineering teams to deploy scalable real-time inference pipelines and batch processing workflows.
Influence technical roadmaps and help sequence data science initiatives based on business value, feasibility, risk, and team capacity.
Mentor data scientists and ML practitioners through design reviews, code reviews, modeling best practices, and knowledge sharing.
Collaborate with ML engineering to productionize models and Gen AI services using AWS-native tools and modern MLOps practices.
Contribute to scalable ML system design, including data pipelines, feature workflows, model serving, observability, monitoring, and lifecycle management.
Apply strong software engineering practices, including version control, CI/CD, testing, reproducibility, containerization, and documentation.
Support deployment patterns for both batch and low-latency inference use cases.
Partner with security, governance, architecture, and legal/privacy stakeholders to ensure AI systems are reliable, secure, compliant, and responsibly deployed.
Work closely with product owners, data scientists, ML engineers, data engineers, architects, and business stakeholders to deliver end-to-end algorithmic products.
Communicate model behavior, limitations, assumptions, risks, and business impact clearly to technical and non-technical audiences.
Define measurable success criteria and help evaluate whether AI solutions are delivering intended outcomes.
Champion responsible AI, inclusive design, and practical experimentation across projects.

Requirements

Master’s degree in computer science, software engineering, or a related field; PhD preferred.
Six or more years of experience in machine learning roles with a focus on NLP/NLU, reinforcement learning, or large language model applications.
Three or more years of people management experience in a tech leadership role.
Experience fine-tuning and deploying LLMs or other Generative AI solutions to production.
Proficiency with AWS cloud services such as SageMaker, ECS/EKS, Step Functions, Lambda, and Glue.
Strong Python programming skills, with familiarity in SQL, PySpark, and containerization using Docker.
Proven experience designing scalable data pipelines and ML systems for both real-time and batch inference.
Deep understanding of responsible AI practices, CI/CD pipelines, Agile development practices, and model lifecycle management.
Excellent interpersonal and communication skills, with a strong bias for action and collaboration.
Familiarity with ML observability and governance tools.

Technologies

Python
SQL
PySpark
Docker
SageMaker
ECS
EKS
Step Functions
Lambda
Glue

Benefits

Annual allotment of free hotel stays at Hyatt hotels globally
Flexible work schedule
Headspace subscription
Discount at on-site fitness center
Paid time off following the birth or adoption of a child
Financial assistance for adoption
Paid Time Off
Medical, Dental, Vision coverage
401K with company match

Qualifications

Master’s degree in computer science, software engineering, or a related field; PhD preferred.
Six or more years of experience in machine learning roles focused on NLP/NLU, reinforcement learning, or LLM applications, including at least three years in a tech leadership role.
Experience fine-tuning and deploying LLMs or other Generative AI solutions to production.
Expertise in AWS cloud services (SageMaker, ECS/EKS, Step Functions, Lambda, Glue).
Strong Python skills with SQL, PySpark, and Docker experience.
Proven ability to design scalable data pipelines and ML systems for real-time and batch inference.
Deep understanding of responsible AI, CI/CD, Agile methods, and model lifecycle management.
Excellent communication and collaboration skills.
Familiarity with ML observability and governance tools.

View full job page →

5d ago

Sr. Data Scientist

Ledgent Technology Sacramento, CA

Sacramento, CA

5d ago

$55 - $72/hr Full time Senior Analytics Big Data

Apply Now

Job Description

Ledgent Technology is actively seeking a Senior Data Scientist for an on-site role in Sacramento, California. The position centers on healthcare data modernization, analytics, dashboards, data validation, and EHR modernization, leveraging lakehouse and medallion architectures on AWS. The role offers collaboration with engineering, program leadership, and clinical operations to drive data-driven decision making.

Benefits

Competitive hourly pay ranging between $55.00 and $72.00 per hour, depending on experience and qualifications.

Position Overview

Ledgent Technology's client is seeking an analytical and independent Senior Data Scientist to join the California team. This role supports a significant data modernization effort within a healthcare environment, utilizing lakehouse and medallion architecture on AWS. The ideal candidate will collaborate with engineering, program leadership, and clinical operations to deliver advanced analytics, validate and reconcile complex datasets, and develop dashboards that inform strategic decisions.

Responsibilities

Create and maintain dashboards with Tableau or Power BI to deliver clear, actionable insights.
Develop and optimize complex SQL queries to analyze large-scale datasets.
Assist data ingestion, pipeline development, and performance monitoring within Databricks.
Conduct large-scale data validation and reconciliation across multiple systems.
Examine healthcare or program data to identify trends, anomalies, and opportunities.
Translate complex data structures into understandable insights for business and clinical stakeholders.
Collaborate with cross-functional teams including engineering, program leadership, and clinical operations.
Support data governance efforts to uphold data accuracy and consistency.
Document data flows across systems and understand data lineage.
Provide data-driven recommendations to inform strategic initiatives.
Offer analytics support for EHR modernization initiatives.

Requirements

Strong SQL skills with the ability to write complex queries.
Hands-on experience with Databricks and lakehouse or medallion architecture.
Experience with AWS or similar cloud platforms.
Practical dashboard development experience using Tableau or Power BI.
Experience building and monitoring data pipelines and ingestion processes.
Extensive experience with data validation and reconciliation at scale.
Ability to analyze large, complex datasets and derive meaningful insights.
Excellent communication skills to convey technical findings to non-technical audiences.

Technologies

Tableau
Power BI
SQL
Databricks
Lakehouse architecture
Medallion architecture
AWS

Work Hours

Monday to Friday

Preferred Qualifications

Experience with machine learning concepts or applied modeling.
Exposure to A/B testing or experimentation frameworks.
Experience with RDS or relational database systems.
Familiarity with DevOps practices.
Healthcare, patient data, or EHR experience.

Functional Skills

Strong understanding of data governance principles.
Ability to interpret ambiguous datasets and extract meaningful insights.
Clear understanding of data lineage and system data flows.
Ability to connect analytics to program outcomes and clinical operations.
Strong troubleshooting and data quality validation skills.

Soft Skills

Excellent communication and executive-level presentation skills.
Collaborative and effective across technical and non-technical teams.
Self-sufficient and able to work independently.
Analytical, solution-oriented mindset.
Comfortable working in fast-paced, complex environments.
Adaptable and calm when navigating ambiguity.

View full job page →

9d ago

Lead Data Engineer

INSPYR Solutions Doraville, GA 🇺🇸 Remote

Doraville, GA

9d ago

$60 - $70/hr Contract Artificial Intelligence Big Data Bigdata

Apply Now

Job Description

Lead Data Engineer at INSPYR Solutions in Doraville, GA (remote) responsible for architecting and optimizing distributed data pipelines, Lakehouse architecture, ML workflows, data quality, and production reliability.

Responsibilities

Design, build, and optimize scalable distributed data pipelines using Apache Spark within a high-volume, mission-critical environment.
Design and sustain an enterprise Lakehouse with Delta Lake, ensuring ACID transactions, data lineage, auditability, and governance.
Create automated ingestion frameworks for batch, streaming, and event-driven data across multiple cloud services and integration points.
Enable machine learning workflows by preparing feature-ready datasets and establishing reproducible deployment patterns for ML models.
Lead platform wide data quality, access control, and data cataloging initiatives.
Apply cost optimization, cluster tuning, and performance engineering techniques to maximize efficiency.
Collaborate with Finance, BI, Operations, and ML teams to translate business needs into scalable data solutions.
Own production reliability, troubleshoot incidents, and perform root cause analysis for data and ML pipelines.

Requirements

7+ years of experience in advanced data engineering with distributed compute technologies.
Expert Spark engineering experience, including performance tuning, cluster configuration, partition strategies, and large dataset optimization.
Hands-on experience with Lakehouse architectures featuring ACID transactions, schema evolution, and governance frameworks.
Advanced Python and SQL proficiency for large-scale data transformations.
Experience supporting machine learning pipelines or model operationalization.
Proven track record architecting cloud-native data platforms on Azure, AWS, or GCP.
Strong ability to integrate diverse, complex data sources at enterprise scale.
Demonstrated capacity to own mission-critical production systems.
Experience with distributed streaming frameworks such as Kafka, Event Hubs, or equivalent.
Experience building or supporting ML platforms, feature stores, or experiment tracking systems.
Background in data security, compliance controls, or audit-ready governance.
Experience automating data operations with CI/CD and infrastructure as code.

Technologies

Databricks, Python, SQL, Apache Spark, Delta Lake, MLflow, Notebooks
Hugging Face Transformers, LangChain, LlamaIndex, Anthropic Claude, Meta LLaMA, Google Gemini
Kafka, Event Hubs, REST APIs, ADLS, S3, GCS
Git, GitHub, GitLab, Azure Repos, Databricks Repos, GitHub Actions, Azure DevOps
Unity Catalog, RBAC

Benefits

Comprehensive medical benefits
Competitive pay
401(k) retirement plan

Core Tools

Databricks (Spark, Delta Lake, MLflow, Notebooks)
Python & SQL
Apache Spark (via Databricks)
Delta Lake for Lakehouse architecture

Cloud Platforms

Azure, AWS, or GCP
Cloud storage options: ADLS, S3, GCS

Data Integration

Kafka or Event Hubs for streaming data
Auto Loader for Databricks file ingestion
REST APIs
AI/ML workflows
MLflow for model tracking and deployment
Hugging Face Transformers
LangChain and LlamaIndex for LLM integration
LLMs: Anthropic Claude, Meta LLaMA, Google Gemini

DevOps

Git ecosystem: GitHub, GitLab, Azure Repos
Databricks Repos
CI/CD: GitHub Actions, Azure DevOps

Security & Governance

Unity Catalog
RBAC

View full job page →

18d ago

Data Engineer

Major League Baseball New York, NY

New York, NY

18d ago

$115k - $140k/yr Full time Apache Airflow Automation Big Data

Apply Now

Job Description

In this hands-on Data Engineer role with Major League Baseball's LAI team, you will design and operate production data pipelines on a Google Cloud Platform lakehouse, using Airflow and dbt to empower league analytics. The position is onsite in New York, NY, placing you at the center of MLB’s data-driven decision making.

Responsibilities

Construct production-grade pipelines with Airflow and dbt to orchestrate batch and streaming transformations across GCP, delivering trusted data to downstream analysts and engineers.
Design clean, layered data models (staging, intermediate, mart) that serve as the single source of truth for league analytics, applying dbt best practices for materialization, testing, and documentation.
Operate the ingestion layer using Pub/Sub, GCS, Dataflow, and Knowledge Catalog DataPlex to land both batch and streaming sources into the lakehouse with integrity.
Establish observability and monitoring standards so data quality issues surface before stakeholders notice them.
Manage code through GitHub-based CI/CD, contributing to deployment workflows that keep the platform reliable and changes safe.
Adhere to data governance practices that keep proprietary baseball data secure and compliant.

Requirements

2–4 years of production data engineering experience.
Expert-level SQL, comfortable writing complex freehand queries (sub-queries, nested logic, window functions) and reading others’ code to spot issues.
Strong Python for data processing, scripting, and automation.
Hands-on dbt experience — built models across staging, intermediate, and mart layers, wrote tests, and shipped to production.
Production Airflow experience — DAG authoring, dependency management, debugging failed runs.
Deep familiarity with Google Cloud Platform (BigQuery, GCS, Pub/Sub) or equivalent depth in AWS/Azure with willingness to convert.
Git-based development workflows — branches, PRs, code review as a daily practice.
Clear communicator with both engineers and non-engineers, receptive to feedback and able to provide it constructively.
Execution mindset with the ability to own a project from requirements to deployment with minimal oversight.

Technologies

Airflow
dbt
BigQuery
GCS
Pub/Sub
Dataflow
Knowledge Catalog DataPlex
GitHub

Benefits

Competitive Benefits Package
Company Contributed 401K Plan
Paid Time Off and Holidays
Paid Parental Leave
Access to Free Tickets to Baseball Games & MLB.TV
Discounts at MLB Store | MLBShop.com
Employee Assistance Programs (EAP)
Onsite/Online Training & Development Programs
Tuition Reimbursement
Disability Benefits (short term and long term)
Life and Accidental Death Insurance
Pet Insurance

Nice-to-Have

A degree in Computer Science, Engineering, or a related field — or non-traditional background with equivalent practical experience.
Experience with Terraform or other Infrastructure-as-Code tools.
Experience with AI-assisted development or enterprise AI tooling (Gemini Enterprise, Vertex AI). The team is early but ambitious about AI as a lever for engineering efficiency.
A passion for baseball or prior experience in sports, media, or entertainment.
Ability to craft creative solutions for unusual problems.

Salary Range

USD 115,000 - 140,000 per year, plus Bonus

Why MLB?

Major League Baseball is one of the most historic professional sports leagues in North America, with a culture that values growth, teamwork, and professionalism.
Employees who thrive at MLB tend to take initiative, identify problems, and deliver solutions while prioritizing the team.
MLB aims to empower its workforce by engineering experiences that position people for success, aligning individual growth with the broader goals of the organization.

View full job page →

19d ago

Data Engineer (Senior Consultant)

Guidehouse Washington, DC

Washington, DC

19d ago

$85k - $141k/yr Full time Senior Big Data Consultant

Apply Now

Job Description

This Senior Consultant data engineering role focuses on CMS data modernization, delivering scalable ETL pipelines and lakehouse architectures to securely process federal health data in a compliant environment.

Compensation

USD 85,000 - 141,000 per year

Location

Washington, DC onsite

Responsibilities

Design, implement, and maintain end-to-end ETL and ELT pipelines within Databricks based environments
Architect and manage medallion style lakehouse architectures spanning bronze, silver, and gold layers
Build scalable data pipelines capable of handling large, complex datasets including structured and semi-structured data
Collaborate with data scientists, analysts, subject matter experts, and CMS stakeholders to translate business requirements into technical solutions
Ensure data quality, lineage, governance, validation, monitoring, and documentation
Optimize performance and efficiency of cloud-based data processing workflows
Support data integration and modernization efforts aligned with CMS EIPM objectives
Participate in Agile development practices, including sprint planning and backlog management

Requirements

U.S. citizenship and ability to obtain a Public Trust clearance
Bachelor’s degree in Computer Science, Data Science, Information Systems, or a related field
Minimum of four years of experience in data engineering, software engineering, or similar technical roles
Proficiency with Python and SQL for data engineering and transformation workloads
Hands-on experience with Databricks and large-scale data processing environments
Experience developing and maintaining ETL/ELT data pipelines
Experience working with medallion lakehouse architectures (bronze, silver, gold)
Experience with relational and/or distributed data systems, including data modeling
Experience with CI/CD practices and Git-based workflows
Familiarity with version control and collaboration tools, including Git, Jira, and Confluence
Experience working in Agile development environments
Ability to work independently in a fast-paced environment
Strong analytical and troubleshooting skills
Excellent verbal and written communication skills, with the ability to explain complex technical concepts to diverse audiences

Technologies

Databricks
Python
SQL
Git
Jira
Confluence

Benefits

Medical, Rx, Dental & Vision Insurance
Personal and Family Sick Time & Company Paid Holidays
Position may be eligible for a discretionary variable incentive bonus
Parental Leave and Adoption Assistance
401(k) Retirement Plan
Basic Life & Supplemental Life
Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
Short-Term & Long-Term Disability
Student Loan PayDown
Tuition Reimbursement, Personal Development & Learning Opportunities
Skills Development & Certifications
Employee Referral Program
Corporate Sponsored Events & Community Outreach
Emergency Back-Up Childcare Program
Mobility Stipend

Job Family

Data Engineering & Architecture Consulting

Travel Required

Up to 10%

Clearance Required

Ability to Obtain Public Trust

Nice To Have

Experience working with healthcare data such as claims, eligibility, or clinical datasets
Experience supporting CMS, HHS, or other federal health programs
Experience working with state government data programs
Familiarity with data governance, privacy, and compliance frameworks (e.g., HIPAA)
Experience supporting data modernization or cloud migration initiatives

View full job page →

25d ago

Senior Data Engineer

Space Telescope Science Institute Baltimore, MD

Baltimore, MD

25d ago

$125k - $150k/yr Full time Senior Big Data Cloud Operations

Apply Now

Job Description

Senior Data Engineer at Space Telescope Science Institute will manage backend data pipelines and MPP database systems for the Mikulski Archive for Space Telescopes (MAST), enabling high-performance data access, with a hybrid work arrangement in Baltimore, MD, residency/ITAR requirements, a salary range of USD 125,000 - 150,000 per year, and a minimum of 8 years of experience plus a Bachelor's or Master's degree in computer science, information technology, or a related discipline.

Responsibilities

Manage PostgreSQL and MPP platform performance and operations
Design, implement, deploy, monitor, and troubleshoot complex data pipelines using Apache Airflow to process, transform, and load large-scale datasets reliably
Build, maintain, and continually improve data systems supporting scientific research, including relational databases and cloud-based lakehouse architectures
Ensure data accuracy, accessibility, observability, and reliability through proactive monitoring, alerting, and incident response
Collaborate with scientists, data engineers, and cross-functional teams to translate requirements into robust, scalable platform solutions

Requirements

Advanced expertise in PostgreSQL and Greenplum MPP
Strong proficiency with Apache Airflow
Hands-on experience with AWS cloud services
Strong Python programming skills and proficiency in SQL and SQL performance tuning
Extensive experience designing, building, and optimizing data pipelines at scale

Technologies

PostgreSQL
Greenplum
Apache Airflow
AWS
Python
SQL
Trino
Apache Iceberg
Lakehouse

Benefits

Employer retirement contribution – direct STScI contribution of 10% of your salary from your first day
12 days sick leave
Up to 24 days of vacation
10 paid holidays
Flexible work schedule with healthy work/life balance
Comprehensive medical, dental, vision, prescription plans, and more

Growth opportunities

Specialize in Trino for distributed querying
Work with Apache Iceberg for lakehouse management
Explore Greenplum or other MPP systems and large-scale performance tuning
Tackle complex challenges with high-volume scientific data, query optimization, and modern data infrastructure

View full job page →

1mo ago

Data Analyst

Security Public Storage San Francisco, CA

San Francisco, CA

1mo ago Closed

$115k - $125k/yr Full time Analyst Analytics Azure Data Lake Storage

This position is no longer accepting applications (closed Jul 21, 2026).

Job Description

The Data Analyst role at Security Public Storage is a full-time, hybrid position in San Francisco, CA, focused on revenue management, pricing strategy, and operational intelligence across a portfolio of 53 self-storage facilities. The role involves building and maintaining the data infrastructure the team relies on and using it to answer questions that shape pricing, promotions, and day-to-day operations.

Responsibilities

Assess property level performance, including occupancy, tenant turnover, pricing movements, and revenue yield.
Support portfolio pricing initiatives to balance revenue with occupancy targets.
Evaluate discounting and promotional programs such as move-in specials, concessions, and rate changes, and advise on deployment or reduction.
Develop frameworks to quantify promotional effectiveness, considering occupancy lift, revenue trade-offs, stay duration, and conversion rates.
Track competitive pricing and market trends within the self-storage sector.
Assist in forecasting revenue and occupancy outcomes.
Collaborate on and maintain existing revenue management models.
Identify opportunities to refine model logic in response to outcomes and evolving market conditions.
Partner with internal stakeholders to design, validate, and maintain the Microsoft Fabric data warehouse.
Create Power BI dashboards and reports to illuminate KPIs and portfolio-wide trends.
Ensure data accuracy and integrity across reporting systems.
Perform ad hoc analyses on pricing, promotions, and customer behavior.
Collaborate with operations and finance teams to translate data insights into actionable strategies.
Communicate findings clearly to non-technical stakeholders through polished reports and presentations.

Requirements

Advanced SQL skills with complex queries using common table expressions, window functions, and joins; T-SQL is preferred.
Strong data modeling fundamentals, including star and dimensional modeling, with clear relationships and cardinality.
ETL or ELT pipeline logic and data quality practices.
Experience in revenue analysis, pricing models, or tracking operational KPIs.
Familiarity with Microsoft Fabric components, including Lakehouses, Warehouses, data pipelines, and OneLake architecture.
Power BI proficiency covering DAX measures, calculated columns, data modeling, report design, and Row-Level Security (RLS).
Power Query / M for data transformation tasks.
Understanding of Direct Lake mode and performance optimization in Power BI.
Ability to translate business questions into data problems and conduct independent analyses.
Effective data storytelling to present findings to non-technical audiences.
Experience designing or validating measurement frameworks such as A/B tests, lift analyses, and conversion analyses.
Commitment to data integrity and documentation practices, including naming conventions and lineage tracking.

Technologies

SQL
T-SQL
Microsoft Fabric
Power BI
DAX
Power Query / M
DirectLake mode
OneLake architecture
Lakehouses
Warehouses
Azure
Python
Git

Benefits

401(k) with 4% employer match
Health insurance
Health savings account
Life insurance
Paid time off

Nice to Have

Python for data processing and notebook-based analysis
Experience with medallion architecture (Bronze to Silver to Gold) or lakehouse design patterns
Azure fundamentals, given Fabric runs on Azure
Git or version control for analytical assets
Background in real estate, self-storage, or asset-intensive industries
Experience with revenue management systems or dynamic pricing tools