Lakehouse

Lakehouse Data Jobs

11 jobs found

8d ago

Data Engineer

Major League Baseball New York, NY

New York, NY

8d ago

$115k - $140k/yr Full time Apache Airflow Automation Big Data

Apply Now

Job Description

In this hands-on Data Engineer role with Major League Baseball's LAI team, you will design and operate production data pipelines on a Google Cloud Platform lakehouse, using Airflow and dbt to empower league analytics. The position is onsite in New York, NY, placing you at the center of MLB’s data-driven decision making.

Responsibilities

Construct production-grade pipelines with Airflow and dbt to orchestrate batch and streaming transformations across GCP, delivering trusted data to downstream analysts and engineers.
Design clean, layered data models (staging, intermediate, mart) that serve as the single source of truth for league analytics, applying dbt best practices for materialization, testing, and documentation.
Operate the ingestion layer using Pub/Sub, GCS, Dataflow, and Knowledge Catalog DataPlex to land both batch and streaming sources into the lakehouse with integrity.
Establish observability and monitoring standards so data quality issues surface before stakeholders notice them.
Manage code through GitHub-based CI/CD, contributing to deployment workflows that keep the platform reliable and changes safe.
Adhere to data governance practices that keep proprietary baseball data secure and compliant.

Requirements

2–4 years of production data engineering experience.
Expert-level SQL, comfortable writing complex freehand queries (sub-queries, nested logic, window functions) and reading others’ code to spot issues.
Strong Python for data processing, scripting, and automation.
Hands-on dbt experience — built models across staging, intermediate, and mart layers, wrote tests, and shipped to production.
Production Airflow experience — DAG authoring, dependency management, debugging failed runs.
Deep familiarity with Google Cloud Platform (BigQuery, GCS, Pub/Sub) or equivalent depth in AWS/Azure with willingness to convert.
Git-based development workflows — branches, PRs, code review as a daily practice.
Clear communicator with both engineers and non-engineers, receptive to feedback and able to provide it constructively.
Execution mindset with the ability to own a project from requirements to deployment with minimal oversight.

Technologies

Airflow
dbt
BigQuery
GCS
Pub/Sub
Dataflow
Knowledge Catalog DataPlex
GitHub

Benefits

Competitive Benefits Package
Company Contributed 401K Plan
Paid Time Off and Holidays
Paid Parental Leave
Access to Free Tickets to Baseball Games & MLB.TV
Discounts at MLB Store | MLBShop.com
Employee Assistance Programs (EAP)
Onsite/Online Training & Development Programs
Tuition Reimbursement
Disability Benefits (short term and long term)
Life and Accidental Death Insurance
Pet Insurance

Nice-to-Have

A degree in Computer Science, Engineering, or a related field — or non-traditional background with equivalent practical experience.
Experience with Terraform or other Infrastructure-as-Code tools.
Experience with AI-assisted development or enterprise AI tooling (Gemini Enterprise, Vertex AI). The team is early but ambitious about AI as a lever for engineering efficiency.
A passion for baseball or prior experience in sports, media, or entertainment.
Ability to craft creative solutions for unusual problems.

Salary Range

USD 115,000 - 140,000 per year, plus Bonus

Why MLB?

Major League Baseball is one of the most historic professional sports leagues in North America, with a culture that values growth, teamwork, and professionalism.
Employees who thrive at MLB tend to take initiative, identify problems, and deliver solutions while prioritizing the team.
MLB aims to empower its workforce by engineering experiences that position people for success, aligning individual growth with the broader goals of the organization.

View full job page →

8d ago

Data Engineer (Senior Consultant)

Guidehouse Washington, DC

Washington, DC

8d ago

$85k - $141k/yr Full time Senior Big Data Consultant

Apply Now

Job Description

This Senior Consultant data engineering role focuses on CMS data modernization, delivering scalable ETL pipelines and lakehouse architectures to securely process federal health data in a compliant environment.

Compensation

USD 85,000 - 141,000 per year

Location

Washington, DC onsite

Responsibilities

Design, implement, and maintain end-to-end ETL and ELT pipelines within Databricks based environments
Architect and manage medallion style lakehouse architectures spanning bronze, silver, and gold layers
Build scalable data pipelines capable of handling large, complex datasets including structured and semi-structured data
Collaborate with data scientists, analysts, subject matter experts, and CMS stakeholders to translate business requirements into technical solutions
Ensure data quality, lineage, governance, validation, monitoring, and documentation
Optimize performance and efficiency of cloud-based data processing workflows
Support data integration and modernization efforts aligned with CMS EIPM objectives
Participate in Agile development practices, including sprint planning and backlog management

Requirements

U.S. citizenship and ability to obtain a Public Trust clearance
Bachelor’s degree in Computer Science, Data Science, Information Systems, or a related field
Minimum of four years of experience in data engineering, software engineering, or similar technical roles
Proficiency with Python and SQL for data engineering and transformation workloads
Hands-on experience with Databricks and large-scale data processing environments
Experience developing and maintaining ETL/ELT data pipelines
Experience working with medallion lakehouse architectures (bronze, silver, gold)
Experience with relational and/or distributed data systems, including data modeling
Experience with CI/CD practices and Git-based workflows
Familiarity with version control and collaboration tools, including Git, Jira, and Confluence
Experience working in Agile development environments
Ability to work independently in a fast-paced environment
Strong analytical and troubleshooting skills
Excellent verbal and written communication skills, with the ability to explain complex technical concepts to diverse audiences

Technologies

Databricks
Python
SQL
Git
Jira
Confluence

Benefits

Medical, Rx, Dental & Vision Insurance
Personal and Family Sick Time & Company Paid Holidays
Position may be eligible for a discretionary variable incentive bonus
Parental Leave and Adoption Assistance
401(k) Retirement Plan
Basic Life & Supplemental Life
Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
Short-Term & Long-Term Disability
Student Loan PayDown
Tuition Reimbursement, Personal Development & Learning Opportunities
Skills Development & Certifications
Employee Referral Program
Corporate Sponsored Events & Community Outreach
Emergency Back-Up Childcare Program
Mobility Stipend

Job Family

Data Engineering & Architecture Consulting

Travel Required

Up to 10%

Clearance Required

Ability to Obtain Public Trust

Nice To Have

Experience working with healthcare data such as claims, eligibility, or clinical datasets
Experience supporting CMS, HHS, or other federal health programs
Experience working with state government data programs
Familiarity with data governance, privacy, and compliance frameworks (e.g., HIPAA)
Experience supporting data modernization or cloud migration initiatives

View full job page →

15d ago

Senior Data Engineer

Space Telescope Science Institute Baltimore, MD

Baltimore, MD

15d ago

$125k - $150k/yr Full time Senior Big Data Cloud Operations

Apply Now

Job Description

Senior Data Engineer at Space Telescope Science Institute will manage backend data pipelines and MPP database systems for the Mikulski Archive for Space Telescopes (MAST), enabling high-performance data access, with a hybrid work arrangement in Baltimore, MD, residency/ITAR requirements, a salary range of USD 125,000 - 150,000 per year, and a minimum of 8 years of experience plus a Bachelor's or Master's degree in computer science, information technology, or a related discipline.

Responsibilities

Manage PostgreSQL and MPP platform performance and operations
Design, implement, deploy, monitor, and troubleshoot complex data pipelines using Apache Airflow to process, transform, and load large-scale datasets reliably
Build, maintain, and continually improve data systems supporting scientific research, including relational databases and cloud-based lakehouse architectures
Ensure data accuracy, accessibility, observability, and reliability through proactive monitoring, alerting, and incident response
Collaborate with scientists, data engineers, and cross-functional teams to translate requirements into robust, scalable platform solutions

Requirements

Advanced expertise in PostgreSQL and Greenplum MPP
Strong proficiency with Apache Airflow
Hands-on experience with AWS cloud services
Strong Python programming skills and proficiency in SQL and SQL performance tuning
Extensive experience designing, building, and optimizing data pipelines at scale

Technologies

PostgreSQL
Greenplum
Apache Airflow
AWS
Python
SQL
Trino
Apache Iceberg
Lakehouse

Benefits

Employer retirement contribution – direct STScI contribution of 10% of your salary from your first day
12 days sick leave
Up to 24 days of vacation
10 paid holidays
Flexible work schedule with healthy work/life balance
Comprehensive medical, dental, vision, prescription plans, and more

Growth opportunities

Specialize in Trino for distributed querying
Work with Apache Iceberg for lakehouse management
Explore Greenplum or other MPP systems and large-scale performance tuning
Tackle complex challenges with high-volume scientific data, query optimization, and modern data infrastructure

View full job page →

21d ago

Data Analyst

Security Public Storage San Francisco, CA

San Francisco, CA

21d ago

$115k - $125k/yr Full time Analyst Analytics Azure Data Lake Storage

Apply Now

Job Description

The Data Analyst role at Security Public Storage is a full-time, hybrid position in San Francisco, CA, focused on revenue management, pricing strategy, and operational intelligence across a portfolio of 53 self-storage facilities. The role involves building and maintaining the data infrastructure the team relies on and using it to answer questions that shape pricing, promotions, and day-to-day operations.

Responsibilities

Assess property level performance, including occupancy, tenant turnover, pricing movements, and revenue yield.
Support portfolio pricing initiatives to balance revenue with occupancy targets.
Evaluate discounting and promotional programs such as move-in specials, concessions, and rate changes, and advise on deployment or reduction.
Develop frameworks to quantify promotional effectiveness, considering occupancy lift, revenue trade-offs, stay duration, and conversion rates.
Track competitive pricing and market trends within the self-storage sector.
Assist in forecasting revenue and occupancy outcomes.
Collaborate on and maintain existing revenue management models.
Identify opportunities to refine model logic in response to outcomes and evolving market conditions.
Partner with internal stakeholders to design, validate, and maintain the Microsoft Fabric data warehouse.
Create Power BI dashboards and reports to illuminate KPIs and portfolio-wide trends.
Ensure data accuracy and integrity across reporting systems.
Perform ad hoc analyses on pricing, promotions, and customer behavior.
Collaborate with operations and finance teams to translate data insights into actionable strategies.
Communicate findings clearly to non-technical stakeholders through polished reports and presentations.

Requirements

Advanced SQL skills with complex queries using common table expressions, window functions, and joins; T-SQL is preferred.
Strong data modeling fundamentals, including star and dimensional modeling, with clear relationships and cardinality.
ETL or ELT pipeline logic and data quality practices.
Experience in revenue analysis, pricing models, or tracking operational KPIs.
Familiarity with Microsoft Fabric components, including Lakehouses, Warehouses, data pipelines, and OneLake architecture.
Power BI proficiency covering DAX measures, calculated columns, data modeling, report design, and Row-Level Security (RLS).
Power Query / M for data transformation tasks.
Understanding of Direct Lake mode and performance optimization in Power BI.
Ability to translate business questions into data problems and conduct independent analyses.
Effective data storytelling to present findings to non-technical audiences.
Experience designing or validating measurement frameworks such as A/B tests, lift analyses, and conversion analyses.
Commitment to data integrity and documentation practices, including naming conventions and lineage tracking.

Technologies

SQL
T-SQL
Microsoft Fabric
Power BI
DAX
Power Query / M
DirectLake mode
OneLake architecture
Lakehouses
Warehouses
Azure
Python
Git

Benefits

401(k) with 4% employer match
Health insurance
Health savings account
Life insurance
Paid time off

Nice to Have

Python for data processing and notebook-based analysis
Experience with medallion architecture (Bronze to Silver to Gold) or lakehouse design patterns
Azure fundamentals, given Fabric runs on Azure
Git or version control for analytical assets
Background in real estate, self-storage, or asset-intensive industries
Experience with revenue management systems or dynamic pricing tools

Pay

Salary: USD 115,000 - 125,000 per year

Ability to Commute

San Francisco, CA 94107 (Required)

Work Location

In person

View full job page →

23d ago

Senior Data Engineer (Chinese Mandarin Speaker)

Bitus Labs Irvine, CA

Irvine, CA

23d ago

$130k+/yr Full time Python Senior AWS

Apply Now

Job Description

Bitus Labs is seeking a Senior Data Engineer who communicates in Chinese Mandarin to design and scale an AWS based data lakehouse, develop production-grade data pipelines in Java and Python, and guide data quality, governance, and platform decisions while mentoring engineers.

Responsibilities

Architect and implement scalable medallion architecture data lakehouses on AWS S3 using Apache Iceberg, covering Bronze, Silver, and Gold layers.
Build and maintain high-throughput ETL and ELT pipelines with AWS Glue, EMR (Spark), and Lambda.
Apply schema evolution, partitioning strategies, and Iceberg table compaction to optimize storage and query performance.
Deliver production-grade pipeline code in Java and Python, selecting the appropriate language for performance and maintainability.
Design and operate event driven data pipelines with Amazon Kinesis Data Streams, Kinesis Firehose, or Apache Kafka (MSK).
Implement streaming semantics including exactly-once and at-least-once processing using Apache Flink or Spark Structured Streaming on EMR.
Manage infrastructure as code using AWS CDK or Terraform to enable repeatable, auditable deployments.
Optimize cost and performance across AWS services such as S3, Glue, Athena, Redshift Spectrum, EMR, Lambda, Step Functions, and EventBridge.
Enforce data security best practices including IAM least-privilege policies, KMS encryption, VPC networking, and Lake Formation access controls.
Establish and maintain CI/CD pipelines for data workloads using AWS CodePipeline, GitHub Actions, or similar tools.
Implement data quality frameworks (for example Great Expectations, Deequ) and integrate validation steps into pipeline orchestration.
Define and enforce data contracts between producers and consumers to ensure data reliability.
Contribute to data cataloging and lineage tracking via AWS Glue Data Catalog or Apache Atlas.
Collaborate with data scientists, ML engineers, and analysts to deliver performant, well-documented datasets.
Mentor mid-level and junior engineers through code reviews, design discussions, and pair programming.
Document architecture decisions (ADRs) and contribute to the internal engineering knowledge base.

Requirements

Minimum 5 years of professional data engineering experience, including at least 3 years on AWS cloud platforms.
Proven track record delivering production data pipelines at scale (TB+ datasets, high throughput SLAs).
Experience with data lakehouse architectures using the medallion pattern and open table formats (Iceberg preferred; Delta Lake or Hudi acceptable).
Java proficiency (8+) for Spark jobs, Iceberg connectors, and performance-critical components; familiarity with Maven or Gradle.
Python proficiency (Python 3) for AWS Glue scripts, orchestration, data quality checks, and automation; experience with pandas, PySpark, boto3, and packaging best practices.
Storage and compute: S3, Glue (jobs, crawlers, Data Catalog), EMR (Spark/Flink), Lambda, EC2.
Streaming: Kinesis Data Streams, Kinesis Firehose, or MSK (Kafka).
Orchestration: Step Functions, MWAA (Managed Airflow), or EventBridge Scheduler.
Querying: Athena, Redshift, or Redshift Spectrum.
Security and governance: IAM, KMS, Lake Formation, Secrets Manager, and VPC.
DevOps: AWS CDK or CloudFormation; CodePipeline or comparable CI/CD tools.
Apache Spark (PySpark and Spark Java API) including distributed transformations, performance tuning, and memory management.
Apache Iceberg capabilities such as table maintenance, time travel, snapshot management, and partition evolution.
SQL expertise with advanced transformations, window functions, CTEs, and query optimization.

Technical Stack

Languages: Java 8+ and Python 3
Cloud: AWS (S3, Glue, EMR, Kinesis, Athena, Lambda, Step Functions, Lake Formation, CDK)
Processing engines: Apache Spark, Apache Flink, Spark Structured Streaming
Table formats: Apache Iceberg (primary); familiarity with Delta Lake or Hudi
Streaming: Kinesis Data Streams, Kinesis Firehose, MSK
Orchestration: MWAA, AWS Step Functions, or EventBridge
IaC and CI/CD: AWS CDK, Terraform; GitHub Actions, CodePipeline
Build tools: Maven, Gradle
Data quality and lineage: Great Expectations, Deequ; AWS Glue Data Catalog, Apache Atlas
Security and governance: IAM, KMS, Lake Formation, Secrets Manager, VPC
Additional: SQL, pandas, boto3

Compensation and Benefits

Compensation: USD 130,000 per year.

401(k) and matching
Dental insurance
Health insurance
Life insurance
Paid time off
Parental leave
Retirement plan
Vision insurance

Location, Language and Work Arrangement

Location: Irvine, California 92618, onsite
Language requirement: Chinese Mandarin is required
Work arrangement: In person

View full job page →

1mo ago

Senior Scientific Data Engineer

Lawrence Berkeley National Laboratory San Francisco Bay Area, CA

San Francisco Bay Area, CA

1mo ago

$139k - $174k/yr Full time Python Senior Data Engineer

Apply Now

Job Description

Lawrence Berkeley National Laboratory's Joint Genome Institute is seeking a Senior Scientific Data Engineer to help design and sustain the data platforms that underpin genomic data generation, metadata management, and laboratory operations. This hybrid role in the San Francisco Bay Area offers the opportunity to influence systems that span LMS/LIMS, data warehouse and lakehouse architectures, and project management workflows, with the possibility to lead major technical initiatives across diverse teams.

Responsibilities

Translate intricate scientific, operational, and user needs into concrete specifications, architecture, and implementation plans for system enhancements, integrations, and shared data platforms.
Design, develop, deploy, and support core systems, APIs, and workflows including the Laboratory Information Management System (LMS/LIMS), Data Warehouse/Data Lakehouse, and Proposal/Project Management platforms to enable genomic data generation, metadata management, and laboratory operations.
Lead the resolution of complex technical challenges, drive system improvements, and ensure production platforms are reliable, scalable, interoperable, and high-performing.
Advocate engineering best practices through technical reviews, documentation, mentorship, and ongoing refinement of development and operational processes.

Requirements

A Bachelor’s degree in Computer Science or a related field with a minimum of 8 years of professional experience developing, integrating, deploying, and supporting production software applications and data systems that enable metadata management, workflow orchestration, data lifecycle operations, and broad user access to scientific and operational data.
Experience with data engineering and event-driven technologies such as Airflow or Kafka (or equivalent).
Hands-on experience with relational databases, object storage, and systems supporting structured, semi-structured, and large-scale datasets.
Solid grounding in software and data engineering principles for large-scale production systems, including system design, APIs, testing, concurrency, reliability, scalability, interoperability, and performance optimization.
Proficiency in Python and experience with one or more additional programming languages.
Experience using AI-assisted development tools with sound judgment to evaluate generated code for production readiness.
Ability to provide technical leadership shaping system architecture and direction across cross-functional engineering teams.
Excellent communication skills, including the ability to present complex technical information to internal teams and stakeholders.
Demonstrated experience collaborating with stakeholders to understand project goals and translate them into automated systems, technical specifications, and implementation plans.

Technologies

Python
Airflow
Kafka
Laboratory Information Management System (LMS/LIMS)
Data Warehouse/Data Lakehouse
AI-assisted development tools

Benefits

Exceptional health benefits
Generous paid time off, sick time off, and holidays
A culture where you’ll belong, with investment in our teams
Relocation assistance
Hybrid work schedule eligibility

Why join Berkeley Lab?

Strong health benefits
Generous paid time off, sick leave, and holidays
Inclusive culture with support for teams

Additional Information

Application deadline: Priority consideration for applications with resume and cover letter by June 22, 2026; applications accepted until the posting is removed.
Appointment type: Full-time, exempt from overtime pay, monthly paid, two-year term with benefits eligibility, with potential extension or conversion to a Career appointment based on performance, funding, and operational needs.
Salary range: USD 139,440 - 174,312 annually, within the broader job code range; final salary determined by qualifications, experience, and alignment with peers.
Background check: A background check is required; convictions will be evaluated for relevance to responsibilities; a history does not automatically disqualify.
Work modality: Hybrid schedule with on-site work at Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA 94720; must reside within 150 miles; rare cases of full remote work may be considered; Real ID or other acceptable identification required for access.
Relocation assistance: Eligible
Work authorization: Must be legally authorized to work in the United States; Berkeley Lab does not provide visa sponsorship for this position.

View full job page →

1mo ago

Data Analytics Engineer

Murdoch's Ranch & Home Supply Bozeman, MT 🇺🇸 Remote

Bozeman, MT

1mo ago Closed

Full time Analytics Azure Business Analytics

This position is no longer accepting applications (closed Jul 10, 2026).

Job Description

The Data Analytics Engineer at Murdoch's Ranch & Home Supply focuses on Microsoft Fabric Lakehouse architecture, Power BI, and data analytics across sales, inventory, customer, and finance to drive insights and decision-making.

Responsibilities

Collaborate with stakeholders to understand business requirements and translate them into scalable data analytics solutions.
Apply in-depth knowledge of the Microsoft Fabric ecosystem, including Lakehouse, PySpark/SQL notebooks, data pipelines, and deployment pipelines; understanding Microsoft Dynamics as a data source is a plus.
Design data models bridging the Fabric Lakehouse, Power BI semantic models, and reporting layers to ensure consistency across raw, curated, and business-facing data.
Optimize data models for storage, retrieval, and analysis while ensuring data integrity, accuracy, and reliability; knowledge of Fabric capacity optimization is advantageous.
Implement data integration pipelines using Microsoft Fabric and other relevant tools to extract, transform, and load data from diverse sources into a unified data platform.
Create interactive dashboards, reports, and visualizations using Power BI and DAX to enable actionable insights.
Identify performance bottlenecks and implement optimizations to improve data processing, query performance, and overall system efficiency.
Collaborate with cross-functional teams, providing technical leadership and guidance to data engineers, analysts, and other stakeholders to ensure successful project delivery.
Stay current with the latest advancements in data analytics technologies and industry trends, applying best practices in solution design and implementation.

Requirements

Experience as a Data Engineer owning core data pipelines, with a primary focus on Microsoft Fabric and Azure technologies.
Extensive knowledge and hands-on experience with Microsoft Fabric, including medallion architecture using Lakehouse, PySpark notebooks, data and deployment pipelines.
Experience designing and maintaining scalable semantic models (Power BI / Tabular / DAX) to provide a consistent, governed analytics layer across the business.
Establish and maintain best practices for version control, CI/CD, testing, and deployment of data pipelines and semantic models.
Proficiency in data visualization techniques using Power BI / DAX.
Strong problem-solving skills and the ability to design scalable and efficient data analytics architectures.
Excellent understanding of data warehousing concepts, data governance, and data quality principles.
Solid knowledge of data security and compliance considerations in the context of data analytics.
Strong customer service focus for assisting the team and business functions with their data analytics needs.
Ability to present, facilitate, and participate in requirements elicitation and brainstorming sessions with internal and external teams in small and medium group environments.
Track record of building impactful solutions that you are proud to stand behind.

Technologies

Microsoft Fabric
Lakehouse
PySpark
SQL notebooks
Data pipelines
Deployment pipelines
Microsoft Dynamics
Power BI
DAX
Azure
Tabular

Benefits

Amazing Employee Discount on everything from clothes to power tools to pet food.
Multiple Health Insurance options to best suit your needs after 60 days.
Dental & Vision options to complete your health plan.
Life Insurance to secure your family’s financial future.
Paid Vacation allows you to relax and recharge.
Holiday Pay throughout the year so you can celebrate with your family.
Paid Sick Time empowers you to stay home and focus on your health without losing pay.
401(k) with a generous 4% company match to help plan for retirement.
Paid Parental Leave to take time to bond with your family’s new addition.
Community Giving Program matches your donations and provides paid volunteer hours.
Wellness Program saves you money by lowering medical premiums with credits earned.
Training Program helps you expand your knowledge and skills with over 250 courses.
Other various Voluntary Insurance Options.

Physical Demands

Sitting for long periods of time.
Repetitive wrist movements on keyboard.
Close vision for PC work.
Bending, carrying, and pushing.
Reasonable accommodations may be made to enable individuals with disabilities to perform the essential job functions.