Data Lakehouse

Data Lakehouse Jobs

9 jobs found

1d ago

Data Analytics Engineer

Apple Cupertino, CA

Cupertino, CA

1d ago

$147k - $272k/yr Full time Analytics Apache Airflow Big Data

Apply Now

Job Description

Apple invites a Data Analytics Engineer to join the Apple Games App Data Engineering team in Cupertino, CA, onsite, to develop analytics solutions and derive insights from large-scale gaming data with emphasis on privacy, scalability, and collaboration with developers and partners.

Responsibilities

In this role within the Apple Games App Data Engineering group, you will help steer the team's strategic priorities.
This position centers on software engineering.
We develop robust production-grade software rather than ad hoc scripts.
Our systems process dozens of terabytes of gaming and player engagement data daily.
Solutions must be efficient, scalable, and reliable.
Data is the primary product and central concern of our work.
We value accurate, high-quality data as much as we value clean, maintainable code.
The insights generated support Apple leadership and game developers with understanding player behavior, tracking innovations, and identifying future gaming trends.
You will contribute to building analytics solutions tailored for game developers, leveraging experience with the third-party gaming analytics ecosystem to create leading tools.
Success here requires championing world-class analytical capabilities.
As a team member, your ideas, concerns, and viewpoints are welcomed in collaborative discussions.
Our culture is highly collaborative.
You will work closely with engineering, product, and partner teams.
Privacy is a core engineering requirement and our solutions are designed to align with Apple privacy principles from the outset.
The team is compact and focused, with engineers addressing a broad range of problems.

Requirements

Strong foundation in software engineering principles, including algorithms and data structures.
Self-driven, motivated, with the ability to produce architecture and design documents.
Ability to translate cross-functional requirements into concrete engineering tasks.
Excellent communication skills and demonstrated experience working across functional teams.
Bachelor's degree in Computer Science, Engineering, or an equivalent combination of education and experience.
Experience building analytics and data solutions targeted at game developers.
Familiarity with third-party analytics tools and platforms used in the gaming industry.
More than five years of hands-on experience building distributed data processing applications with Apache Spark or Apache Flink.
More than five years of programming experience in Scala (preferred) or Java.
Experience with big data technologies including Hadoop, HDFS, Spark, SQL, and Kafka.
Proven ability to design scalable, highly available distributed systems using Kafka, Iceberg, Kubernetes, Airflow, and Cassandra.

Technologies

Apache Spark
Apache Flink
Scala
Java
Hadoop
HDFS
Spark
SQL
Kafka
Iceberg
Kubernetes
Airflow
Cassandra

Compensation

Base pay range: USD 147,400 - 272,100 per year. This range depends on skills, qualifications, experience, and location.

Benefits

Discretionary employee stock programs, including RSUs.
Employee Stock Purchase Plan allowing purchase of Apple stock at a discount.
Comprehensive medical and dental coverage.
Retirement benefits.
A range of discounted products and free services.
Educational expense reimbursement, including tuition.
Discretionary bonuses or commission payments.
Relocation eligibility.

View full job page →

3d ago

Senior Data Engineer - Financial Transactions & Automation

NVIDIA Santa Clara, CA

Santa Clara, CA

3d ago

$184k - $288k/yr Full time Senior Azure Databricks Cloud

Apply Now

Job Description

Join NVIDIA in Santa Clara, California, onsite, as a Senior Data Engineer within the Financial Systems Engineering group. You will help build foundational, scalable data platforms that support NVIDIA’s high performance financial infrastructure and cloud services, enabling automated operations across global finance by integrating diverse systems. The package includes equity and a comprehensive benefits program, with a salary range of USD 184,000 - 287,500 per year.

Responsibilities

Pipeline Architecture & Integrity: design event driven pipelines using Kafka and craft data models that preserve transactional ACID properties for invoices, payments, and adjustments.
Scalable Data Solutions: automate scalable ETL processes and refactor modern data architectures to enhance quality, security, and coverage for rapidly growing business needs.
Operational Excellence: work cross functionally to codify business processes into self measuring systems and troubleshoot complex challenges to maintain reliable financial operations.

Requirements

Bachelor’s degree (or equivalent experience); Master’s preferred.
10+ years of industry experience delivering scalable financial services.
Expertise in building scalable REST APIs backed by PostgreSQL.
Proficiency in Python, Golang, Scala, or NodeJS.
Hands on experience architecting robust ETL/ELT pipelines using Databricks and dbt to manage large scale Delta Lake or Apache Iceberg tables.
Deep understanding of high scale distributed systems, Linux, and cloud infrastructure (AWS/GCP/Azure) using container technologies like Docker and Kubernetes.

Technologies

Kafka
PostgreSQL
Python
Golang
Scala
NodeJS
Databricks
dbt
Delta Lake
Apache Iceberg
Linux
AWS
GCP
Azure
Docker
Kubernetes

Benefits

Equity
Benefits

Ways to Stand Out from the Crowd

Advanced Data Modeling: experience developing data ontology or semantic data layers that map disparate raw data into interconnected objects with defined properties and functions.
Financial Domain Scale: a track record of managing financial or ERP data at internet scale, ensuring transactional integrity and zero data loss.
Modern Operations: an Infrastructure as Code mindset with hands on experience building self healing systems in distributed environments.

View full job page →

3d ago

Senior Data Engineer - Vice President

Citi Irving, TX

Irving, TX

3d ago

$126k - $189k/yr Full time Senior Big Data Bigdata

Apply Now

Job Description

Join Citi in Irving, TX for an on-site Senior Data Engineer - Vice President role that centers on collaborative, client-focused data solutions. The position offers a comprehensive benefits package including medical, dental, and vision coverage; a 401(k); life, accident, and disability insurance; wellness programs; paid time off, sick leave, and holidays. The role is full-time with a salary range of USD 125,760 to 188,640 per year. Requires a bachelor’s degree and a minimum of 6 years of hands-on data engineering experience, with leadership responsibilities in an enterprise setting.

Benefits

Medical, dental, and vision coverage
401(k)
Life, accident, and disability insurance
Wellness programs
Planned time off (vacation)
Unplanned time off (sick leave)
Paid holidays

Responsibilities

Architect and sustain scalable ETL/ELT pipelines using PySpark, Spark SQL, and Delta Lake on Databricks to efficiently ingest, transform, and integrate large-scale datasets across cloud environments.
Cloud Data Platform Management: design, implement, and manage data solutions on cloud platforms (AWS, GCP, Azure) leveraging cloud-native services for storage, processing, and analytics.
Big Data Technologies: work with Databricks, Snowflake, and open table formats such as Apache Iceberg to process petabyte-scale datasets.
Optimize Spark workloads and Databricks clusters by tuning jobs, partitioning strategies, caching, and autoscaling to boost performance and manage costs.
Implement and govern Lakehouse architecture with Delta Lake, enforcing data quality, schema evolution, and governance via Unity Catalog for reliable analytics and downstream use.
Lead the design and architecture of Starburst-based data solutions to ensure enterprise-level scalability, performance, and reliability.
Execute data federation strategies using Starburst connectors to unify access across data lakes, RDBMS, NoSQL, and cloud storage.
Performance Optimization: identify bottlenecks in pipelines and queries, improving storage and processing efficiency.
Develop and optimize robust data pipelines with a strong emphasis on data governance, ensuring quality, lineage, and compliant data flow from ingestion to consumption.
Data Modeling and Architecture: design data models that support BI, analytics, and ML use cases, building a robust, secure, scalable data architecture.
AI and Machine Learning Collaboration: partner with data scientists to support AI model development and deployment, contributing to RAG and Agentic AI initiatives through data infrastructure support.
Agile Methodology: operate effectively in an Agile environment, participating in sprint planning, daily stand-ups, and retrospectives to deliver milestones.
Leadership and Project Guidance: provide technical leadership, mentor junior engineers, and influence architectural decisions aligned with client needs and strategic goals.
Stakeholder and Client Interaction: act as a primary point of contact for stakeholders and clients, communicating progress and translating business requirements into actionable technical tasks.

Requirements

Python: expert-level proficiency with Python and its data ecosystem (Pandas, NumPy, Dask); experience delivering production-grade data processing, automation, and API development.
PySpark: extensive hands-on experience with the Spark framework, deep knowledge of the DataFrame API, Spark SQL, and performance tuning for distributed processing.
Databricks: proven work on the Databricks Lakehouse Platform, including Delta Lake, structured streaming, and optimizing Spark jobs within the environment.
Ab Initio: strong practical experience with the Ab Initio suite (GDE, Co>Operating System, Conduct>It) for enterprise-grade ETL workflows.
Snowflake: hands-on experience designing and maintaining Snowflake data warehouses, including modeling, RBAC, performance tuning, Snowpipe, and Time Travel.
Starburst/Trino: experience with federated query engines to provide unified access across diverse data sources and data systems.
Apache Iceberg: familiarity with open table formats for managing large analytic datasets.
Cloud provider experience: multi-year, in-depth experience with at least one major cloud provider (AWS, GCP, or Azure).
Cloud-native data pipelines: practical experience building pipelines with cloud-native services such as AWS Glue, Lambda, S3, Redshift; Azure Data Factory, Synapse; or Google Cloud Composer, Dataflow, and BigQuery.
Data lifecycle for ML: solid understanding of the data lifecycle required for machine learning projects.
AI data pipelines: experience building data pipelines to support AI/ML models, with interest or exposure to preparing data for advanced AI applications such as vector databases for RAG and Agentic AI.
Agile Proficiency: strong familiarity with Agile and Scrum methodologies and iterative delivery.
Leadership & Influence: demonstrated ability to provide technical leadership and guide architectural decisions aligned with client needs and long-term strategy.
Client Engagement: excellent communication and interpersonal skills, capable of articulating complex concepts to diverse audiences and building stakeholder relationships.
6-10 years of hands-on data engineering experience, preferably in a large-scale enterprise or financial services environment.
Experience leading project work streams and mentoring junior team members.
Relevant industry certifications (e.g., AWS Certified Big Data, Google Professional Data Engineer, Snowflake SnowPro).
Containerization and orchestration: experience with Docker and Kubernetes.
Deep understanding of data governance, data quality, and data security principles.
Strong analytical and problem-solving skills with the ability to work independently or as part of a team.
Experience as Applications Development Manager and senior-level experience in an Applications Development role, with stakeholder and people management experience.
Proven project management skills and a solid grasp of industry practices and standards.
Clear and concise written and verbal communication.

View full job page →

3d ago

Databricks Data Engineer

Deloitte Arlington, VA

Arlington, VA

3d ago

$87k - $171k/yr Full time Artificial Intelligence Azure Databricks Big Data

Apply Now

Job Description

Deloitte's Databricks Data Engineer role is based onsite in Arlington, VA, within the Core AI & Data practice. The position centers on designing, building, and optimizing cloud-based data engineering solutions that leverage Databricks to support large-scale transformations, analytics, and AI initiatives. The salary range is USD 86,700 to 170,900 per year.

Responsibilities

Design, build, and sustain data pipelines and engineering solutions using Databricks, Apache Spark, Python, and SQL.
Support modernization of enterprise data platforms by implementing cloud-based ingestion, transformation, and storage patterns.
Develop and manage batch and streaming data workflows that support reporting, analytics, and AI use cases.
Apply data architecture, integration, and data modeling practices to improve data quality, scalability, and platform performance.
Collaborate with client and Deloitte teams to troubleshoot issues, implement enhancements, and support delivery of large-scale data transformation programs.
Demonstrate the following skills expected for the role:
Ability to work independently and as part of a team
Effective written and verbal communication skills
Meticulous attention to detail and quality of work product
Ability to build and sustain professional relationships
Ability to lead projects or workstreams
Ability to manage and prioritize multiple tasks in a fast-paced and dynamic environment
Strong interpersonal skills and professional demeanor
Ability to meet deadlines
Ability to provide clear guidance to others

Requirements

Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, Mathematics, or Statistics
3+ years of experience in data engineering, ETL development, or data platform engineering
2+ years of experience building data pipelines in Databricks using Apache Spark, Python, and SQL
Experience implementing data models, data integration patterns, and cloud-based data processing workflows
Experience working with at least one cloud platform: AWS, Azure, or GCP
Ability to travel 50% on average depending on client engagements
Limited immigration sponsorship may be available

Technologies

Databricks
Apache Spark
Python
SQL
Delta Lake
Databricks Workflows
Unity Catalog
Git-based version control
CI/CD pipelines
AWS
Azure
GCP

Preferred

Experience with Delta Lake, Databricks Workflows, and Unity Catalog
Experience using Git-based version control and CI/CD pipelines
Experience supporting batch and streaming data workloads
Experience with data warehousing, lakehouse architecture, and performance optimization
Experience supporting analytics or machine learning data pipelines
Experience working in consulting or client service delivery environments

Accommodation Information

Information for applicants with a need for accommodation: https://www2.deloitte.com/us/en/pages/careers/articles/join-deloitte-assistance-for-disabled-applic...

View full job page →

7d ago

Senior Analytics Engineer, People Data

Anduril Industries Costa Mesa, CA

Costa Mesa, CA

7d ago

$166k - $220k/yr Full time Senior Analytics Apache Airflow

Apply Now

Job Description

Anduril Industries is seeking a Senior Analytics Engineer focused on People Data to design and maintain the data infrastructure, models, and pipelines that power analytics across HR systems. This onsite role in Costa Mesa, CA centers on turning disparate people data into reliable, accessible insights to support data-driven decisions within the People organization. The position offers a salary range of USD 166,000 to 220,000 per year and requires a strong background in data engineering to scale our analytics capabilities.

Responsibilities

Design, build, and optimize robust ETL/ELT pipelines to reliably ingest, integrate, and transform diverse people data from various HR systems (HRIS, ATS, LMS, etc.) into our data platform.
Develop, maintain, and govern scalable and secure data models, schemas, and ontologies specifically for people analytics, ensuring data quality, consistency, and accessibility for downstream consumption.
Contribute to the strategic design, development, and evolution of our people data platform and tooling, advocating for engineering best practices, automation, and a scalable analytics ecosystem (e.g., leveraging SQLMesh, Iceberg, Flyte).
Partner closely with People Analysts, HR Business Partners, and other stakeholders to understand their analytical needs and translate them into robust data solutions, providing well-structured, documented, and reliable datasets.
Implement and monitor data quality checks, identify discrepancies, troubleshoot data issues, and ensure the reliability and integrity of people data across all systems.
Continuously monitor the performance of data pipelines and models, identifying bottlenecks and implementing solutions to ensure the efficiency and scalability of our people data infrastructure.
Create and maintain comprehensive documentation for data pipelines, models, and processes, and champion data engineering best practices (e.g., version control, testing, CI/CD) within the team.
Implement and enforce strict data security measures and ensure all data handling practices comply with internal policies and external regulations (e.g., GDPR, CCPA) related to employee data privacy.
Collaborate with broader enterprise analytics and data engineering teams to align on data architecture standards, integrate people data with other business domains, and contribute to the overall evolution of the company's data platform.

Requirements

5+ years of progressive experience in Data Engineering, Analytics Engineering, or a similar role focused on building and optimizing data pipelines and data infrastructure.
Expert-level proficiency in SQL for complex data manipulation and querying, and advanced Python for scripting, data processing, and automation.
Extensive experience with cloud-based data warehousing solutions (e.g., Snowflake, Google BigQuery, AWS Redshift, Databricks/Delta Lake) and data lake technologies (e.g., AWS S3, Azure Data Lake Storage).
Deep understanding and proven experience in designing, implementing, and maintaining robust data models (e.g., dimensional modeling, Kimball methodology) for analytical purposes.
Hands-on experience building and optimizing complex ETL/ELT processes and data pipelines using modern tools such as dbt, Apache Airflow, Flyte, Dagster, or similar orchestration platforms.
Excellent communication skills, both written and verbal, with the ability to translate technical concepts for non-technical stakeholders and collaborate effectively across diverse teams.
Bachelor's degree in Computer Science, Engineering, Data Science, Information Systems, or a related field.
Strong technical foundation in data engineering, with expertise in SQL, Python, and experience working with cloud-based data platforms (AWS, GCP, Azure).

Technologies

SQL, Python
Snowflake
Google BigQuery
AWS Redshift
Databricks/Delta Lake
AWS S3
Azure Data Lake Storage
dbt
Apache Airflow
Flyte
Dagster
Iceberg
SQLMesh
Palantir Foundry
Terraform
CloudFormation
Docker
Kubernetes
Tableau
Power BI
Looker
Rippling
Workday
Oracle HCM Cloud
Apache Spark
Flink

Benefits

Equity grants included in the majority of full-time offers and are part of total compensation
Comprehensive benefits package for full-time employees, including health and recovery support

Preferred Qualifications

Experience with big data processing frameworks (e.g., Apache Spark, Flink) and advanced data warehousing features like schema evolution or time travel (e.g., Iceberg, Delta Lake).
Direct hands-on experience with tools mentioned in our stack like Palantir Foundry, SQLMesh, Flyte, or similar cutting-edge data orchestration and transformation platforms.
Relevant professional certifications from major cloud providers (e.g., AWS Certified Data Analytics - Specialty, Google Cloud Professional Data Engineer).
Experience with Terraform, CloudFormation, Docker, or Kubernetes for managing data infrastructure and deploying applications.
Familiarity with integrating data pipelines with leading BI tools (Tableau, Power BI, Looker) to optimize dashboard performance and data accessibility for end-users.
Deep expertise working with data from specific enterprise HRIS systems like Rippling, Workday, and Oracle HCM Cloud including their data models and APIs.
Solid understanding of HR data concepts, metrics, and common HR systems (HRIS, ATS, LMS), with a strong interest in People Analytics.

View full job page →

10d ago

Forward Deployed Data Engineer

Spear AI Groton, CT 🇺🇸 Remote

Groton, CT

10d ago

Full time Apache Airflow Automation AWS

Apply Now

Job Description

Benefits and culture

Based in Groton, CT with remote options, this role involves on-site collaboration with customers and requires a U.S. Secret clearance. Spear AI offers a benefits rich, growth oriented environment that values practical impact, direct leadership involvement, and a collaborative, autonomous culture.

Unlimited paid time off to recharge and maintain work life balance
Dedicated sick time to support health and wellbeing
Comprehensive medical, dental, and vision coverage
Eleven paid holidays
Professional development opportunities and certification reimbursement
Collaborative environment with direct access to leadership in a flat structure
Mission driven projects with real world impact
Growth opportunities during a period of expansion
401(k) plan with company match
Onsite, remote, or hybrid work arrangements depending on the role
Relocation assistance where applicable
Referral bonuses and performance bonuses
Life insurance and disability coverage
Home office setup stipend
Professional certification reimbursement, position dependent

Responsibilities

Build real-time data pipelines using MQTT and Redpanda for stream processing
Develop offline data pipelines with Dagster for batch workflows
Parse and process binary message formats from diverse data sources
Construct data warehouses leveraging Postgres, Apache Iceberg, Parquet, and S3
Design data models optimized for high performance queries
Validate and normalize incoming data sources
Improve local development and CI/CD with modern tooling and GitHub Actions

Requirements

Current or active U.S. Secret clearance
Expertise in time-series data processing and analysis (windowing, resampling, interpolation)
Proficiency in Python and Rust for data engineering workflows
Experience with binary message parsing
Familiarity with row-based and columnar data formats
Experience with OLTP and OLAP databases
Knowledge of distributed systems, streaming architectures, and batch processing patterns
Hands-on experience with batch orchestrators such as Dagster or Airflow
Hands-on experience with streaming platforms such as Redpanda or Kafka
Hands-on experience with binary message formats such as Protobuf

Technology stack

MQTT
Redpanda
Dagster
Apache Iceberg
Parquet
Postgres
S3
Python
Rust
Protobuf
Airflow
GitHub Actions
Kafka

Nice to have

Experience with IoT devices and sensors
Digital signal processing experience
Geospatial analysis and GIS experience
Familiar with working in monorepos

Why work with us

We ship quickly on meaningful projects rather than long, inconsequential cycles
Our work has tangible impact, including deployments related to subsystems and integrations with associated hardware
Responsible, values driven growth during an expansion phase
Remote friendly with real time collaboration on Slack and asynchronous work via GitHub
Profitability and investor backing support sustainable success
Autonomy to focus on quality work without excessive gatekeeping
Light hearted culture while building safety focused products

View full job page →

12d ago

Senior Software Engineer - Data & Analytics- Contingent

Experis Charlotte, NC

Charlotte, NC

12d ago

$76 - $76/hr Contract Senior Apache Spark Bash Scripting

Apply Now

Job Description

This contingent Senior Software Engineer role centers on data and analytics within a large enterprise, based onsite in Charlotte, NC. The position is structured as a contract engagement with an hourly rate of USD 76. The successful candidate will design, build, and maintain high‑performance data pipelines and translate business requirements into technical solutions using SQL, PySpark, and Python.

Responsibilities

Design, develop, and maintain scalable, high‑performance data pipelines within a large enterprise environment
Create and optimize data processing solutions using SQL, PySpark, and Python
Own assigned deliverables end to end, from design through production support
Drive prior initiatives to completion, clearly communicating status, risks, and dependencies
Collaborate closely with product managers and upstream/downstream teams to translate business requirements into technical solutions

Requirements

Seven or more years of overall experience as a Software Engineer
Four or more years of experience building large data pipelines
Hands‑on experience with Apache Spark, Hive, and Iceberg tables
Strong programming proficiency in Python, PySpark, SparkSQL, Scala, and Bash scripting
Experience with CI/CD pipelines, code quality improvement, and vulnerability remediation