Sr Data Engineer

The Sr Data Engineer will join the founding team of Baker Tilly’s AI Product Group (APG) to build scalable, secure, and reliable data infrastructure that supports AI applications and intelligent agents.

Role Overview

This position centers on governed data foundations and production-grade data pipelines designed for AI use cases. You will work across lakehouse architecture, knowledge graph and retrieval capabilities, and evaluation-driven data engineering, with an emphasis on data quality, lineage, and operational reliability.

Responsibilities

• Design, develop, implement, and maintain scalable batch and incremental data pipelines that transform structured and unstructured information into clean, validated, business-ready datasets within a modern lakehouse environment.
• Build and optimize lakehouse architecture, including data processing and transformation layers that apply modern principles such as medallion architecture, data modeling, and integration of enterprise and legacy data sources.
• Implement automated validation, reconciliation, lineage tracking, and metadata management to support data accuracy, consistency, traceability, and reliability across the data lifecycle.
• Develop data models, data contracts, and entity-resolution capabilities to establish consistent client, engagement, and document identities across enterprise data sources.
• Support data governance and access controls through data cataloging, classification, permissions, and access management for sensitive information used by authorized users and AI agents.
• Build and maintain knowledge graph structures, data relationships, search indexes, and retrieval pipelines that enable AI applications to access relevant, accurate, and properly governed enterprise information.
• Create pipelines that capture AI agent activity, practitioner feedback, reviews, and corrections to support ground-truth datasets, AI evaluation, and continuous improvements in data and retrieval quality.
• Establish evaluation-based data engineering practices, including quality standards, testing frameworks, benchmarks, and validation processes for extraction, retrieval effectiveness, and grounding accuracy.
• Maintain data freshness and reliability using incremental processing, change data capture, versioning, and monitoring so data products, knowledge graphs, and search indexes remain current as sources evolve.
• Deploy, monitor, troubleshoot, and maintain production data pipelines using engineering best practices such as automated testing, code reviews, and operational documentation.
• Collaborate with external engineering partners to transition data solutions into internally owned environments, including reusable code, tests, and runbooks.
• Partner with AI Engineering, Platform & Production Engineering, enterprise data teams, subject matter experts, and engineering teams in India to align solutions with enterprise standards and business objectives.
• Participate in special projects and other assigned duties to support the firm’s AI and data platform capabilities.

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related field; advanced degree preferred.
• 3 years of hands-on data engineering or software engineering experience, including demonstrated success developing, deploying, and maintaining production data pipelines.
• Experience independently building and operating production pipelines on a modern lakehouse platform, with ownership of data processing, transformation, and operational reliability.
• Experience implementing data quality, lineage, or data contracts in production pipelines, including automated testing, reconciliation, validation, or traceability practices.
• Strong proficiency in Python and SQL, including the ability to develop, optimize, test, and maintain production-quality data engineering solutions.
• Strong understanding of data modeling principles and the ability to design scalable, maintainable data structures for evolving requirements and source systems.
• Experience designing and developing batch and incremental data pipelines for structured and unstructured enterprise data.
• Solid software engineering fundamentals, including version control, automated testing, code reviews, debugging, and reliable, repeatable data processing workflows.
• Ability to validate technical solutions with measurable outcomes such as automated tests, benchmarks, experiments, performance metrics, or comparative evaluations.
• Strong preference for experience with modern lakehouse technologies such as Databricks, Apache Spark, Delta Lake, or medallion architecture.
• Preference for experience with cloud-based data platforms, preferably Microsoft Azure.
• Preference for familiarity with enterprise data governance capabilities including data catalogs, classification, access controls, lineage, and technologies such as Unity Catalog or comparable solutions.
• Preference for knowledge graphs and related tooling such as graph databases, graph data modeling, vector search, hybrid search, or retrieval-based applications.
• Preference for familiarity with unstructured data processing, including document extraction, metadata management, content chunking, and search indexing.
• Preference for experience with entity resolution, master data management, change data capture, or incremental data processing.
• Understanding of data provenance, traceability, and governance requirements in complex enterprise or regulated environments.
• Excellent written and verbal communication skills, including documentation and collaboration with technical and nontechnical stakeholders.
• Demonstrated ability to work with business subject matter experts and globally distributed stakeholders to translate requirements into reliable data solutions.
• Ability to manage competing priorities and adapt quickly in a fast-paced, collaborative environment.
• Experience using AI-assisted and agentic development tools to improve engineering productivity, or willingness to incorporate them into daily development practices.
• No professional certifications or licenses required; prior experience with AI, machine learning, LLM systems, or knowledge graphs is not required.

Preferred Skills & Tools

• Python, SQL
• Databricks, Apache Spark, Delta Lake
• Medallion architecture
• Microsoft Azure
• Unity Catalog
• Knowledge graphs, graph databases, graph data modeling
• Vector search, hybrid search, retrieval-based applications
• Document extraction, content chunking, search indexing
• Entity resolution, master data management, change data capture, incremental data processing
• LLM systems

Location

Santa Clara, CA (onsite)

Compensation

USD 185,000 - 230,000 per year

National pay rate range: $185,000 – $230,000. In California: $185,000 – $230,000.