Azure Data Engineer

Skywalk Global is seeking an Azure Data Engineer for an onsite role supporting reliable data products across integration, pipelines, modeling, governance, and documentation.

Responsibilities

• Support data technology strategy, including resolving data integration issues related to data quality and stewardship across secure systems
• Provide direction and oversight for design and implementation of data solutions for integration and synthesis across multiple secure systems
• Assist in defining, implementing, and maintaining data management policies and engineering design standards
• Advise, train, and provide technical support for projects
• Participate in requirements gathering, solution reviews, and explaining technical complexities to non-technical stakeholders
• Collaborate with solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation
• Extract, transform, and load data from a variety of sources using data engineering tools and methods
• Design, build, and maintain batch and streaming Azure Databricks pipelines using PySpark, Spark SQL, and Delta Lake following medallion (bronze/silver/gold) architecture
• Develop, load, and tune dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboards
• Orchestrate and monitor workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery
• Apply data governance and security controls to protect PHI/PII in line with HIPAA and agency policy (including Unity Catalog permissions, role-based access, row- and column-level security, and data masking)
• Document and demonstrate solutions through documentation, flowcharts, layouts, diagrams, charts, code comments, and clear code
• Support operations by performing systems analysis and recommending changes in policies and procedures

Requirements

• 4+ years experience required, supported by one of the following education/experience paths: a Master’s degree in a related field (data science, analytics, informatics, computer science, mathematics, engineering, or related) OR a Bachelor’s degree plus 2+ years relevant experience OR 4+ years hands-on experience in a similar data engineering role
• Relevant certifications and coursework may substitute for formal education/experience requirements (still requiring 4 years)
• Developing with Databricks notebooks using PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions
• Experience with CID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance such as OPTIMIZE, Z-order, and VACUUM
• Working with medallion architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT
• Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials
• Right-sizing clusters and SQL warehouses and diagnosing partitioning, shuffle, skew, and caching issues using Spark UI
• Scheduling and managing dependencies with ADF Pipelines and Databricks Workflows (Jobs), including using Git folders (Repos) for source control
• Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning
• Querying Parquet, Delta, and CSV in ADLS Gen2 using OPENROWSET, external tables, and views
• Building star-schema facts and dimensions, plus stored procedures and views, including SCD Type 1 and SCD Type 2
• Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)
• Experience with workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing
• ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault
• Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles
• Monitoring pipelines and workloads with Azure Monitor/Log Analytics, alerting, and troubleshooting failed jobs
• Strong written and verbal communication skills
• Understanding Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks

Location

• Indianapolis, IN (onsite)
• Local candidates only will be considered

Technologies

• Azure Databricks, PySpark, Spark SQL, Delta Lake
• Medallion (bronze/silver/gold) lake house architecture
• Azure Synapse Analytics, SQL pools
• Databricks Workflows/Jobs, Azure Data Factory
• Unity Catalog, HIPAA, Unity Catalog permissions, role-based access, row- and column-level security, data masking
• Databricks notebooks, R
• CID transactions, MERGE upserts, schema enforcement/evolution, time travel, OPTIMIZE, Z-order, VACUUM
• Auto Loader, Structured Streaming, Lakeflow/DLT
• Spark UI, ADF Pipelines
• MPP warehouse tables, Parquet, Delta, CSV, ADLS Gen2, OPENROWSET, external tables, views
• Star-schema, SCD Type 1, SCD Type 2, stored procedures
• COPY INTO, PolyBase, CETAS, data flows, triggers
• Materialized views, result-set caching, Azure Monitor, Log Analytics
• Entra ID, managed identities, Key Vault
• Azure DevOps, GitHub, Git (Repos), Databricks Asset Bundles
• Microsoft lakehouse patterns, Power BI