Data Engineer - Databricks / Azure Consultant
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Integration Tools
Data Lakehouse
Data Management
Data Pipeline
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Mlflow
ETL
Informatica
Information Technology (IT)
Pyspark
Software Development
Spark
SQL
Job Description
Yoh, A Day & Zimmermann Company is seeking a Databricks and Azure Consultant to help design, implement, and support an enterprise-grade Databricks data platform on Microsoft Azure. This hybrid role in Jersey City brings together data engineering with cloud platform, security, automation, and operational best practices.
What you will do
- Design, implement, and support the Databricks data platform, including ongoing development
- Partner with technology and data teams to build and improve scalable data solutions using Databricks on Microsoft Azure
- Concentrate on data engineering, integration, security, automation, and platform best practices
- Collaborate across application development, infrastructure, cloud, security, architecture, and data teams
What you bring
- Azure, Databricks, and Apache Spark
- CI/CD experience
- Hands-on experience with Databricks Workspace, Unity Catalog, and Delta Lake
- Hands-on experience with Spark UI and MLflow
- Strong development experience with Python, PySpark, SQL, and/or Java
- Experience developing and optimizing data processing and ETL/ELT pipelines
- Strong Microsoft Azure experience, especially Azure Databricks
- Strong Microsoft Azure experience with Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Data Factory
- Experience with Git and CI/CD pipelines
- Experience with OpenShift and Kubernetes
- Experience with Terraform and Infrastructure as Code (IaC)
- Experience implementing or supporting Databricks security, access controls, secrets management, and identity integration
- Experience implementing or supporting data governance and Unity Catalog
- Experience with Databricks workflows, job orchestration, and monitoring
- Experience with troubleshooting and performance optimization, including production support
- Ability to collaborate across technical teams and operate within enterprise architecture, security, change-management, and governance standards
- Knowledge of Apache Airflow
- Desirable: experience with AWS (including S3 and IAM)
Technologies involved
- Azure, Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory
- Databricks, Databricks Workspace, Unity Catalog, Delta Lake, Databricks workflows
- Apache Spark, Spark UI, Python, PySpark, SQL, Java
- MLflow, ETL/ELT
- CI/CD
- Git
- OpenShift, Kubernetes
- Terraform, Infrastructure as Code (IaC)
- Databricks security, Databricks access controls
- Apache Airflow
- AWS, S3, IAM
Location and schedule
- Jersey City, NJ (hybrid)
- Ideally 5 days onsite, though 3-4 days onsite may be acceptable and will be discussed during the interview
- Candidates locals to NJ/NY area
- USC & GC ONLY
Compensation
- Estimated rate range: $70.00 to $85.00 per hour (USD)
- Pay ranges shown are estimations; actual pay depends on experience, technical expertise, and other qualifications in the job description
Benefits
- Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
- Health Savings Account (HSA) (for employees working 20+ hours per week)
- Life & Disability Insurance (for employees working 20+ hours per week)
- MetLife Voluntary Benefits
- Employee Assistance Program (EAP)
- 401K Retirement Savings Plan
- Direct Deposit & weekly epayroll
- Referral Bonus Programs
- Certification and training opportunities
Preferred experience
- Experience designing and delivering production-grade Databricks solutions in a large enterprise environment, especially where security, data governance, resiliency, automation, and operational support are key requirements
Benefit eligibility is subject to applicable laws and client requirements.