Data Engineer- Azure Databricks- Financial Domain Exp Must
Backend Developer
Application Security
Azure
Azure Data Engineer
Azure Databricks
Azure Functions
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Data Warehouse
Cloud Data Warehouse
Cloud Engineer
Cloud Infrastructure
Cloud Platform
Cloud Platforms
Cloud Platforms Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Analytics
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Data Warehousing
Database
Databases
Databricks
Databricks Pyspark
Databricks Spark
DevOps
DevSecOps
Distributed Systems
Engineer
Engineering
ETL
GraphQL
Informatica
Information Technology (IT)
Infrastructure As Code
Microsoft Azure
Programming
Programming Language
Programming Languages
Security Automation
Security Devops
Snowflake
Software Architecture
Software Development
Software Engineering
Spark
SQL
Job Description
Josh Pros LLC is hiring a Data Engineer to support data engineering delivery in a financial domain setting, with a focus on Azure Databricks and cloud-based analytics and AI/ML enablement. The role centers on building secure, scalable data pipelines and ETL/ELT solutions, while partnering with stakeholders and technical teams to translate requirements into verified functional changes.
Responsibilities
- Develop services, controls, and reusable patterns that help the team deliver value safely, quickly, and sustainably in a public cloud environment.
- Build highly scalable Azure cloud applications using microservices and Azure Functions, including public cloud patterns with security enablement.
- Drive high-performance, collaborative technical work that results in high-quality output.
- Review new product capabilities and break down implementations into specific functional changes for verification.
- Gather and analyze data to support informed decision-making and provide detailed, realistic estimates.
- Partner effectively with business stakeholders and third-party technical organizations.
- Design and develop ETL/ELT solutions on Azure Databricks, LakeBase, and Spark.
- Develop, implement, and deploy large-scale data pipelines that enable machine learning algorithms, insights generation, and business intelligence dashboards.
- Design, build, optimize, and manage modern large-scale data pipelines to support analytics and machine learning feature data integration.
- Write advanced and complex SQL, including performance tuning and optimization.
- Build AI-based solutions for business needs, process automation, and operational efficiency improvements.
- Identify opportunities to improve data reliability, data integrity, system efficiency, and overall data quality.
- Contribute to the architectural evolution of data engineering patterns, frameworks, systems, and platforms, including defining best practices and standards.
- Mentor other data engineers and provide technical guidance on how to use cloud data platforms effectively.
Requirements
- 7+ years of experience in Python and Scala.
- 5+ years of experience in Apache Spark (including PySpark and Spark SQL).
- 3+ years of DevOps automation using Terraform.
- 3+ years of experience with API design and lifecycle management (for example, GraphQL and REST).
- 3+ years building and deploying cloud-based solutions using Azure Databricks with UC, Snowflake, Azure Functions, and Service Bus.
- 5+ years of SQL experience, including designing complex data schemas and optimizing query performance.
- Experience building LLM integrations for workflow automation and business needs.
- Experience with CI/CD processes and tools such as GitHub Actions, GIT, Artifactory, and Sonar.
Technology Stack
- Python, Scala
- Apache Spark, PySpark, Spark SQL
- Terraform
- GraphQL, REST
- Azure Databricks (with UC), Snowflake
- Azure Functions, Service Bus
- SQL, LLM integrations
- CI/CD with GitHub Actions, GIT, Artifactory, Sonar
- Microservices, LakeBase, ETL/ELT, AI/ML
Pay and Schedule
- $70.00 - $80.00 per hour
- 40.0 hours per week
Location
- Eden Prairie, MN (in person)
Minimum experience: 3 years.