DataJobs.io
← Back to all jobs

Job Description

Josh Pros LLC is hiring a Data Engineer to support data engineering delivery in a financial domain setting, with a focus on Azure Databricks and cloud-based analytics and AI/ML enablement. The role centers on building secure, scalable data pipelines and ETL/ELT solutions, while partnering with stakeholders and technical teams to translate requirements into verified functional changes.

Responsibilities

  • Develop services, controls, and reusable patterns that help the team deliver value safely, quickly, and sustainably in a public cloud environment.
  • Build highly scalable Azure cloud applications using microservices and Azure Functions, including public cloud patterns with security enablement.
  • Drive high-performance, collaborative technical work that results in high-quality output.
  • Review new product capabilities and break down implementations into specific functional changes for verification.
  • Gather and analyze data to support informed decision-making and provide detailed, realistic estimates.
  • Partner effectively with business stakeholders and third-party technical organizations.
  • Design and develop ETL/ELT solutions on Azure Databricks, LakeBase, and Spark.
  • Develop, implement, and deploy large-scale data pipelines that enable machine learning algorithms, insights generation, and business intelligence dashboards.
  • Design, build, optimize, and manage modern large-scale data pipelines to support analytics and machine learning feature data integration.
  • Write advanced and complex SQL, including performance tuning and optimization.
  • Build AI-based solutions for business needs, process automation, and operational efficiency improvements.
  • Identify opportunities to improve data reliability, data integrity, system efficiency, and overall data quality.
  • Contribute to the architectural evolution of data engineering patterns, frameworks, systems, and platforms, including defining best practices and standards.
  • Mentor other data engineers and provide technical guidance on how to use cloud data platforms effectively.

Requirements

  • 7+ years of experience in Python and Scala.
  • 5+ years of experience in Apache Spark (including PySpark and Spark SQL).
  • 3+ years of DevOps automation using Terraform.
  • 3+ years of experience with API design and lifecycle management (for example, GraphQL and REST).
  • 3+ years building and deploying cloud-based solutions using Azure Databricks with UC, Snowflake, Azure Functions, and Service Bus.
  • 5+ years of SQL experience, including designing complex data schemas and optimizing query performance.
  • Experience building LLM integrations for workflow automation and business needs.
  • Experience with CI/CD processes and tools such as GitHub Actions, GIT, Artifactory, and Sonar.

Technology Stack

  • Python, Scala
  • Apache Spark, PySpark, Spark SQL
  • Terraform
  • GraphQL, REST
  • Azure Databricks (with UC), Snowflake
  • Azure Functions, Service Bus
  • SQL, LLM integrations
  • CI/CD with GitHub Actions, GIT, Artifactory, Sonar
  • Microservices, LakeBase, ETL/ELT, AI/ML

Pay and Schedule

  • $70.00 - $80.00 per hour
  • 40.0 hours per week

Location

  • Eden Prairie, MN (in person)

Minimum experience: 3 years.

Similar Jobs