Data Engineer II
Big Data
Clickhouse
Cloud
Cloud Infrastructure
Cloud Native
Cloud Native Technologies
Cloud Platform
Cloud Platforms
Cloud Platforms Cloud Platforms
Cloud Technology
Data
Data Analysis
Data Analytics
Data Architecture
Data Build Tool
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Management
Data Modeling
Data Operations
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Transforms
Data Warehouse
Database
Databases
DevOps
Devops Tools
DevSecOps
Engineer
Engineering
Engineering Software
ETL
Exasol
Informatica
Kubernetes
Lakehouse
Platform Engineering
Programming Languages
Reporting and Analytics
Security Devops
Software Development
SQL
Job Description
Vylor is seeking a full-time Data Engineer II based in Des Moines, IA (onsite). The role focuses on designing and delivering scalable lakehouse and analytics data solutions for Seeds and Biotech domains, alongside support for an enterprise data platform.
What You’ll Do
- Design, develop, and maintain lakehouse data pipelines, APIs, and data products for seed models, biotech research assets, enterprise applications, dashboards, and analytics.
- Build scalable ETL/ELT pipelines using Python, SQL, dbt, Trino, and shell scripting to process and integrate large-scale structured and unstructured agricultural data from enterprise systems, laboratory platforms, geographic datasets, drones, satellites, and IoT devices.
- Create and maintain Bronze, Silver, and Gold Layer data assets by integrating sources such as Exasol, ClickHouse, and MSSQL Server, applying transformation, validation, reconciliation, and quality-control processes.
- Design lakehouse datasets and transformation pipelines for seed, biotech, pathology, and disease analytics, including integration of laboratory, geographic, and historical research data.
- Develop and optimize incremental and full-refresh dbt models and complex Trino SQL queries to improve performance, scalability, and maintainability of Seeds and Biotech data products.
- Modernize legacy data applications by documenting business rules and source-to-target mappings, migrating ETL logic from legacy and Exasol-based environments to Trino and lakehouse architectures, and deploying curated datasets to ClickHouse and Gold Layer data products.
- Develop and enhance PRISM and FRIP data integrations by implementing Trino connectors and end-to-end comparison pipelines with reusable validation and reconciliation logic across PRISM, FRIP, lakehouse, Exasol, and downstream application datasets.
- Build reusable frameworks for logging, error handling, scheduling, automated testing, monitoring, and data-quality validation; troubleshoot integration issues; and create analytics-ready datasets for statistical analysis, predictive modeling, regression, classification, clustering, dashboards, and business-intelligence reporting.
Required Qualifications
- Master’s degree (or equivalent) in Computer Science, Statistics, or a related field, plus 2 years of related experience.
- 24 months of experience designing and developing scalable ETL/ELT pipelines using Python, SQL, and shell scripting to extract, transform, validate, reconcile, and integrate large-volume structured, semi-structured, and unstructured data used by enterprise R&D applications, analytical models, dashboards, and reporting systems.
- 24 months of experience developing ETL design mappings, source-to-target transformation rules, data reconciliation processes, data-quality controls, and complex SQL transformations to ensure accuracy and consistency across platforms.
- 24 months of experience developing and optimizing complex Trino SQL queries and integrations among Lakehouse, Exasol, ClickHouse, MSSQL Server, and other database platforms to deliver analytics-ready curated datasets for R&D models, scientific analysis, and enterprise applications such as phenomic pillar, dashboards, and reporting.
- 24 months of experience building reusable Python- and Linux shell-based pipeline reliability and observability frameworks for job scheduling, structured logging, exception handling, automated testing, monitoring, data-quality validation, cross-platform reconciliation, and application integration; using version-control, containerization, and automated-deployment tools to deliver production data products.
- 24 months of experience analyzing and modernizing legacy data applications by documenting business rules and data mappings, migrating ETL logic to cloud-based or lakehouse architecture, deploying curated data products, and using Git and Docker for source control, automated deployment, and containerization.
Technologies
- Lakehouse, Python, SQL, dbt, Trino, shell scripting
- Exasol, ClickHouse, MSSQL Server
- Kubernetes, Docker, Git, Linux
Benefits
- Health benefits for you and your family on your first day of employment
- Four weeks of paid time off and two weeks of well-being pay per year, plus paid holidays
- Excellent parental leave which includes a minimum of 16 weeks for mother and father
- Future planning with a competitive retirement savings plan and tuition reimbursement program
- Numerous development opportunities to build skills
- Opportunity to contribute to making the world a better place
- Learn more about the total rewards package: Vylor Benefits
Telecommute Benefit
The position includes a telecommute benefit within commuting distance to the Johnston, IA Pioneer Hi-Bred/Vylor office, as directed.