Slurm Data Jobs

1 job found

Today

Shield Consulting Solutions Annapolis Junction, MD

Annapolis Junction, MD

Today

$230k - $240k/yr Full time Senior Apache Airflow Big Data

This Senior Data Engineer role focuses on building and optimizing analytics data pipelines for efficient, scalable, and reliable processing.

Design, implement, and optimize data ingress and egress pathways to support analytics processing
Apply expertise in dataflow design and data transport mechanisms to ensure dependable pipeline performance
Use Apache Spark for distributed processing aligned to organizational analytics workloads
Manage workflow orchestration for complex processing by using Apache Airflow

7+ years of experience
Bachelor’s degree in a technical discipline
Experience using the Linux CLI and Linux tools
Experience developing Bash scripts to automate manual processes
Recent software development experience using Python and Java
Experience with Apache Airflow, including:
- DAG design
- Scheduling
- Operators and sensors
- Orchestrating, scheduling, and monitoring complex workflows
Experience with distributed big data processing engines, including Apache Spark
Familiarity with SQL technologies: MySQL, MariaDB, and PostgreSQL for querying, joining, and aggregating large datasets
Experience using Jupyter Notebook
Experience with data wrangling and preprocessing using pandas and NumPy
Experience working with structured, semi-structured, and unstructured data formats: Parquet, JSON, CSV, XML
Familiarity with data quality concepts, data validation, and anomaly detection
Experience using Git source control
Familiarity with HPC job scheduling tools including Slurm
Experience with the Atlassian Tool Suite including JIRA and Confluence

25 days PTO
11 paid holidays
100% employer-paid healthcare for employees and dependents (available day 1)
8% 401(k) employer match with immediate vesting

New Slurm Data Jobs, daily.