S
Senior Data Engineer
Backend Developer
Senior
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Data Warehouse
Cloud Data Warehouse
Cloud Operations
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Warehouse
Data Warehousing
Database
Databases
Databricks
Engineer
ETL
Event Driven Architecture
Informatica
Information Technology (IT)
Kafka
Programming Language
Programming Languages
Pyspark
Snowflake
Software Development
Spark
SQL
Stream Processing
Streaming Data
Job Description
SkillStart is hiring a Senior Data Engineer (hybrid, New York) to architect and optimize high-throughput batch and streaming pipelines for a Capital Markets post-trade data environment.
Responsibilities
- Design, develop, and maintain high-throughput batch and streaming ETL/ELT pipelines using Python, SQL, Spark, and streaming platforms
- Build post-trade data processing flows for trade allocations, confirmations, settlement, risk management, finance, and regulatory reporting
- Develop and optimize relational data models, database schemas, security master structures, reference data models, and time-series financial datasets
- Implement and manage scalable data repositories across Data Warehouses, Data Lakes, and Data Lakehouses on cloud platforms (preferably AWS, Snowflake, Redshift, Databricks)
- Define data quality rules, data lineage, automated validation checks, and reconciliation processes across complex end-to-end data flows
- Integrate real-time messaging architectures including Kafka, RabbitMQ, IBM MQ for ingestion, processing, and distribution of high-volume transactional trade data
- Apply data security, privacy controls, and regulatory compliance across ingestion and storage layers
- Use monitoring and observability tools such as Datadog, Prometheus, Grafana to track pipeline health, support SLA compliance, and enable rapid incident response
Requirements
- Strong understanding of post-trade data flows across allocations, confirmations, settlement, risk, finance, and regulatory reporting
- In-depth experience with securities data modeling and financial data structures, including reference data, security master, and transactional trade data
- Strong SQL skills, including designing, tuning, and optimizing relational data models and database schemas
- Proficiency in Python with hands-on experience in data processing libraries and distributed frameworks such as Pandas and PySpark
- Operational understanding of Data Warehouse, Data Lake, and Data Lakehouse architectures and best practices
- Hands-on experience with PostgreSQL, SQL Server, Snowflake, Redshift, Databricks, or similar platforms on cloud infrastructure, preferably AWS
- Experience with stream and batch processing technologies such as Kafka, Spark, Flink, or similar platforms
- Proven track record building robust batch and streaming ETL/ELT pipelines using Python and SQL
- Strong background in data quality assurance, data lineage tracking, data validation frameworks, and automated reconciliation
- Experience with messaging and event-driven platforms including Kafka, RabbitMQ, IBM MQ, or similar technologies
- Clear understanding of data security principles, data privacy guidelines, and financial regulatory compliance requirements
- Familiarity with Datadog, Prometheus, and Grafana for monitoring and observability
Preferred & Bonus Qualifications
- Understanding of time-series data management and financial market data is a strong plus
- Exposure to data science, machine learning, and AI/GenAI data preparation concepts is a plus
Pay
- USD 122,000 - 148,000 per year
Work Location
- Hybrid remote in New York, NY 10114
Technologies
- Python, SQL, Spark, Pandas, PySpark
- AWS, Snowflake, Redshift, Databricks
- PostgreSQL, SQL Server
- Kafka, RabbitMQ, IBM MQ, Flink
- Datadog, Prometheus, Grafana