DataJobs.io
← Back to all jobs

Job Description

SkillStart is hiring a Senior Data Engineer (hybrid, New York) to architect and optimize high-throughput batch and streaming pipelines for a Capital Markets post-trade data environment.

Responsibilities

  • Design, develop, and maintain high-throughput batch and streaming ETL/ELT pipelines using Python, SQL, Spark, and streaming platforms
  • Build post-trade data processing flows for trade allocations, confirmations, settlement, risk management, finance, and regulatory reporting
  • Develop and optimize relational data models, database schemas, security master structures, reference data models, and time-series financial datasets
  • Implement and manage scalable data repositories across Data Warehouses, Data Lakes, and Data Lakehouses on cloud platforms (preferably AWS, Snowflake, Redshift, Databricks)
  • Define data quality rules, data lineage, automated validation checks, and reconciliation processes across complex end-to-end data flows
  • Integrate real-time messaging architectures including Kafka, RabbitMQ, IBM MQ for ingestion, processing, and distribution of high-volume transactional trade data
  • Apply data security, privacy controls, and regulatory compliance across ingestion and storage layers
  • Use monitoring and observability tools such as Datadog, Prometheus, Grafana to track pipeline health, support SLA compliance, and enable rapid incident response

Requirements

  • Strong understanding of post-trade data flows across allocations, confirmations, settlement, risk, finance, and regulatory reporting
  • In-depth experience with securities data modeling and financial data structures, including reference data, security master, and transactional trade data
  • Strong SQL skills, including designing, tuning, and optimizing relational data models and database schemas
  • Proficiency in Python with hands-on experience in data processing libraries and distributed frameworks such as Pandas and PySpark
  • Operational understanding of Data Warehouse, Data Lake, and Data Lakehouse architectures and best practices
  • Hands-on experience with PostgreSQL, SQL Server, Snowflake, Redshift, Databricks, or similar platforms on cloud infrastructure, preferably AWS
  • Experience with stream and batch processing technologies such as Kafka, Spark, Flink, or similar platforms
  • Proven track record building robust batch and streaming ETL/ELT pipelines using Python and SQL
  • Strong background in data quality assurance, data lineage tracking, data validation frameworks, and automated reconciliation
  • Experience with messaging and event-driven platforms including Kafka, RabbitMQ, IBM MQ, or similar technologies
  • Clear understanding of data security principles, data privacy guidelines, and financial regulatory compliance requirements
  • Familiarity with Datadog, Prometheus, and Grafana for monitoring and observability

Preferred & Bonus Qualifications

  • Understanding of time-series data management and financial market data is a strong plus
  • Exposure to data science, machine learning, and AI/GenAI data preparation concepts is a plus

Pay

  • USD 122,000 - 148,000 per year

Work Location

  • Hybrid remote in New York, NY 10114

Technologies

  • Python, SQL, Spark, Pandas, PySpark
  • AWS, Snowflake, Redshift, Databricks
  • PostgreSQL, SQL Server
  • Kafka, RabbitMQ, IBM MQ, Flink
  • Datadog, Prometheus, Grafana

Similar Jobs