Senior Data Engineer
Job Description
ScienceSoft USA Corporation is hiring a Senior Data Engineer for a full-time, on-site role in Dallas, TX (McKinney, TX).
Responsibilities
- Own data pipelines end-to-end, covering ingestion, processing, validation, and publication
- Build and maintain batch and streaming data pipelines
- Implement reliable processing patterns for idempotency, backfills, late-arriving data, and partitioning
- Develop automated data quality checks for completeness, timeliness, anomalies, and cross-source consistency
- Monitor production pipelines and configure effective alerting
- Investigate and resolve data quality and pipeline reliability issues
- Optimize pipeline performance and resource usage
- Onboard new datasets and data providers
- Work with large volumes of financial and market-related data
- Communicate with data consumers about incidents, data issues, changes, and availability
- Collaborate on APIs and internal libraries for data access and distribution
- Improve reliability, maintainability, and observability of the data platform
Requirements
- 5+ years of professional experience in Data Engineering
- Strong practical experience with Python
- Strong knowledge of SQL
- Strong hands-on experience with Apache Spark
- Experience with Hadoop / Hive or modern lakehouse formats such as Delta Lake or Apache Iceberg
- Experience building and maintaining batch and streaming pipelines
- Strong understanding of idempotent data processing
- Strong understanding of backfills
- Strong understanding of late-arriving data
- Strong understanding of partitioning strategies
- Experience implementing automated data quality checks
- Experience with monitoring and alerting for production data pipelines
- Experience working with large volumes of data
- Strong understanding of reliable data ingestion and publication processes
- Strong analytical and troubleshooting skills
- High attention to detail and data correctness
- Strong ownership and ability to independently manage data engineering tasks
- Strong communication skills with data consumers and technical stakeholders
- Valid US work authorization
- Ability to work on-site full-time in Dallas, TX
Technology
- Python, SQL
- Apache Spark
- Hadoop, Hive
- Delta Lake, Apache Iceberg
- Kafka
- Airflow
- Snowflake, Databricks
- Java, Scala
Nice to Have
- Experience working with market data, including equities, macroeconomic data, or market data vendors
- Experience with Kafka or other streaming technologies
- Experience with Airflow or another workflow orchestration platform
- Experience with Snowflake or Databricks
- Experience developing APIs or reusable libraries for data distribution
- Experience with Java or Scala
- Previous experience in Investment Management / Financial Services