Senior Big Data Engineer

Systems Development and Analysis Inc seeks a Senior Big Data Engineer to design, implement, and maintain scalable data ingestion, integration, and analytics solutions. This full-time, hybrid role based in Arlington, VA focuses on Java, Kafka, Spark, Elasticsearch, and cloud platforms. The position requires U.S. citizenship with CBP BI eligibility.

Responsibilities

• Architect and implement enterprise data ingestion, ETL, and transformation pipelines supporting both batch and real-time workloads.
• Establish and sustain scalable Kafka streaming architectures to process and distribute high-volume data.
• Develop Java-based applications and services to support enterprise data integration and transformation workflows.
• Create Spark and Spark Streaming solutions for processing, enriching, and analyzing large datasets.
• Design and support enterprise search capabilities with Elasticsearch to enable data discovery, investigative workflows, analytics, and informed decision-making.
• Analyze source and target data structures and craft source-to-target mapping specifications.
• Integrate data from multiple structured and semi-structured sources while ensuring data integrity, consistency, traceability, and performance.
• Develop REST APIs and system integration services to enable cross-enterprise data exchange.
• Support data migration, modernization, and cloud transformation initiatives.
• Perform data validation, reconciliation, profiling, and quality assurance activities.
• Implement and oversee data quality, governance, lineage, and validation processes across ingestion and transformation pipelines.
• Optimize SQL queries, database procedures, and distributed data pipelines for performance and reliability.
• Troubleshoot production issues and enhance platform reliability, scalability, and performance.
• Collaborate with architects, developers, analysts, and stakeholders to design effective technical solutions.
• Assist with onboarding data consumers and integration partners.
• Evaluate emerging technologies and develop proof-of-concept solutions.
• Mentor junior engineers and contribute to technical leadership efforts.

Requirements

• U.S. Citizenship required.
• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field, or equivalent experience.
• 5-10+ years of experience in Data Engineering, Software Engineering, ETL Development, or Enterprise Data Integration.
• Strong hands-on Java development experience.
• Experience designing and implementing ETL and data integration solutions.
• Experience building and supporting Apache Kafka producers, consumers, and streaming applications.
• Experience developing applications using Apache Spark and/or Spark Streaming.
• Strong SQL skills and relational database experience, including Oracle.
• Experience creating and maintaining source-to-target data mappings.
• Experience with data transformation, data migration, and data validation activities.
• Experience developing REST APIs and integration services.
• Experience working with XML, JSON, CSV, and other structured or semi-structured formats.
• Experience supporting Linux/Unix environments.
• Strong analytical, troubleshooting, and problem-solving abilities.

Technologies

• Java
• Kafka
• Spark
• Elasticsearch
• Oracle
• SQL
• ETL
• Data Integration
• Data Mapping
• REST APIs
• MongoDB
• AWS
• Linux
• Scala
• Spring Framework
• Spring Boot
• Apache NiFi
• AWS Glue
• Informatica
• Talend
• Hadoop
• HBase
• Delta Lake
• PostgreSQL
• DocumentDB
• Redshift
• Docker
• Kubernetes
• Azure
• Google Cloud Platform
• CI/CD Pipelines
• Spark Streaming

Benefits

• Exposure to modern Big Data, cloud, and enterprise integration technologies.
• Opportunity to design and build large-scale data platforms supporting mission-critical operations.
• Collaboration with experienced engineers, architects, and data professionals.
• Influence over technical direction and solution architecture.
• Competitive compensation and benefits package.
• Flexible hybrid work environment.
• Paths for professional growth and technical leadership.

Preferred Qualifications

• Experience with Elasticsearch and enterprise search platforms.
• Experience with Hadoop ecosystem technologies.
• Experience with Scala development.
• Experience with MongoDB, DocumentDB, HBase, Delta Lake, Redshift, or similar Big Data technologies.
• Experience with ETL tools such as Apache NiFi, AWS Glue, Talend, Informatica, Pentaho, or SSIS.
• Experience with event-driven architectures and enterprise messaging technologies.
• Experience with Docker and Kubernetes.
• Experience supporting AWS, Azure, or Google Cloud Platform environments.
• Experience supporting enterprise data migration or modernization initiatives.
• Experience working in Agile software development environments.
• Experience supporting large-scale analytics, risk assessment, investigative, intelligence, law enforcement, or operational decision-support systems.

Technologies You’ll Work With

• Java
• Scala
• SQL
• Spring Framework
• Spring Boot
• REST APIs
• ETL Development
• Data Mapping & Transformation
• Data Migration
• Apache NiFi
• AWS Glue
• Informatica
• Talend
• Apache Kafka
• Apache Spark
• Spark Streaming
• Hadoop
• HBase
• Delta Lake
• Elasticsearch
• Oracle
• PostgreSQL
• MongoDB
• DocumentDB
• Redshift
• Relational & NoSQL Databases
• AWS
• Azure
• Google Cloud Platform
• Docker
• Kubernetes
• CI/CD Pipelines

Why Join SDA

• Work with modern Big Data, cloud, and enterprise integration technologies.
• Design and build large-scale data platforms supporting mission-critical operations.
• Collaborate with highly skilled engineers, architects, and data professionals.
• Influence technical direction and solution architecture.
• Competitive compensation and benefits package.
• Flexible hybrid work environment.
• Opportunities for professional growth and technical leadership.