Senior Data Engineer

The Senior Data Engineer at IvoryCloud will design and optimize data pipelines and analytics platforms using Databricks, focusing on large-scale data engineering and analytics workloads for a Department of Energy client.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Apache Spark.
• Build and optimize batch and streaming data processing solutions.
• Develop robust ETL/ELT frameworks for structured and semi‑structured data from SQL, APIs, SaaS apps, and Data Live Lake Storage.
• Implement transformation logic using PySpark, SQL, and Delta Lake within a Lakehouse / Medallion architecture.
• Integrate and model data from SQL, SaaS systems, and cloud storage into unified, analytics‑ready and AI‑ready structures.
• Partner with analytics, BI, and AI/ML teams to deliver trusted datasets, semantic layers, and AI‑ready data (features, vector‑ready patterns, embeddings).
• Improve data quality, observability, lineage, governance, and security in alignment with Azure and enterprise standards.
• Optimize data performance, storage costs, and processing efficiency in cloud environments.
• Participate in architecture discussions and contribute to the evolution of the organization’s data platform and engineering best practices.

Requirements

• U.S. Citizenship is required and non-negotiable
• 2 days a week in the Rockville, MD office are required and non-negotiable
• Ability to pass multiple background and employment checks - federal and local
• Bachelor’s degree Technology, Business, or Related Field

Technologies

• Databricks
• Apache Spark
• PySpark
• SQL
• Delta Lake
• Data Live Lake Storage
• Python
• dbt
• Airflow
• Azure Data Factory
• Kafka
• Snowflake
• Git
• CI/CD Pipelines
• Azure
• AWS
• GCP
• Unity Catalog
• Delta Live Tables
• Workflows
• Lakeflow
• SQL Warehouse
• Mosaic AI
• Synapse pipelines
• Dagster
• Azure OpenAI
• Event Hubs
• Kinesis
• Redshift
• Synapse

Benefits

• 401(k)
• Health insurance
• Paid time off
• Vision insurance

What Success Looks Like

• Deliver reliable, production-ready data pipelines supporting critical business use cases.
• Improve data processing performance and scalability.
• Establish reusable engineering patterns and best practices.
• Enable analytics and AI teams with high-quality, trusted datasets.
• Contribute to the modernization and growth of the organization’s data platform capabilities.

Technical Environment

• Databricks
• Apache Spark
• PySpark
• SQL
• Delta Lake
• Python
• dbt
• Airflow
• Azure Data Factory
• Kafka
• Snowflake
• Git
• CI/CD Pipelines
• Cloud Platforms (Azure, AWS, or GCP)

Education & Certifications

• B.S. in Business/Technology-related field

Legal

• IvoryCloud is an Equal Opportunity Employer (EEO)

Pay

Salary: USD 155,000 - 200,000 per year

Location

Rockville, MD (hybrid)

Experience

• Databricks: 4 years (Required)

Application Questions

• This role requires U.S. Citizenship. Do you meet this requirement?
• Can you meet the hybrid requirement of 2 days a week in office in Rockville, Maryland?
• Are you able to come into the office for in-person interviews?