Consultant, Data Engineer
AWS
Azure Data Lake
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Lake
Data Lakehouse
Data Management
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Security
Data Warehouse
Database
Databases
Databricks
Databricks Workflows
Delta Lake
ETL
Informatica
Information Technology (IT)
Integration
Pyspark
Software Development
Spark
Spark Sql
SQL
Job Description
Join Infinitive Inc in a hybrid data engineering consulting role building production pipelines on Databricks for client teams in McLean, Virginia.
Responsibilities
- Build and maintain batch and incremental ETL pipelines in Databricks using Python, PySpark, and Spark SQL
- Ingest data from relational databases, APIs, flat files, and cloud storage into Delta Lake tables
- Load data into Delta Lake following medallion architecture patterns (bronze, silver, gold layers)
- Write and tune SQL for transformations, data validation, and reporting tables
- Add data quality checks, manage schema changes, and troubleshoot failed jobs
- Schedule and monitor workloads with Databricks Workflows using Unity Catalog permissions
- Optimize Spark for performance and cost (partitioning, file sizing, and cluster configuration)
- Use Git for version control, participate in code reviews, and contribute to CI/CD pipelines
- Document pipelines, source-to-target mappings, and runbooks so client teams can operate what you build
- Attend client meetings to clarify requirements, share status, and walk through your work
Requirements
- 1 to 3 years of professional experience in data engineering, software engineering, or a related technical role
- Bachelor’s degree in computer science, information systems, engineering, mathematics, or a related field, or equivalent experience
- Hands-on experience with Databricks in a professional role or substantial academic project
- Working proficiency in Python for data processing
- Experience building transformations with Apache Spark and PySpark (DataFrames and Spark SQL)
- Strong SQL skills including joins, window functions, CTEs, and aggregations
- Understanding of ETL concepts such as incremental loads, slowly changing dimensions, dimensional modeling, and data quality checks
- Familiarity with at least one cloud platform: AWS, Azure, or Google Cloud
- Clear written and verbal communication; ability to explain technical work to non-technical stakeholders
- Authorization to work in the United States (confirm sponsorship policy)
Technologies
- Databricks
- Python, PySpark, Spark SQL
- Delta Lake, Medallion architecture
- Amazon S3, Azure Data Lake Storage
- Databricks Workflows, Unity Catalog
- Apache Spark, SQL, Git
- CI/CD, DataFrames, ETL
- AWS, Azure, Google Cloud
Nice to Have
- Databricks Certified Data Engineer Associate certification
- Experience with Delta Lake features such as MERGE and time travel, or with Lakeflow Declarative Pipelines (formerly Delta Live Tables)
- Exposure to Unity Catalog, dbt, Apache Airflow, or Azure Data Factory
- Experience with Terraform or CI/CD tools such as GitHub Actions or Azure DevOps
- Prior consulting or client-facing experience
- Background in financial services, healthcare, or media data
Role Details
- Location: McLean, VA (hybrid)
- Compensation: USD 75,000 - 120,000 per year
- Project teams: work with senior engineers, architects, and business analysts
- Core focus: write production Python, PySpark, and SQL that moves data from source systems into tables used by client analysts and data scientists
Similar Jobs
