Senior Databricks Data Engineer

Senior Databricks Data Engineer role focused on building scalable enterprise ETL/ELT pipelines end to end.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Apache Spark, Python/PySpark, and SQL
• Build batch and incremental data integration pipelines from files, databases, and enterprise data sources
• Create reusable, maintainable data processing solutions
• Optimize Spark workloads for performance, scalability, reliability, and cost efficiency
• Implement data quality controls, reconciliation, monitoring, and error handling
• Own the complete engineering lifecycle: requirements analysis, solution design, technical review, implementation, and deployment
• Prepare and present technical solution designs for architecture and engineering review
• Incorporate review feedback and implement approved designs
• Make appropriate technical decisions and proactively resolve engineering challenges
• Take accountability for deliverables, quality, timelines, and production readiness
• Develop complex SQL transformations for cleansing, enrichment, and aggregation
• Design and maintain curated datasets, dimensional models, and reporting views
• Support enterprise analytics, reporting, and downstream data consumers
• Implement and maintain pipeline orchestration using Airflow, Databricks Workflows, or comparable tools
• Apply CI/CD practices for version control, automated deployment, and release management
• Troubleshoot pipeline failures and production performance issues
• Collaborate with data architects, integration teams, engineering teams, and business stakeholders
• Document technical designs, implementation decisions, and operational procedures

Requirements

• Minimum 5 years of relevant experience in data engineering, data integration, or related disciplines
• Strong, recent hands-on development experience with Databricks
• Proven experience designing, building, and implementing production ETL/ELT pipelines
• Advanced proficiency in Apache Spark, PySpark, and Python with demonstrated hands-on engineering experience
• Strong SQL skills, including complex joins, window functions, data transformations, and query performance tuning
• Experience with at least one major cloud platform: AWS, Azure, or Google Cloud Platform (GCP)
• Experience with workflow orchestration tools such as Apache Airflow, Databricks Workflows, or equivalent platforms
• Experience with CI/CD practices, source control, and automated deployment of data pipelines
• Demonstrated ability to independently own the full development lifecycle, including solution design, technical review, implementation, and production support
• Proven ability to evaluate requirements, make sound technical design decisions, and implement approved solutions
• Strong problem-solving skills with a track record of independent execution, accountability, and ownership in fast-paced environments

Technologies

• Databricks
• Apache Spark
• Python
• PySpark
• SQL
• Airflow
• Databricks Workflows
• CI/CD
• AWS
• Azure
• Google Cloud Platform (GCP)
• Delta Lake
• Unity Catalog
• REST API

Nice to Haves

• Calling external APIs from Databricks notebooks, including REST API integration, authentication, request handling, and ingestion of API responses (building or exposing APIs is not required)
• Streaming and event driven data processing
• Delta Lake and medallion architecture
• Unity Catalog, governance, and lineage
• Metadata driven and reusable pipeline frameworks
• Enterprise scale Databricks implementations
• Experience with complex enterprise data integration environments
• Spark performance tuning and cost optimization
• Batch and incremental data processing
• Data quality, reconciliation, monitoring, and observability
• Data warehousing, fact/dimension modeling, and curated data layers
• Production troubleshooting and operational reliability
• Technical documentation and design review participation

Benefits

• Competitive compensation
• Comprehensive insurance options
• Matching contributions through the 401(k) plan and the share purchase plan
• Paid time off for vacation, holidays, and sick time
• Paid parental leave
• Learning opportunities and tuition assistance
• Wellness and Well being programs

Location: Miami, FL (hybrid)

Salary: USD 91,800 - 203,800 per yearly

Education: Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Additional info: CGI anticipates accepting applications through November 13, 2026. Compensation range estimate for the U.S.: $91,800.00 - $203,800.00.