Sr Data Engineer - Databricks

Senior data engineer specializing in Databricks and Spark, delivering production-grade data pipelines and data products in a hybrid Miami environment.

Responsibilities

• Architect and implement ETL and ELT pipelines on Databricks Spark to support both batch and incremental workloads.
• Establish reliable data ingestion from diverse sources including files, databases, APIs, and streaming when applicable.
• Tune Spark jobs for performance, scalability, and cost control.
• Enforce data quality, reconciliation, and observability across end-to-end pipelines.
• Create advanced SQL transformations for cleansing, enrichment, and aggregation.
• Build and maintain analytical data models such as fact and dimension schemas, curated layers, and reporting views.
• Provide clean, well-structured datasets to support downstream reporting, analytics, and data science use cases.
• Configure and manage data exposure via APIs for internal and external consumers.
• Collaborate with application and integration teams to define API contracts and data payload formats.
• Establish secure, scalable, and high-performance data access patterns.
• Support versioning and backward compatibility for published data services.

Requirements

• Minimum five years of experience in data engineering or related roles.
• Hands-on expertise with Databricks and Apache Spark.
• Advanced SQL skills, including complex joins, window functions, and performance tuning.
• Proven experience building and managing ETL and ELT pipelines.
• Experience configuring and maintaining data exposure via APIs.
• Solid understanding of data warehousing and analytics concepts.
• Strong problem-solving and analytical abilities.
• Ability to work independently on complex data problems.
• Clear communication with both technical and non-technical stakeholders.
• Comfortable working in an agile, product-oriented environment.
• Excellent verbal and written communication; ability to document processes and frameworks clearly for future reference; dependable, accurate, and professional.
• Self-motivated and capable of working without close supervision.

Technologies

• Databricks
• Apache Spark
• SQL

Compensation

Salary range: USD 89,600 - 198,400 per year.

Benefits

• Competitive compensation and total rewards
• Comprehensive insurance options
• 401(k) plan with company matching and an employee share purchase plan
• Paid time off for vacation, holidays, and sick time
• Paid parental leave
• Learning opportunities and tuition assistance
• Wellness and wellbeing programs

Skills

• Cloud data platform design
• Data engineering
• Databricks
• ETL

What You Can Expect From Us

We operate with a sense of ownership and collaboration. Life at CGI Partners emphasizes teamwork, respect, and belonging, with opportunities to grow and contribute from day one as we pursue shared goals. Your work will create value, and you will develop innovative solutions as part of a collective effort to shape the company’s strategy and direction.