Senior Data Engineer

The Senior Data Engineer role at NTT Ltd focuses on designing and optimizing scalable data pipelines with Databricks and related technologies to support AI-driven document processing. The position emphasizes structured extraction from unstructured and semi-structured documents, ontology-based modeling, and delivery of standardized JSON outputs for downstream underwriting workflows.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Build ingestion, extraction, classification, validation, and transformation pipelines for documents including PDFs, emails, attachments, and Excel/XLSX files, as well as insurance applications and Statements of Values (SOVs).
• Integrate LLMs and AI models into data-processing workflows to extract structured information from unstructured and semi-structured content.
• Troubleshoot false positives, incorrect metadata, and inaccurate field values produced by LLM-based extraction pipelines.
• Implement document classification, source-specific extraction rules, data validation, reconciliation, and quality-control mechanisms.
• Transform extracted information into standardized JSON structures for downstream applications and underwriting workflows.
• Design and implement ontology-based data models representing insurance entities, business concepts, and relationships.
• Map extracted document attributes to standardized ontology concepts and metadata structures.
• Optimize processing for large and complex Excel/XLSX and SOV files, including performance, scalability, and concurrency challenges.
• Implement reliable data persistence using PostgreSQL, Lakebase, or related database technologies.
• Develop and maintain APIs and data products to support downstream application integration.
• Collaborate with brokerage, underwriting, AI, and data engineering teams to align extraction outputs with business requirements.
• Monitor production pipelines, troubleshoot data-processing issues, and continuously improve reliability, performance, and data quality.

Required Qualifications

• Minimum 7+ years of experience in data engineering, software development, or production data pipeline development.
• Minimum 4+ years of hands-on experience with Databricks and scalable data-processing pipelines.
• Minimum 5+ years of experience with Python, PySpark, and SQL.
• Minimum 3+ years of experience processing unstructured and semi-structured data, including PDFs, emails, Excel files, JSON, and business documents.
• Minimum 2+ years of experience integrating LLMs or AI models into document-processing or data-extraction pipelines.
• Minimum 3+ years of experience implementing structured data extraction, document classification, validation, reconciliation, and data-quality controls.
• Strong understanding of data modeling, schemas, metadata, entity relationships, and ontology or semantic modeling concepts.
• Minimum 2+ years of experience designing JSON outputs, APIs, and data products for downstream consumption.
• Minimum 2+ years of experience with PostgreSQL, Lakebase, or similar relational database technologies.
• Minimum 2+ years of experience troubleshooting production performance, scalability, concurrency, and data-processing issues.
• Strong understanding of cloud-based data architectures, distributed processing, and pipeline optimization.
• Strong analytical, problem-solving, communication, and cross-functional collaboration skills.

Technologies

Databricks, Python, PySpark, SQL, LLMs, AI models, PostgreSQL, Lakebase, JSON, APIs, Excel/XLSX, PDFs, emails

Education

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Engineering, or a related field, or equivalent.

Location

Charlotte, North Carolina, United States (onsite)

Travel

NA

Nice to Have

• Experience in insurance, commercial insurance, brokerage, underwriting, or risk management.
• Experience processing insurance applications, Statements of Values (SOVs), policy documents, and underwriting submissions.
• Experience with LLM frameworks, prompt engineering, structured outputs, and AI evaluation techniques.
• Experience building custom ontologies, knowledge graphs, or semantic data layers.
• Experience with document AI, OCR, information extraction, and document intelligence platforms.
• Experience with Delta Lake, Lakehouse architecture, Databricks Workflows, and Delta Live Tables.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Experience optimizing large-scale document ingestion and distributed processing workloads.