The Senior Data Engineer role at NTT Ltd focuses on designing and optimizing scalable data pipelines with Databricks and related technologies to support AI-driven document processing. The position emphasizes structured extraction from unstructured and semi-structured documents, ontology-based modeling, and delivery of standardized JSON outputs for downstream underwriting workflows.
Key Responsibilities
- Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
- Build ingestion, extraction, classification, validation, and transformation pipelines for documents including PDFs, emails, attachments, and Excel/XLSX files, as well as insurance applications and Statements of Values (SOVs).
- Integrate LLMs and AI models into data-processing workflows to extract structured information from unstructured and semi-structured content.
- Troubleshoot false positives, incorrect metadata, and inaccurate field values produced by LLM-based extraction pipelines.
- Implement document classification, source-specific extraction rules, data validation, reconciliation, and quality-control mechanisms.
- Transform extracted information into standardized JSON structures for downstream applications and underwriting workflows.
- Design and implement ontology-based data models representing insurance entities, business concepts, and relationships.
- Map extracted document attributes to standardized ontology concepts and metadata structures.
- Optimize processing for large and complex Excel/XLSX and SOV files, including performance, scalability, and concurrency challenges.
- Implement reliable data persistence using PostgreSQL, Lakebase, or related database technologies.
- Develop and maintain APIs and data products to support downstream application integration.
- Collaborate with brokerage, underwriting, AI, and data engineering teams to align extraction outputs with business requirements.
- Monitor production pipelines, troubleshoot data-processing issues, and continuously improve reliability, performance, and data quality.
Required Qualifications
- Minimum 7+ years of experience in data engineering, software development, or production data pipeline development.
- Minimum 4+ years of hands-on experience with Databricks and scalable data-processing pipelines.
- Minimum 5+ years of experience with Python, PySpark, and SQL.
- Minimum 3+ years of experience processing unstructured and semi-structured data, including PDFs, emails, Excel files, JSON, and business documents.
- Minimum 2+ years of experience integrating LLMs or AI models into document-processing or data-extraction pipelines.
- Minimum 3+ years of experience implementing structured data extraction, document classification, validation, reconciliation, and data-quality controls.
- Strong understanding of data modeling, schemas, metadata, entity relationships, and ontology or semantic modeling concepts.
- Minimum 2+ years of experience designing JSON outputs, APIs, and data products for downstream consumption.
- Minimum 2+ years of experience with PostgreSQL, Lakebase, or similar relational database technologies.
- Minimum 2+ years of experience troubleshooting production performance, scalability, concurrency, and data-processing issues.
- Strong understanding of cloud-based data architectures, distributed processing, and pipeline optimization.
- Strong analytical, problem-solving, communication, and cross-functional collaboration skills.
Technologies
Databricks, Python, PySpark, SQL, LLMs, AI models, PostgreSQL, Lakebase, JSON, APIs, Excel/XLSX, PDFs, emails
Education
Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Engineering, or a related field, or equivalent.
Location
Charlotte, North Carolina, United States (onsite)
Travel
NA
Nice to Have
- Experience in insurance, commercial insurance, brokerage, underwriting, or risk management.
- Experience processing insurance applications, Statements of Values (SOVs), policy documents, and underwriting submissions.
- Experience with LLM frameworks, prompt engineering, structured outputs, and AI evaluation techniques.
- Experience building custom ontologies, knowledge graphs, or semantic data layers.
- Experience with document AI, OCR, information extraction, and document intelligence platforms.
- Experience with Delta Lake, Lakehouse architecture, Databricks Workflows, and Delta Live Tables.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud.
- Experience optimizing large-scale document ingestion and distributed processing workloads.