DataJobs.io
← Back to all jobs

Job Description

Scotiabank is hiring a Senior AI Engineer in Dallas, TX (onsite) to design, build, and run AI and agentic solutions across Canadian Banking, International Banking, GBM, GWM, Fraud, and AML. This role is hands-on engineering work that spans AI applications, data pipelines, software services, MLOps, and secure, reliable production delivery using Databricks and Azure-native tooling.

What you’ll do

  • Join discovery sessions with business stakeholders to understand workflows, data sources, pain points, and opportunities for AI-enabled improvements.
  • Convert requirements into technical designs, implementation tasks, estimates, and testable acceptance criteria with product owners and technical leads.
  • Evaluate data readiness, integration dependencies, and technical feasibility, and build focused prototypes to validate assumptions before production delivery.
  • Define boundaries for agent autonomy, including human review, approvals, and deterministic business rules.
  • Develop LLM-powered applications and agentic workflows using tool calling, enterprise APIs, retrieval-augmented generation, structured outputs, and state management.
  • Implement document ingestion, parsing, chunking, embeddings, indexing, retrieval, and reranking to provide relevant, permission-aware context.
  • Integrate predictive models and business rules with AI workflows in collaboration with data scientists and domain experts.
  • Harden workflows with validation, timeouts, retries, idempotency, and fallback paths to prevent unintended repeated actions.
  • Compare model and orchestration options based on task quality, latency, cost, and operational needs.
  • Build and maintain batch and streaming data pipelines using Azure Databricks, Apache Spark, Delta Lake, Python, and SQL.
  • Apply data quality checks, schema validation, lineage, and access controls using enterprise standards and Unity Catalog where applicable.
  • Create tested APIs and services that connect AI applications to enterprise data, business systems, and approved tools.
  • Write maintainable code with clear interfaces, unit and integration tests, documentation, and peer review, and troubleshoot complex issues across application and data layers.
  • Use MLflow to track experiments, register models, version prompts and configurations, and retain evaluation results and deployment evidence.
  • Build evaluation datasets and automated checks for model performance, retrieval relevance, grounded responses, tool execution, and end-to-end task completion.
  • Implement CI/CD pipelines, automated regression testing, environment configuration, and controlled releases with rollback support.
  • Containerize services with Docker and deploy to Kubernetes, including Azure Kubernetes Service (AKS), using approved infrastructure-as-code patterns.
  • Instrument systems for tracing, logs, and metrics; monitor AI quality, data and model drift where applicable, latency, availability, and consumption costs.
  • Investigate production incidents, perform root-cause analysis, maintain runbooks, and support operations under the team’s agreed support model.
  • Implement least-privilege access, secure secrets management, data protection, and audit logging using enterprise security and Azure identity standards.
  • Apply safeguards for prompt injection, sensitive data disclosure, unsafe tool execution, and unauthorized agent actions, including human approval and escalation steps where needed.
  • Partner with cybersecurity, model risk, compliance, and data governance teams to address required controls and remediation items.
  • Work with Principal Engineers and architects to align designs with enterprise architecture and reusable platform capabilities.
  • Collaborate with data engineers, data scientists, platform engineers, and business teams to resolve dependencies and deliver integrated solutions.
  • Mentor engineers through code reviews, pairing, and knowledge sharing; contribute reusable components and practical engineering guidance.
  • Communicate progress, blockers, technical trade-offs, and delivery risks clearly and early.

Requirements

  • Strong experience in software, data, or machine learning engineering, including hands-on delivery of production AI or machine learning solutions.
  • Proficiency in Python and SQL, API development, automated testing, Git, and modern software engineering practices.
  • Hands-on experience building generative AI applications or agentic workflows, including retrieval, tool integration, orchestration, and evaluation.
  • Experience with Databricks, Apache Spark, Delta Lake, and scalable data processing pipelines.
  • Experience with MLflow and MLOps practices such as experiment tracking, model registration, versioning, deployment, and monitoring.
  • Practical experience with Microsoft Azure services and cloud application security, identity, networking, and observability.
  • Experience with Docker, Kubernetes, CI/CD pipelines, and infrastructure as code, including ability to deploy and troubleshoot services.
  • Ability to independently deliver well-defined engineering work, investigate ambiguous problems, and collaborate across business and technical teams.
  • Understanding of AI evaluation, data privacy, secure tool use, human oversight, and the limitations of probabilistic model outputs.

Technologies

  • Databricks, Azure, Azure Databricks, Apache Spark, Delta Lake, Unity Catalog
  • MLflow, tool calling, retrieval-augmented generation, structured outputs, state management
  • Document ingestion, chunking, embeddings, indexing, reranking, vector search
  • Kubernetes, Azure Kubernetes Service (AKS), Docker, REST APIs, Python, SQL
  • Azure Data Lake Storage, Azure Data Factory, Azure Event Hubs, Azure Functions, Azure API Management
  • Azure DevOps, GitHub Actions, Terraform, Bicep
  • Azure Monitor, Application Insights, Key Vault, Microsoft Entra ID
  • Tracing, logs, metrics, audit logging
  • Prompt injection, human approval
  • Model Context Protocol (MCP), event-driven architectures, agent orchestration frameworks, semantic models, knowledge graph integrations

Benefits

  • Holistic well-being
  • Flexible benefit programs to support unique family, financial, physical, mental, and social health needs

Preferred qualifications

  • Experience in financial services, particularly banking, capital markets, wealth management, fraud detection, or AML.
  • Experience implementing retrieval and vector search, document processing, semantic models, or knowledge graph integrations.
  • Familiarity with Model Context Protocol (MCP), event-driven architectures, and agent orchestration frameworks.
  • Experience working in regulated environments with model risk, audit, and data governance requirements.
  • Bachelor’s or master’s degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.

Hiring location

United States: Texas: Dallas

Hiring manager: Prabhu Gururaj

Similar Jobs