AI Engineer
Agentic Ai
Ai Agent
Ai Agent Platform
Ai Platform
Ai Security
Application Security
Artificial Intelligence
Artificial Intelligence Engineer
Automation
Azure
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Data Platform
Cloud Engineer
Cloud Infrastructure
Cloud Native
Cloud Native Technologies
Cloud Platform
Cloud Platforms
Cloud Platforms Cloud Platforms
Cloud Technology
Data Analysis
Data Analytics
Data Engineer
Data Engineering
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Science
Data Warehouse
Database
Databases
Databricks
Databricks Mlflow
DevOps
DevSecOps
Engineering
Engineering Software
Generative AI
Information Technology (IT)
Infrastructure As Code
Kubernetes
Llm Operations
Platform Engineering
Programming
Risk Management
Security Automation
Security Devops
Software Development
Software Engineering
Software Security
Spark
SQL
Job Description
Scotiabank is hiring a Senior AI Engineer in Dallas, TX (onsite) to design, build, and run AI and agentic solutions across Canadian Banking, International Banking, GBM, GWM, Fraud, and AML. This role is hands-on engineering work that spans AI applications, data pipelines, software services, MLOps, and secure, reliable production delivery using Databricks and Azure-native tooling.
What you’ll do
- Join discovery sessions with business stakeholders to understand workflows, data sources, pain points, and opportunities for AI-enabled improvements.
- Convert requirements into technical designs, implementation tasks, estimates, and testable acceptance criteria with product owners and technical leads.
- Evaluate data readiness, integration dependencies, and technical feasibility, and build focused prototypes to validate assumptions before production delivery.
- Define boundaries for agent autonomy, including human review, approvals, and deterministic business rules.
- Develop LLM-powered applications and agentic workflows using tool calling, enterprise APIs, retrieval-augmented generation, structured outputs, and state management.
- Implement document ingestion, parsing, chunking, embeddings, indexing, retrieval, and reranking to provide relevant, permission-aware context.
- Integrate predictive models and business rules with AI workflows in collaboration with data scientists and domain experts.
- Harden workflows with validation, timeouts, retries, idempotency, and fallback paths to prevent unintended repeated actions.
- Compare model and orchestration options based on task quality, latency, cost, and operational needs.
- Build and maintain batch and streaming data pipelines using Azure Databricks, Apache Spark, Delta Lake, Python, and SQL.
- Apply data quality checks, schema validation, lineage, and access controls using enterprise standards and Unity Catalog where applicable.
- Create tested APIs and services that connect AI applications to enterprise data, business systems, and approved tools.
- Write maintainable code with clear interfaces, unit and integration tests, documentation, and peer review, and troubleshoot complex issues across application and data layers.
- Use MLflow to track experiments, register models, version prompts and configurations, and retain evaluation results and deployment evidence.
- Build evaluation datasets and automated checks for model performance, retrieval relevance, grounded responses, tool execution, and end-to-end task completion.
- Implement CI/CD pipelines, automated regression testing, environment configuration, and controlled releases with rollback support.
- Containerize services with Docker and deploy to Kubernetes, including Azure Kubernetes Service (AKS), using approved infrastructure-as-code patterns.
- Instrument systems for tracing, logs, and metrics; monitor AI quality, data and model drift where applicable, latency, availability, and consumption costs.
- Investigate production incidents, perform root-cause analysis, maintain runbooks, and support operations under the team’s agreed support model.
- Implement least-privilege access, secure secrets management, data protection, and audit logging using enterprise security and Azure identity standards.
- Apply safeguards for prompt injection, sensitive data disclosure, unsafe tool execution, and unauthorized agent actions, including human approval and escalation steps where needed.
- Partner with cybersecurity, model risk, compliance, and data governance teams to address required controls and remediation items.
- Work with Principal Engineers and architects to align designs with enterprise architecture and reusable platform capabilities.
- Collaborate with data engineers, data scientists, platform engineers, and business teams to resolve dependencies and deliver integrated solutions.
- Mentor engineers through code reviews, pairing, and knowledge sharing; contribute reusable components and practical engineering guidance.
- Communicate progress, blockers, technical trade-offs, and delivery risks clearly and early.
Requirements
- Strong experience in software, data, or machine learning engineering, including hands-on delivery of production AI or machine learning solutions.
- Proficiency in Python and SQL, API development, automated testing, Git, and modern software engineering practices.
- Hands-on experience building generative AI applications or agentic workflows, including retrieval, tool integration, orchestration, and evaluation.
- Experience with Databricks, Apache Spark, Delta Lake, and scalable data processing pipelines.
- Experience with MLflow and MLOps practices such as experiment tracking, model registration, versioning, deployment, and monitoring.
- Practical experience with Microsoft Azure services and cloud application security, identity, networking, and observability.
- Experience with Docker, Kubernetes, CI/CD pipelines, and infrastructure as code, including ability to deploy and troubleshoot services.
- Ability to independently deliver well-defined engineering work, investigate ambiguous problems, and collaborate across business and technical teams.
- Understanding of AI evaluation, data privacy, secure tool use, human oversight, and the limitations of probabilistic model outputs.
Technologies
- Databricks, Azure, Azure Databricks, Apache Spark, Delta Lake, Unity Catalog
- MLflow, tool calling, retrieval-augmented generation, structured outputs, state management
- Document ingestion, chunking, embeddings, indexing, reranking, vector search
- Kubernetes, Azure Kubernetes Service (AKS), Docker, REST APIs, Python, SQL
- Azure Data Lake Storage, Azure Data Factory, Azure Event Hubs, Azure Functions, Azure API Management
- Azure DevOps, GitHub Actions, Terraform, Bicep
- Azure Monitor, Application Insights, Key Vault, Microsoft Entra ID
- Tracing, logs, metrics, audit logging
- Prompt injection, human approval
- Model Context Protocol (MCP), event-driven architectures, agent orchestration frameworks, semantic models, knowledge graph integrations
Benefits
- Holistic well-being
- Flexible benefit programs to support unique family, financial, physical, mental, and social health needs
Preferred qualifications
- Experience in financial services, particularly banking, capital markets, wealth management, fraud detection, or AML.
- Experience implementing retrieval and vector search, document processing, semantic models, or knowledge graph integrations.
- Familiarity with Model Context Protocol (MCP), event-driven architectures, and agent orchestration frameworks.
- Experience working in regulated environments with model risk, audit, and data governance requirements.
- Bachelor’s or master’s degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.
Hiring location
United States: Texas: Dallas
Hiring manager: Prabhu Gururaj