Lead AI Engineer

Vytwo is building production-ready AI solutions and is hiring a Lead AI Engineer to help design and deploy end-to-end data and AI pipelines on Azure Databricks and other Azure cloud services. This onsite role in Dallas, TX focuses on moving AI from experimentation to reliable production systems, including deployment, monitoring, and optimization across ML and Generative AI use cases.

What you will do

• Design and build end-to-end data and AI pipelines using Azure Databricks.
• Develop robust ETL/ELT workflows with Python (including PySpark) and SQL.
• Implement CI/CD pipelines for Databricks deployments, including jobs, notebooks, and workflows.
• Integrate Databricks with Azure services such as Data Lake, Blob Storage, Key Vault, Azure OpenAI, and Azure Functions.
• Optimize Databricks jobs for performance, cost, and reliability.
• Create reusable, modular code to support maintainable production systems.
• Collaborate with data scientists and platform teams to move models from experimentation to production deployment.
• Implement logging, monitoring, and error handling for production pipelines.
• Develop and deploy ML and Generative AI models, including LLMs, embeddings, and RAG pipelines for NLP, computer vision, and predictive analytics.
• Fine-tune LLMs using LoRA or QLoRA, and integrate with Azure OpenAI or Hugging Face models.
• Build vector search and retrieval pipelines using FAISS or Azure Cognitive Search.
• Support responsible AI practices, including bias detection and model governance.

What you bring

• Hands-on experience with Azure Databricks, including jobs, workflows, and clusters (with Unity Catalog preferred).
• Python skills with strong emphasis on PySpark rather than only pandas.
• SQL experience, including complex joins, window functions, and analytical queries.
• Working knowledge of Azure cloud components such as ADLS Gen2, ADF, Key Vault, and IAM concepts.
• Experience with pipeline orchestration and deployment, including CI/CD and environment promotion.
• Experience with Azure DevOps.
• Strong understanding of the ML lifecycle and MLOps best practices.
• Experience deploying models using MLflow or similar frameworks.

Technologies you will work with

• Azure Databricks; Databricks workflows, jobs, and notebooks
• Azure cloud infrastructure including Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions
• Python, PySpark, SQL; ETL/ELT
• CI/CD pipelines; Logging and Monitoring
• ML and Generative AI; MLflow
• LoRA, QLoRA, Hugging Face models
• FAISS and Azure Cognitive Search; vector search and RAG pipelines
• Unity Catalog; ADLS Gen2; ADF; IAM concepts; Azure DevOps
• NLP, computer vision, predictive analytics

Strong advantage

• Experience with ML and Generative AI workloads on Databricks.
• Experience with RAG, embeddings, or inference pipelines.
• Terraform / ARM / Bicep experience for infrastructure.
• Databricks Asset Bundles experience.
• Airflow or ADF orchestration experience.
• Production monitoring and cost optimization experience.
• Knowledge of LangChain or similar frameworks for AI application development.
• Experience with Azure AI services such as Azure Machine Learning or Azure Cognitive Services.