Principal Machine Learning Engineer (Advanced Threat Protection)
Job Description
Palo Alto Networks is building next-generation, AI-driven cloud security capabilities, with a focus on security telemetry and Security-as-a-Service (SECaaS). In this Principal Machine Learning Engineer role based in Santa Clara, CA, you will provide technical leadership to architect and deliver production-grade machine learning and GenAI solutions from prototype to enterprise deployment.
You will work across advanced model development and systems engineering, including agents that can plan, execute multi-step workflows, self-correct, and collaborate with both multi-agent setups and human-in-the-loop processes. The work also includes diagnosing performance regressions and implementing targeted fine-tuning strategies to continuously improve safety and quality.
Key Responsibilities
- Lead the design and implementation of ML and GenAI-powered agents with dynamic planning, multi-step execution, self-correction, and seamless multi-agent and human-in-the-loop collaboration.
- Build, evaluate, and fine-tune LLMs and SLMs, including work with multimodal architectures.
- Diagnose model regressions or sub-par performance by identifying root causes such as data distribution shift, alignment failures, and loss anomalies, then implement targeted fine-tuning approaches including SFT, DPO/RLHF, and PEFT/LoRA for continuous recovery and improvement.
- Profile, debug, and optimize model training and inference on GPU clusters, including memory allocation bottlenecks (OOMs), distributed communication overhead, and compute utilization.
- Architect scalable data preprocessing and feature extraction pipelines that feed real-time endpoint telemetry into agentic and analytical workflows.
- Drive end-to-end engineering from prototype to enterprise-grade production, establishing best practices for prompt architecture, agent memory persistence, context window optimization, tool-use execution, and safety guardrails.
- Continuously evaluate state-of-the-art agent frameworks, model serving engines, and accelerator architectures to inform long-term technical roadmaps.
Requirements
- 7+ years in professional software engineering, with a proven track record deploying production-grade ML, GenAI, or agentic architectures at enterprise scale.
- Advanced proficiency in Python (AI/ML development) and production-level Go or Java for distributed microservices.
- Strong hands-on experience building and scaling data pipelines.
- Proficiency with Apache Spark, core data tools, SQL, BigQuery schema design, and Google Cloud infrastructure including Dataproc, GKE, and GCP SDKs.
- Practical operational experience with both LLMs and SLMs.
- Deep expertise diagnosing model performance issues and executing remedial fine-tuning strategies, including dataset curation, parameter-efficient fine-tuning, loss monitoring, alignment, and evaluation benchmarks.
- Practical experience with GPU profiling and debugging, such as CUDA memory optimization, multi-GPU synchronization, batch throughput tuning, and handling GPU compute crashes or stalls.
- Solid understanding of Retrieval-Augmented Generation (RAG), vector storage, prompt optimization, model evaluation metrics, and applied statistical modeling.
- BS or MS in Computer Science, Data Science, Software Engineering, or a related technical discipline.
Technologies
- Python, Go, Java, Apache Spark, SQL, BigQuery
- Google Cloud: Dataproc, GKE, GCP SDKs
- LLMs, SLMs, Retrieval-Augmented Generation (RAG), vector storage
- SFT, DPO/RLHF, PEFT/LoRA
- CUDA, GPU clusters, multi-GPU synchronization
Preferred Qualifications
- Background in cybersecurity, threat detection, or cloud security principles.
- Working knowledge of L3–L7 networking protocols, packet inspection, and network security topologies.
- Experience handling high-volume cybersecurity telemetry, endpoint detection data, or real-time threat intelligence feeds.
Compensation
- Salary range: $163,200.00 - $264,000.00/yr
- Compensation is based on qualifications, experience, and work location, and may include restricted stock units and a bonus.
Equal Opportunity and Accommodations
- Palo Alto Networks is an equal opportunity employer.
- Reasonable accommodations are available for qualified individuals with a disability.