DataJobs.io
← Back to all jobs

Job Description

Palo Alto Networks is building next-generation, AI-driven cloud security capabilities, with a focus on security telemetry and Security-as-a-Service (SECaaS). In this Principal Machine Learning Engineer role based in Santa Clara, CA, you will provide technical leadership to architect and deliver production-grade machine learning and GenAI solutions from prototype to enterprise deployment.

You will work across advanced model development and systems engineering, including agents that can plan, execute multi-step workflows, self-correct, and collaborate with both multi-agent setups and human-in-the-loop processes. The work also includes diagnosing performance regressions and implementing targeted fine-tuning strategies to continuously improve safety and quality.

Key Responsibilities

  • Lead the design and implementation of ML and GenAI-powered agents with dynamic planning, multi-step execution, self-correction, and seamless multi-agent and human-in-the-loop collaboration.
  • Build, evaluate, and fine-tune LLMs and SLMs, including work with multimodal architectures.
  • Diagnose model regressions or sub-par performance by identifying root causes such as data distribution shift, alignment failures, and loss anomalies, then implement targeted fine-tuning approaches including SFT, DPO/RLHF, and PEFT/LoRA for continuous recovery and improvement.
  • Profile, debug, and optimize model training and inference on GPU clusters, including memory allocation bottlenecks (OOMs), distributed communication overhead, and compute utilization.
  • Architect scalable data preprocessing and feature extraction pipelines that feed real-time endpoint telemetry into agentic and analytical workflows.
  • Drive end-to-end engineering from prototype to enterprise-grade production, establishing best practices for prompt architecture, agent memory persistence, context window optimization, tool-use execution, and safety guardrails.
  • Continuously evaluate state-of-the-art agent frameworks, model serving engines, and accelerator architectures to inform long-term technical roadmaps.

Requirements

  • 7+ years in professional software engineering, with a proven track record deploying production-grade ML, GenAI, or agentic architectures at enterprise scale.
  • Advanced proficiency in Python (AI/ML development) and production-level Go or Java for distributed microservices.
  • Strong hands-on experience building and scaling data pipelines.
  • Proficiency with Apache Spark, core data tools, SQL, BigQuery schema design, and Google Cloud infrastructure including Dataproc, GKE, and GCP SDKs.
  • Practical operational experience with both LLMs and SLMs.
  • Deep expertise diagnosing model performance issues and executing remedial fine-tuning strategies, including dataset curation, parameter-efficient fine-tuning, loss monitoring, alignment, and evaluation benchmarks.
  • Practical experience with GPU profiling and debugging, such as CUDA memory optimization, multi-GPU synchronization, batch throughput tuning, and handling GPU compute crashes or stalls.
  • Solid understanding of Retrieval-Augmented Generation (RAG), vector storage, prompt optimization, model evaluation metrics, and applied statistical modeling.
  • BS or MS in Computer Science, Data Science, Software Engineering, or a related technical discipline.

Technologies

  • Python, Go, Java, Apache Spark, SQL, BigQuery
  • Google Cloud: Dataproc, GKE, GCP SDKs
  • LLMs, SLMs, Retrieval-Augmented Generation (RAG), vector storage
  • SFT, DPO/RLHF, PEFT/LoRA
  • CUDA, GPU clusters, multi-GPU synchronization

Preferred Qualifications

  • Background in cybersecurity, threat detection, or cloud security principles.
  • Working knowledge of L3–L7 networking protocols, packet inspection, and network security topologies.
  • Experience handling high-volume cybersecurity telemetry, endpoint detection data, or real-time threat intelligence feeds.

Compensation

  • Salary range: $163,200.00 - $264,000.00/yr
  • Compensation is based on qualifications, experience, and work location, and may include restricted stock units and a bonus.

Equal Opportunity and Accommodations

  • Palo Alto Networks is an equal opportunity employer.
  • Reasonable accommodations are available for qualified individuals with a disability.

Similar Jobs