DataJobs.io
← Back to all jobs

Job Description

Palo Alto Networks is building resilient AI-driven security across cloud environments, and this role sits at the center of that effort. As a Senior Principal Machine Learning Engineer, you will help set the technical direction for advanced threat protection initiatives, combining architectural strategy with production-ready engineering rigor. The impact is direct: you will partner across Product, Security Research, Data Science, and Infrastructure to deliver Security-as-a-Service capabilities designed to withstand real-world, high-stakes threat landscapes.

Location: Santa Clara, CA (onsite)

Compensation: $190,400 - $308,000 per year. Compensation depends on qualifications, experience, and work location. RSUs and a bonus may also be included.

Immigration sponsorship: Yes

Responsibilities

  • Act as the organization’s primary technical compass by evaluating frontline AI research, separating short-lived hype from production-viable approaches, and steering engineering investments toward high-leverage architectural paths.
  • Drive engineering rigor by implementing RFC standards, leading architectural review boards (ARBs), and mentoring Staff and Principal engineers to scale technical excellence across teams.
  • Translate ambiguous business objectives and complex threat landscapes into concrete technical requirements, milestone architectures, and executable designs in close collaboration with Product Management, Security Researchers, Data Science, and Infrastructure.
  • Own macro- and micro-level architecture for highly distributed, fault-tolerant cloud systems that process real-time security data streams at petabyte scale.
  • Architect production-grade agentic frameworks with dynamic multi-step planning, stateful memory persistence, secure tool execution, and deterministic guardrails.
  • Lead ingestion, preprocessing, and feature engineering pipeline architecture using Apache Spark, SQL, and BigQuery across Google Cloud (Dataproc, GKE, and GCP SDKs).
  • Maintain hands-on mastery of evolving model options and guide architectural decisions across LLMs, SLMs, and multimodal systems with focus on cost, latency, privacy, and domain efficacy.
  • Set up systematic frameworks to diagnose underperforming models in production, including drift, context dilution, alignment collapse, and hallucination spikes, then lead remediation through fine-tuning methods such as SFT, DPO, RL, and LoRA/PEFT.
  • Troubleshoot low-level compute bottlenecks on distributed GPU clusters, including distributed training or inference issues, CUDA out-of-memory (OOM) conditions, NCCL overhead, and kernel performance stalls.

Requirements

  • 10+ years of progressive software engineering and systems design experience, with a documented record of technical leadership delivering high-availability, enterprise-scale platforms.
  • Proven ability to author complex system designs from scratch, establish data contracts, manage architectural trade-offs, and lead large-scale legacy-to-modern migrations.
  • Demonstrated experience aligning cross-functional teams (Product, Data, Applied Research, SecOps, Infrastructure) around a unified technical vision and resolving competing priorities.
  • Deep hands-on knowledge of modern AI architectures, including Transformers, attention mechanisms, and tokenization strategies, plus operational experience deploying and serving LLMs and SLMs.
  • Expert capability in diagnosing why models degrade and executing targeted recovery pipelines using data curation, synthetic data generation, supervised fine-tuning, preference alignment, and parameter-efficient tuning.
  • Practical skills profiling and debugging distributed GPU workloads, including memory leaks, concurrency locks, and cluster communication bottlenecks.
  • Advanced proficiency in Python for AI/ML pipelines and scientific computing.
  • Production-level fluency in Go or Java for low-latency, concurrent backend microservices.
  • Strong data engineering foundation with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration (Dataproc, GKE).
  • BS, MS, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline.

Preferred Qualifications (Strong Bonus)

  • Deep background in threat modeling, endpoint detection and response (EDR), SIEM architectures, or cloud security posture management.
  • Working knowledge of L3–L7 network protocols, packet telemetry, and deep packet inspection systems.
  • Contributions to open-source foundation model tooling, published research, or active participation in the broader AI systems community.

Technologies

Python, Go, Java, Apache Spark, SQL, BigQuery, Dataproc, GKE, GCP SDKs, Large Language Models (LLMs), Small Language Models (SLMs), multimodal systems, Transformers, attention mechanisms, tokenization strategies, SFT, DPO, RL, LoRA, PEFT, CUDA, NCCL, agentic frameworks, supervised fine-tuning, preference alignment, parameter-efficient tuning

Similar Jobs