How Palo Alto Networks built a multi tenant scalable Unified Data Platform

Enterprises across the world are processing significant amounts of data. Palo Alto Networks processes thousands of firewall logs, telemetry signals and threat events every second across its product portfolio. To support this scale, Palo Alto Networks had 30,000 individual data pipelines, each with its own operational load. And while this single tenant architecture model worked originally, it  had recently started to slow innovation, limit further scale, and made onboarding new analytics use cases increasingly costly.

To support the next generation of security products, Palo Alto Networks partnered with Google Cloud to modernize their data processing landscape into a unified multi-tenant platform powered by Dataflow, Pub/Sub and BigQuery. This transformation became the foundation of Palo Alto Network's Unified Data Platform (UDP), which now processes billions of events every day with improved agility, simpler operations and meaningful cost efficiency.

The challenge: A single tenant architecture could not keep pace

Before migrating, Palo Alto Network’s data platform was built around a “one pipeline per tenant” model. Each tenant pipeline required its own configuration, troubleshooting, on-call rotations and capacity tuning. As Palo Alto Networks usage grew, so did the friction:

• Brittle alerting and weekly operational overhead to support more than 30,000 pipelines that were processing a combined throughput of roughly 30 GB per second.

• Slow deployment cycles made onboarding new tenants harder.

• Significant compute resources were dedicated to each tenant, regardless of load.

• Engineering time was spent managing infrastructure instead of building new analytics.

This model hindered operational agility and made it challenging to scale,as new product lines expanded and data volumes increased.

The transformation: Embracing a new architectural paradigm with Dataflow

The turning point came when the team recognized that Google Cloud Dataflow’s serverless auto scaling architecture could support a completely different operating model. Instead of maintaining thousands of individual pipelines, Palo Alto Networks could unify workloads into a multi-tenant system where resources are shared intelligently across tenants.

Several core capabilities made this possible:

1. The architectural shift
Dataflow allowed the team to move from “one job per tenant” to a “shared resource pool” that can handle multiple tenants within a single architecture. This shift dramatically simplified operations and unlocked new efficiencies.

2. Unlocking multi tenancy at scale
Dataflow’s autoscaling engine manages fluctuating workloads with ease, accommodating the unpredictable spikes that are common in cybersecurity environments. This eliminated the need for manual capacity planning.

3. Operational freedom
By using Flex Templates and Dataflow’s managed service model, the team transformed their CI/CD process from week-long deployment cycles into a single day workflow. Engineers no longer spend time managing infrastructure and can instead focus on analytics, threat detection and product innovation.

4. Unified execution
With all jobs running on a shared Dataflow based platform, the team gained flexibility to move workloads across real time and batch systems without maintaining different codebases.

5. Observability
With Dataflow, the team relies on built in logging and metrics to monitor pipeline health across both real time and batch workloads, providing clear visibility into performance without additional tooling. Dataflow exposes the full set of metrics required for on-call alerting, eliminating the need to build or maintain custom metrics in the PANW codebase. When alerts trigger, the Dataflow UI enables engineers to quickly identify performance bottlenecks and take corrective action.

Architecture overview

The impact: A meaningful shift in value, cost and engineering focus

The migration to Dataflow did not just modernize the old system. It fundamentally changed how the engineering teams work, delivering impact across several dimensions.

• The economic win: By consolidating pipelines and relying on Dataflow’s autoscaling, Palo Alto Networks achieved around 30 percent compute cost savings. These savings were driven by a reduction in redundant pipelines, better utilization of shared resources and elimination of manual capacity tuning.
• The platform win: The Unified Data Platform became the long term standard for real time data processing across the company. It provides a “Dataflow native” blueprint that is scalable, repeatable and ready to support new product lines without duplicating engineering effort.
• The people win: With Dataflow handling operational complexity, engineers now focus on building new analytics features instead of managing infrastructure. This shift improved morale, accelerated delivery cycles and reduced alert fatigue.
  
  “The real differentiator for us was Dataflow’s ability to handle true multi-tenancy at massive scale. Its autoscaling engine is sophisticated enough to manage resources across thousands of tenants in a single job, which was key to unlocking around 30 percent cost savings. We moved from a world of managing more than 30,000 jobs to just a handful & that has fundamentally changed how our team operates.” — Palo Alto Networks Engineering Team

Extending the model: Use cases beyond cybersecurity

The architectural pattern Palo Alto Networks adopted is broadly applicable to any organization dealing with multi-tenant real time data at scale. Examples include:

• E-commerce: powering real time dashboards for thousands of merchants on a single marketplace

• Gaming: processing telemetry signals from millions of players to update leaderboards and detect fraud

• Fintech: monitoring transactions from hundreds of banks to flag suspicious behavior in real time

• IoT and logistics: analyzing data from fleets of vehicles to optimize routing and maintenance schedules

The same principles of multi-tenancy, shared execution and autoscaling can accelerate efficiency across many industries.

Building a sustainable data future

By standardizing on Dataflow, Palo Alto Networks has laid the foundation for long term agility in their security analytics platform. The Unified Data Platform now serves as the cornerstone of their real time data strategy, helping them innovate faster and operate with greater economic efficiency.

Their journey highlights how a flexible high performance data processing engine like Dataflow can give enterprises the confidence to scale without increasing operational overhead. It also provides a reusable playbook for teams that want to modernize their real time architectures using Google Cloud.

To learn more about how you can modernize your data pipelines, visit the Dataflow product page.