The open AI-native stack on Kubernetes: From open models to distributed inference and agents

Over the past decade, Kubernetes established itself as the standard for orchestrating microservices and containerized applications. Today, we're witnessing the next major architectural shift in distributed computing: AI workloads moving from isolated research prototypes into large-scale, multi-tenant production fleets. In this post, we'll walk through the open-source stack that powers this shift and show you the sample manifests that you can try on your own Kubernetes clusters. You can also join us at KubeCon + CloudNativeCon North America 2026 to see these projects in action through live demos and technical sessions.

For modern engineering teams, production AI is rapidly moving beyond generic calls to closed, black-box APIs. Organizations are realizing that their long-term competitive advantage lies in domain specialization. Platform engineers, ML researchers, and AI builders want the autonomy to run high-performing open-weight models (such as the Gemma family). They want to adapt them to proprietary enterprise datasets by using reinforcement learning (RL) and serve them with predictable latency, high throughput, and maximum hardware efficiency.

Whether it's scientific initiatives training domain models like DolphinGemma (an LLM helping scientists study how dolphins communicate) on acoustic data, legal platforms evaluating complex contracts, or engineering teams building autonomous coding agents, the objective is the same: achieving frontier-level task performance at a fraction of the serving cost and latency.

Yet, scaling RL post-training, stateful distributed inference, and autonomous agent execution on traditional container infrastructure introduces substantial hurdles:

• Accelerator underutilization in RL: When you alternate between training and rollout generation, expensive GPUs and TPUs remain idle for most of the training run.

• Cache-blind load balancing: When you use standard Kubernetes round-robin and least-connections routing, multi-turn agent calls are scattered across arbitrary pods and wipe out key-value (KV) cache locality.

• Framework fragmentation: When you operate parallel, siloed compute clusters (such as bespoke Ray or Slurm setups) alongside Kubernetes, you fragment capacity and multiply maintenance toil.

• Agent security in multi-tenant clusters: When you execute untrusted, model-generated shell commands and scripts in shared environments, you expose clusters to sandbox escapes and privilege escalation.

At KubeCon + CloudNativeCon North America 2026, Google Cloud is addressing these challenges by introducing and expanding a unified, open-source AI-native stack across Kubernetes and Google Kubernetes Engine (GKE) based on these components:

• OpenRL (CNCF Sandbox): A Kubernetes-native post training service that simplifies RL post-training and maximizes accelerator utilization.

• llm-d 1.0 (CNCF Sandbox): An open distributed inference control plane that provides prefix-cache-aware routing, disaggregated serving, and multi-tier KV cache management.

• Ray on GKE with KubeRay (CNCF Sandbox): A framework that provides enterprise lifecycle management, elastic autoscaling, and Kueue-driven scheduling for distributed Ray compute and data processing.

• Agent Substrate (CNCF Sandbox): An open orchestration runtime that provides lifecycle management and execution targets for multi-turn AI agent workflows.

• gVisor (CNCF Sandbox): A kernel-level container sandbox that delivers strong system call isolation when you execute untrusted agent code in multi-tenant clusters.

• TorchTPU: A backend that provides first-class, native PyTorch support across Cloud TPUs and GPUs inside upstream vLLM and SGLang.

Bridge reinforcement learning and Kubernetes with OpenRL

While supervised fine-tuning (SFT) has become standard practice, domain excellence especially in multi-step reasoning, mathematical proofing, and autonomous tool use requires RL.

Historically, it's been difficult to bridge the world of ML experimentation with production Kubernetes fleets. ML researchers want to iterate rapidly in Python and PyTorch by using standard reward functions, while platform teams need multi-tenant governance, quota enforcement, and predictable scheduling.

To eliminate this friction, Google created OpenRL and contributed it to the CNCF Sandbox.

Architectural highlights of OpenRL

• Clean 4-primitive Python abstraction: Researchers write standard Python training code that's centered around four core primitives (forward_backward, optim_step, sample, and save_state). OpenRL translates these primitives into Kubernetes scheduling workflows without requiring developers to learn Kubernetes internals or manage infrastructure code.

• Cooperative accelerator time-slicing: OpenRL interleaves training steps and rollout inferences on the same physical accelerators. This approach avoids the need to dedicate separate, static clusters to compute-bound training and latency-sensitive rollout generation.

• Delta-weight synchronization: Shipping full multi-gigabyte model weights over cluster networking after every optimization step creates massive network saturation. OpenRL transmits only parameter deltas to rollout workers, eliminating rollout cold starts.

• Zero-config dynamic resource allocation (DRA): OpenRL automatically profiles model context lengths, batch dimensions, and LoRA versus full fine-tuning configurations. It then dynamically bin-packs workloads across a heterogenous accelerator fleet by using native Kubernetes DRA claims.

Distributed inference at scale with llm-d 1.0

After you post-train a model, serving it serving it under interactive, bursty agent traffic presents a new set of distributed systems challenges. Standard ingress controllers treat LLM traffic as stateless HTTP requests, scattering sequential turns across different replicas and constantly rebuilding KV cache.

Google Cloud, Red Hat, IBM Research, CoreWeave, and NVIDIA co-founded the project, with contributions from over 500 engineers across 25+ organizations. Today, llm-d 1.0 is the CNCF Sandbox distributed inference control plane that transforms standalone inference engines (like vLLM and SGLang) into a coordinated, cluster-wide serving fabric.

Core capabilities of llm-d 1.0

1. Prefix-cache-aware routing (Endpoint Picker): The Endpoint Picker integrates into the Kubernetes Gateway API Inference Extension, and GKE manages it natively through GKE Inference Gateway. The Endpoint Picker evaluates real-time KV cache residency across worker pods. In production deployments, it's doubled KV cache hit rates from 35% to 70% and cut median time-to-first-token (TTFT) by 70-85% on multi-turn agent workloads.

2. 3-Tier KV cache offload: When accelerator high-bandwidth memory (HBM) nears capacity during long-context interactions, llm-d cascades KV cache blocks from HBM → host CPU RAM → local NVMe SSD / Google Cloud Managed Lustre / Cloud Storage. This offload prevents expensive context recalculations.

3. EPD Disaggregation with DisaggregatedSet: Decouples Encoding, Prefill, and Decode (EPD) onto specialized hardware pools that are connected by high-speed interconnects. DisaggregatedSet is built built on Kubernetes LeaderWorkerSet (LWS). It enables independent horizontal scaling and coordinated, zero-downtime rolling upgrades.

4. Native TorchTPU upstream support: PyTorch models run natively on Google Cloud TPUs inside upstream vLLM and SGLang through TorchTPU. Developers get day-zero parity across TPU v5e, v6e, and v7x alongside NVIDIA GPUs without the need to maintain private forks or rewrite pipelines.

5. Latency-tolerant async processing & flow control: The llm-d-async-processor component pulls background, latency-tolerant tasks from Pub/Sub or Redis into cluster slack capacity. It applies strict traffic prioritization so background processing never degrades interactive user experiences.

6. Fleet observability with Prism: Track cluster-wide throughput, inter-token latency (ITL), TTFT, and cache hit ratios by using Prism, the llm-d open-source benchmarking and performance dashboard.

Distributed Python & ML orchestration with KubeRay

While specialized runtimes handle inference and RL loops, distributed data preparation, embedding generation, and distributed PyTorch training depend heavily on Ray.

KubeRay provides an open-source, Kubernetes-native operator that manages the complete lifecycle of Ray clusters, jobs, and services (RayCluster, RayJob, and RayService custom resources) on Kubernetes.

Why KubeRay matters for platform teams

• Declarative cluster lifecycle: Platform engineers can spin up ephemeral, self-healing Ray clusters on demand for ad-hoc training runs, or run long-lived clusters for batch pipelines.

• Native Kueue queueing & multi-tenancy: KubeRay integrates directly with Kueue for multi-tenant fair sharing, priority preemption, and quota management across shared GPU and TPU capacity.

• Elastic autoscaling: KubeRay dynamically scales worker pods based on active Ray tasks and actor placement, preventing idle resource consumption.

Strong isolation for AI agents: Agent Substrate & gVisor

As multi-turn AI agents advance from conversational assistants to autonomous systems that generate code, execute shell commands, and orchestrate external APIs running untrusted, model-generated code in shared clusters introduces serious security risks.

Google Cloud is contributing Agent Substrate and continuing core contributions to gVisor in the CNCF Sandbox:

• Kernel-level sandboxing: gVisor provides a user-space kernel that intercepts and virtualizes system calls, which helps ensure that untrusted agent code can't compromise the host kernel or access neighboring tenant workloads.

• Sub-second ephemeral execution: Sandboxes spin up in milliseconds per agent turn, providing isolated execution environments for both RL rollout generation in OpenRL and live production agent workflows.

Proven in practice: Mastering domain reasoning

To evaluate this complete open-source stack on a rigorous enterprise workload, we tested Gemma 4 against the open-source Legal Agent Benchmark (LAB) developed by Harvey.ai. This benchmark spans 24 complex legal practice areas from mergers and acquisitions (M&A)M&A due diligence to contract dispute analysis. The evaluation produced these results:

• Benchmark results: While standard SFT plateaued around 50% accuracy on multi-step reasoning rubrics, when we applied reproducible RL loops with OpenRL, we lifted model accuracy to 70%.

• Operational efficiency: The entire training pipeline dynamically shared GPU and TPU capacity with concurrent LoRA and inference jobs on GKE without manual partition management.

How teams are adopting this stack

Connect with us at KubeCon North America 2026

Attending KubeCon NA in Salt Lake City? Come see these open-source technologies in action:

• Join Google Cloud at KubeCon + CloudNativeCon North America 2026:

• Come say hi to us at booth #122 in Hall B.

• Discover Googlers talks on our Google Cloud at KubeCon North America 2026 event site.

• Attend featured sessions & keynotes:

• Kubernetes with Conviction: How Customization is Shaping the OSS AI Ecosystem's Future - Salt Palace | Level 1 | Hall DE.

• The Latest in GPU, TPU, NIC and Other Device Support (WG Device Management) - Salt Palace | Level 2 | 250 D-F

• Kubernetes Solutions for Agent-Shaped Problems - Salt Palace | Level 1 | Hall DE

• How Anthropic Almost Melted GKE (And the PRs That Saved It) - Salt Palace | Level 1 | Ballroom H

• Gateway API Project Update: Building the Future of Traffic - Salt Palace | Level 2 | 250 D-F

• Watch live demos at the Google Cloud booth:

• Demo | Inferencing with ARM processors on GKE (Tue 1:30-4:30PM MST)

• Sandbox your agent workflows with gVisor (Tue 4:30-7:30PM MST)

• Securely running fleets of agents with OpenShell (Wed 10:30-1:30PM MST)

• Cost optimization via GKE spot nodes & node auto-provisioning (Wed 12:30-2:30PM MST)

• From Reasoning to Action: Arm-Powered Coding Agents with GKE Agent Sandbox and Axion (Thu 10:30-1:30PM MST)

Get started

1. Explore OpenRL: Visit openrl.dev and check out the OpenRL GitHub repository for Kubernetes-native RL fine-tuning recipes.

2. Deploy llm-d 1.0: Head to llm-d.ai to test Gateway API Inference extensions and deploy prefix-aware inference pools on your clusters.

3. Run KubeRay: Follow the KubeRay documentation to manage distributed Ray workloads alongside Kueue.

4. Deploy Gemma on GKE: Explore step-by-step guides for deploying Gemma models on TPUs and GPUs in the Google Kubernetes Engine documentation.