Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets

Google team have released llm-d 1.0, the first stable version of its distributed inference stack.This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool.They released it at the PyTorch Conference North America.

It targets self-hosted agent and RL workloads, where round-robin load balancing breaks down.TL;DR A serving stack: Google reports a published benchmark on a 17,000-node multi-region GKE deployment.Runs on: Kubernetes with NVIDIA, AMD or Intel GPUs or Google TPUs, over vLLM or SGLang.

Performance: Vendor-reported gains come from cache-aware routing, tiered KV cache offload and prefill/decode disaggregation.Best: Median TTFT cut by more than 85%, with 2.4x throughput, on an Ironwood TPU coding workload.

Worst: RL time-slicing lifts accelerator duty cycle only to above 70%, so roughly 30% idle time remains.What is llm-d 1.0?llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA.Red Hat, IBM and Google are the lead maintainers.

The project counts more than 500 contributors from over 25 organizations.It is a CNCF Sandbox project, accepted on March 12, 2026, and is licensed under Apache 2.0.Prefix-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference already shipped.

Version 1.0 is about running them reliably in production, which IBM calls day 2 operations.What is new in llm-d 1.0?Multi-tenant quality of service: Priority and fairness-aware flow control stops bursty agents from starving latency-sensitive requests.

Stable foundation: v1 APIs and a hardened request path, validated on multiple clouds and accelerators.Zero-downtime upgrades: Router high availability with hot standby, plus DisaggregatedSet, which rolls prefill and decode pools together.

Agent-aware serving: Session-aware routing, turn-priority scheduling and native Responses API state.Async inference: llm-d-async-processor pulls jobs from Google Cloud Pub/Sub or Redis into slack capacity.Expanded TPU support: TorchTPU runs PyTorch models natively inside upstream vLLM and SGLang.

How does llm-d route requests?The llm-d router builds on the Kubernetes Gateway API Inference Extension.It tracks which replicas hold each cached prefix.It then weighs load and predicted latency before picking a replica.Agents call one endpoint, and each step lands where its context already lives.

When accelerator memory fills, KV cache moves to host memory, then SSD or Managed Lustre.It uses vLLM’s OffloadingConnector, Mooncake or SGLang’s HiCache for this.Prefill and decode run as separate pools, because prefill is compute-heavy and decode is memory-bound.Agent request1 endpoint

llm-d routerprefix + load aware

Prefill / decodeseparate pools

Tiered KV cacheHBM to Lustre Accelerator HBM

Host CPU RAM

Local SSD

Managed Lustre Back Next step Auto-play Step 1 of 4 Latency and cache Throughput All figures self-reported by Google, Snap and Zenteiq.Source: Google Cloud blog, Oct 20, 2026.Multi-tenant QoSFlow controlPriority and fairness-aware routing keeps bursty agent loops from starving interactive users.

UpgradesDisaggregatedSetBuilt on LeaderWorkerSet.Rolls prefill and decode pools together on matching versions without dropping sessions.ReliabilityRouter HAHot standby and rolling updates keep traffic flowing through upgrades and maintenance.

AgentsSession-aware routingTurn-priority scheduling and native Responses API state help agents reuse context across turns.Background jobsAsync processorPulls requests from Google Cloud Pub/Sub or Redis and runs them in slack capacity.TPUTorchTPU time-slicingInterleaves inference into idle RL cycles.

Duty cycle rises from about 40% to above 70%.Click a card to reveal details.Stable v1 APIs ship across all components.llm-d 1.0NVIDIA DynamoAIBrix Version1.0 (Oct 2026)v1.5.1 (Oct 2026)v0.7.

0 (Jun 2026) GovernanceCNCF SandboxNVIDIA-ledvLLM project org EnginesvLLM, SGLangSGLang, TRT-LLM, vLLMvLLM, SGLang, TRT-LLM AcceleratorsNVIDIA, AMD, Intel, TPUNVIDIA GPUsMixed GPUs KV tiersHBM, CPU, SSD, LustreGPU, CPU, SSD, remoteDistributed KV cache LicenseApache 2.0Apache 2.0</td