Lessons from bringing up RDMA-backed Prefill/Decode serving across H200, B200, GB200, and GB300 on GKE

Prefill/Decode (P/D) disaggregation splits LLM serving into two pools of workers: prefill workers process the prompt, and decode workers generate the output tokens. Between the two, every request hands its KV cache over the network. The path those bytes take (which NIC, which switches, how many hops) is set by three layers that don't coordinate by default: the platform decides which network paths exist at all, the resource allocator picks each worker's GPU and NIC, and the router picks which prefill worker pairs with which decode worker.

We learned how much that matters while bringing up llm-d's P/D path on four GCP GPU platforms: A3 Ultra (H200), A4 (B200), A4X (GB200 NVL72), and A4X Max (GB300 NVL72). The bring-up itself was a first: llm-d's P/D guide had no high-performance GKE RDMA recipe before this work built one on A3 Ultra, then carried the same allocation-and-transport contract through A4, A4X, and A4X Max (allocation). The problems we hit shared one shape: one layer lost what another knew, and the symptom appeared far from the cause.

The lesson is that topology is not only a transport optimization; it is an end-to-end contract spanning allocator, kernel, NIC, transport, engine, and router. Keeping RDMA fast depends on every layer holding its part: a mismatched GPU and NIC, a missing route, a silent fallback to TCP, or a topology-blind pairing anywhere along the way degrades the whole path. Concretely, the platform must discover topology, publishing each device's PCIe root, NIC, and rail as attributes Kubernetes can see. Allocation must preserve it, giving each worker a GPU and a NIC that belong together. The runtime must verify it, confirming that KV really moves over RDMA. And the router must consume it, pairing prefill and decode workers that are close on the network. Two results from our clusters show what that is worth:

• A silent fallback from RDMA to TCP is not a slowdown but an outage. Forcing one on purpose under benchmark load made 436 of 5,400 requests time out; eliminating one on A4X cut mean KV-transfer time by more than 93%.
• Mean time-to-first-token (TTFT) fell from 2.15 s to 1.157 s once each worker had a matched GPU and NIC and a verified RDMA channel, and then to 0.33 s, a further 3.5× reduction, once the router also paired workers by topology (evaluation).

An earlier post, Networking for Distributed Inference in llm-d, covered the transport stack itself: NIXL (the KV-transfer library), UCX (the communication library underneath it), and backend performance. This post follows topology from the platform through allocation and transport into routing.

llm-d v1.0 is here, and it is the project's most important release so far. In under a year and a half, llm-d has gone from a shared idea among a handful of contributors to the open standard for distributed inference, built by more than 500 contributors from over 25 organizations: proven in production across every major accelerator, running on public clouds and sovereign on-prem fleets alike, and serving open models through both vLLM and SGLang. The founding vision, any model, any accelerator, any cloud, is now something you can deploy.

v1.0 is the release where that stack reaches maturity. llm-d already delivered hardware heterogeneity and advanced inference capabilities out of the box: prefix-aware routing, KV cache offloading, prefill/decode disaggregation, and wide expert parallelism. v1.0 brings that performance to the realities of Day 2 operations: serving multiple tenants, balancing competing priorities, upgrading without downtime, and supporting emerging agentic workloads, all within one suite of co-designed components that compose into your platform. Serving a frontier model fast is no longer the hard part. Running it reliably, for many tenants, through upgrades, under real load, is. That is what v1.0 is built for.

When we introduced co-operative time-slicing in llm-d, we made a claim: if RL phases become schedulable units, independent jobs can share accelerators with near-zero waste. Today we're backing that claim with a measured, end-to-end proof. For research teams, the claim cashes out as one thing: more experiments per week on the GPUs you already have.

OpenRL, an open source, Kubernetes-native, Tinker-compatible, self-hosted fine-tuning service built on the llm-d time-slicing stack, runs many post-training jobs, SFT and reinforcement learning alike, concurrently on the same GPUs.

Text inference has a comfortable shape in llm-d. A request arrives as a few thousand tokens, prefill and decode run back to back on one replica, and a prefix-aware router keeps the KV cache warm. Every layer of the stack — scheduler, load balancer, autoscaler — is calibrated against that shape.

Then someone attaches an image, and it is tempting to think of the result as the same request with a picture bolted on. It is not. A multimodal request differs from a text-only one in almost every property a serving system actually reads: how big it is, how much it costs, how long it takes before the first output token, what identity its cache entries have, and how many stages it has to move through. The parts of the stack that were calibrated on text keep working, silently, on numbers that no longer mean what they used to.

This post walks those differences deliberately. First what makes media content different from text content. Then what that does to an inference request, difference by difference. Then, for each one, what llm-d actually does about it — the token estimator, the content-hash routing path, the retuned affinity gate, and the encode tier. Model server like vllm is the engine throughout; its per-replica mechanisms are the floor, not the answer.

If v0.8 was about declaring llm-d an inference control plane, v0.9 is about proving it at production scale. The router gains high availability so that no single endpoint picker failure drops traffic. Flow control ships with bounded defaults that protect clusters out of the box. A formal plugin lifecycle gates experimental extensions behind explicit opt-in. And KEDA begins replacing Prometheus Adapter as the autoscaling foundation, aligning llm-d with the Kubernetes-native scaling ecosystem.

The v0.8 release previewed four themes for v0.9 — agentic inference, RL workflow maturity, scaling and hardening, and accelerator expansion — and this release advances all four. DisaggregatedSet revision routing makes P/D rolling updates safe for the first time. GPU utilization-aware routing scores endpoints by actual compute usage, not just queue depth. Diffusion model routing extends the control plane to image generation. And new well-lit paths land for GB200, GB300, Intel XPU, ROCm, and TPU v7 hardware. The reinforcement learning story advances from feasibility proof to architectural foundation, with time-slicing entering incubation.

Production llm-d deployments usually already expose Prometheus metrics, GPU utilization, and latency histograms. Those signals answer whether the fleet is healthy. They rarely answer the question that dominates incident response and optimization work:

For this slow or failed request, which hop decided what, and where did the time actually go?

That gap is structural. llm-d’s value sits in control-plane decisions: precise prefix-cache scoring, load-aware selection, prefill/decode (P/D) disaggregation, multi-hop proxies. Fleet aggregates cannot reconstruct those decisions after the fact. Distributed tracing can.

This post walks through two complementary capabilities now available across the llm-d stack:

1. End-to-end (E2E) distributed tracing: one trace_id across Gateway → EPP → KV-cache → P/D proxy → vLLM
2. Fine-grained tracing: first-party spans at the decision points that make well-lit paths real

Together they move observability from “the cluster looks busy” to “this request chose this pod for this cache score, ran prefill here, and spent most of its time to first token waiting in the decode queue.”

How the llm-d router balances KV-cache affinity against token load

The llm-d router's default configuration has changed to something an operator can reason about, built on a single methodology: token-aware routing. The scheduler prioritizes KV-cache affinity, keeping each request on the endpoint that already holds its prefix, unless the load on that endpoint exceeds a calibrated limit; past the limit it ignores affinity and picks the endpoint by load alone. Load is measured in tokens matched to the workload's bottleneck, giving two configurations: for prefill-bound traffic (long prompts), prefix-cache affinity + token load (prefix-cache-affinity-filter + token-load-scorer), routing on uncached prefill tokens in flight; for decode-bound traffic (long outputs), prefix-cache affinity + active requests (prefix-cache-affinity-filter + active-request-scorer), routing on active streams. We call each pairing of the affinity filter with a load signal a bottleneck-matched configuration (or matched configuration for short).

Two failure modes motivated the change. The first is the previous default itself: a four-signal weighted blend (prefix-cache match, queue depth, KV utilization, LRU) whose emergent behavior was hard to predict and harder to tune. The second is the hot spotting most routers suffer from over-indexing on KV-cache affinity: affinity concentrates traffic on cache-warm endpoints, and without an explicit saturation release the warm endpoint keeps absorbing load past the point where a cold endpoint would serve the request faster. Token-aware routing pairs one signal with one bottleneck and one calibrated limit, so the scheduler is legible: an operator can predict what it will do, why it will do it, and how it will degrade under load.

The results back the change. On the benchmarks below, the matched configurations sustain 2–3× the throughput of Kubernetes Service round-robin on prefill-bound workloads while holding TTFT through the operating range, and hold parity or better everywhere else. The stack now ships as the default across llm-d's optimized-baseline, agentic-serving, multimodal-serving, and P/D-disaggregation guides, and it is running on production serving fleets, including Google Cloud Vertex AI, Red Hat, and Mistral.

A request arrives and its 48K-token prefix is already sitting in a KV cache, but the endpoint holding it - one schedulable serving instance: either a complete engine instance or a rank within a multi-rank deployment - is busy. Route the request to the cache owner and it queues behind that backlog; route it to an idle endpoint and it spends seconds recomputing work the cluster already did. Waiting and recomputing are both wrong answers. P2P (peer-to-peer) KV cache sharing adds the third option: send the request to the best endpoint for load, and copy the finished KV to it.

A busy owner is only one way locality breaks. Load balancing may select another endpoint, or the required KV may have been generated by a different serving role. In every such case the llm-d router already knows how much of a request's prefix each candidate endpoint holds. With P2P, that knowledge becomes a transfer instruction: route the request to the best endpoint overall, then tell it where to fetch the missing prefix.

This is not a universal speedup. When routing already produces a local hit, P2P correctly does nothing. When locality conflicts with load balance or serving topology, it can replace seconds of repeated computation with a peer transfer. The operating principle is simple: local hits first; portable reuse when locality breaks.

The rest of this post answers four questions: how expensive is a pull; whether it improves load-balanced serving; whether it preserves session history across prefill/decode (P/D) roles; and when it should stay inactive.

Vision-language model (VLM) inference involves several stages: vision encoding, language model prefill, and language model decode. Each stage has different compute and memory characteristics, making heterogeneous disaggregation a natural fit. This gives us the flexibility to place different stages to different accelerator resources based on their compute and memory characteristics.

In this post, we evaluate a vision-heavy Kimi-VL serving workload on llm-d and SGLang. We place vision encoding stage in 4 Intel Arc Pro B60 (Intel B60) GPUs while keeping prefill and decode stage on 1 NVIDIA H200 GPU. Compared with the collocated baseline, the measured heterogeneous E/PD configuration achieved 2.4x-2.8x higher throughput and reduced mean Time To First Token (TTFT) by roughly 69%-80% under load.

llm-d's well-lit paths are tested, benchmarked recipes for serving large language models in production. The agentic-serving path starts with the workload and combines the capabilities it needs into a deployment for each accelerator. This post applies that pattern to GLM-5.2-FP8 on H200, using the 219 production Claude Code sessions in SemiAnalysis's Weka trace corpus.

GLM-5.2 is a 744B-parameter (39B active) mixture-of-experts model with 78 layers. Its Multi-head Latent Attention (MLA) stores one compressed latent instead of separate KV heads, so tensor parallelism cannot shard it: TP=8 keeps eight copies of the KV cache. Tensor-parallel all-reduce is also expensive across nodes: every layer performs the column-parallel, row-parallel, and all-reduce sequence twice. llm-d's Wide-EP path avoids both costs with data-parallel attention and expert-parallel MoE, making the KV budget viable for long-context agentic serving.

At a measured output rate of roughly 37-41 tokens per second per user, a 64-GPU llm-d deployment works out to $0.05 per million input tokens and $3.55 per million output tokens at an H200 rental rate of $2.71 per GPU-hour. Z.ai's public GLM-5.2 row lists $1.40 input and $4.40 output; Claude Opus 4.8 lists $5 input and $25 output. These figures assume sustained load and assign prefill rental to input and decode rental to output.