Production llm-d deployments usually already expose Prometheus metrics, GPU utilization, and latency histograms. Those signals answer whether the fleet is healthy. They rarely answer the question that dominates incident response and optimization work:

For this slow or failed request, which hop decided what, and where did the time actually go?

That gap is structural. llm-d’s value sits in control-plane decisions: precise prefix-cache scoring, load-aware selection, prefill/decode (P/D) disaggregation, multi-hop proxies. Fleet aggregates cannot reconstruct those decisions after the fact. Distributed tracing can.

This post walks through two complementary capabilities now available across the llm-d stack:

1. End-to-end (E2E) distributed tracing: one trace_id across Gateway → EPP → KV-cache → P/D proxy → vLLM
2. Fine-grained tracing: first-party spans at the decision points that make well-lit paths real

Together they move observability from “the cluster looks busy” to “this request chose this pod for this cache score, ran prefill here, and spent most of its time to first token waiting in the decode queue.”

How the llm-d router balances KV-cache affinity against token load

The llm-d router's default configuration has changed to something an operator can reason about, built on a single methodology: token-aware routing. The scheduler prioritizes KV-cache affinity, keeping each request on the endpoint that already holds its prefix, unless the load on that endpoint exceeds a calibrated limit; past the limit it ignores affinity and picks the endpoint by load alone. Load is measured in tokens matched to the workload's bottleneck, giving two configurations: for prefill-bound traffic (long prompts), prefix-cache affinity + token load (prefix-cache-affinity-filter + token-load-scorer), routing on uncached prefill tokens in flight; for decode-bound traffic (long outputs), prefix-cache affinity + active requests (prefix-cache-affinity-filter + active-request-scorer), routing on active streams. We call each pairing of the affinity filter with a load signal a bottleneck-matched configuration (or matched configuration for short).

Two failure modes motivated the change. The first is the previous default itself: a four-signal weighted blend (prefix-cache match, queue depth, KV utilization, LRU) whose emergent behavior was hard to predict and harder to tune. The second is the hot spotting most routers suffer from over-indexing on KV-cache affinity: affinity concentrates traffic on cache-warm endpoints, and without an explicit saturation release the warm endpoint keeps absorbing load past the point where a cold endpoint would serve the request faster. Token-aware routing pairs one signal with one bottleneck and one calibrated limit, so the scheduler is legible: an operator can predict what it will do, why it will do it, and how it will degrade under load.

The results back the change. On the benchmarks below, the matched configurations sustain 2–3× the throughput of Kubernetes Service round-robin on prefill-bound workloads while holding TTFT through the operating range, and hold parity or better everywhere else. The stack now ships as the default across llm-d's optimized-baseline, agentic-serving, multimodal-serving, and P/D-disaggregation guides, and it is running on production serving fleets, including Google Cloud Vertex AI, Red Hat, and Mistral.

A request arrives and its 48K-token prefix is already sitting in a KV cache, but the endpoint holding it - one schedulable serving instance: either a complete engine instance or a rank within a multi-rank deployment - is busy. Route the request to the cache owner and it queues behind that backlog; route it to an idle endpoint and it spends seconds recomputing work the cluster already did. Waiting and recomputing are both wrong answers. P2P (peer-to-peer) KV cache sharing adds the third option: send the request to the best endpoint for load, and copy the finished KV to it.

A busy owner is only one way locality breaks. Load balancing may select another endpoint, or the required KV may have been generated by a different serving role. In every such case the llm-d router already knows how much of a request's prefix each candidate endpoint holds. With P2P, that knowledge becomes a transfer instruction: route the request to the best endpoint overall, then tell it where to fetch the missing prefix.

This is not a universal speedup. When routing already produces a local hit, P2P correctly does nothing. When locality conflicts with load balance or serving topology, it can replace seconds of repeated computation with a peer transfer. The operating principle is simple: local hits first; portable reuse when locality breaks.

The rest of this post answers four questions: how expensive is a pull; whether it improves load-balanced serving; whether it preserves session history across prefill/decode (P/D) roles; and when it should stay inactive.

Vision-language model (VLM) inference involves several stages: vision encoding, language model prefill, and language model decode. Each stage has different compute and memory characteristics, making heterogeneous disaggregation a natural fit. This gives us the flexibility to place different stages to different accelerator resources based on their compute and memory characteristics.

In this post, we evaluate a vision-heavy Kimi-VL serving workload on llm-d and SGLang. We place vision encoding stage in 4 Intel Arc Pro B60 (Intel B60) GPUs while keeping prefill and decode stage on 1 NVIDIA H200 GPU. Compared with the collocated baseline, the measured heterogeneous E/PD configuration achieved 2.4x-2.8x higher throughput and reduced mean Time To First Token (TTFT) by roughly 69%-80% under load.

llm-d's well-lit paths are tested, benchmarked recipes for serving large language models in production. The agentic-serving path starts with the workload and combines the capabilities it needs into a deployment for each accelerator. This post applies that pattern to GLM-5.2-FP8 on H200, using the 219 production Claude Code sessions in SemiAnalysis's Weka trace corpus.

GLM-5.2 is a 744B-parameter (39B active) mixture-of-experts model with 78 layers. Its Multi-head Latent Attention (MLA) stores one compressed latent instead of separate KV heads, so tensor parallelism cannot shard it: TP=8 keeps eight copies of the KV cache. Tensor-parallel all-reduce is also expensive across nodes: every layer performs the column-parallel, row-parallel, and all-reduce sequence twice. llm-d's Wide-EP path avoids both costs with data-parallel attention and expert-parallel MoE, making the KV budget viable for long-context agentic serving.

At a measured output rate of roughly 37-41 tokens per second per user, a 64-GPU llm-d deployment works out to $0.05 per million input tokens and $3.55 per million output tokens at an H200 rental rate of $2.71 per GPU-hour. Z.ai's public GLM-5.2 row lists $1.40 input and $4.40 output; Claude Opus 4.8 lists $5 input and $25 output. These figures assume sustained load and assign prefill rental to input and decode rental to output.

In Reinforcement Learning (RL) post-training for Large Language Models using algorithms like GRPO, optimizing the ratio of generator (sampler) to trainer throughput is the single largest driver of infrastructure Total Cost of Ownership (TCO). Because typical RL post-training loops alternate sequentially between generation and optimization phases, GPU and TPU clusters sit completely idle for 40% to 60% of their lifecycle.

Today, llm-d introduces a new well-lit path for Co-operative Time-Slicing: the Snapshot Agent. Rather than forcing physical hardware to wait on upstream phases or sit idle during blocking operations, this platform-level capability dynamically interleaves independent RL jobs onto shared hardware blocks, driving aggregate accelerator duty cycles from the 40% baseline up to 70%+ efficiency without altering underlying model convergence or accuracy.

Key Takeaways​

• Near Zero Idle Accelerators: Co-operative time-slicing multiplexes concurrent RL jobs onto shared GPU and TPU hardware, reclaiming stranded compute capacity and boosting duty cycles from ~40% up to 70%+.
• No Model Degradation: Because active jobs retain exclusive access to physical accelerators during their compute windows, there is no loss in token generation throughput or training step convergence.
• Low Code Changes: A lightweight two-call client API (acquire and yield) integrates seamlessly into existing RL loops, while the Snapshot Agent automatically handles CUDA context evacuation and restoration to host DRAM under the hood.

If v0.7 was about making features deployable — standalone mode, Kustomize, documentation from scratch — then v0.8 is the payoff. Flow Control, Batch Gateway, and multi-modal serving graduate from experimental to production. The routing layer reaches beyond Kubernetes into RL training loops. And the image pipeline simplifies from a growing matrix of custom builds to upstream vLLM images with llm-d as the control plane on top. Three axes define this release: graduating capabilities that were introduced in v0.7 to production readiness, extending llm-d into non-Kubernetes environments for reinforcement learning and Slurm-based research, and aligning the project's identity around what it actually is — an inference control plane, not a fork of any engine underneath it.

The scope of what llm-d orchestrates has broadened considerably. A single deployment can now serve interactive chat, batch processing, multi-modal requests, and agentic multi-step workflows — with routing intelligence that understands the differences between them. SGLang joins vLLM as a first-class inference engine across most well-lit paths, and the first agentic serving guide lands as a new workload-level well-lit path. Forty-eight new contributors joined the project since v0.6, more than doubling v0.7's twenty-three, reflecting broadening adoption across the industry.

Networking: The Critical Path in P/D Disaggregation​

llm-d's prefill-decode disaggregation unlocks significant efficiency gains by separating compute-heavy prefill from memory-bandwidth-heavy decode onto dedicated GPU pools. But it introduces a hard dependency on the network: the KV Cache must be transferred from prefill to decode before the first token can be generated. This transfer time lands directly on the Time to First Token (TTFT) — making networking a first-order concern for end-to-end inference latency.

This post dives into llm-d's networking stack — how it works today and how it's evolving in collaboration with NVIDIA.

For most of the transformer era, the KV cache rested on a quiet assumption: one model, one uniform cache. Every layer attended the same way, every block was the same size, and everything built on top of the cache (allocators, offload connectors, schedulers) could treat it as a single pool.

Hybrid models broke this assumption. Many recent frontier and open-weight models increasingly mix attention types within a single model (full attention next to sliding-window, linear, or Mamba layers), making the cache heterogeneous: different layers now hold different amounts of state, in different shapes, with different reuse rules. A cache block that used to be allocated as one uniform unit is now constituted of several distinct parts.

To serve a hybrid model efficiently, an AI inference platform has to handle that heterogeneity in at least three aspects of the stack:

• GPU Memory Allocation: How the cache is laid out and allocated on the GPU. vLLM solved this with its Hybrid Memory Allocator (HMA), rebuilt around a unified allocator (see Hybrid Models as First-Class Citizens in vLLM).
• KV Offloading: Extending the KV cache to CPU and storage. Without HMA awareness, an offloading connector turns the HMA off and therefore discards the GPU memory improvements or potential data movement savings.
• KV-Aware Routing: Sending each request to the right model-server replica. Ignoring hybrid memory structure may erroneously list nodes as having or not having the required KV data based on information stemming from just part of the layers.

vLLM's HMA solved hybrid GPU memory allocation when handling a single vLLM instance. This post shows how llm-d extends that to tiered KV cache management - including KV offloading to CPU and storage, and KV-aware request routing - significantly improving throughput and latency at scale for hybrid models.

Most production inference clusters today are single-vendor because that is often the simplest way to configure and operate a cluster.

That is starting to change. Procurement cycles bring new generations alongside older ones, supply planning spans multiple accelerator options, and cost/performance profiles differ by workload. Real production fleets are accumulating heterogeneity whether or not the architecture planned for it.

This is an opportunity to unlock real value: different accelerator classes can be matched to workload requirements, stranded capacity gets reclaimed, and operators gain more flexibility in capacity planning. The case is stronger still for sovereign and on-premise deployments, where data residency, regulatory alignment, and the long-term economics of high-volume inference make local fleet optimization especially important.

Making that work in practice is a non-trivial systems problem. Each accelerator stack brings its own optimized drivers, firmware, container images, runtime settings, and attention kernels. A coherent serving layer needs to preserve those platform-specific optimizations while still giving operators one control plane for routing, observability, and policy.