Skip to main content

llm-d v0.9: Hardened for Scale

· 12 min read
llm-d maintainers
llm-d maintainers

If v0.8 was about declaring llm-d an inference control plane, v0.9 is about proving it at production scale. The router gains high availability so that no single endpoint picker failure drops traffic. Flow control ships with bounded defaults that protect clusters out of the box. A formal plugin lifecycle gates experimental extensions behind explicit opt-in. And KEDA begins replacing Prometheus Adapter as the autoscaling foundation, aligning llm-d with the Kubernetes-native scaling ecosystem.

The v0.8 release previewed four themes for v0.9 — agentic inference, RL workflow maturity, scaling and hardening, and accelerator expansion — and this release advances all four. DisaggregatedSet revision routing makes P/D rolling updates safe for the first time. GPU utilization-aware routing scores endpoints by actual compute usage, not just queue depth. Diffusion model routing extends the control plane to image generation. And new well-lit paths land for GB200, GB300, Intel XPU, ROCm, and TPU v7 hardware. The reinforcement learning story advances from feasibility proof to architectural foundation, with time-slicing entering incubation.

Production-Hardened Router​

The most consequential changes in v0.9 are about what happens when things go wrong.

Router high availability introduces a multi-EPP topology where non-leader replicas keep their datastores populated and route traffic rather than returning 503. Requests that reach a standby replica route normally rather than failing, so a load balancer distributing traffic across EPP instances provides resilience without additional coordination. The new --drain-timeout flag enables graceful shutdown — the endpoint picker stops accepting new requests while draining in-flight traffic, preventing the connection resets that operators reported during rolling updates.

Plugin stability lifecycle formalizes the maturity of every router plugin as Alpha, Beta, or Stable. Alpha plugins require --allow-experimental-plugins to activate, preventing accidental use of unproven extensions in production. Automated dependency validation ensures that a plugin's prerequisites are satisfied before the router starts, replacing runtime crashes with clear startup errors.

Flow control ships with production-safe defaults for the first time. defaultRequestTTL defaults to 60 seconds and per-band maxRequests to 5000, bounding queue growth without operator intervention. Rejection codes now distinguish an empty endpoint pool (503) from retryable backpressure (429), giving load balancers and clients the signal they need to back off correctly. Priority-aware stepped gating and soft-reflective-ceiling policies give operators finer control over how traffic sheds under saturation.

Disaggregation: Production-Grade Operations​

Disaggregated prefill/decode has been one of llm-d's deepest investments. v0.9 focuses on making it operationally robust.

DisaggregatedSet revision routing solves the rolling update problem. During a model version rollout, the router understands which pods belong to which revision and routes accordingly, preventing the request failures that occurred when a prefill pod sent KV cache to a decode pod running a different model checkpoint. This is the kind of operational safety that P/D disaggregation needs before platform teams will run it in production with confidence.

Topology-aware routing extends this with a new filter and scorer that understands the physical P/D topology for placement decisions. CPU-tier-aware pull-source selection distributes KV transfer load proportionally across available sources, avoiding the hot spots that emerged when all decode pods pulled from the same prefill replica. An early offloading connector on the disaggregation sidecar begins exploring KV cache transfer without RDMA fabric, using vLLM's OffloadingConnector and ZMQ transport — an incubation-stage capability that may broaden where P/D disaggregation can run in future releases.

Routing Intelligence​

The router's scheduling pipeline gains several capabilities that move it from heuristic-based to measurement-based decisions.

GPU utilization-aware routing uses the DCGM extractor and a generic endpoint-attribute filter and scorer to route requests based on actual GPU utilization metrics rather than proxy signals like queue depth. Operators can now express routing policies in terms of the resource they actually care about — GPU compute — rather than inferring it from request counts.

Session affinity via the session_id strategy keeps multi-turn conversations on the same backend where prior KV cache is resident. The header-based profile handler selects a scheduling profile per request from a header, enabling a single router deployment to apply different routing strategies to different traffic classes without separate endpoint picker instances.

Sticky-until-saturated routing is now the default strategy across all guides. Requests stick to the endpoint with the best prefix cache match until that endpoint reaches a saturation threshold, then spill to the next-best option. This balances cache reuse against load distribution without manual tuning.

The utilization filter replaces the former active-request-filter with a unified mechanism that caps endpoints on any combination of active, running, waiting, and KV-cache utilization. This consolidates what was previously multiple independent filters into a single, composable policy.

Multimodal and Diffusion Model Support​

v0.9 extends the router to workload types that previous releases did not address.

Image generation routing adds OpenAI-compatible /v1/images/generations endpoint handling for diffusion backends such as vLLM-Omni. This extends the router's API surface to cover generative image workloads alongside language models. Video input support in approximate prefix match enables the router to compute cache affinity for video-bearing multimodal requests, extending the prefix-cache optimization to vision-language models processing video.

E/P/D heterogeneous serving with SGLang enables encode, prefill, and decode stages to run on mixed GPU and XPU hardware, with the router managing request flow across the heterogeneous topology. Multimodal predicted latency extends the latency predictor to account for the additional compute cost of processing image and video tokens alongside text, improving routing accuracy for multimodal workloads.

KEDA-Native Autoscaling​

v0.9 begins the migration from Prometheus Adapter to KEDA as the autoscaling foundation.

New guides land for KEDA + EPP pool-level saturation metrics, where the endpoint picker exposes queue depth, in-flight request counts, and KV-cache pressure as KEDA-compatible scaling signals. The router becomes the source of truth for scaling decisions rather than relying on Prometheus queries against model server metrics. SLO-driven autoscaling combines KEDA with the predicted latency scorer, scaling based on whether the deployment can meet its latency targets rather than reacting to utilization thresholds after they are breached.

Queue-based KEDA + EPP autoscaling now runs in nightly CI on OpenShift with GPU, validating that the scaling path works end-to-end on every commit. The Workload Variant Autoscaler continues to support the existing Prometheus Adapter path for operators who have not yet migrated.

Observability and Security​

Production operations require visibility. v0.9 closes the observability gaps that operators reported during v0.8 deployments.

End-to-end tracing adds OpenTelemetry spans over the scoring path and propagates trace context through the full P/D transfer chain — from request arrival through prefill scheduling, KV transfer, and decode completion. For the first time, operators can trace a single request across the entire disaggregated inference pipeline in their existing tracing infrastructure.

mTLS for the EPP metrics endpoint and scrape sources hardens the security posture of the control plane. Configurable ext_proc TLS minimum version and cipher suites give security teams the controls they need. Bounded metric cardinality prevents label explosion on model-name and fairness-id dimensions, addressing the Prometheus memory issues that operators with many models or tenants encountered. The Grafana dashboard moves from the router repository to the main llm-d repo, and per-guide troubleshooting documentation ships for workload autoscaling and P/D disaggregation.

New Well-Lit Paths and Accelerator Coverage​

Each release expands the set of tested, documented deployment patterns. v0.9 adds new well-lit paths, new hardware coverage, and new deployment topologies.

Fast Model Actuation (FMA) is a new well-lit path that addresses cold-start latency. Combined with ModelExpress P2P weight transfer, model weights load via GPU-to-GPU RDMA rather than disk — one pod fetches from HuggingFace, and every other pod pulls directly from that pod's HBM over the RDMA fabric. This can significantly reduce cold-start time for large models by bypassing disk I/O entirely.

Multi-model routing introduces the Inference Payload Processor (IPP), enabling multiple LLMs to serve behind a single gateway endpoint. IPP extracts the model name from request bodies and sets routing headers, letting HTTPRoutes direct traffic to the appropriate InferencePool. P2P KV cache sharing adds a new well-lit path where vLLM instances pull cached prefix KV blocks directly from a peer's CPU offload tier instead of recomputing them — extending cache reuse beyond a single pod without requiring a centralized cache store.

NVIDIA GB200 (A4X) and GB300 (A4X-Max) receive P/D disaggregation overlays with DRA and RDMA recipes, bringing disaggregated inference to NVIDIA's latest data center hardware. Intel XPU expands to wide expert parallelism, tiered prefix cache with the lmcache connector, and multimodal optimized baseline — making XPU a viable platform for all three well-lit paths. ROCm gains predicted latency routing and P/D disaggregation with the MoRI connector. TPU v7 adds disaggregated P/D benchmarks for the agentic serving path.

LiteLLM and Kong AI Gateway integration guides land for operators who want to front llm-d with an API management layer that handles authentication, rate limiting, and model routing across multiple providers. SGLang adds managed Lustre L3 offloading support and native filesystem offloading, expanding the tiered caching story to SGLang backends. The default vLLM image updates to v0.26.0.

Reinforcement Learning: Incubation and Direction​

RL rollout generation is an inference problem — the same prefix-cache-aware, load-balanced scheduling that llm-d provides for production serving applies directly to the thousands of generation requests per training step. v0.9 establishes the early integration points, with active work in the llm-d-rl and llm-d-rl-time-slicing incubation repositories.

Framework integrations with veRL and vime replace each framework's built-in load balancer with llm-d's scoring and filtering pipeline — bringing prefix-cache affinity, P2P KV cache sharing, and flow control to rollout generation without changes to the training code. Time-slicing enters incubation with an architecture proposal and an early orchestrator that enables a single GPU fleet to alternate between training and inference phases within an RL step, targeting the 45–66% of accelerator time that synchronous RL loops typically leave idle.

This work is incubation-stage — APIs are evolving and production hardening is ahead. The full RL story targets v0.10 and beyond.

What This Means for You​

Platform teams running llm-d in production gain true HA support, flow control defaults that protect clusters without manual tuning, and a plugin lifecycle that prevents experimental code from reaching production unless explicitly enabled. The KEDA migration provides a standard autoscaling interface that integrates with existing Kubernetes scaling infrastructure. The metric renames in this release (EPP KV metrics move to the llm_d_router_epp subsystem) require dashboard and alert updates — deprecated aliases are available during the transition.

ML infrastructure teams building RL training pipelines can follow the incubation-stage veRL and vime integrations, which bring llm-d's routing intelligence to rollout generation — with time-slicing on the horizon for GPU-efficient training. New accelerator overlays for GB200, GB300, and expanded XPU coverage give teams deploying across hardware generations tested recipes for each platform.

Application architects building multimodal and generative applications get routing support for diffusion models alongside language models on shared infrastructure. Session affinity and header-based profile selection enable sophisticated traffic management for agentic and multi-turn workloads. End-to-end OTel tracing through the disaggregated pipeline provides the observability needed to debug latency issues in production.

What Is Next?​

The v0.10 milestone targets September and continues along four directions.

RL integration deepens. v0.9 established early veRL and vime integrations and introduced the time-slicing orchestrator. Next steps include end-to-end validation across more RL frameworks, weight synchronization hardening, and maturing time-slicing from prototype to deployable infrastructure.

A standardized model server interface. The control plane needs a common API to query engine capabilities, manage lifecycle, and configure runtime parameters — independent of whether the engine underneath is vLLM, SGLang, or TensorRT-LLM. This is a prerequisite for the engine lifecycle management that time-slicing and fast model actuation require.

KV-cache observability and intelligence. As the multi-tier cache hierarchy deepens, operators need visibility into hit rates, tier utilization, and eviction patterns. The KV cache layer targets a unified observability framework alongside smarter cache management policies.

Autoscaling maturity. The KEDA foundation laid in v0.9 opens the door to proactive, SLO-driven scaling policies that anticipate demand rather than reacting to it.

Community and Contribution​

The project's contributor base continues to grow, with contributions spanning the router, KV cache layer, guides, and CI infrastructure across multiple hardware platforms.

To get started with llm-d v0.9:

Follow @llm_d on Twitter/X, llm-d on Bluesky, or llm-d on LinkedIn for updates. Check llm-d.ai/community/events for upcoming community events and llm-d on YouTube for talks and demos. Come build with us.