llm-d v1.0: State-of-the-art inference, ready for Day 2

llm-d v1.0 is here, and it is the project's most important release so far. In under a year and a half, llm-d has gone from a shared idea among a handful of contributors to the open standard for distributed inference, built by more than 500 contributors from over 25 organizations: proven in production across every major accelerator, running on public clouds and sovereign on-prem fleets alike, and serving open models through both vLLM and SGLang. The founding vision, any model, any accelerator, any cloud, is now something you can deploy.

v1.0 is the release where that stack reaches maturity. llm-d already delivered hardware heterogeneity and advanced inference capabilities out of the box: prefix-aware routing, KV cache offloading, prefill/decode disaggregation, and wide expert parallelism. v1.0 brings that performance to the realities of Day 2 operations: serving multiple tenants, balancing competing priorities, upgrading without downtime, and supporting emerging agentic workloads, all within one suite of co-designed components that compose into your platform. Serving a frontier model fast is no longer the hard part. Running it reliably, for many tenants, through upgrades, under real load, is. That is what v1.0 is built for.

Delivering state-of-the-art performance

The defining goal of the 0.x series was reach: inference infrastructure that is not locked to one vendor, one cloud, or one engine. llm-d got there. It runs state-of-the-art inference across NVIDIA, AMD, and Intel GPUs and Google TPUs, on managed clouds and sovereign on-prem clusters, and across open-source model servers including vLLM and SGLang, all behind the same routing and scheduling layer: prefix-cache and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism, and batch and async inference. The founding promise of any model, any accelerator, any cloud stopped being a slogan and became a deployment target.

The proof came from running it in production across clouds, accelerator vendors, and enterprises.

• Google Cloud builds the GKE Inference Gateway on the llm-d Endpoint Picker, cutting time-to-first-token by more than 35% in Vertex AI production while doubling its prefix-cache hit rate from 35% to 70%, and is integrating llm-d 1.0 into GKE across its TPU and NVIDIA portfolio.
• AWS runs disaggregated inference on Amazon SageMaker HyperPod with llm-d, pairing prefix-cache-aware routing with prefill/decode disaggregation and NIXL over Elastic Fabric Adapter to report up to a 70% increase in tokens per second versus standard vLLM.
• DigitalOcean builds its distributed inference stack on llm-d for native GPU-type heterogeneity, serving the trillion-parameter Kimi K3 model across both AMD and NVIDIA hardware.
• Oracle Cloud Infrastructure runs llm-d on OCI bare-metal AMD MI300X with RDMA networking, and developed and validated the upstream AMD deployment guide on OCI.
• Tesla and Red Hat run Llama 3.1 70B on llm-d across AMD MI300X GPUs and, in a coauthored KServe case study, report a 3x increase in output tokens per second and a 2x reduction in time to first token from prefix-cache-aware routing.
• Prime Intellect integrates llm-d into its large-scale reinforcement learning stack, pairing the router with prefill/decode disaggregation and KV cache offload for multi-node rollouts.
• IBM Research self-hosts frontier open models on llm-d for coding agents and chip-design agents, serving up to 3,000 concurrent coding agents on 544 NVIDIA H100 GPUs and reaching a roughly 98% prefix-cache hit rate at about $0.11 per million tokens, around 11.5x lower cost than a proprietary stack.
• Mistral AI serves open models in production on llm-d as part of its regional inference offering.
• JetBrains runs llm-d behind Junie, its AI coding agent, using the scheduler's predicted-latency routing to keep responses fast under heavy agent load.
• Red Hat ships llm-d as the distributed inference stack in Red Hat AI, generally available on OpenShift and on managed Kubernetes including Microsoft Azure and CoreWeave.

Different clouds, different accelerators, different engines, one stack. These highlights come from a growing roster of more than 25 organizations adopting llm-d, and that breadth, proven in production, is the foundation v1.0 is built on.

Why Day 2 is hard

Getting a model to serve fast is Day 1. Keeping it serving, for many tenants, through upgrades, under unpredictable load, is Day 2, and it is where most inference stacks show their seams.

A production fleet carries mixed traffic: latency-sensitive interactive requests next to bursty agent workloads that reuse enormous amounts of context and spawn parallel work without warning. Priorities compete, and under load something has to give, so a premium tier has to stay protected while best-effort traffic absorbs the pressure. Upgrades and maintenance cannot mean downtime: you need to replace or fail over any component with no dropped requests and a cache that stays warm. Disaggregated deployments, which split prefill and decode across pools for efficiency, multiply the number of moving parts that have to roll forward together. Agentic traffic spends most of its compute on context rather than final tokens, so a stack that does not understand sessions and turns pays to recompute the same work again and again. And underneath all of it, inference has to be run like a service, with the observability to see what is happening and an owner who gets paged when it is not.

We know this because we run it. These are the requirements that showed up when the community, ourselves included, took llm-d from benchmarks to a standing production service. v1.0 addresses each of them directly.

Multi-Tenant Quality of Service

A shared inference fleet has to keep latency-sensitive traffic responsive while absorbing bursty, unpredictable agent workloads. v1.0 makes that a first-class property of the router through centralized flow control.

Priority- and fairness-aware flow control classifies every request by a flow key that combines a fairness ID (the tenant or workload) and a priority band, maintains a separate queue per flow, and dispatches by priority first, then cycles fairly across tenants within a band. Under saturation it does no-regret scheduling: it holds requests in the router rather than committing them to already-overloaded backend queues, so a burst from one tenant does not blow up latency for everyone else. Program-aware fairness lets operators express those policies per workload, and a probabilistic admission controller sheds load gracefully when demand exceeds capacity instead of failing hard. All of it now lives under a single flowControl configuration surface, and the defaults still mimic first-come-first-served, so existing deployments see no change until operators opt into multi-tenant policy. The result is that behind a single endpoint, a premium tier stays protected under load, measurably and not just by policy, while best-effort and batch traffic absorb the bursts.

Zero-Downtime Operations

Day 2 is where most serving stacks show their seams, and disaggregated serving multiplies the risk. v1.0 is built so upgrades and maintenance do not interrupt traffic.

Router high availability runs the Endpoint Picker with a hot standby that takes over scheduling if the active instance is replaced or fails, so routing never goes dark during a rollout. For disaggregated serving, DisaggregatedSet manages prefill and decode replicas as one logical unit: it gates readiness on a snapshot of the underlying pods and does revision-aware routing, so requests reach a prefill/decode pair only once both sides are fully rolled out and compatible. That makes coordinated, lockstep upgrades of a split prefill/decode deployment safe to run in place, instead of requiring a parallel fleet and a manual cutover. On the request path, a coordination layer in the router, consolidated into llm-d-router in this release, drives each request through its disaggregated encode, prefill, and decode stages, so a multi-phase pipeline is orchestrated as one flow rather than stitched together by the caller. Disaggregation is no longer only a performance technique; it is now something you can operate.

A Stable Foundation to Build On

The configuration API graduates to v1, a stable, versioned surface you can standardize a platform on and build against without chasing breaking changes between releases. Underneath it, the Endpoint Picker request path has been hardened across reliability, performance, and scalability and validated across clouds and accelerators, so the scheduler behaves the same whether it fronts a handful of replicas or a large fleet. Topology-aware accelerator scheduling, through a Dynamic Resource Allocation (DRA) and DRANET well-lit path, places work with awareness of device and network locality, so wide expert parallelism and disaggregated serving land on accelerators that are actually close together. The result is a foundation that behaves consistently from a single node to a large heterogeneous cluster.

Agent-Aware Serving

Agentic workloads reuse enormous amounts of context across turns, and most of the compute goes to handling that context rather than producing the final tokens. v1.0 serves this traffic natively.

Session-aware routing keeps a multi-turn conversation pinned to the replica that already holds its KV cache: a session-state-producer plugin tags requests with session state, and the router sends follow-ups back to the same backend. Affinity is sticky until saturated, so a session stays put until its replica fills up and then rebalances, trading a little cache locality for headroom instead of dropping requests. Turn-priority scheduling lets the router reason about where a request sits in an ongoing exchange, keeping an active agent loop responsive rather than treating every turn as an isolated request. And native Responses API state, including previous_response_id, lets the server carry context across turns so an agent does not resend its full history and the stack does not recompute it. For agent builders, that is less latency and lower cost without changing how they write their application.

Any hardware, any engine, any cloud

The founding promise, any model on any accelerator in any cloud, only holds if the same stack genuinely runs everywhere, and that is not something any single vendor could deliver alone. llm-d keeps the routing, scheduling, and disaggregation layers the same across accelerators, and each platform has a well-lit path that uses its native capabilities without forking the stack. Getting there took deep collaboration: AMD, Google, Intel, NVIDIA, and the wider community contributed the data-movement libraries, communication primitives, and serving-engine integrations that make a hardware-independent control plane real. The platforms below are listed alphabetically.

AMD. With v1.0, llm-d brings inference to AMD Instinct MI355X with AINIC across vLLM, SGLang, and AMD ATOM, with validated deployment guides. Wide expert parallelism spans several MI355X nodes, with MoRI-EP handling expert communication, and prefix-cache and predicted-latency routing direct requests across them. For long, context-heavy agent sessions, tiered prefix caching over vLLM native offload and LMCache extends effective KV cache capacity, and flow control and KEDA-based autoscaling handle multi-tenant QoS and scaling. To power disaggregated serving, llm-d integrates directly with MoRI-IO and NIXL on ROCm, scaling prefill and decode independently and moving KV blocks between the two phases.

Google TPU. With v1.0, llm-d brings native PyTorch model serving to Google Cloud TPUs inside upstream vLLM and SGLang via TorchTPU, alongside validated GKE deployment guides. On an agentic coding workload on TPU 7x, llm-d's prefix-aware routing plus host memory KV cache offload cut median time-to-first-token (TTFT) by more than 85% and raised total throughput 2.4x versus a plain Kubernetes Service. To power disaggregated serving and tiered KV cache offloading, llm-d integrates directly with TPU Sync, the open-source data movement library for TPUs, to move KV blocks between TPU HBM, host memory, and disaggregated prefill/decode pools while inference keeps running and across rolling updates.

Intel. Intel Xeon processors and GPU platforms deliver comprehensive support for core llm-d capabilities—including Intelligent Routing, Advanced KV-Cache Management, PD Disaggregation, and Traffic Control—backed by deep integration and validation across vLLM, SGLang, LMCache, and NIXL. To accelerate enterprise adoption, Intel collaborated with the llm-d community to contribute llm-d-lens, streamlining production deployments and observability. Looking ahead, this momentum will extend to Intel’s next-generation D-series GPUs (codenamed Crescent Island), which will deliver out-of-the-box support for llm-d to further advance the ecosystem.

NVIDIA. llm-d brings inference to NVIDIA GPUs with CUDA-based images and validated deployment guides, with well-lit paths exercised on Hopper (H100 and H200), Blackwell (B200), and Grace Blackwell (GB200 NVL72 and GB300). Wide expert parallelism uses DeepEP for expert communication, and topology-aware scheduling places tightly coupled parallelism within the same NVLink domain, including a GB200 NVL72 rack's cross-node NVLink (MNNVL) fabric via the NVIDIA DRA ComputeDomain. To power disaggregated serving and tiered KV cache offloading, llm-d integrates directly with NIXL, contributed through NVIDIA Dynamo, to move KV blocks between GPU HBM, host memory, and prefill/decode pools, while ModelExpress speeds cold starts with peer-to-peer weight transfer. Support for the upcoming NVIDIA Vera Rubin platform is in progress.

And more. Because llm-d's control plane is accelerator-independent and each platform needs only a thin native path, the supported set keeps widening. Community-contributed well-lit paths already run on Rebellions NPUs, Iluvatar BI-V150, and MetaX C500X GPUs, each with its own device allocation and KV-transfer connector but the same routing, scheduling, and disaggregation layer underneath. More backends are in progress, and the well-lit paths grow to meet new devices as they arrive.

What This Means for You

Platform teams get a release they can standardize on: a v1 configuration API, high-availability routing, and rolling upgrades that do not interrupt traffic, validated across clouds and accelerators, with the observability to run inference like a service rather than a science project.

ML infrastructure teams get multi-tenant serving that stays predictable under load, and disaggregated deployments that are finally practical to operate, so a single fleet can carry mixed priorities without over-provisioning.

Application and agent architects get serving that understands sessions and turns, reuses context instead of recomputing it, and keeps interactive traffic responsive even while heavy agent workloads run alongside it.

What Is Next?

v1.0 is a foundation, not a finish line. The community is already working toward active-active router configurations, inference request fail-over and migration, wider upstream vLLM wide-expert-parallelism coverage on managed Kubernetes, and expanded multimodal and audio serving. The library of model-centric guides, tuned recipes for serving a specific frontier model on a specific accelerator, will keep growing, and deeper integration with GitOps control planes (e.g. ArgoCD) will make llm-d deployments first-class citizens in the platform pipelines teams already run. As new hardware and models arrive, the well-lit paths will grow to meet them, and the stack will keep evolving for peak performance across new architectures.

Community and Contribution

llm-d is built in the open as a Cloud Native Computing Foundation (CNCF) sandbox project, with more than 500 contributors across 25-plus organizations spanning model labs, clouds, accelerator vendors, and enterprises.

• Docs and well-lit paths: llm-d.ai walk you from a single node to a production fleet.
• Code and release notes: github.com/llm-d.
• Join us: the Slack community, the weekly community call, and across X, Bluesky, LinkedIn, and YouTube.

If you are at PyTorch Conference this week, come find us and join our keynote, Open Source Inference at Production Scale. We would love to show you what v1.0 can do.

Come build the future of distributed inference with us.