Bringing First-Class SGLang Serving to llm-d
In Serving GLM-5.2 for Agentic Workloads on llm-d, we showed how llm-d serves long-context coding agents on NVIDIA H200 GPUs with vLLM—combining disaggregated prefill and decode pools, wide expert parallelism, prefix- and load-aware routing, and tiered KV caching. Since then, contributors from the llm-d and SGLang communities have made SGLang a first-class, Kubernetes-native inference engine in llm-d's GLM-5.2 well-lit path, the project's tested reference deployment. We also measured SGLang's multi-tier KV cache on single-replica and multi-replica clusters. This post tells that story in three parts:
- Engine-agnostic
llm-darchitecture: howllm-ddecouples Kubernetes workload orchestration, prefix-affinity routing, and disaggregated serving from individual model servers so vLLM and SGLang share a common operational model. - First-class SGLang integration: how the SGLang and
llm-dcommunities adapted SGLang's Wide-EP runtime to work natively withllm-d's Kubernetes routing and disaggregated serving. - Tiered KV-cache offloading: how pairing SGLang's hierarchical KV cache with
llm-d's prefix-aware routing and Kubernetes failover keeps long agent sessions cached across host RAM and shared storage—surviving high concurrency, pod evictions, and cold restarts.






















