Skip to main content

One post tagged with "GKE"

Google Kubernetes Engine deployments and benchmarks

View All Tags

Bringing First-Class SGLang Serving to llm-d

· 12 min read
Yuchen Tian
Software Engineer, Google
Sage Ahrac
Research Engineer, IBM Research
Nishtha Jain
Software Engineering Manager, Google
Bob Tian
Software Engineer, Google
Shangming Cai
Core Maintainer, SGLang
Rain Jiang
Core Maintainer, SGLang
Abdullah Gharaibeh
Senior Staff Software Engineer, Google

In Serving GLM-5.2 for Agentic Workloads on llm-d, we showed how llm-d serves long-context coding agents on NVIDIA H200 GPUs with vLLM—combining disaggregated prefill and decode pools, wide expert parallelism, prefix- and load-aware routing, and tiered KV caching. Since then, contributors from the llm-d and SGLang communities have made SGLang a first-class, Kubernetes-native inference engine in llm-d's GLM-5.2 well-lit path, the project's tested reference deployment. We also measured SGLang's multi-tier KV cache on single-replica and multi-replica clusters. This post tells that story in three parts:

  1. Engine-agnostic llm-d architecture: how llm-d decouples Kubernetes workload orchestration, prefix-affinity routing, and disaggregated serving from individual model servers so vLLM and SGLang share a common operational model.
  2. First-class SGLang integration: how the SGLang and llm-d communities adapted SGLang's Wide-EP runtime to work natively with llm-d's Kubernetes routing and disaggregated serving.
  3. Tiered KV-cache offloading: how pairing SGLang's hierarchical KV cache with llm-d's prefix-aware routing and Kubernetes failover keeps long agent sessions cached across host RAM and shared storage—surviving high concurrency, pod evictions, and cold restarts.