Skip to main content
Version: dev
Accelerator
Engine
community

GLM-5.2-FP8 on H200

Overview​

This guide deploys GLM-5.2-FP8 (753B MoE) on H200 nodes, P/D-disaggregated with wide expert parallelism via a DisaggregatedSet (a pair of LeaderWorkerSets). It is optimized for agentic workloads: the configuration follows the analysis in the GLM-5.2 agentic serving blog post, which sizes each choice against production agentic traces (median main-agent request: ~195K input tokens and ~317 output tokens; 96% of turns reuse at least 90% of the input from their best earlier main-agent turn). Each layer addresses a specific pressure of that workload:

  • Wide EP (DEP) — the MoE experts are spread across ranks (TP=1, DP=EP) instead of tensor-parallel sharding, and GLM-5.2's MLA keeps the per-token KV footprint small; together they preserve HBM for KV cache, and cache hit rate — not FLOPs — sets throughput here.
  • P/D disaggregation (NIXL KV transfer) — input processing and token generation use separate pools, so each side can use the backend and replica count suited to its work. Prefill runs DeepEP high-throughput all-to-all; decode runs low-latency (IBGDA + NVSHMEM).
  • Token-aware, GPU+CPU prefix-cache routing — the EPP keeps prefix groups on ranks that hold them in GPU or CPU memory with the prefix-cache affinity filter, and picks by queued prefill token load, so resumed sessions land where their context can be restored rather than recomputed.
  • CPU+NVMe tiered KV offloading — CPU memory keeps evicted prefixes reusable after HBM fills. Local NVMe supplies a lower tier when needed.
  • MTP speculative decoding (3 draft tokens) — cuts median time between output tokens by about half on this workload (see the blog post).

The guide composes the wide expert parallelism, P/D disaggregation and tiered prefix cache foundations. Serving uses the DeepGemm MoE backend with tool-calling (glm47) and reasoning (glm45) parsers. Tested on CoreWeave (CKS) with InfiniBand networking.

An SGLang variant (MODEL_SERVER=sglang) runs the same wide-EP P/D layout on GKE A3 Ultra (RoCE) with the SGLang Rust frontend, which gives every DP rank its own HTTP listener. It serves the OpenAI routes and SGLang's native /generate API, and has no benchmark numbers yet. It uses the same deploy, verify and cleanup steps below. Where a section differs per engine, pick the SGLang tab. The rest of this page describes the vLLM recipe unless it says otherwise.

Default Configuration​

The default deployment is the latency-leaning Pareto configuration from the blog post: p3w2d1w2 — 3 DEP16 prefill replicas and 1 DEP16 decode replica, 8 nodes / 64 GPUs — with MTP and tiered KV offloading enabled.

ParameterValue
Modelzai-org/GLM-5.2-FP8
AcceleratorNVIDIA H200 (8 GPUs per node)
Serving topologyP/D disaggregated — 3 prefill replicas + 1 decode replica, each TP=1, DP=16, EP=16 (DEP16, 2 nodes)
DP modelSupervisor (--data-parallel-multi-port-external-lb)
All-to-alldeepep_high_throughput (prefill), deepep_low_latency with IBGDA + NVSHMEM (decode)
MoE backendDeepGemm
KV transferNixlConnector
KV cache offloadingCPU + NVMe tiered on prefill (offloading-tiered component)
MTP speculative decodingOn (3 draft tokens)
gpu-memory-utilization0.935 (prefill), 0.95 (decode)
RoutingGPU+CPU prefix-cache affinity + token load (glm-5.2-overrides.values.yaml)
Reasoning / tool-callglm45 / glm47

If your SLO target is throughput rather than TTFT, p2w2d2w2 (2 prefill + 2 decode, also 64 GPUs) is the second Pareto point measured in the blog post. The full topology range — from the 2-node p1w1d1w1 for functional validation up to the 10-node p3w2d2w2 — is in Scaling and Alternative Topologies.

Routing Modes​

This guide supports two prefix-cache routing modes (ROUTING), selected with tabs in Deploy the llm-d Router:

  • Approximate prefix routing (ROUTING=approximate, default) — gates on prefix-cache affinity across GPU- and CPU-resident KV and on token load, using an approximate prefix index (glm-5.2-overrides.values.yaml), paired with the default (p3w2d1w2 + tiered offloading) or any raw topology overlay.
  • Precise prefix-cache routing (ROUTING=precise) — replaces the approximate prefix index with exact KV-event-backed routing (precise-routing.values.yaml) on the 3-node p2w1d1w1-precise deployment (24 GPUs, 64-token KV blocks, wide-ep-render tokenizer Service, and per-rank ZMQ KV-event subscriptions), validated on three 8-GPU H200 nodes with InfiniBand on CoreWeave.

Supported Accelerators and Model Servers​

This guide includes configurations for the following accelerator and model server combinations:

AcceleratorACCELERATOR_TYPEServed modelvLLMSGLangNotes
NVIDIA H200gpuzai-org/GLM-5.2-FP8🟡 community🟡 communityP/D disaggregated wide-EP on 8-GPU nodes; needs an all-to-all RDMA fabric · vLLM: CoreWeave InfiniBand, default 8 nodes / 64 GPUs (p3w2d1w2 + tiered KV offloading), 2 to 10 nodes by DEPLOYMENT · SGLang: GKE A3 Ultra RoCE, default 2 nodes / 16 GPUs (p1w1d1w1), 2 to 8 nodes by DEPLOYMENT

✅ validated: covered by a nightly E2E workflow · 🟡 community: maintained by the hardware vendor or community, not covered by nightly E2E · ❌ not supported: tracked in the linked issue · — no configuration.

Prerequisites​

  • Have the proper client tools installed on your local system to use this guide.

  • Use a Gateway API Inference Extension bundle >= v1.5.0-rc.2 (required because the InferencePool exposes all eight DP rank ports as targetPorts). For ROUTING=precise, use an EPP build with per-rank KV-event attribution (llm-d-router#2233).

  • Deploy the LeaderWorkerSet controller v0.11.1 or newer. When installing with Helm, pass --set enableDisaggregatedSet=true to enable the DisaggregatedSet validating webhook and RBAC used by the model server.

  • Provide H200 nodes on an all-to-all RDMA fabric: DeepEP requires every NIC on a host to reach every NIC on all other hosts (rail-only networks fail). See RDMA and Networking Configuration and the model server orchestration guide.

  • For MODEL_SERVER=sglang on GKE A3 Ultra, prepare the cluster as described in the GKE provider docs (GPUDirect RDMA, DRA and DRANET drivers) so the gke-rdma-template resource claims resolve.

  • Create a HuggingFace token and export it as HF_TOKEN in your shell.

  • (Optional) Install the monitoring stack if you plan to enable Prometheus monitoring.

Get the guide​

Every command below runs from a local clone of the llm-d repository: the manifests, Helm values, and Kustomize overlays it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Configure the environment​

Set the guide-specific environment variables (DEPLOYMENT picks the recommended default overlay or one of the alternative topologies; ROUTING=precise overrides it with p2w1d1w1-precise in the router step):

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=glm-5-2
export NAMESPACE=llm-d-glm-5-2
export MONITORING=false # options: false, true
export MONITORING_VALUES=
export ACCELERATOR_TYPE=gpu # options: gpu
export MODEL_SERVER=vllm # options: vllm, sglang
export ROUTING=approximate # options: approximate, precise
export DEPLOYMENT=default # options: default, p1w1d1w1, p1w1d1w2, p1w2d1w2, p2w1d1w1, p2w1d1w1-precise, p2w1d1w2, p2w2d1w2, p2w2d2w2, p3w2d1w2, p3w2d2w2
export MODEL=zai-org/GLM-5.2-FP8 # the model every deployment serves
source ${REPO_ROOT}/guides/env.sh # defines GAIE_VERSION, ROUTER_CHART_VERSION, router chart URLs, and CURL_TEST_IMAGE

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models:

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Installation Instructions​

1. Deploy the llm-d Router​

All 8 DP rank ports (8000-8007) are exposed as targetPorts on the InferencePool (glm-5-2.values.yaml) for per-rank routing. Select the Helm values overrides for your routing mode (ROUTING):

Layers glm-5.2-overrides.values.yaml on top of glm-5-2.values.yaml, sizing the prefix index to cover CPU-offloaded KV as well as GPU-resident KV:

  • Prefix-cache affinity filter (prefill) — keeps prefix groups on ranks that hold them in GPU or CPU memory, falling back to all ranks when the warm ones are overloaded (peakPrefillThroughput: 3585, measured on an H200 DP8 cell). The approximate prefix index has a fixed capacity of 31,000 blocks per rank: ~3,700 GPU-resident plus ~27,300 for the 100 GiB per-rank CPU offload tier.
  • Token-load scorer (prefill) — picks by queued prefill tokens.
  • Active-request scorer (decode) — load balancing.

With MODEL_SERVER=sglang, sglang.values.yaml is layered instead: it parses OpenAI bodies with the openai-parser and /generate bodies with the sglanghttp-parser, and routes like the vLLM path: one approximate prefix index sized for GPU and CPU (HiCache) KV feeds the prefix-cache affinity filter, then the token-load scorer on prefill and the active-request scorer on decode. None of these need /metrics, which the SGLang Rust listeners do not expose. The index size in the router values is for a DEP8 prefill rank on H200.

vLLM
# Paths to values files: the wide-EP router values, then the GLM-5.2
# GPU+CPU prefix-cache affinity overrides on top
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/models/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml"
export ROUTER_OVERRIDES_VALUES="${REPO_ROOT}/guides/models/${GUIDE_NAME}/router/glm-5.2-overrides.values.yaml"

(Optional) Enable Prometheus monitoring on the llm-d router by defining the helm values file (requires installing the monitoring stack mentioned in Prerequisites):

# only when MONITORING=true:
export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml"

Deploy the router in Standalone Mode, with an Envoy sidecar in front of the router. Keep the release name ${GUIDE_NAME}: the verification steps look up the ${GUIDE_NAME}-epp Service by that name. To front the router with a Kubernetes Gateway instead, see Gateway Mode in the Optimized Baseline.

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  -f ${ROUTER_OVERRIDES_VALUES} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}
note

The published benchmarks ran through an Istio Gateway (Gateway Mode with provider.name=istio) rather than the standalone proxy.

2. Deploy the Model Server​

Apply the Kustomize overlay for your deployment (DEPLOYMENT=default deploys p3w2d1w2 + tiered offloading for approximate routing; ROUTING=precise set DEPLOYMENT=p2w1d1w1-precise in the router step, which includes per-rank KV-event publishing and the wide-ep-render Service). With MODEL_SERVER=sglang, DEPLOYMENT=default deploys p1w1d1w1:

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/deployments/${DEPLOYMENT}

Do not deploy the vLLM and SGLang recipes in the same namespace: the InferencePool selects pods by llm-d.ai/guide only and would mix both engines into one pool.

For vLLM, two hardware notes: prefill workers request 1500Gi DRAM for the CPU offload tier, and the NVMe tier uses a host-path volume at /mnt/local/kv-cache. On nodes without local NVMe, swap the overlay's offloading-tiered component for offloading-cpu (DRAM only).

Wait for pods to become ready (model load takes time; the startup probe allows up to 45 minutes):

kubectl get pods -n ${NAMESPACE} -l llm-d.ai/model=GLM-5.2-FP8 -w

(Optional) Deploy the monitoring resources for model servers (requires the Prometheus Operator from the monitoring stack mentioned in Prerequisites): PodMonitors that scrape each DP rank's port (dp0-dp7 on prefill, rank0-rank7 on decode). See Monitoring for node-exporter sidecars and the plain-Prometheus alternative.

# only when MONITORING=true:
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/monitoring

Verification​

1. Get the IP of the Proxy​

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send Test Requests​

Send a completion request from a temporary pod inside the cluster:

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"Explain how a simple agent loop works in 3 sentences.\"}"'

Check that the router reaches a model server that serves MODEL: the response must name the model and carry generated tokens:

# The router must reach a model server that serves MODEL and returns tokens
kubectl run serve-check --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'R=$(curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"Say hello.\", \"max_tokens\": 16}")
    echo "${R}" | grep -q "\"model\":\"${MODEL}\"" && echo "${R}" | grep -q "\"choices\"" \
      && echo "OK: ${MODEL} served through the router" \
      || { echo "FAIL: ${R}"; exit 1; }'

With MODEL_SERVER=sglang, the same OpenAI requests work, and so do native /generate requests with text or input_ids.

3. Verify Precise Prefix-Cache Routing (ROUTING=precise)​

When running ROUTING=precise with DEPLOYMENT=p2w1d1w1-precise, verify that the wide-ep-render Service returns token IDs and that each engine pod maintains eight active ZMQ subscriptions (5557–5564) to the EPP. A 403 or 404 from the render Service means the prefill pods run without --enable-scale-out, so precise routing is inert while requests still return 200 from the remaining scorers:

# only when ROUTING=precise and MODEL_SERVER=vllm:
# The render Service must return token IDs for MODEL
kubectl run render-check --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'R=$(curl -sS -X POST "http://wide-ep-render:8000/v1/completions/render" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"render check\", \"max_tokens\": 1}")
    echo "${R}" | grep -q "\"token_ids\"" \
      && echo "OK: render returns token IDs" \
      || { echo "FAIL: ${R}"; exit 1; }'

# only when ROUTING=precise and MODEL_SERVER=vllm:
# Each engine pod must hold 8 established ZMQ subscriptions (ports 5557-5564) from the EPP
EPP_IP=$(kubectl -n ${NAMESPACE} get endpointslices \
  -l kubernetes.io/service-name=${GUIDE_NAME}-epp \
  -o jsonpath='{.items[0].endpoints[0].addresses[0]}')
ZMQ_FAILED=0
for POD in $(kubectl -n ${NAMESPACE} get pods \
  -l llm-d.ai/inference-serving=true -o name); do
  N=$(kubectl -n ${NAMESPACE} exec "${POD}" -c vllm -- python3 -c "
hx=''.join(f'{int(o):02X}' for o in reversed('${EPP_IP}'.split('.')))
print(sum(1 for line in open('/proc/net/tcp')
          if len(line.split()) > 3 and line.split()[3] == '01'
          and line.split()[2].split(':')[0] == hx
          and 5000 <= int(line.split()[1].split(':')[1], 16) < 6000))")
  if [ "${N:-0}" -ge 8 ]; then echo "OK: ${POD} ${N} subscriptions"
  else echo "FAIL: ${POD} ${N:-0} subscriptions (want 8)"; ZMQ_FAILED=1; fi
done
[ "${ZMQ_FAILED}" = 0 ]

Send the same long prompt twice: the second request routes to the same prefill rank, with vllm:prefix_cache_hits_total increasing by the prompt length aligned to 64-token blocks.

Scaling and Alternative Topologies​

Every topology below is a Kustomize overlay under modelserver/gpu/${MODEL_SERVER}/deployments/: set DEPLOYMENT to its name and apply it with the model server step. vLLM ships every topology; SGLang ships the ones marked in the SGLang column.

DeploymentPrefillDecodeNodes / GPUsSGLang
p1w1d1w11 replica, 1 node, DEP81 replica, 1 node, DEP82 / 16✓
p1w1d1w21 replica, 1 node, DEP81 replica, 2 nodes, DEP163 / 24✓
p1w2d1w21 replica, 2 nodes, DEP161 replica, 2 nodes, DEP164 / 32✓
p2w1d1w12 replicas, 1 node, DEP81 replica, 1 node, DEP83 / 24
p2w1d1w1-precise2 replicas, 1 node, DEP8 (kv-events + render)1 replica, 1 node, DEP8 (kv-events)3 / 24
p2w1d1w22 replicas, 1 node, DEP81 replica, 2 nodes, DEP164 / 32✓
p2w2d1w22 replicas, 2 nodes, DEP161 replica, 2 nodes, DEP166 / 48
p2w2d2w22 replicas, 2 nodes, DEP162 replicas, 2 nodes, DEP168 / 64✓
p3w2d1w23 replicas, 2 nodes, DEP161 replica, 2 nodes, DEP168 / 64
p3w2d2w23 replicas, 2 nodes, DEP162 replicas, 2 nodes, DEP1610 / 80
note

The alternatives are to be hardened until the coming llm-d release.

The SGLang overlays all build on the GKE provider overlay (modelserver/gpu/sglang/providers/gke), the only one so far; other providers need their own providers/<name> overlay (RDMA device requests, HCA selection, cache volumes). The SGLang components are no-mtp and hicache. Every SGLang topology includes hicache, which enables HiCache (host-memory KV cache) on prefill with HICACHE_SIZE_GB_PER_RANK=64.

Raw topology overlays ship with MTP on and offloading off; opt in or out per deployment by adding Kustomize components from modelserver/gpu/vllm/components/ to the overlay's kustomization.yaml under components: (the default overlay is exactly p3w2d1w2 + offloading-tiered). Component env entries merge by name, so their values replace the base defaults.

ComponentTargetsEffect
offloading-cpuprefill onlyCPU-only KV cache offloading (OFFLOADING_MODE=cpu)
offloading-tieredprefill onlyCPU + NVMe tiered KV cache offloading (OFFLOADING_MODE=tiered)
no-mtpprefill + decodeDisables MTP speculative decoding (ENABLE_MTP=0)
kv-eventsprefill + decodeKV-event publishing for the precise prefix-cache routing mode

Manifest Reference​

The KV cache offloading, InfiniBand and KV cache evictor sections apply to the vLLM recipe.

MTP Speculative Decoding​

On by default (3 tokens) for both prefill and decode. Disable with the no-mtp component or ENABLE_MTP=0. Token count: MTP_NUM_TOKENS (default 3).

KV Cache Offloading (Prefill)​

Off in the raw topology overlays; enabled via the offloading-cpu or offloading-tiered component.

  • offloading-cpu — CPU-only offloading via OffloadingConnector. Uses mmap in /dev/shm. The pod allocates 1500Gi memory and 1500Gi dshm to accommodate 8 DP ranks' mmap regions. cpu_bytes_to_use is per-rank — total CPU KV cache = value x 8.
  • offloading-tiered — CPU + NVMe tiered offloading via TieringOffloadingSpec. Same CPU tier as above, plus NVMe as a secondary eviction target. Host-path volume at /mnt/local/kv-cache mounted as /mnt/nvme-cache.

Decode pods do not use offloading (256Gi dshm, 512Gi memory).

InfiniBand Networking​

Both prefill and decode configure IB for multi-node communication:

VariableValuePurpose
NCCL_IB_HCAibpFilter IB HCAs for NCCL collectives
NVSHMEM_HCA_PREFIXibpFilter IB HCAs for NVSHMEM (decode low-latency)
NVSHMEM_REMOTE_TRANSPORTibgdaGPUDirect Async for NVSHMEM
rdma/ib8Request 8 RDMA/IB devices per pod

KV Cache Evictor​

modelserver/gpu/vllm/base/kv-cache-evictor.yaml deploys a DaemonSet that evicts stale KV cache data from NVMe when utilization exceeds 90%, targeting 70%.

Monitoring (Optional)​

This section applies to the vLLM recipe; the SGLang Rust listeners expose no /metrics yet.

Node-exporter sidecars on each pod collect InfiniBand, CPU, memory pressure, and network retransmission metrics. Merge the per-DP-rank Prometheus scrape configs into an existing prometheus-server configmap with:

NAMESPACE=${NAMESPACE} bash ${REPO_ROOT}/guides/models/${GUIDE_NAME}/monitoring/apply-scrape-configs.sh

If you run the Prometheus Operator instead, apply the PodMonitors with MONITORING=true in the model server step.

DCGM custom metrics: modelserver/gpu/vllm/base/dcgm-custom-metrics.yaml.

Benchmarking​

The benchmark configuration below uses requests derived from production agentic traces and replayed with aiperf on CoreWeave H200 nodes with InfiniBand between nodes and NVLink within a node. Limited-input-length runs filter aiperf to only use shorter requests for apples-to-apples comparisons with EP8 deployments that cannot handle the entire dataset due to some entries containing close to 1M tokens. See the blog post for the full analysis and figures. An inference-perf preset via llm-d-benchmark is not yet published for this deployment.

For a quick synthetic throughput run, inference-perf.yaml runs an inference-perf job (2K input / 2K output tokens, 2048 concurrent requests, 8192 total) against the standalone glm-5-2-epp Service:

kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/models/${GUIDE_NAME}/inference-perf.yaml

Benchmark Overlays​

Pre-built overlays under modelserver/gpu/vllm/deployments/benchmark/<config>/<topology>/ combine components with topology patches. Each matches a tested configuration on CoreWeave H200:

ConfigurationDirectoryComponents
Baselinebenchmark/baseline/no-mtp
MTP + Offloadingbenchmark/mtp-offloading/offloading-tiered
Offloadingbenchmark/offloading/no-mtp + offloading-tiered
Full ISL + MTP + Offloadingbenchmark/full-isl-mtp-offloading/offloading-tiered

Deploy a benchmark config:

kubectl apply -n ${NAMESPACE} \
    -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/modelserver/gpu/vllm/deployments/benchmark/<config>/<topology>

Example overlay (mtp-offloading/p1w1d1w1):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../../providers/coreweave
components:
  - ../../../../components/offloading-tiered
patches:
  - target:
      group: disaggregatedset.x-k8s.io
      kind: DisaggregatedSet
    patch: |-
      # Roles are ordered [prefill, decode] in the base DisaggregatedSet.
      - op: replace
        path: /spec/roles/0/spec/replicas
        value: 1
      - op: replace
        path: /spec/roles/0/spec/leaderWorkerTemplate/size
        value: 1
      - op: replace
        path: /spec/roles/1/spec/replicas
        value: 1
      - op: replace
        path: /spec/roles/1/spec/leaderWorkerTemplate/size
        value: 1

aiperf Command​

Every reported number comes from the same aiperf profile invocation, run from inside the cluster against the Kubernetes Gateway Service (llm-d-inference-gateway-istio, from a Gateway Mode router installation with provider.name=istio; see the Optimized Baseline) and swept across concurrency. Against the standalone proxy, use http://${GUIDE_NAME}-epp:80/v1 instead. The dataset used is the semianalysis_cc_traces_weka_with_subagents aiperf preset, backed by semianalysisai/cc-traces-weka-062126 on HuggingFace.

aiperf profile \
    --scenario 'inferencex-agentx-mvp' \
    --url 'http://llm-d-inference-gateway-istio:80/v1' \
    --model 'zai-org/GLM-5.2-FP8' \
    --max-context-length <142000|10000000> \
    --endpoint-type 'chat' \
    --streaming \
    --use-server-token-count \
    --public-dataset 'semianalysis_cc_traces_weka_with_subagents' \
    --concurrency <16|32|64|128|256|512> \
    --random-seed 42 \
    --benchmark-duration 900 \
    --server-metrics 'http://llm-d-inference-gateway-istio:80/metrics' \
    --no-gpu-telemetry \
    --output-artifact-dir <path> \
    --ui 'simple'

Only --max-context-length and the --concurrency sweep differ across the four reported configurations:

Configuration--max-context-length--concurrency sweep
Baseline14200016, 32, 64, 128
MTP + Offloading14200016, 32, 64, 128
Offloading14200016, 32, 64, 128
Full ISL + MTP + Offloading1000000016, 32, 64, 128, 256, 512

142000 truncates the trace dataset to requests that fit an EP8 (1-node) prefill deployment. 10000000 is effectively unbounded and replays full traces (up to ~1M input tokens) — this is the "Full ISL" config. At concurrency 256/512, some Full ISL runs on smaller topologies exceeded server capacity (warmup failures) and were excluded from the blog post results.

Benchmark Results​

note

TODO: results for the current router config (prefix-cache affinity filter and token-load scorer on prefill, active-request scorer on decode) are not yet published. Earlier results, measured with the previous dual GPU+CPU prefix-cache and active-request scorers, are in the blog post.

What Each Optimization Changes​

These matched comparisons were measured with the previous router config, which was the same in both runs of each comparison. They compare model-server and topology choices, not routing.

  • CPU offloading keeps evicted prefixes reusable. In matched 24-GPU runs at concurrency 32, offloading raises output 13%, cuts median TTFT 32%, and cuts median turn completion 27%, while median ITL is unchanged. At concurrency 128, baseline output remains near 951 tok/s while offloading reaches 2,776 tok/s.
  • MTP reduces decode work. On matched 32-GPU p1w2d1w2 deployments with CPU offloading enabled in both runs, MTP raises aggregate output 1.6-2.0x and average output per user 1.6-1.8x through concurrency 128.
  • Prefill and decode replicas change different parts of the turn. At concurrency 128, p2w2d2w2 produces 8% more output and completes the median turn 10% sooner than p3w2d1w2 at the same 64-GPU budget. p3w2d1w2 reaches the first token 19% sooner and restores 63% less KV from CPU memory.

Cleanup​

To remove the deployed components:

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/deployments/${DEPLOYMENT}
helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/monitoring --ignore-not-found=true
kubectl delete namespace ${NAMESPACE}