Skip to main content
Version: dev
Accelerator
Provider
Engine
validated

P/D Disaggregation

E2E (CKS GPU) E2E (GKE GPU) E2E (GKE GPU SGLang) E2E (GKE TPU) E2E (OCP GPU) E2E (Intel XPU) E2E (AMD ROCM NIXL) E2E (AMD ROCM MORI)

Overview​

This guide splits inference into separate prefill and decode pools. Prefill pods process the prompt and produce its KV cache; decode pods pull that KV cache over a KV transfer connector (NIXL by default) and generate the output tokens. Both pools belong to one InferencePool, and the llm-d Router schedules every request twice: once onto a prefill pod and once onto a decode pod. Because disaggregation is built into the router, it composes with its other scorers:

  • Prefill — the prefix-cache-affinity-filter keeps prefix groups on cache-warm prefill pods (gated by a calibrated peakPrefillThroughput), and the token-load-scorer picks the prefill pod with the least queued prompt work.
  • Decode — the active-request-scorer picks the decode pod with the fewest in-flight requests, since pure decode is bound by concurrency rather than prompt throughput.

The default deployment serves openai/gpt-oss-120b on NVIDIA GPUs with 1 prefill pod (TP=1) and 1 decode pod (TP=4), 5 GPUs in total: the smallest topology that exercises the full P/D path. Production deployments scale the two pools independently (see When to use P/D and how to tune it).

How the two pools are deployed depends on the accelerator:

  • All accelerators: one DisaggregatedSet (pd-disagg-vllm or pd-disagg-sglang on NVIDIA GPU, pd-disagg-tpu-vllm on TPU7x dynamic sub-slices, pd-disagg-<accelerator> elsewhere) with a prefill and a decode role, in a single slice. The set rolls both roles out as one version and can replicate the whole topology into independent copies (slices); it requires the LeaderWorkerSet controller (see Prerequisites) and is covered in Operating the DisaggregatedSet.

For why P/D disaggregation helps, how requests flow between the two pools, and tuning guidance, see Disaggregated Serving.

Supported Accelerators and Model Servers​

This guide includes configurations for the following accelerator and model server combinations (set ACCELERATOR_TYPE and MODEL_SERVER accordingly). Each accelerator serves one model; its platform overlays are selected with INFRA_PROVIDER (see Configure the environment):

AcceleratorACCELERATOR_TYPEServed modelvLLMSGLangNotes
NVIDIA GPUgpuopenai/gpt-oss-120b✅ validated✅ validatedH200 · 1P (TP=1) + 1D (TP=4), 5 GPUs; SGLang 1P (TP=2) + 1D (TP=2) · NIXL
AMD GPUamdopenai/gpt-oss-120b✅ validated—AMD Instinct · 1P (TP=1) + 1D (TP=4), 5 GPUs · NIXL
Intel XPUxpuQwen/Qwen3-0.6B✅ validated—Intel GPU via DRA · 1P + 1D, 1 GPU each · NIXL
Google TPU v6etpu/v6Qwen/Qwen3-32B✅ validated—2x4 v6e nodes on GKE · 1P + 1D, 8 chips each (TP=8) · TPUConnector
Google TPU v7tpu/v7Qwen/Qwen3.5-397B-A17B-FP8🟡 community—2x2x1 TPU7x nodes on GKE · 1P + 1D, 4 chips each (TP=8) · TPUConnectorHMA
Google TPU v7 (dynamic slicing)tpu/v7-dynamic-sliceQwen/Qwen3-Coder-480B-A35B-Instruct-FP8🟡 community—TPU7x sub-slices via dynamic slicing + Kueue · DisaggregatedSet with 1P + 1D, one 2x2x1 sub-slice each · TPUConnectorHMA
Iluvatar GPUiluvatarQwen/Qwen3-32B🟡 community—BI-V150 · 1P + 1D, 2 boards each (TP=4) · IluNixlConnector
MetaX GPUmetaxQwen/Qwen3-32B🟡 community—C500X · 1P (TP=2) + 1D (TP=4) · NIXL over TCP
Biren GPUbirenQwen/Qwen2.5-72B-Instruct-GPTQ-Int8🟡 community—166M · 1P+1D, each TP=4 · NixlConnector over RDMA (UCX_TLS=rc_v)
Moore Threads GPUmthreadsQwen/Qwen3-32B🟡 community🟡 communityMTT S5000 · vLLM 1P TP=4 + 1D TP=4, Mooncake TCP; SGLang 1P+1D TP=8, Mooncake RDMA
Rebellions NPUnpuMiniMaxAI/MiniMax-M2.7🟡 community—4 NPUs + 1 RoCE VF per pod via DRA · 1P (PP=4) + 1D (DP=4, EP) · RblnNixlConnector

✅ validated: covered by a nightly E2E workflow · 🟡 community: maintained by the hardware vendor or community, not covered by nightly E2E · ❌ not supported: tracked in the linked issue · — no configuration.

Community rows are compatibility configurations maintained by their hardware vendors, often with reduced sizing (smaller models, fewer accelerators); review and adjust them for your environment before production use.

KV Transfer Connectors​

P/D disaggregation requires a KV transfer connector to move KV cache blocks from prefill workers to decode workers. On vLLM it is configured with the --kv-transfer-config flag:

ConnectorOverlaysTransportNotes
NixlConnectordefault on NVIDIA GPU, AMD GPU, Intel XPU and MetaXUCX (RDMA / TCP)Supports heterogeneous TP across P/D.
RblnNixlConnectornpu/vllm/baseNIXL over a RoCE VFRebellions' NIXL connector with kv_buffer_device=rbln. See the Rebellions NPU tab.
MooncakeConnectorgpu/vllm/cks-mooncakeRDMA via Mooncake Transfer EngineCKS with InfiniBand. See the platform table in Prerequisites.
MoRIIOConnectoramd/vllm/moriio/*RDMA via MoRI-IOAMD GPU.
TPUConnector / TPUConnectorHMAtpu/*TPU ICI / DCNFrom tpu_inference; HMA on TPU7x.
IluNixlConnectoriluvatar/vllm/baseUCX with CUDA-aware transportsIluvatar's fork of NixlConnector. See the Iluvatar tab.
NixlConnectorbiren/vllm/baseRDMA (UCX_TLS=rc_v)Biren 166M, kv_buffer_device=cpu. See the Biren tab.
MooncakeConnectormthreads/vllm/baseTCP via Mooncake MUSA transportMTT S5000 compatibility path; requires the same TP on prefill and decode. See the Moore Threads tab.
SGLang mooncakemthreads/sglang/baseRDMA via Mooncake Transfer EngineSGLang --disaggregation-transfer-backend mooncake (not vLLM MooncakeConnector). Same TP on prefill and decode.

NIXL works over TCP, but RDMA networking (InfiniBand, RoCE, EFA) is highly recommended for production.

vLLM vs SGLang​

Both engines run on NVIDIA GPU with the same router configuration and move KV cache over NIXL, but they disaggregate differently:

AspectvLLMSGLang
P/D deciderCan disaggregate conditionally (prefix-based-pd-decider); this guide's router uses always-disagg-pd-deciderMust use always-disagg-pd-decider: a decode worker has no local-prefill path. See Unconditional Disaggregation
Peer discoveryNIXL side-channel handshake (TCP 5600)Prefill bootstrap server on port 8998. To change it, set SGLANG_BOOTSTRAP_PORT on the sidecar and --disaggregation-bootstrap-port on the engine so the two match
Request cancellationPrefill-side free notification releases the KV cacheNo prefill-side free notification or reclaim timeout: a request cancelled before decode pulls the KV cache can strand it on the prefill until the pod restarts
Engine flags--kv-transfer-config (see above)--disaggregation-mode={prefill,decode}, --disaggregation-transfer-backend=nixl; the decode routing sidecar runs with --kv-connector=sglang
OperationsDisaggregated Serving: Operations (vLLM)Disaggregated Serving: Operations (SGLang)

Prerequisites​

  • Have the proper client tools installed on your local system to use this guide.

  • Ensure your cluster has enough accelerators for your configuration (default NVIDIA GPU configuration: 1 prefill pod with tensor parallelism 1 and 1 decode pod with tensor parallelism 4, 5 GPUs in total). The prefill and decode pods exchange KV cache over the network, so place them on nodes connected by a high-bandwidth fabric.

  • Create a HuggingFace token and export it as HF_TOKEN in your shell.

  • (Optional) Set up the monitoring stack if you plan to enable Prometheus monitoring.

  • The LeaderWorkerSet controller v0.11.1 or newer with the DisaggregatedSet API enabled (--set enableDisaggregatedSet=true when installing with Helm, which also installs its validating webhook and RBAC). The environment step below installs it; skip that step if your cluster already runs it. For TPU7x dynamic sub-slices, install or upgrade it through the GKE dynamic slicing cluster preparation instead, whose GKE-documented minimum LWS version is older.

  • Prepare the cluster for your platform (INFRA_PROVIDER). base, nixl/base, moriio/base, and the TPU and community accelerators need nothing beyond the accelerator's device plugin (see the model server tabs); the other platforms need:

    AcceleratorINFRA_PROVIDERCluster requirementsSetup
    NVIDIA GPUgke, gke/a4x, gke/a4xmaxGPU DRA (manual node labels and driver install, not yet GKE-managed) and managed DRANET for RoCE; gke targets A3/A4, gke/a4x / gke/a4xmax target A4X / A4X Max (GB200 / GB300); DRANet must support hairpin (same-node) and cross-rail (inter-node multi-rail) routingGKE: GPU DRA and DRANET, GKE A4X setup
    coreweaveAn RDMA device plugin exposing rdma/ib inside pods (one per pod); a default StorageClass supporting ReadWriteMany for the model cache PVCCoreWeave, RDMA resources
    cks-mooncakevLLM only. GPUs via nvidia.com/gpu; InfiniBand devices exposed inside pods via an rdma/ib device plugin (NVIDIA Network Operator, Multus with SR-IOV, or equivalent); a vLLM image with mooncake-transfer-engine (the standard image may not include it); VLLM_MOONCAKE_BOOTSTRAP_PORT (default 8998) unique per co-located instance. Configures MooncakeConnector transport only, not Mooncake StoreRDMA resources, verify RDMA in a pod, Mooncake installation
    awsp5en.48xlarge nodes with the EFA device plugin exposing vpc.amazonaws.com/efaAWS EFA notes
    AMD GPUnixl/ociOKE BM.GPU.MI300X.8 nodes; NVIDIA Network Operator SR-IOV VFs as nvidia.com/sriov-rdma-vf, with a sriov-rdma-vf SriovNetwork in namespace defaultRDMA resources
    nixl/tensorwaveAn RDMA device plugin exposing rdma/ib inside podsRDMA resources
    Intel XPUrdmaA network DRA driver publishing a dranet-rdma DeviceClass with RDMA-capable NICs (dra.net rdma attribute); each claim aligns the NIC with its GPU's PCIe rootRDMA resources

Get the guide​

Every command below runs from a local clone of the llm-d repository: the manifests, Helm values, and Kustomize overlays it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Configure the environment​

INFRA_PROVIDER selects the platform overlay for your ACCELERATOR_TYPE and, where an accelerator offers more than one, the KV transfer connector. Valid values per accelerator:

ACCELERATOR_TYPEINFRA_PROVIDER values
gpubase, gke, gke/a4x, gke/a4xmax, coreweave, aws, cks-mooncake (SGLang: base, gke, coreweave, aws)
amdnixl/base, nixl/oci, nixl/tensorwave, moriio/base
xpubase, rdma
tpu/v6, tpu/v7, tpu/v7-dynamic-slicegke
iluvatar, metax, biren, mthreads, npubase
  • base: generic Kubernetes cluster.
  • gke: on NVIDIA GPU, GKE A3/A4 with GPU DRA and DRANet RoCE (see the platform table in Prerequisites); gke/a4x and gke/a4xmax target GKE A4X / A4X Max (GB200 / GB300). On TPU, the GKE TPU node pools described in the model server step.
  • coreweave: CoreWeave with rdma/ib. aws: AWS with EFA.
  • cks-mooncake: CoreWeave CKS with MooncakeConnector over InfiniBand, vLLM only (see the platform table in Prerequisites).
  • nixl/*: NixlConnector on AMD; nixl/oci targets OCI OKE BM.GPU.MI300X.8 with SR-IOV RDMA VFs, and nixl/tensorwave targets TensorWave.
  • moriio/base: MoRIIOConnector instead of NIXL, serving Qwen/Qwen3-32B (set MODEL accordingly).
  • rdma (Intel XPU): NIXL over RDMA instead of TCP.

Set the guide-specific environment variables:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=pd-disaggregation
export NAMESPACE=llm-d-pd-disaggregation
export MONITORING=false # options: false, true
export MONITORING_VALUES=
export ACCELERATOR_VALUES=
export ACCELERATOR_TYPE=gpu # options: gpu, amd, xpu, tpu/v6, tpu/v7, tpu/v7-dynamic-slice, iluvatar, metax, biren, mthreads, npu
export MODEL_SERVER=vllm # options: vllm, sglang
export INFRA_PROVIDER=base # options: base, gke, gke/a4x, gke/a4xmax, coreweave, aws, cks-mooncake, nixl/base, nixl/amd-ci, nixl/oci, nixl/tensorwave, moriio/base, moriio/amd-ci, moriio/amd-ci-1p1d-tp8, rdma; valid values per accelerator: table above
export MODEL=openai/gpt-oss-120b # set to the model your accelerator serves (Supported Accelerators and Model Servers table); the AMD MoRIIO overlays (INFRA_PROVIDER=moriio/*) serve Qwen/Qwen3-32B
source ${REPO_ROOT}/guides/env.sh # defines GAIE_VERSION, ROUTER_CHART_VERSION, router chart URLs, and CURL_TEST_IMAGE

Install the LeaderWorkerSet controller with the DisaggregatedSet API (skip if your cluster already runs LWS v0.11.1 or newer with enableDisaggregatedSet=true):

# helm upgrade never updates CRDs, so apply the chart's CRDs first (see LWS
# "Upgrade by Helm"); this upgrades them on clusters that run an older LWS.
helm pull oci://registry.k8s.io/lws/charts/lws --version=${LWS_VERSION} --untar
kubectl apply --server-side --force-conflicts -f lws/crds
helm upgrade --install lws oci://registry.k8s.io/lws/charts/lws \
  --version=${LWS_VERSION} \
  --namespace lws-system --create-namespace \
  --set enableDisaggregatedSet=true \
  --wait --timeout 300s

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

On CoreWeave, create the model cache PVC before applying the model server overlay, which mounts the HuggingFace cache from an RWX PVC named model-pvc rather than a node-local hostPath. See Caching model weights on a shared PVC for why, and for the cold-cache caveat with multiple replicas.

# only when INFRA_PROVIDER=coreweave:
kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/recipes/modelserver/components/model-cache/model-cache-pvc.yaml

Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models:

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Installation Instructions​

1. Deploy the llm-d Router​

Prepare the paths to the helm values files for the llm-d router (used in the deployment command below):

# Paths to values files
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml"

The router configuration in router/pd-disaggregation.values.yaml disaggregates every request (always-disagg-pd-decider) and schedules it with a prefill and a decode profile, which select pods by their llm-d.ai/role label.

note

The prefix-cache-affinity-filter uses a peakPrefillThroughput of 33821, calibrated for gpt-oss-120b on TP=1 H200 prefill workers through the full P/D path (including the NIXL KV transfer). On a different model or accelerator, measure and set your value with the calibration guide before performance work.

(Optional) Enable Prometheus monitoring on the llm-d router by defining MONITORING_VALUES (see Collect Metrics):

# only when MONITORING=true:
export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml"

(Rebellions NPU only) Layer the NPU router values, which list every decode rank port:

export ACCELERATOR_VALUES="-f ${REPO_ROOT}/guides/${GUIDE_NAME}/router/npu.rbln.values.yaml"

Deploy the router in Standalone Mode, with an Envoy sidecar in front of the router. The release name ${GUIDE_NAME} is mandatory: the InferencePool selector matches a guide label that pairs with this release. To front the router with a Kubernetes Gateway instead, see Gateway Mode in the Optimized Baseline.

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  ${ACCELERATOR_VALUES} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

2. Deploy the Model Server​

For model sources, caching, and startup optimization, see the Model Loading and Startup Acceleration operations guide.

Apply the Kustomize overlay for your accelerator and model server. Each overlay deploys a prefill role (pods labeled llm-d.ai/role=prefill) and a decode role (llm-d.ai/role=decode, with the routing sidecar in front of the engine): the prefill and decode roles of a DisaggregatedSet on NVIDIA GPU and TPU7x dynamic sub-slices, two Deployments elsewhere. See Configure the environment for the INFRA_PROVIDER values of each accelerator.

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}/

The GPU overlays enable tool calling and reasoning parsing with --enable-auto-tool-choice --tool-call-parser=openai --reasoning-parser=openai_gptoss.

3. Enable Monitoring (optional)​

(Optional) Deploy the monitoring resources for model servers (see Collect Metrics):

# only when MONITORING=true:
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring-pd

Operating the DisaggregatedSet​

The overlays run prefill and decode as one DisaggregatedSet (pd-disagg-vllm or pd-disagg-sglang on NVIDIA GPU, pd-disagg-<accelerator> elsewhere) with groupIdentity: Hash. This guide ships slices: 1 with one prefill and one decode replica; each role of each slice runs as its own LeaderWorkerSet (of size 1 here, so every pod is a leader). Raise the per-role replicas to change the xPyD ratio, or slices to add complete, independently rolled copies of the topology. Scaling, rollouts, placement policy, router slice affinity, and per-role autoscaling are covered in Disaggregated Serving: Operations (DisaggregatedSet).

The TPU7x dynamic sub-slice overlay (ACCELERATOR_TYPE=tpu/v7-dynamic-slice) runs pd-disagg-tpu-vllm the same way, with groupIdentity: Ordinal so that Kueue admits the generated LeaderWorkerSets; see Kueue-Scheduled Sets.

Verification​

1. Get the IP of the Proxy​

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send Test Requests​

Send a completion request from a temporary pod inside the cluster:

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\"}"'

3. Verify P/D disaggregation​

A served request only proves the stack is up: the decode pod's routing sidecar can also run a request end to end on its own. To confirm that requests are actually split, send a few requests with a long prompt, then check that the prefill pod processed them and the decode pod received their KV cache from it.

Send 5 requests with a ~1k-token prompt:

# 5 requests with a ~1k-token prompt: each is prefilled on a prefill
# pod, and its KV cache is transferred to a decode pod
kubectl run pd-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'P=$(for i in $(seq 1 60); do printf "The prefill pod computes the KV cache and the decode pod pulls it to generate tokens. "; done)
    for i in $(seq 1 5); do
      curl -sS -o /dev/null -w "request ${i}: HTTP %{http_code}\n" -X POST "http://${IP}/v1/completions" \
        -H "Content-Type: application/json" \
        -d "{\"model\": \"${MODEL}\", \"prompt\": \"${P} Question ${i}: what is llm-d?\", \"max_tokens\": 16}"
    done'

Read the request and KV-transfer counters of the prefill and decode pods:

# Request and KV-transfer counters of the prefill and decode pods, read
# through the Kubernetes API server proxy (no port-forward needed).
# Prefill engines serve on port 8000; decode engines on 8200, behind
# the routing sidecar on 8000.
for role in prefill decode; do
  port=8000; [ "${role}" = decode ] && port=8200
  for pod in $(kubectl get pods -n ${NAMESPACE} -l llm-d.ai/guide=${GUIDE_NAME},llm-d.ai/role=${role} -o jsonpath='{.items[*].metadata.name}'); do
    echo "== ${role}: ${pod}"
    kubectl get --raw "/api/v1/namespaces/${NAMESPACE}/pods/${pod}:${port}/proxy/metrics" \
      | grep -E '^(vllm:(request_success_total|prompt_tokens_total|external_prefix_cache_hits_total)|sglang:(num_requests_total|prompt_tokens_total))' || true
  done
done

On Rebellions NPU the decode pod runs one API server per data-parallel rank (ports 8200-8203); the loop reads rank 0, which serves a share of the requests.

What to expect:

The prefill pods' vllm:request_success_total counts the requests (prefill runs them with max_tokens=1), and the decode pods' vllm:external_prefix_cache_hits_total grows by roughly the prompt length of each request: those tokens were loaded through the KV transfer connector instead of being recomputed.

If the prefill pods count no requests, the router is not disaggregating: check that the pods carry the llm-d.ai/role labels (on DisaggregatedSet paths, also that its LeaderWorkerSets are ready: kubectl get leaderworkerset -n ${NAMESPACE}) and the router logs (kubectl logs -n ${NAMESPACE} deploy/${GUIDE_NAME}-epp). If prefill counts the requests but decode reports no external prefix-cache hits, the KV transfer is failing and decode recomputes the prompt: check the decode pod's logs for connector errors.

(Optional) Read the router's P/D decisions (MONITORING=true only: without the monitoring values the router's metrics endpoint requires authentication). Each request disaggregated by the router counts under decision_type="prefill-decode":

# only when MONITORING=true:
# P/D decisions taken by the router, read through the API server service proxy
kubectl get --raw "/api/v1/namespaces/${NAMESPACE}/services/${GUIDE_NAME}-epp:9090/proxy/metrics" \
  | grep -E '^llm_d_epp_disagg_decision_total' || true

Performance benchmarks for this configuration are not part of this guide: they live with the model-specific guides.

Observability & Troubleshooting​

Once monitoring is enabled, use the signals below to operate P/D disaggregation under sustained load in production or benchmark scenarios (note that the small number of requests in Verification is only a smoke check and does not produce steady-state pool utilization or saturation signals). This section covers the metrics that matter for this path and how to read them; full metric definitions live in the metric reference and ready-to-run queries in the PromQL reference.

In a P/D deployment the prefill and decode pools scale and fail independently, and every decode step depends on a KV transfer from a prefill worker over the configured KV-transfer backend (NIXL, Mooncake, or MoRIIO). Most problems show up as an imbalance between the two pools or as KV-transfer stalls, so watch them as a pair rather than as a single aggregate.

Key metrics for this path​

SignalWhy it matters for P/DWhere to look
Prefill worker utilization (vllm:num_requests_running{pod=~".*prefill.*"})Prefill is short and bursty. Sustained saturation here means prompts queue before decode can even start, inflating TTFTPromQL → Prefill/Decode
Decode KV cache utilization (vllm:kv_cache_usage_perc{pod=~".*decode.*"})Decode holds KV for the full generation. Above ~0.9 the decode pool preempts or rejects, so decode — not prefill — is usually the scaling bottleneckPromQL → Prefill/Decode
P/D decision ratio (llm_d_epp_disagg_decision_total)Confirms the router is actually splitting prefill and decode (decision_type="prefill-decode"). A ratio drifting toward decode-only means requests are falling back to aggregated servingPromQL → Prefill/Decode
EPP scheduler e2e latency (llm_d_epp_scheduler_e2e_duration_seconds)Rising scheduler latency with healthy pools points at the routing layer, not the model serversPromQL → Tier 1
TTFT vs. ITL split (vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds)TTFT regressions localize to prefill or KV transfer; ITL regressions localize to decode. Splitting them tells you which pool to investigateMetrics → vLLM

SGLang deployments expose the equivalent signals under sglang:* (sglang:num_running_reqs, sglang:token_usage); the PromQL reference lists both.

Common failure modes​

  • TTFT regression, decode healthy — prefill pool is saturated or KV transfer is stalling. Check prefill utilization and TTFT together; if prefill is idle but TTFT is high, suspect the configured KV-transfer backend (NIXL, Mooncake, or another backend; see the SGLang operations doc for the prefill-side KV-strand caveat).
  • ITL regression, prefill healthy — decode pool is the bottleneck. Check decode KV cache utilization; sustained values near 1.0 mean the decode role needs more replicas or a larger TP degree.
  • Both pools underutilized but latency high — routing problem. Check the P/D decision ratio (llm_d_epp_disagg_decision_total) and EPP scheduler e2e latency (llm_d_epp_scheduler_e2e_duration_seconds) before touching the model servers.
  • cks-mooncake: no RDMA in pod — check that rdma/ib appears in the node's allocatable resources (kubectl describe node) and that the RDMA device plugin is running.
  • cks-mooncake: MooncakeConnector fails to initialize — verify that mooncake-transfer-engine is installed in the image (pip show mooncake-transfer-engine inside the pod) and that the bootstrap port is free.
  • cks-mooncake: KV transfer failures — confirm that the prefill and decode pods can reach each other over the RDMA network.

For alert rules covering these signals, see Alerting.

Cleanup​

To remove the deployed components:

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}

On DisaggregatedSet paths, deleting the set cascades to its LeaderWorkerSets, their pods, and per-slice Services. For TPU dynamic sub-slices, delete the overlay before removing any node pools so that Kueue releases the Slice resources it created.

helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring-pd --ignore-not-found=true
kubectl delete namespace ${NAMESPACE}