Version: dev

Accelerator

Provider

Engine

validated

Wide Expert Parallelism

Overview

This guide deploys a large Mixture-of-Experts (MoE) model in a wide expert-parallel (DP/EP) pattern across nodes, with vLLM prefill/decode (P/D) disaggregation and DP-aware routing by the llm-d Router. The NVIDIA GPU and AMD configurations deploy a single DisaggregatedSet that manages the prefill and decode LeaderWorkerSets as one versioned unit; the Intel XPU configuration uses two plain LeaderWorkerSets.

The default deployment serves deepseek-ai/DeepSeek-R1-0528 on 32 NVIDIA H200 GPUs: a prefill and a decode role, each one DP16/EP group across two nodes, with DeepEP for the expert all-to-all and NIXL for the KV transfer over RDMA. Every DP rank runs its own API server port, and the router's InferencePool targets all of them, so the router picks a DP rank, not just a pod.

Each accelerator serves one model, chosen to teach the mechanics: multi-node LeaderWorkerSet/DisaggregatedSet deployment, DP/EP configuration in vLLM, and routing to individual DP ranks. For state-of-the-art, benchmarked MoE recipes, see the Models guides, such as DeepSeek-V4 and GLM-5.2.

For why wide expert parallelism helps, how a DP/EP forward pass works, why it pairs with P/D disaggregation, and how requests flow to individual DP ranks, see Wide Expert Parallelism.

Supported Accelerators and Model Servers

This guide includes configurations for the following accelerator and model server combinations (set ACCELERATOR_TYPE and MODEL_SERVER accordingly). Each accelerator serves exactly one model:

Accelerator	`ACCELERATOR_TYPE`	Served model	vLLM	Notes
NVIDIA GPU	`gpu`	`deepseek-ai/DeepSeek-R1-0528`	✅ validated	Default. 32× H200 · `DisaggregatedSet`: prefill DP16 + decode DP16, 2 nodes each · DeepEP + NIXL · `INFRA_PROVIDER`: `base`, `gke`, `coreweave`
AMD GPU	`amd`	`deepseek-ai/DeepSeek-V3`	✅ validated	32× MI355X · `DisaggregatedSet`: prefill DP16 + decode DP16, 2 nodes each · MoRI-EP + MoRI-IO · `INFRA_PROVIDER`: `base`, `amd-ci`
Intel XPU	`xpu`	`deepseek-ai/DeepSeek-V2-Lite-Chat`	✅ validated	4 XPUs via DRA · `LeaderWorkerSet`: prefill TP2 + decode TP2 with EP · NIXL · `INFRA_PROVIDER`: `base`

✅ validated: covered by a nightly E2E workflow · 🟡 community: maintained by the hardware vendor or community, not covered by nightly E2E · ❌ not supported: tracked in the linked issue · — no configuration.

NVIDIA GPU (DeepSeek-R1-0528): prefill DP16 and decode DP16 on 32 H200, with DeepEP (high-throughput prefill, low-latency decode) and NIXL KV transfer. Validated on a 32×H200 cluster with InfiniBand networking (coreweave) and a 32×H200 cluster on GKE with RoCE networking (gke); base is the provider-neutral overlay the others patch.

note

The NVIDIA GPU overlays run the upstream vllm/vllm-openai image pinned by the gpu-vllm/release image component. The custom llm-d vLLM image that earlier releases needed for DeepEP on RoCE is no longer used.

AMD GPU (DeepSeek-V3): prefill and decode each run 1 replica of 2 nodes, DP16 (TP=1), on 32 MI355X, with the mori_high_throughput all-to-all backend and MoRI-IO KV transfer. amd-ci adapts base to the AMD CI cluster (MI355X, Pensando AINIC, rail-isolated RoCE fabric).

The AMD decode routing sidecar reaches its MoRI-IO peers by their LeaderWorkerSet pod DNS names, built from the pod's disaggregatedset.x-k8s.io/revision label, so no extra Services are needed. The peer list is fixed to this 2P2D shape (one two-pod group per role) and resolved once at startup: changing replicas or size means editing it and redeploying. This is a temporary workaround until multi-pod peer discovery is supported.

Intel XPU (DeepSeek-V2-Lite-Chat): prefill and decode each run TP2 with expert parallelism on 4 XPUs in total, with the allgather_reducescatter all-to-all backend, NIXL KV transfer (kv_buffer_device=xpu), and the tcp,ze_copy UCX transport for the validated non-RDMA configuration.

Prerequisites

Have the proper client tools installed on your local system to use this guide.
Have a cluster with RDMA-capable accelerator nodes. For the networking stack and how to verify it, see RDMA and Networking Configuration and the model server orchestration guide. For GKE, see the provider setup doc and the GKE overlay cluster prerequisites.

info

The NVIDIA GPU and AMD configurations use an RDMA all-to-all backend (DeepEP, MoRI) for inter-node EP and require all-to-all RDMA connectivity: every NIC on a host must be able to communicate with every NIC on all other hosts. Networks restricted to communicating only between matching NIC IDs (rail-only connectivity) will fail. The Intel XPU configuration uses XCCL and allgather_reducescatter; it does not use DeepEP, but still requires full-mesh pod network connectivity between decode and prefill workers.

Deploy the LeaderWorkerSet controller v0.11.1 or newer. When installing with Helm, pass --set enableDisaggregatedSet=true to enable the DisaggregatedSet validating webhook and RBAC used by the NVIDIA GPU and AMD configurations.
For Intel XPU, install the Intel Resource Drivers for Kubernetes and verify that the gpu.intel.com DRA DeviceClass is available.
Create a HuggingFace token and export it as HF_TOKEN in your shell.
(Optional) Set up the monitoring stack if you plan to enable Prometheus monitoring.

Get the guide

Every command below runs from a local clone of the llm-d repository: the manifests, Helm values, and Kustomize overlays it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Configure the environment

Set the guide-specific environment variables:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=wide-ep
export NAMESPACE=llm-d-wide-ep
export MONITORING=false # options: false, true
export MONITORING_VALUES=
export ACCELERATOR_TYPE=gpu # options: gpu, amd, xpu
export MODEL_SERVER=vllm # options: vllm
export ACCELERATOR=${ACCELERATOR_TYPE} # keep equal to ACCELERATOR_TYPE
export INFRA_PROVIDER=base # options: base, gke, coreweave, amd-ci; valid values per accelerator: table above
export MODEL=deepseek-ai/DeepSeek-R1-0528 # set to the model your accelerator serves (table above)
source ${REPO_ROOT}/guides/env.sh # defines GAIE_VERSION, ROUTER_CHART_VERSION, router chart URLs, and CURL_TEST_IMAGE

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models:

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Installation Instructions

1. Deploy the llm-d Router

Prepare the paths to the helm values files for the llm-d router (used in the deployment command below). router/wide-ep.values.yaml configures P/D disaggregation and targets all eight DP rank ports of every pod; the per-accelerator file layered on top re-targets the router at the AMD or Intel XPU model servers:

# Paths to values files
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml"
# Per-accelerator overrides: the AMD and Intel XPU ones re-target the router
# at their model servers; the NVIDIA GPU one is empty
export ROUTER_ACCELERATOR_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/${ACCELERATOR}.values.yaml"

(Optional) Enable Prometheus monitoring on the llm-d router by defining MONITORING_VALUES (see Collect Metrics):

OSS Prometheus / OpenShift
GKE (Managed Prometheus)

# only when MONITORING=true:
export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml"

# only when MONITORING=true and INFRA_PROVIDER=gke:
export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring-gke.values.yaml"

Deploy the router in Standalone Mode, with an Envoy sidecar in front of the router. The release name ${GUIDE_NAME} is mandatory: the InferencePool selector matches a guide label that pairs with this release. To front the router with a Kubernetes Gateway instead, see Gateway Mode in the Optimized Baseline.

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  -f ${ROUTER_ACCELERATOR_VALUES} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

2. Deploy the Model Server

For model sources, caching, and startup optimization, see the Model Loading and Startup Acceleration operations guide.

Apply the Kustomize overlay for your accelerator and infrastructure provider (see the table above for the valid INFRA_PROVIDER values):

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}/

3. (Optional) Enable Monitoring

On the NVIDIA GPU deployment each DP rank is available at podIP:port, where the ports are named rank0-rank7, and this guide ships a monitoring overlay that scrapes each rank's port. The overlay selects the DeepSeek-R1-0528 pods only; the AMD and Intel XPU overlays expose a single modelserver port and are not covered by it:

# only when MONITORING=true:
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/monitoring

Verification

1. Get the IP of the Proxy

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send Test Requests

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\"}"'

Mechanism Verification

These checks confirm that requests are spread across DP ranks and that every request is disaggregated: prefilled on a prefill rank, with its KV cache transferred to a decode rank.

Send a batch of requests with distinct prompts through the proxy:

# 32 requests, 8 at a time, each with a distinct ~1k-token prompt (no shared
# prefix, so prefix-cache affinity does not pin them to one DP rank): each is
# prefilled on a prefill DP rank, and its KV cache is transferred to a decode
# DP rank
kubectl run dp-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'P=$(for i in $(seq 1 60); do printf "Prefill ranks compute the KV cache and decode ranks pull it to generate tokens. "; done)
    for i in $(seq 1 32); do
      curl -sS -o /dev/null -w "%{http_code}\n" -X POST "http://${IP}/v1/completions" \
        -H "Content-Type: application/json" \
        -d "{\"model\": \"${MODEL}\", \"prompt\": \"Request ${i}. ${P} What is wide expert parallelism?\", \"max_tokens\": 32}" &
      [ $((i % 8)) -eq 0 ] && wait
    done | sort | uniq -c'

Count the DP ranks that served them, from each pod's vLLM metrics:

# Requests and connector (KV transfer) prefix-cache hits per DP rank, read from
# every running model server pod through the Kubernetes API server proxy (no
# port-forward needed). NVIDIA GPU runs one API server per DP rank (prefill on
# ports 8000-8007, decode on 8200-8207 behind the routing sidecar); AMD and
# Intel XPU serve all of a pod's ranks on one port (prefill 8000, decode 8200)
for role in prefill decode; do
  first=8000; [ "${role}" = decode ] && first=8200
  ports=${first}; [ "${ACCELERATOR_TYPE}" = gpu ] && ports=$(seq ${first} $((first + 7)))
  pods=$(kubectl get pods -n ${NAMESPACE} -l llm-d.ai/guide=${GUIDE_NAME},llm-d.ai/role=${role} \
    --field-selector=status.phase=Running -o jsonpath='{.items[*].metadata.name}')
  if [ -z "${pods}" ]; then echo "${role}: no running pods"; continue; fi
  for pod in ${pods}; do
    for port in ${ports}; do
      kubectl get --raw "/api/v1/namespaces/${NAMESPACE}/pods/${pod}:${port}/proxy/metrics" \
        | awk -v k="${pod}:${port}" '/^vllm:(request_success|external_prefix_cache_hits)_total\{/ {
            e = match($0, /engine="[^"]*"/) ? substr($0, RSTART + 8, RLENGTH - 9) : "0"
            if ($0 ~ /^vllm:request_success/) r[k "/" e] += $NF; else h[k "/" e] += $NF
          } END { for (x in r) print x, r[x], h[x] + 0 }'
    done
  done | awk -v role="${role}" '$2 > 0 { n++; req += $2; hits += $3 }
    END { printf "%s: %d DP rank(s) served %d requests; connector prefix-cache hits: %d tokens\n", role, n, req, hits }'
done

The script prints one line per role: the number of DP ranks that served requests, the requests they served, and the connector prefix-cache hits.

DP/EP: on NVIDIA GPU and AMD, where each role spans 16 DP ranks, both roles report more than one DP rank. The router picks a rank per request, so the exact counts vary from run to run. On Intel XPU, each role is one TP2 engine with expert parallelism, so each reports one rank.
P/D: the prefill pods served every request, and the decode pods report connector prefix-cache hits: tokens whose KV cache was transferred from a prefill rank instead of being recomputed. Without disaggregation, the prefill counters stay at zero.

If you enabled monitoring (MONITORING=true), the router also counts its P/D decisions:

# only when MONITORING=true:
# P/D decisions taken by the router, read through the API server service proxy
kubectl get --raw "/api/v1/namespaces/${NAMESPACE}/services/${GUIDE_NAME}-epp:9090/proxy/metrics" \
  | grep -E '^llm_d_epp_disagg_decision_total' || true

Cleanup

To remove the deployed components:

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}

helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/monitoring --ignore-not-found=true

kubectl delete namespace ${NAMESPACE}

Overview​

Supported Accelerators and Model Servers​

Prerequisites​

Get the guide​

Configure the environment​

Installation Instructions​

1. Deploy the llm-d Router​

2. Deploy the Model Server​

3. (Optional) Enable Monitoring​

Verification​

1. Get the IP of the Proxy​

2. Send Test Requests​

Mechanism Verification​

Cleanup​