Skip to main content
Version: dev
Accelerator
Engine
community

DeepSeek-V4-Pro on GB200

Overview​

This guide deploys DeepSeek-V4-Pro on NVIDIA GB200 NVL72 with vLLM prefill/decode disaggregation (NIXL KV transfer) in a wide expert-parallel pattern, managed as a single DisaggregatedSet (a pair of LeaderWorkerSets). Wide-EP spans multiple nodes over GB200's cross-node NVLink (MNNVL) fabric, provisioned through an NVIDIA DRA ComputeDomain.

It composes the wide expert parallelism and P/D disaggregation foundations with P/D-aware, prefix-cache-aware routing, and ships a range of prefill : decode operating points, from a 16-GPU low-latency layout up to a 56-GPU high-throughput layout.

These manifests were tested on Oracle Cloud Infrastructure (OCI, BM.GPU.GB200.4 nodes). Storage and DRA may need to be adapted to your environment.

Default Configuration​

ParameterValue
Modeldeepseek-ai/DeepSeek-V4-Pro
AcceleratorNVIDIA GB200 NVL72 (4 GPUs per node), cross-node NVLink via ComputeDomain
Serving topologyP/D disaggregated DisaggregatedSet; prefill DEP8, decode TP=8, DEP8 or DEP16 (see Deployments)
DP modelHybrid load balancing (--data-parallel-hybrid-lb), expert parallelism enabled
MoE backenddeep_gemm_mega_moe
KV transferNixlConnector
KV cacheFP8, block size 256
Max model length9280 tokens
Model storageNode-local NVMe (hostPath /mnt/numa0/hf-cache, ~850 GB model)

Supported Accelerators and Model Servers​

This guide includes configurations for the following accelerator and model server combinations:

AcceleratorACCELERATOR_TYPEServed modelvLLMNotes
NVIDIA GB200 NVL72gpudeepseek-ai/DeepSeek-V4-Pro🟡 communityOCI BM.GPU.GB200.4 (4 GPUs per node) · 4 to 14 nodes (16 to 56 GPUs) by DEPLOYMENT · P/D disaggregated wide-EP; needs the NVIDIA DRA driver (ComputeDomain)

✅ validated: covered by a nightly E2E workflow · 🟡 community: maintained by the hardware vendor or community, not covered by nightly E2E · ❌ not supported: tracked in the linked issue · — no configuration.

Prerequisites​

Get the guide​

Every command below runs from a local clone of the llm-d repository: the manifests, Helm values, and Kustomize overlays it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Configure the environment​

Set the guide-specific environment variables (DEPLOYMENT picks one of the deployments):

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=deepseek-v4
export NAMESPACE=llm-d-deepseek-v4
export MONITORING=false # options: false, true
export MONITORING_VALUES=
export ACCELERATOR_TYPE=gpu # options: gpu
export MODEL_SERVER=vllm # options: vllm
export DEPLOYMENT=oci-low-latency # options: oci-low-latency, oci-low-latency-scaled, oci-mid-curve, oci-balanced, oci-high-tpt, oci-high-tpt-dep16, oci-max-tpt, oci-ultra-tpt, oci-3p2d-dep8-dep16-flashinfer
export MODEL=deepseek-ai/DeepSeek-V4-Pro # the model every deployment serves
source ${REPO_ROOT}/guides/env.sh # defines GAIE_VERSION, ROUTER_CHART_VERSION, router chart URLs, and CURL_TEST_IMAGE

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Check that the ComputeDomain CRD is present (if this returns NotFound, install the NVIDIA DRA driver before continuing):

# The NVIDIA DRA driver must be installed: the deployments claim a ComputeDomain
kubectl get crd computedomains.resource.nvidia.com

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models:

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Installation Instructions​

1. Deploy the llm-d Router​

Prepare the paths to the helm values files for the llm-d router. The router values (deepseek-v4.values.yaml) run separate prefill and decode scheduling profiles (prefix-cache affinity filtering and token-load scoring for prefill; active-request scoring for decode) and pin each request to one DisaggregatedSet revision during rollouts:

warning

peakPrefillThroughput is an estimate for DeepSeek-V4-Pro on GB200, not a measurement. The prefix-cache-affinity-filter ships with 20000 tokens/s per prefill pod. Each pod is one router endpoint fronting 4 DP ranks, and the value is ~5000 tokens/s per rank x 4, estimated by scaling measured MoE prefill rates to this model (plausible range 12000 to 25200). Measure your own value with the router calibration tool using CHUNK_SIZE=8192, because a request must fit --max-model-len 9280. A single calibration request runs on one rank, so multiply the result by 4 before setting it in router/deepseek-v4.values.yaml.

# Paths to values files
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/models/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml"

(Optional) Enable Prometheus monitoring on the llm-d router by defining the helm values file (requires installing the monitoring stack mentioned in Prerequisites):

# only when MONITORING=true:
export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml"

Deploy the router in Standalone Mode, with an Envoy sidecar in front of the router. Keep the release name ${GUIDE_NAME}: the verification steps look up the ${GUIDE_NAME}-epp Service by that name. To front the router with a Kubernetes Gateway instead, see Gateway Mode in the Optimized Baseline.

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

2. Deploy the Model Server​

Each deployment is a different prefill : decode operating point. Set DEPLOYMENT to one of them (default oci-low-latency, the smallest):

DeploymentLayoutNodes / GPUs
oci-low-latency1 prefill (DEP8) : 1 decode (TP=8)4 / 16
oci-low-latency-scaled1 prefill (DEP8) : 4 decode (TP=8)10 / 40
oci-mid-curve1 prefill : 1 decode (DEP8 each)4 / 16
oci-balanced2 prefill (DEP8) : 1 decode (DEP16)8 / 32
oci-high-tpt2 prefill : 1 decode (DEP8 each)6 / 24
oci-high-tpt-dep163 prefill (DEP8) : 1 decode (DEP16)10 / 40
oci-max-tpt3 prefill : 1 decode (DEP8 each)8 / 32
oci-ultra-tpt4 prefill (DEP8) : 1 decode (DEP16)12 / 48
oci-3p2d-dep8-dep16-flashinfer3 prefill (DEP8) : 2 decode (DEP16), flashinfer14 / 56

Apply the Kustomize overlay for your deployment. The deployments compose the providers/oci overlay (node-local NVMe model cache and the ComputeDomain claim), or base-low-latency with the same OCI settings inlined for the two low-latency variants, with a topology component from modelserver/gpu/vllm/components/:

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/deployments/${DEPLOYMENT}

Wait for the pods to become ready (the ~850 GB model takes a while to download and load):

kubectl get pods -n ${NAMESPACE} -l llm-d.ai/model=DeepSeek-V4-Pro -w

(Optional) Deploy the monitoring resources for model servers (requires installing the monitoring stack mentioned in Prerequisites):

# only when MONITORING=true:
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/monitoring

Verification​

1. Get the IP of the Proxy​

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send Test Requests​

Send a completion request from a temporary pod inside the cluster:

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\"}"'

Check that the router reaches a model server that serves MODEL: the response must name the model and carry generated tokens:

# The router must reach a model server that serves MODEL and returns tokens
kubectl run serve-check --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'R=$(curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"Say hello.\", \"max_tokens\": 16}")
    echo "${R}" | grep -q "\"model\":\"${MODEL}\"" && echo "${R}" | grep -q "\"choices\"" \
      && echo "OK: ${MODEL} served through the router" \
      || { echo "FAIL: ${R}"; exit 1; }'

Benchmarking​

note

TODO: benchmark profiles and results for this guide are not yet published. The first published results should be measured with the current router config (prefix-cache affinity filter + token-load scorer on prefill, active-request scorer on decode), after peakPrefillThroughput has been measured for DeepSeek-V4-Pro on GB200 (see the warning in Deploy the llm-d Router).

Cleanup​

To remove the deployed components:

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/deployments/${DEPLOYMENT}
helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/monitoring --ignore-not-found=true
kubectl delete namespace ${NAMESPACE}