Version: dev

Accelerator

Engine

community

Not supported on NVIDIA GB300 NVL72: SGLang

Kimi-K3.1-Pro on NVIDIA GB300 GPU (vLLM) and Google TPU 7x (SGLang)

Kimi-K3.1-Pro on NVIDIA GB300 GPUs via vLLM using 3P1D P/D disaggregation (3× TP=8 × EP=8 Prefill + 1× DP=16 × EP=16 Decode), DSpark speculative decoding, NIXL + Mooncake RDMA KV cache sharing, and ZMQ precise prefix-cache routing, alongside TPU 7x via SGLang using 480K long-context multi-host parallelism (TP=32 × EP=8) with prefix-aware routing and HiCache host-DRAM KV offloading.

Overview

This guide deploys moonshotai/Kimi-K3.1-Pro (MXFP4 experts, 1.56 TB; 69 KDA + 24 sparse-MLA layers with a DSA indexer) behind the llm-d router:

3P1D P/D disaggregated GPU serving (vLLM on NVIDIA GB300): Kimi-K3.1-Pro runs as a 10-node (40× GB300 GPU) P/D disaggregated pool (modelserver/gpu/vllm/) connected over cross-node NVLink (ComputeDomain MNNVL/IMEX) and 8-NIC RoCEv2 RDMA:
- Prefill Pool (3 replicas × 2 nodes = 6 nodes / 24× GB300 GPUs, TP=8 × EP=8): Each prefill replica spans 2 nodes × 4 GPUs (--nnodes 2 --tensor-parallel-size 8 --enable-expert-parallel --enable-ep-weight-filter --kv-cache-dtype nvfp4_ds_mla), publishes ZMQ KV-cache events on :5557 (PYTHONHASHSEED=42), and runs a co-located vllm launch render sidecar on :8089 for EPP's token-producer.
- Decode Pool (1 replica × 4 nodes = 4 nodes / 16× GB300 GPUs, DP=16 × EP=16): Spans 4 nodes × 4 GPUs (--data-parallel-size 16 --data-parallel-size-local 4 --enable-expert-parallel --enable-ep-weight-filter --kv-cache-dtype nvfp4_ds_mla --use-replayssm) with MXFP8 attention/expert projections and llm-d-router-disagg-sidecar (--kv-connector=nixlv2) fronting ports 8000..8003 -> 8200..8203.
- DSpark speculative decoding: Both pools use the Inferact/Kimi-K3.1-Pro-DSpark-preview EAGLE3 drafter (num_speculative_tokens=5, method=dspark, draft_sample_method=probabilistic) and fastsafetensors weight loading.
- NIXL + Mooncake RDMA KV cache sharing: MultiConnector combines NixlConnector (kv_producer -> kv_consumer) with MooncakeStoreConnector (protocol=rdma, global_segment_size=160GB/worker, plus optional 500 GB/node mooncake-donor DRAM donors) coordinated by mooncake-master.
Multi-host TPU serving (SGLang on Google TPU 7x): Kimi-K3.1-Pro runs as one TP=32 × EP=8 engine per tpu7x 2x2x4 slice (4 hosts × 4 chips), as a LeaderWorkerSet of 4 pods (modelserver/tpu/sglang/). SGLang HiCache over TPU Sync keeps evicted prefixes (sparse-MLA / DSA pages and KDA recurrent-state slots) in a host-DRAM pool twice the size of the HBM pool.
Precise & prefix-aware routing: On GPU, the llm-d router (kimi-k3-1-gpu.values.yaml) runs disagg-profile-handler (always-disagg-pd-decider), token-producer (:8089), and precise-prefix-cache-producer (blockSizeTokens=2048, hashSeed="42", ZMQ :5557) to route prefill turns to cache-warm replicas (prefix-cache-scorer) and balance decode across the 16 DP ranks. On TPU (kimi-k3-1.values.yaml), approx-prefix-cache-producer + prefix-cache-affinity-filter + token-load-scorer route turns to warm tpu7x slices.

Default Configuration

NVIDIA GB300 GPU (vLLM)
Google TPU 7x (SGLang)

Parameter	Value
Model	moonshotai/Kimi-K3.1-Pro (MXFP4 experts, 1.56 TB; 69 KDA + 24 sparse-MLA layers with a DSA indexer)
Speculative drafter	Inferact/Kimi-K3.1-Pro-DSpark-preview (`num_speculative_tokens: 5`, `method: dspark`, `draft_sample_method: probabilistic`)
Topology (3P1D)	10 nodes (40× NVIDIA GB300 GPUs): 3× 2-node Prefill LeaderWorkerSet groups (24 GPUs) + 1× 4-node Decode LeaderWorkerSet group (16 GPUs)
Accelerator	NVIDIA GB300 NVL72 (4 GPUs per node) with NVIDIA DRA `ComputeDomain` (`numNodes: 10`, MNNVL / IMEX) and 8-NIC RoCEv2 RDMA
Prefill parallelism	`TP=8 × EP=8` (`--nnodes 2 --tensor-parallel-size 8 --enable-expert-parallel --enable-ep-weight-filter --kv-cache-dtype nvfp4_ds_mla --max-num-seqs 32`) + co-located `vllm launch render` on `:8089`
Decode parallelism	`DP=16 × EP=16` (`--data-parallel-size 16 --data-parallel-size-local 4 --enable-expert-parallel --enable-ep-weight-filter --kv-cache-dtype nvfp4_ds_mla --use-replayssm` + MXFP8 projections) behind `llm-d-router-disagg-sidecar` (`--kv-connector=nixlv2`)
KV cache transfer & sharing	`MultiConnector`: `NixlConnector` (`kv_producer` -> `kv_consumer`, port `5600..5615`) + `MooncakeStoreConnector` (`protocol: rdma`, `mooncake-master` on `:8700`)
Routing	Disaggregated P/D (`always-disagg-pd-decider`) + ZMQ `precise-prefix-cache-producer` (`blockSizeTokens: 2048`, `hashSeed: "42"`, `:5557`) + `token-producer` (`:8089`) (`kimi-k3-1-gpu.values.yaml`)
Model server	vLLM (`docker.io/vllm/vllm-openai:kimi-k31`, pinned by digest)

Parameter	Value
Model	moonshotai/Kimi-K3.1-Pro (MXFP4 experts, 1.56 TB; 69 KDA + 24 sparse-MLA layers with a DSA indexer)
Replicas	2 LeaderWorkerSet groups (4 pods each)
Accelerator	Google TPU 7x (`tpu7x`)
Topology	`2x2x4` (4 hosts × 4 chips, 32 TensorCores) per replica; 32 chips for the default 2 replicas
Parallelism	TP=32 × EP=8 (SGLang's Kimi-K3.1 TPU recipe)
Context length	491,520 tokens (480K). The batch-64 decode kernel's SMEM caps the context at about 496K tokens; the HBM KV pool holds 532,352 tokens
KV cache	fp8, `--page-size 128`, radix cache; HiCache host tier with `--hicache-ratio 2` (1,064,832 host tokens per rank)
Model weights	`ReadOnlyMany` Hyperdisk ML volume hydrated once from a GCS copy of the Hugging Face repo
Model server	SGLang on Torch TPU (`docker.io/lmsysorg/sglang:dev-kimi-k31-pro-tpu`, pinned by digest)

note

On TPU 7x (ACCELERATOR_TYPE=tpu), serving is text only, and OpenAI tool calling does not work in this configuration yet. This SGLang TPU recipe runs with --grammar-backend none, so requests that carry an OpenAI tools array are rejected with HTTP 400, and coding-agent clients that send tools will fail. The kimi_k3 reasoning parser works. The server keeps --tool-call-parser=kimi_k3 so tool calling works without a manifest change once SGLang on TPU supports it; until then it has no effect.

Supported Accelerators and Model Servers

This guide includes configurations for the following accelerator and model server combinations:

Accelerator	`ACCELERATOR_TYPE`	Served model	vLLM	SGLang	Notes
NVIDIA GB300 NVL72	`gpu`	`moonshotai/Kimi-K3.1-Pro`	🟡 community	—	vLLM (`vllm/vllm-openai:kimi-k31`) · 3P1D P/D disaggregated LeaderWorkerSets on 10 nodes / 40× GB300 GPUs: 3× 2-node Prefill (`TP=8 × EP=8`) + 1× 4-node Decode (`DP=16 × EP=16`, MXFP8 projections, ReplaySSM) · MXFP4 experts, `nvfp4_ds_mla` KV cache, `Inferact/Kimi-K3.1-Pro-DSpark-preview` speculative decoding (`num_speculative_tokens=5`, `dspark`), fastsafetensors · NIXL + MooncakeStore RDMA KV cache sharing (`MultiConnector`) and ZMQ precise prefix-cache routing (`blockSizeTokens=2048`)
Google TPU v7x	`tpu`	`moonshotai/Kimi-K3.1-Pro`	—	🟡 community	GKE only · SGLang on Torch TPU · 2 replicas, each a LeaderWorkerSet of 4 pods on one `tpu7x` `2x2x4` slice (16 chips): TP=32 × EP=8, MXFP4 experts, sparse-MLA (DSA) attention, fp8 KV cache, 480K context, host-DRAM HiCache over TPU Sync (2× the HBM pool) · text only (`--grammar-backend=none` in this SGLang recipe; OpenAI `tools` requests are not yet enabled) · weights on a ReadOnlyMany Hyperdisk ML volume hydrated from GCS

✅ validated: covered by a nightly E2E workflow · 🟡 community: maintained by the hardware vendor or community, not covered by nightly E2E · ❌ not supported: tracked in the linked issue · — no configuration.

Prerequisites

Have the proper client tools installed on your local system to use this guide.
For NVIDIA GB300 GPU (ACCELERATOR_TYPE=gpu, MODEL_SERVER=vllm):
- A Kubernetes cluster with 10× 4-GPU NVIDIA GB300 NVL72 nodes (40 GPUs total: 6 nodes for the 3× 2-node TP=8 × EP=8 Prefill pool and 4 nodes for the 1× 4-node DP=16 × EP=16 Decode pool).
- On GKE A4X-MAX (modelserver/gpu/vllm/gke/), enable the NVIDIA DRA driver (ComputeDomain CRD computedomains.resource.nvidia.com for Multi-Node NVLink MNNVL / IMEX) and DRANET GPUDirect RDMA so kimi-k3-1-compute-domain and the 8-NIC mrdma.google.com ResourceClaimTemplate (kimi-k3-1-gpu-vllm-rdma-nic) resolve.
For Google TPU 7x (ACCELERATOR_TYPE=tpu, MODEL_SERVER=sglang):
- A GKE cluster with two TPU 7x (tpu7x) 2x2x4 node pools (4 nodes of 4 chips each), with Workload Identity Federation for GKE enabled. Each node needs about 800 GiB of allocatable host memory (weight loading peaks at about 660 GiB per pod, plus the HiCache host pools).
The LeaderWorkerSet (LWS) controller.
A Google Cloud Storage bucket in the cluster's region, exported as MODEL_BUCKET, and a Hyperdisk ML quota of at least 2 TB in the cluster's zone (or a pre-populated /models volume).
Create a HuggingFace token and export it as HF_TOKEN in your shell.
(Optional) Install the monitoring stack if you plan to enable Prometheus monitoring.

Get the guide

Every command below runs from a local clone of the llm-d repository: the manifests, Helm values, and Kustomize overlays it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Configure the environment

Set the guide-specific environment variables (defaults to ACCELERATOR_TYPE=gpu and MODEL_SERVER=vllm; set ACCELERATOR_TYPE=tpu and MODEL_SERVER=sglang for the TPU 7x recipe):

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=kimi-k3-1
export NAMESPACE=llm-d-kimi-k3-1
export MONITORING=false # options: false, true
export MONITORING_VALUES=
export ACCELERATOR_TYPE=gpu # options: gpu, tpu
export MODEL_SERVER=vllm # options: vllm, sglang
export MODEL=moonshotai/Kimi-K3.1-Pro # the served model name
export MODEL_BUCKET=my-model-bucket # GCS bucket that holds the moonshotai/Kimi-K3.1-Pro copy
source ${REPO_ROOT}/guides/env.sh # defines GAIE_VERSION, ROUTER_CHART_VERSION, router chart URLs, and CURL_TEST_IMAGE

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Install the LeaderWorkerSet controller (skip if your cluster already runs LWS):

# helm upgrade never updates CRDs, so apply the chart's CRDs first (see LWS
# "Upgrade by Helm"); this upgrades them on clusters that run an older LWS.
helm pull oci://registry.k8s.io/lws/charts/lws --version=${LWS_VERSION} --untar
kubectl apply --server-side --force-conflicts -f lws/crds
helm upgrade --install lws oci://registry.k8s.io/lws/charts/lws \
  --version=${LWS_VERSION} \
  --namespace lws-system --create-namespace \
  --wait --timeout 300s

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models:

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Stage the weights

At 1.56 TB the checkpoint is far larger than a node's boot disk, and reading it over a network filesystem dominates start-up. Copy it once into MODEL_BUCKET (in the cluster's region):

# One-time copy of the 1.56 TB checkpoint and DSpark speculative drafter into a bucket in
# the cluster's region. The download needs about 1.6 TB of free local disk; run it on a GCE
# VM in the same region with a large enough disk if your workstation does not have it.
hf download moonshotai/Kimi-K3.1-Pro --local-dir ./Kimi-K3.1-Pro
HF_HOME=./hf-cache hf download Inferact/Kimi-K3.1-Pro-DSpark-preview
gcloud storage cp -r ./Kimi-K3.1-Pro ./hf-cache gs://${MODEL_BUCKET}/moonshotai/

Then create the model volume: a ReadOnlyMany Hyperdisk ML PVC that the GKE Volume Populator hydrates from the bucket once. Every model-server pod mounts it read-only at /models. Give the populator's transfer job read access to the bucket:

# The GKE Volume Populator's transfer job reads the bucket as this Kubernetes ServiceAccount.
kubectl create serviceaccount kimi-k3-1-model-populator -n ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -
export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format='value(projectNumber)')
gcloud storage buckets add-iam-policy-binding gs://${MODEL_BUCKET} \
  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/kimi-k3-1-model-populator" \
  --role=roles/storage.objectViewer

Create the StorageClass, data source and PVC. The PVC hydrates when the first pod that mounts it is scheduled:

# A ReadOnlyMany Hyperdisk ML volume that every model-server pod mounts at /models,
# hydrated once from gs://${MODEL_BUCKET}/moonshotai by the GKE Volume Populator.
kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: kimi-k3-1-hyperdisk-ml
provisioner: pd.csi.storage.gke.io
parameters:
  type: hyperdisk-ml
  provisioned-throughput-on-create: "6400Mi"
allowVolumeExpansion: false
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: datalayer.gke.io/v1
kind: GCPDataSource
metadata:
  name: kimi-k3-1-models
spec:
  cloudStorage:
    serviceAccountName: kimi-k3-1-model-populator
    uri: gs://${MODEL_BUCKET}/moonshotai/
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kimi-k3-1-models
spec:
  accessModes: [ReadOnlyMany]
  storageClassName: kimi-k3-1-hyperdisk-ml
  resources:
    requests:
      storage: 2000Gi
  dataSourceRef:
    apiGroup: datalayer.gke.io
    kind: GCPDataSource
    name: kimi-k3-1-models
EOF

Installation Instructions

1. Deploy the llm-d Router

Prepare the paths to the helm values files for the llm-d router:

NVIDIA GB300 NVL72 · vLLM

export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/models/${GUIDE_NAME}/router/${GUIDE_NAME}-gpu.values.yaml"

(Optional) Enable Prometheus monitoring on the llm-d router by defining the helm values file (requires installing the monitoring stack mentioned in Prerequisites):

# only when MONITORING=true:
export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml"

note

On GPU (kimi-k3-1-gpu.values.yaml), the router configures disagg-profile-handler (always-disagg-pd-decider), token-producer (targeting prefill leaders' vllm launch render sidecar on :8089), and precise-prefix-cache-producer (blockSizeTokens: 2048, hashSeed: "42", ZMQ kvEventsConfig on :5557) with prefix-cache-scorer, active-request-scorer, and token-load-scorer on the prefill profile and active-request-scorer, queue-scorer, and kv-cache-utilization-scorer across ports 8000..8003 on the decode profile. On TPU (kimi-k3-1.values.yaml), blockSizeTokens: 128 matches SGLang's --page-size, maxPrefixTokensToMatch: 491520 matches --context-length, lruCapacityPerServer: 12478 covers the HBM + HiCache host pools of one TP rank, and peakPrefillThroughput: 4209 was calibrated on tpu7x 2x2x4.

Deploy the router in Standalone Mode, with an Envoy sidecar in front of the router. Keep the release name ${GUIDE_NAME}: the verification steps look up the ${GUIDE_NAME}-epp Service by that name. To front the router with a Kubernetes Gateway instead, see Gateway Mode in the Optimized Baseline.

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

2. Deploy the Model Server

Apply the Kustomize overlay (ACCELERATOR_TYPE=gpu and MODEL_SERVER=vllm deploys the 3P1D P/D disaggregated prefill + decode LeaderWorkerSets and mooncake-master; ACCELERATOR_TYPE=tpu and MODEL_SERVER=sglang deploys the 4-host tpu7x 2x2x4 LeaderWorkerSet):

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/

3P1D Disaggregated GPU Architecture and GKE Overlay (`ACCELERATOR_TYPE=gpu`)

Under modelserver/gpu/vllm/, the 3P1D Disaggregated + Mooncake RDMA + Precise Prefix Cache Routing recipe deploys:

Component	Replicas × Size	Parallelism & Role	Key vLLM Flags & Sidecars
`prefill-lws.yaml`	`3` × `2` nodes (`24`× GB300 GPUs)	`TP=8 × EP=8` (`kv_both`)	`--nnodes 2 --tensor-parallel-size 8 --enable-expert-parallel --enable-ep-weight-filter --enable-cumem-allocator --enable-prefix-caching --load-format fastsafetensors --gpu-memory-utilization 0.95 --kv-cache-dtype nvfp4_ds_mla --language-model-only --max-num-seqs 32 --speculative-config '{"model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"method":"dspark","draft_sample_method":"probabilistic"}'` + `MultiConnector` (`NixlConnector` + `MooncakeStoreConnector`), ZMQ KV events (`:5557`), and co-located `vllm launch render` (`:8089`)
`decode-lws.yaml`	`1` × `4` nodes (`16`× GB300 GPUs)	`DP=16 × EP=16` (`kv_consumer`)	`--data-parallel-size 16 --data-parallel-size-local 4 --data-parallel-multi-port-external-lb --data-parallel-supervisor-port 8208 --enable-expert-parallel --enable-ep-weight-filter --enable-cumem-allocator --load-format fastsafetensors --gpu-memory-utilization 0.95 --kv-cache-dtype nvfp4_ds_mla --max-num-seqs 32 --max-num-batched-tokens 512 --max-cudagraph-capture-size 512 --no-enable-prefix-caching --use-replayssm` + MXFP8 quantization config, `MultiConnector`, and `llm-d-router-disagg-sidecar` (`--kv-connector=nixlv2`, ports `8000..8003` -> `8200..8203`)
`mooncake.yaml`	`1` Deployment	Mooncake RDMA coordinator	`mooncake_master --port=8700 --enable_http_metadata_server=true --http_metadata_server_port=8080 --rpc_thread_num=32` + `mooncake-store-config` ConfigMap (`protocol: "rdma"`, `device_name: ""` for RDMA topology auto-discovery)
`modelserver/gpu/vllm/gke/`	Overlay	GKE A4X-MAX DRA + DRANET + Donors	Adds NVIDIA DRA `ComputeDomain` (`compute-domain.yaml`, `numNodes: 10`), 8-NIC `mrdma.google.com` `ResourceClaimTemplate` (`rdma-nic.yaml`), and optional `mooncake-donor` DaemonSet (`--global_segment_size=500GB`)

On GKE A4X-MAX clusters with NVIDIA DRA (ComputeDomain) and DRANET RDMA NIC claims (mrdma.google.com), apply the gke/ overlay instead:

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/modelserver/gpu/vllm/gke/

Wait for the model server to become ready:

kubectl wait --for=condition=Available leaderworkerset/kimi-k3-1-${ACCELERATOR_TYPE}-${MODEL_SERVER}-decode -n ${NAMESPACE} --timeout=7200s

(Optional) Deploy the monitoring resources for model servers (requires installing the monitoring stack mentioned in Prerequisites):

# only when MONITORING=true:
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring

Verification

1. Get the IP of the Proxy

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send Test Requests

Send a completion request from a temporary pod inside the cluster:

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"Explain how a simple agent loop works in 3 sentences.\"}"'

Check that the router reaches a model server that serves MODEL: the response must name the model and carry generated tokens:

# The router must reach a model server that serves MODEL and returns tokens
kubectl run serve-check --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'R=$(curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"Say hello.\", \"max_tokens\": 16}")
    echo "${R}" | grep -q "\"model\":\"${MODEL}\"" && echo "${R}" | grep -q "\"choices\"" \
      && echo "OK: ${MODEL} served through the router" \
      || { echo "FAIL: ${R}"; exit 1; }'

Benchmarking

This guide comes with an inference-perf benchmark preset (defined in inference-perf.yaml): multi-turn coding-style sessions over a shared system prompt, a large per-session repository context, and simulated pauses between turns (tool_call_latency_sec, client-side think time only; the requests carry no tools array). The configuration parameters include:

Workload Characteristic	Metric / Distribution Type	Min	Max	Mean / Constant	Std Dev	Description
Shared System Prompt	Constant	-	-	3,000 tokens	-	Common base instructions, libraries, and API schemas shared across all sessions. Highly cacheable.
Dynamic System Prompt	Lognormal	32,000	420,000	220,000 tokens	160,000	Repository context, file indexes, and session-specific code context, sized so sessions reach ~460K tokens and active KV spills into the host-DRAM HiCache tier.
Turns per Conversation	Lognormal	20	200	50 turns	30	Depth of the multi-turn conversational loop, avoiding premature Turn-0 resets during 20-turn benchmark stages.
Input Tokens per Turn	Lognormal	100	10,000	1,500 tokens	1,200	Incremental prompt extensions (test logs, follow-ups, modified code blocks) on each turn.
Output Tokens per Turn	Lognormal	50	4,096	425 tokens	825	Model generations per turn.
Tool Call Latency	Lognormal	1s	100s	15 seconds	55	Simulated client-side delay between turns (compilation, unit tests, search).

# CONCURRENCY_LEVEL is the number of concurrent coding sessions (at 16, active KV fills the
# 1.60M-token/replica HBM + HiCache tier); NUM_REQUESTS is 20 per session; SEED is varied per
# concurrency level so prompts don't overlap across runs.
export CONCURRENCY_LEVEL=16
export NUM_REQUESTS=$((20 * CONCURRENCY_LEVEL))
export SEED=$((7 + CONCURRENCY_LEVEL))

kubectl delete job inference-perf -n ${NAMESPACE} --ignore-not-found
envsubst < ${REPO_ROOT}/guides/models/${GUIDE_NAME}/inference-perf.yaml | kubectl apply -n ${NAMESPACE} -f -
kubectl logs -n ${NAMESPACE} job/inference-perf -f

Benchmark Results

Results for moonshotai/Kimi-K3.1-Pro on multi-node NVIDIA GB300 GPUs (vLLM) and Google TPU 7x (2x2x4, SGLang) are being collected with the workload above and will be published here with the image digest and benchmark parameters of each run.

Cleanup

To remove the deployed components:

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/models/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/

helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring --ignore-not-found=true

# The Hyperdisk ML volume holds a copy of the weights; delete it when you no longer need it.
kubectl delete pvc kimi-k3-1-models -n ${NAMESPACE} --ignore-not-found=true
kubectl delete gcpdatasource kimi-k3-1-models -n ${NAMESPACE} --ignore-not-found=true
kubectl delete serviceaccount kimi-k3-1-model-populator -n ${NAMESPACE} --ignore-not-found=true
kubectl delete storageclass kimi-k3-1-hyperdisk-ml --ignore-not-found=true
export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format='value(projectNumber)')
gcloud storage buckets remove-iam-policy-binding gs://${MODEL_BUCKET} \
  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/kimi-k3-1-model-populator" \
  --role=roles/storage.objectViewer

kubectl delete namespace ${NAMESPACE}

Overview​

Default Configuration​

Supported Accelerators and Model Servers​

Prerequisites​

Get the guide​

Configure the environment​

Stage the weights​

Installation Instructions​

1. Deploy the llm-d Router​

2. Deploy the Model Server​

3P1D Disaggregated GPU Architecture and GKE Overlay (ACCELERATOR_TYPE=gpu)​

Verification​

1. Get the IP of the Proxy​

2. Send Test Requests​

Benchmarking​

Benchmark Results​

Cleanup​