Skip to main content
Version: dev
Accelerator
Provider
Engine
validated

Not supported on NVIDIA GPU: Atom

Optimized Baseline

E2E (AMD ROCM) E2E (AMD ROCM SGLang) E2E (AMD ROCM ATOM) E2E (CKS GPU) E2E (GKE GPU) E2E (GKE TPU) E2E (OCP GPU) E2E (Intel XPU)

Overview​

Traditional HTTP requests are fast, uniform, and cheap. Standard round-robin request scheduling strategies balance this load well.

LLM requests break all three assumptions. They are:

  • Multi-turn - conversations and agentic tool loops send the same growing prefix repeatedly
  • Slow - a single request can take over a minute generating tokens
  • Non-uniform - range from 1000s of reasoning tokens to a 100k+ context tokens

The llm-d Router injects awareness of the LLM-workload into the load-balancing layer considering prefix-cache affinity and server load metrics.

This guide deploys the recommended out of the box configuration for most vLLM and SGLang deployments, reducing tail latency and increasing throughput through load-aware and prefix-cache aware balancing.

The optimized-baseline defaults to two main routing criteria:

  • Prefix-cache aware using the prefix cache affinity filter, which narrows candidates to "sticky" endpoints with high estimated prompt prefix cache reuse, with a saturation-aware override that spreads load when endpoints get hot.

  • Load-aware using the token load scorer, which scores endpoints based on the total prefill token load handled by each model server.

Both plugins are used with their built-in defaults — no per-deployment tuning is required for this guide's reference setup (Qwen3-32B on H100 80 GB, TP=2). If you deploy a different model or accelerator, the saturation-aware override gate keys off the peakPrefillThroughput of the filter, which is hardware- and model-specific; measure your own with the shared calibration recipe and set it on the filter.

Supported Accelerators and Model Servers​

This guide includes configurations for the following accelerator and model server combinations (set ACCELERATOR_TYPE and MODEL_SERVER accordingly). Each accelerator serves exactly one model:

AcceleratorACCELERATOR_TYPEServed modelvLLMSGLangTensorRT-LLMAtomNotes
NVIDIA GPUgpuQwen/Qwen3-32B✅ validated✅ validated🟡 community—Default. H100 80 GB reference · 2 replicas × TP=2 (4 GPUs) · INFRA_PROVIDER: base, gke
AMD GPUamdQwen/Qwen3-32B✅ validated✅ validated—✅ validatedInstinct MI355X · 2 replicas × TP=2 (4 GPUs)
Intel XPUxpuQwen/Qwen3-0.6B✅ validated🟡 community——vLLM: Data Center GPU Max 1550+ · SGLang: Arc Pro B60 · 2 replicas × 1 XPU via DRA · fp16
Google TPU v5ptpu/v5pQwen/Qwen3-32B✅ validated———GKE only · 2 replicas × 4 chips (2x2x1, TP=4) · nightly runs on Spot capacity
Google TPU v6etpu/v6Qwen/Qwen3-32B🟡 community———GKE only · 2 replicas × 8 chips (2x4, TP=8) · nightly moved to tpu/v5p while v6e capacity is unavailable
Google TPU v7tpu/v7Qwen/Qwen3-32B🟡 community———GKE only · 2 replicas × 4 chips (2x2x1, TP=8)
Google TPU v7 (dynamic slicing)tpu/v7-dynamic-sliceQwen/Qwen3-Coder-480B-A35B-Instruct-FP8🟡 community———GKE dynamic slicing + Kueue · one sub-slice per replica (TPU_SLICE_TOPOLOGY: 2x2x1, 2x2x2); see below
Rebellions NPUnpuopenai/gpt-oss-120b🟡 community———RBLN-CR series · 2 replicas × 1 NPU via DRA
CPUcpumeta-llama/Llama-3.2-3B-Instruct🟡 community———x86 with AMX or AVX512-BF16 (Sapphire Rapids+, GCP C3, AMD Zen 4+) · 2 replicas × 64 cores / 64 GiB (CPUs without AMX/AVX512-BF16, e.g. Cascade/Ice Lake, need --dtype=float32 for the bf16 model)
Iluvatar GPUiluvatardeepseek-ai/DeepSeek-V4-Flash🟡 community———BI-V150 (dual-die) · 1 replica × 4 GPUs
MetaX GPUmetaxdeepseek-ai/DeepSeek-R1-Distill-Llama-70B🟡 community———2 replicas × TP=8 (16 GPUs)
Biren GPUbirenQwen/Qwen3-32B🟡 community———BR166M · 2 replicas × TP=8 (16 GPUs)
Moore Threads GPUmthreadsQwen/Qwen3-32B🟡 community🟡 community——MTT S5000 · vLLM Qwen3-32B TP=8 (1 replica); SGLang colocated DeepSeek-V4-Flash (no PD)

✅ validated: covered by a nightly E2E workflow · 🟡 community: maintained by the hardware vendor or community, not covered by nightly E2E · ❌ not supported: tracked in the linked issue · — no configuration.

The MThreads vLLM overlay is a single-replica TP validation profile: one Qwen/Qwen3-32B vLLM server, tensor parallelism 8, and eight mthreads.com/gpu resources in the pod. It uses the standalone Router/EPP path and the llm-d-hf-token Secret for model download. Set ACCELERATOR_TYPE=mthreads, MODEL_SERVER=vllm, and keep MODEL=Qwen/Qwen3-32B when following the deployment commands. The MThreads profile is not covered by the default H100 calibration value; measure peakPrefillThroughput on the target model and hardware before performance tuning.

The MThreads SGLang overlay colocates prefill and decode for DeepSeek-V4-Flash-0731-FP8-mt (--tp 8 --ep 8) on one 8-GPU node. See Deploying on Moore Threads S5000 (SGLang, colocated, no PD).

Prerequisites​

  • Have the proper client tools installed on your local system to use this guide.

  • Ensure your cluster has enough accelerators for your configuration (default NVIDIA GPU configuration: 2 replicas with tensor parallelism 2, 4 GPUs in total). If your cluster has fewer resources, adjust replicas and --tensor-parallel-size in the model server patch for your environment (e.g., patch for vLLM on GPU).

  • For DRA-based overlays (Intel XPU, Rebellions NPU), install the accelerator's resource driver and verify its DeviceClass before deployment. These manifests use the resource.k8s.io/v1 DRA APIs, which require Kubernetes 1.34+; on 1.32/1.33 the driver may install but these overlays will still fail to apply.

  • Create a HuggingFace token and export it as HF_TOKEN in your shell.

  • (Optional) Set up the monitoring stack if you plan to enable Prometheus monitoring.

Get the guide​

Every command below runs from a local clone of the llm-d repository: the manifests, Helm values, and Kustomize overlays it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Configure the environment​

Set the guide-specific environment variables:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=optimized-baseline
export NAMESPACE=llm-d-optimized-baseline
export MONITORING=false # options: false, true
export MONITORING_VALUES=
export ACCELERATOR_TYPE=gpu # options: gpu, amd, xpu, tpu/v5p, tpu/v6, tpu/v7, tpu/v7-dynamic-slice, npu, cpu, iluvatar, metax, biren, mthreads
export MODEL_SERVER=vllm # options: vllm, sglang, trtllm, atom
export INFRA_PROVIDER=base # options: base, gke, amd-ci
export TPU_SLICE_TOPOLOGY=2x2x1 # options: 2x2x1, 2x2x2
export MODEL=Qwen/Qwen3-32B # set to the Served model for your accelerator (table above)
source ${REPO_ROOT}/guides/env.sh # defines GAIE_VERSION, ROUTER_CHART_VERSION, router chart URLs, and CURL_TEST_IMAGE

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models:

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Installation Instructions​

1. Deploy the llm-d Router​

Prepare the paths to the helm values files for the llm-d router (used in the deployment commands below):

vLLM / SGLang / Atom
# Paths to values files
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"

export ROUTER_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml"
note

The prefix-cache-affinity-filter in router/optimized-baseline.values.yaml defaults to peakPrefillThroughput tuned for the reference setup (Qwen3-32B on H100 80 GB, TP=2). On a different model or accelerator, miscalibrated peakPrefillThroughput can prevent the saturation override from spreading load across pods — measure and set your value with the calibration guide (or check the configuration matrix for pre-measured values).

(Optional) Enable Prometheus monitoring on the llm-d router by defining MONITORING_VALUES (see Collect Metrics):

# only when MONITORING=true:
export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml"

(Rebellions NPU only) Layer the NPU router values, which drop the prefix-cache affinity filter because the NPU overlay serves with prefix caching off:

export ACCELERATOR_VALUES="-f ${REPO_ROOT}/guides/${GUIDE_NAME}/router/npu.rbln.values.yaml"

Deploy the router in one of two modes: Standalone Mode (the default, used by every other guide) or Gateway Mode, which fronts the InferencePool with a Kubernetes Gateway–managed proxy. The Verification section below has matching steps for each mode:

This deploys the llm-d Router in Standalone Mode with an Envoy sidecar (default):

# Assuming base-directory is the root of the llm-d repo
helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  ${ACCELERATOR_VALUES:-} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

To use agentgateway as the sidecar proxy instead of Envoy, see router recipes.

2. Deploy the Model Server​

For model sources, caching, and startup optimization, see the Model Loading and Startup Acceleration operations guide.

The NVIDIA GPU, AMD, CPU, Intel XPU, and TPU v5p/v6/v7 (including dynamic-slice) vLLM overlays drain in-flight requests during rollouts and scale-down: --shutdown-timeout=45, plus a 15s preStop sleep and a 75s termination grace period from the graceful-shutdown component. The Iluvatar, MetaX, Biren, Moore Threads, and Rebellions NPU overlays do not drain yet. To tune these, see Graceful Shutdown & Request Draining.

Apply the Kustomize overlays for your specific backend (INFRA_PROVIDER=gke applies only to accelerators available on GKE: NVIDIA GPU, TPU, and CPU; use base elsewhere):

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}/

(Optional) Deploy the monitoring resources for model servers (see Collect Metrics):

# only when MONITORING=true:
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring

Verification​

1. Get the IP of the Proxy​

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send Test Requests​

Send a completion request from a temporary pod inside the cluster:

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\"}"'

3. Verify prefix-cache aware routing​

A served request only proves the stack is up. To confirm the routing mechanism this guide deploys is engaged, send a burst of requests that share a long prompt prefix, then check that the router kept them on the model server that already caches that prefix.

Send 10 requests that share a ~2k-token prefix:

# 10 requests that share a ~2k-token prefix and differ only in the last sentence
kubectl run prefix-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'P=$(for i in $(seq 1 100); do printf "The llm-d router keeps a conversation on the server that already caches its prefix. "; done)
    for i in $(seq 1 10); do
      curl -sS -o /dev/null -w "request ${i}: HTTP %{http_code}\n" -X POST "http://${IP}/v1/completions" \
        -H "Content-Type: application/json" \
        -d "{\"model\": \"${MODEL}\", \"prompt\": \"${P} Question ${i}: what is llm-d?\", \"max_tokens\": 16}"
    done'

Read the prefix-cache and request counters of every model server pod:

# Prefix-cache and request counters of every model server pod, read
# through the Kubernetes API server proxy (no port-forward needed)
for pod in $(kubectl get pods -n ${NAMESPACE} -l llm-d.ai/guide=${GUIDE_NAME} -o jsonpath='{.items[*].metadata.name}'); do
  echo "== ${pod}"
  kubectl get --raw "/api/v1/namespaces/${NAMESPACE}/pods/${pod}:8000/proxy/metrics" \
    | grep -E '^(vllm:prefix_cache_(hits|queries)_total|vllm:request_success_total|sglang:(cache_hit_rate|cached_tokens_total|num_requests_total))' || true
done

What to expect:

One pod reports most of the 10 requests in vllm:request_success_total, and its vllm:prefix_cache_hits_total is a large fraction of vllm:prefix_cache_queries_total (every request after the first reuses the cached prefix). The other pods show few or none of these requests.

If the requests are spread evenly and hit rates stay near zero, prefix-cache affinity is not taking effect; see Common failure modes. Performance benchmarks for this configuration are not part of this guide: they live with the model-specific guides.

Observability & Troubleshooting​

Once metric collection is enabled, use the signals below to operate the optimized baseline under sustained load in production or benchmark scenarios (note that the small number of requests in Verification is only a smoke check and does not produce steady-state load or saturation signals):

View the imported Grafana dashboards or query Prometheus directly.

This section covers the metrics that matter for this path and how to read them; full metric definitions live in the metric reference and ready-to-run queries in the PromQL reference.

This path is defined by its two routing objectives: prefix-cache affinity (route to endpoints that already hold the prompt prefix) and load-aware balancing (spread work by token load), with a saturation override that trades cache locality for spread once endpoints get hot. Most issues show up as those two objectives pulling against each other, so watch load balance and cache hit rate together rather than either one alone.

Key metrics for this path​

SignalWhy it matters for the optimized baselineWhere to look
Per-pod load (llm_d_epp_request_total, vllm:num_requests_running)The load-aware scorer should keep QPS and active requests roughly even across pods. A persistently hot pod next to idle ones means balancing is not taking effectPromQL → Routing & Load Balancing
Prefix cache hit rate (vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total)The prefix-affinity filter is only helping if hit rate stays high. A falling ratio means requests are not landing on sticky endpointsPromQL → Prefix Caching
Per-pod KV cache utilization and queue depth (vllm:kv_cache_usage_perc, vllm:num_requests_waiting)These drive the saturation-aware override. If one pod sits near saturation while others are cold, the override is either not firing or mis-tunedPromQL → Basic Model Serving
Routing decision latency (llm_d_epp_plugin_duration_seconds)Rising scheduler latency with healthy model servers localizes the problem to the routing layer, not the podsPromQL → Routing & Load Balancing
TTFT and ITL (vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds)The user-facing SLO signals this path is tuned to protect. Regressions here are the trigger to inspect the balance/cache split aboveMetrics → vLLM

Common failure modes​

  • Uneven load across pods (some hot, some idle) — the load-aware scorer or the saturation override is not spreading work. On non-default hardware this usually means peakPrefillThroughput is miscalibrated, so the override never gates in; measure it with the calibration recipe and set it on the filter.
  • Low prefix cache hit rate — either the prompt mix is not prefix-sticky, or the saturation override is spreading so aggressively that it defeats affinity. Compare hit rate against per-pod saturation; if pods are cold but hit rate is still low, the issue is the prompt pattern, not the override.
  • TTFT/ITL regression with balanced load and healthy cache — look at routing decision latency (llm_d_epp_plugin_duration_seconds) and model-server queue depth before touching the routing config; the bottleneck is likely the model servers, not the scheduler.

For alert rules covering these signals, see Alerting.

Cleanup​

To remove the deployed components:

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}
helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring --ignore-not-found=true
kubectl delete namespace ${NAMESPACE}