Version: dev

Accelerator

Provider

Engine

community

Omni Model Serving

Overview

This guide serves models whose output is not only text (speech, images, or both) behind the llm-d Router, on vLLM-Omni (and, for the image tasks, SGLang). The TASK variable picks what the pool serves:

`TASK`	Model	Endpoints	Engines	Router scorer
`omni` (default)	`Qwen/Qwen3-Omni-30B-A3B-Instruct`	`/v1/chat/completions` (text and spoken answers)	vLLM-Omni	`queue-scorer`
`text-to-image`	`Qwen/Qwen-Image` (GPU); `black-forest-labs/FLUX.2-klein-4B` (XPU)	`/v1/images/generations`	vLLM-Omni, SGLang	`active-request-scorer`
`image-to-image`	`Qwen/Qwen-Image-Edit-2511` (GPU); `black-forest-labs/FLUX.2-klein-4B` (XPU)	`/v1/images/edits` (`multipart/form-data`)	vLLM-Omni, SGLang	`active-request-scorer`
`text-to-speech`	`Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`	`/v1/audio/speech`	vLLM-Omni	`active-request-scorer`

The tasks fall into two shapes:

An omni model: one checkpoint, many endpoints. Qwen3-Omni takes text, images, audio, or video in and answers with text and speech from a multi-stage pipeline (thinker → talker → code2wav). One InferencePool fronts every OpenAI endpoint the checkpoint serves. The router's request parser understands the chat, speech, and image-generation request bodies, so a single router and pool serve all of them; when the pool shares a Kubernetes Gateway with other pools, exact-path HTTPRoute rules send the modality endpoints to it (see Split endpoints across pools).
A single-endpoint generator: one model per pool. A diffusion model (Qwen-Image, Qwen-Image-Edit, or, on Intel XPU, FLUX.2-klein-4B) or a dedicated TTS model answers one endpoint. Each TASK deploys its own pool: 3 single-GPU replicas, or, for Intel XPU's text-to-image and image-to-image, 2 single-XPU replicas.

Either way the router routes on load. Diffusion models generate by iterative denoising rather than autoregressive token generation, and the Qwen3-Omni pipeline runs with prefix caching off, so prefix-cache and token-load scorers have no signal. The omni router picks the pod with the shortest queue (queue-scorer). The single-endpoint routers track how many requests they have dispatched to each pod and not yet seen complete, and prefer the least busy one (active-request-scorer); they need no model-server metrics, which SGLang's diffusion backend does not publish. Image requests log a token-estimation error (unsupported request body type) because the token producer cannot parse image bodies; this does not disable active-request-scorer, which tracks in-flight request counts independently of token estimation. The Qwen3-TTS talker is autoregressive, so prefix-aware TTS routing is possible in principle and is tracked in llm-d-router#2483.

Why vLLM-Omni

Standard vLLM generates text output only: even when a model accepts images or audio (for example Qwen3-VL, see Multimodal Model Serving), the response is text. Models whose output is audio or images need vLLM-Omni, which extends vLLM with multi-stage pipelines and the endpoints that return audio or images. Which endpoints a vLLM-Omni pod answers depends on the checkpoint: the TASK table above lists them for the models this guide deploys.

Engine	Input	Output	Example models
vLLM	Text, images, audio, video	Text only	Qwen3-VL, Llama
vLLM-Omni	Text, images, audio, video	Text, audio, images	Qwen3-Omni, Qwen3-TTS, Qwen-Image, FLUX
SGLang (`sglang serve` → `sglang.multimodal_gen`)	Text, images	Images	Qwen-Image, Qwen-Image-Edit

Architecture

Client → Proxy (Envoy sidecar) → Router (EPP) → pod of the TASK's pool
  TASK=omni            /v1/chat/completions (and the checkpoint's other endpoints)   queue-scorer
  TASK=text-to-image   /v1/images/generations                                       active-request-scorer
  TASK=image-to-image  /v1/images/edits                                             active-request-scorer
  TASK=text-to-speech  /v1/audio/speech                                             active-request-scorer

The model server overlay creates one Deployment whose pods carry the llm-d.ai/guide=omni-serving label that the InferencePool selects. Every pod serves the same model and endpoints, so the router needs no filter: each request goes to the least loaded pod. That spreads long-running audio and image generations across the pool instead of round-robining them.

Supported Accelerators and Model Servers

This guide includes configurations for the following accelerator and model server combinations:

Accelerator	`ACCELERATOR_TYPE`	Served model	vLLM-Omni	SGLang	Notes
NVIDIA GPU	`gpu`	`Qwen/Qwen3-Omni-30B-A3B-Instruct`	🟡 community	🟡 community	Served model is `TASK=omni`'s (others in the Overview table) · H100 80 GB reference · `omni`: 1 replica × 2 GPUs · `text-to-image`, `image-to-image`, `text-to-speech`: 3 replicas × 1 GPU · SGLang: `text-to-image` and `image-to-image` only
Intel XPU	`xpu`	`black-forest-labs/FLUX.2-klein-4B`	—	🟡 community	`text-to-image`, `image-to-image` · Arc Pro B60 · 2 replicas × 1 XPU via DRA · memory offload (`--performance-mode=memory`)

✅ validated: covered by a nightly E2E workflow · 🟡 community: maintained by the hardware vendor or community, not covered by nightly E2E · ❌ not supported: tracked in the linked issue · — no configuration.

Prerequisites

Have the proper client tools installed on your local system to use this guide.
Ensure your cluster has enough accelerators for your configuration: 2 NVIDIA GPUs with 80 GB of HBM (e.g. H100) for omni (1 replica: thinker on the first GPU, talker and code2wav on the second), 3 for the other GPU tasks (3 replicas × 1 GPU), or 2 Intel XPUs (Arc Pro B60) for Intel XPU's text-to-image or image-to-image (2 replicas × 1 XPU, 64 GiB host memory requested per replica). Qwen-Image is a ~20B MMDiT plus a ~8B text encoder, about 57 GB in bf16, and fits one 80 GB GPU at full precision; Qwen3-TTS is small, so the H100 is a default there, not a requirement.
For the Intel XPU overlay, install the Intel GPU DRA driver and verify the gpu.intel.com DeviceClass is available; it requires the resource.k8s.io/v1 API (Kubernetes 1.34+).
Create a HuggingFace token and export it as HF_TOKEN in your shell.
(Optional) Install the monitoring stack if you plan to enable Prometheus monitoring.

Get the guide

Every command below runs from a local clone of the llm-d repository: the manifests, Helm values, and Kustomize overlays it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Configure the environment

Set the guide-specific environment variables. Set TASK to what the pool serves and MODEL to that task's model (listed in the comment). MODEL_SERVER=sglang applies to text-to-image and image-to-image only:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=omni-serving
export NAMESPACE=llm-d-omni-serving
export MONITORING=false # options: false, true
export MONITORING_VALUES=
export TASK=omni # options: omni, text-to-image, image-to-image, text-to-speech
export ACCELERATOR_TYPE=gpu # options: gpu, xpu
export MODEL_SERVER=vllmomni # options: vllmomni, sglang
export INFRA_PROVIDER=base # options: base, gke
export MODEL=Qwen/Qwen3-Omni-30B-A3B-Instruct # the model the TASK's overlay serves: omni Qwen/Qwen3-Omni-30B-A3B-Instruct, text-to-image Qwen/Qwen-Image (GPU) or black-forest-labs/FLUX.2-klein-4B (XPU SGLang), image-to-image Qwen/Qwen-Image-Edit-2511 (GPU) or black-forest-labs/FLUX.2-klein-4B (XPU SGLang), text-to-speech Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice
source ${REPO_ROOT}/guides/env.sh # defines GAIE_VERSION, ROUTER_CHART_VERSION, router chart URLs, and CURL_TEST_IMAGE

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models:

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Installation Instructions

1. Deploy the llm-d Router

Prepare the paths to the helm values files for the llm-d router (used in the deployment command below). Select your task (TASK), which layers its router values on the shared base:

Omni (Default)
Text-to-Image
Image-to-Image
Text-to-Speech

The omni router values schedule every request with the queue-scorer and max-score-picker only.

# only when TASK=omni:
# Paths to values files
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/omni.values.yaml"

The text-to-image router values schedule on the active-request-scorer and max-score-picker for both the GPU and XPU overlays, and turn off the router's default metrics scraping (injectDefaults: false), because the diffusion backend publishes no metrics.

# only when TASK=text-to-image:
# Paths to values files
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/text-to-image.values.yaml"

The image-to-image router values are the text-to-image configuration on a pool of their own: active-request-scorer, max-score-picker, no metrics scraping.

# only when TASK=image-to-image:
# Paths to values files
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/image-to-image.values.yaml"

The text-to-speech router values are the same configuration on a pool of their own: active-request-scorer, max-score-picker, no metrics scraping.

# only when TASK=text-to-speech:
# Paths to values files
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/text-to-speech.values.yaml"

note

The router parses /v1/audio/speech requests since llm-d-router#2484, which landed after the v0.10.0 release. The default chart version (${ROUTER_CHART_VERSION} = v0) deploys the main-tagged router image, which includes it.

(Optional) Enable Prometheus monitoring on the llm-d router by defining the helm values file (requires installing the monitoring stack mentioned in Prerequisites):

# only when MONITORING=true:
export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml"

Deploy the router in Standalone Mode, with an Envoy sidecar in front of the router. Keep the release name ${GUIDE_NAME}: the verification steps look up the ${GUIDE_NAME}-epp Service by that name (and httproutes.yaml references the ${GUIDE_NAME} InferencePool). A single pool needs no HTTPRoute: the standalone proxy sends every path to the router. To front the router with a Kubernetes Gateway instead, see Gateway Mode in the Optimized Baseline and, for omni, Split endpoints across pools below.

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

2. Deploy the Model Server

For model sources, caching, and startup optimization, see the Model Loading and Startup Acceleration operations guide.

Apply the Kustomize overlay for your task. The overlays live at modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${TASK}/${INFRA_PROVIDER}/; the gke variants add the disable-gke-nccl-tuner-patch component on top of base.

Omni (Default)
Text-to-Image
Image-to-Image
Text-to-Speech

Runs vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni on the vllm/vllm-omni image (the shared gpu-vllm-omni image component), 1 replica × 2 GPUs:

# only when TASK=omni:
kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/omni/${INFRA_PROVIDER}/

To serve another vLLM-Omni checkpoint, change the model in patch-vllm-omni.yaml and size the GPU count to its deploy config; set MODEL to match.

Serves an image-generation checkpoint, one model server per accelerator:

NVIDIA GPU
Intel XPU

Qwen/Qwen-Image on 3 replicas × 1 GPU, with vLLM-Omni (patch-vllm-omni.yaml) or SGLang (patch-sglang.yaml).

Select ACCELERATOR_TYPE=xpu, MODEL_SERVER=sglang, TASK=text-to-image, INFRA_PROVIDER=base, and MODEL=black-forest-labs/FLUX.2-klein-4B. SGLang serves it on 2 replicas × 1 Arc Pro B60 with --performance-mode=memory (patch-sglang.yaml); this is community support, not nightly E2E coverage. Qwen/Qwen-Image does not fit a single B60 (~57 GB bf16), and a two-B60 --tp-size=2 run fails on the published v0.10.0 image's pinned SGLang, which predates an upstream XPU layer-norm fix (sgl-project/sglang#36825).

# only when TASK=text-to-image:
kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/text-to-image/${INFRA_PROVIDER}/

Serves an edit-capable checkpoint, one model server per accelerator. An edit-capable checkpoint is required: Qwen/Qwen-Image reports task type T2I and rejects edits (input_reference is not supported for T2I models), and Qwen/Qwen-Image-Edit-2511 reports I2I and rejects /v1/images/generations.

NVIDIA GPU
Intel XPU

Qwen/Qwen-Image-Edit-2511 on 3 replicas × 1 GPU, with vLLM-Omni (patch-vllm-omni-edit.yaml) or SGLang (patch-sglang.yaml).

Select ACCELERATOR_TYPE=xpu, MODEL_SERVER=sglang, TASK=image-to-image, INFRA_PROVIDER=base, and MODEL=black-forest-labs/FLUX.2-klein-4B. SGLang serves it on 2 replicas × 1 Arc Pro B60 with --performance-mode=memory (patch-sglang.yaml), the same overlay component text-to-image uses; this is community support, not nightly E2E coverage. FLUX.2-klein-4B supports both generation and reference-image editing, so the same checkpoint serves text-to-image and image-to-image. Qwen/Qwen-Image-Edit-2511 does not fit a single B60 (~57 GB bf16), even with --performance-mode=memory.

# only when TASK=image-to-image:
kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/image-to-image/${INFRA_PROVIDER}/

Serves Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice (a ~1.7B autoregressive talker plus a 12 Hz codec decoder) on vLLM-Omni, 3 replicas × 1 GPU (patch-vllm-omni-tts.yaml):

# only when TASK=text-to-speech:
kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/text-to-speech/${INFRA_PROVIDER}/

note

Deploy one task and one engine at a time per namespace. Every overlay carries the same llm-d.ai/guide=omni-serving label, which is what the InferencePool selects on, so two overlays in one namespace would join one pool.

3. Enable Monitoring (optional)

(Optional) Deploy the monitoring resources for model servers (requires installing the monitoring stack mentioned in Prerequisites and MONITORING=true when deploying the router). SGLang's diffusion backend publishes no metrics, so with MODEL_SERVER=sglang only the router's metrics are scraped:

# only when MONITORING=true and MODEL_SERVER=vllmomni:
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring

Split endpoints across pools (Gateway Mode)

This section applies to TASK=omni. Path-based routing across pools is the one part of this path that needs a Kubernetes Gateway: a standalone router fronts a single pool. A typical setup puts a text model and a speech or image model behind one Gateway, sending the modality endpoints to the vLLM-Omni pool and everything else to the text pool:

/v1/audio/speech         (Exact)                  → omni-serving pool
/v1/audio/transcriptions (Exact)                  → omni-serving pool
/v1/images/generations   (Exact)                  → omni-serving pool
/v1/chat/completions     (PathPrefix / catch-all) → text pool

Exact-path matches take precedence over PathPrefix matches per the Gateway API specification, so the modality requests reach the vLLM-Omni pool regardless of the catch-all. To set it up:

Deploy the shared llm-d-inference-gateway and the text pool with Gateway Mode in the Optimized Baseline. Its router chart creates the PathPrefix: / catch-all HTTPRoute.
Install this guide's router with the Gateway chart (${ROUTER_GATEWAY_CHART}, --set provider.name=<your gateway provider>) instead of the standalone chart, leaving out --set httpRoute.create=true so it does not add a second catch-all, and deploy the model server as above.
Remove the path matches your model does not serve from httproutes.yaml, then apply it:

kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/${GUIDE_NAME}/httproutes.yaml

note

Path matching splits traffic by endpoint, not by model. It fits TTS and diffusion checkpoints, whose endpoints differ from the text pool's. Qwen3-Omni returns spoken answers through /v1/chat/completions, the path the text pool owns, so behind a shared Gateway give it its own endpoint (a separate namespace and Gateway listener, with the Gateway chart's own HTTPRoute) rather than a path rule. The same applies to two models that serve the same endpoint, such as two TTS models on /v1/audio/speech.

Verification

1. Get the IP of the Proxy

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send Test Requests

Send a request to your task's endpoint and check that a valid response comes back:

Omni (Default)
Text-to-Image
Image-to-Image
Text-to-Speech

Send a text chat request:

# only when TASK=omni:
# Text in, text out
kubectl run chat-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/chat/completions" -H "Content-Type: application/json" \
    -d "{\"model\": \"${MODEL}\", \"messages\": [{\"role\": \"user\", \"content\": \"What can you do, in one sentence?\"}], \"modalities\": [\"text\"], \"max_tokens\": 64}" \
    | jq -r ".choices[0].message.content"'

The text request only shows that the thinker stage answers. Request text and audio output to exercise the whole pipeline:

# only when TASK=omni:
# Text in, text and speech out: the thinker writes the answer, the
# talker and code2wav stages turn it into audio (base64 WAV)
kubectl run chat-audio-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/chat/completions" -H "Content-Type: application/json" \
    -d "{\"model\": \"${MODEL}\", \"messages\": [{\"role\": \"user\", \"content\": \"Say hello to llm-d in one short sentence.\"}], \"modalities\": [\"text\", \"audio\"]}" \
    | jq -r ".choices[].message | (if .content then \"text: \(.content)\" else empty end), (if .audio then \"audio: \(.audio.data | length) base64 characters\" else empty end)"'

Expect two lines: text: with the answer, and audio: with the size of the base64-encoded WAV the talker and code2wav stages generated. A missing audio: line means the speech stages did not run: check the model server logs for the talker stage.

Generate an image (response_format: b64_json returns the image inline):

# only when TASK=text-to-image:
# Text in, image out: report the format of the returned image
# (PNG from vLLM-Omni, JPEG from SGLang) and its size
kubectl run t2i-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/images/generations" -H "Content-Type: application/json" \
    -d "{\"model\": \"${MODEL}\", \"prompt\": \"a red apple on a wooden table\", \"size\": \"512x512\", \"n\": 1, \"response_format\": \"b64_json\"}" \
    | jq -r "if .data[0].b64_json then (.data[0].b64_json | (if startswith(\"iVBORw0KGgo\") then \"PNG\" elif startswith(\"/9j/\") then \"JPEG\" else \"unexpected\" end) + \" image, \(length) base64 characters\") else . end"'

Expect PNG image, … base64 characters from vLLM-Omni or JPEG image, … from SGLang. Anything else prints the full response body, which carries the server's error. SGLang's response also carries peak_memory_mb, inference_time_s, and file_path (a path inside the pod); neither engine returns token usage.

Edit an image. The request is multipart/form-data: an input image (a 64×64 red PNG embedded in the command) and an instruction:

# only when TASK=image-to-image:
# Image and instruction in (multipart/form-data), edited image out.
# The input is a 64x64 red PNG embedded here, so no download is needed
kubectl run i2i-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'echo "iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAAS0lEQVR42u3PQQkAAAgAsetfWiP4FgYrsKZeS0BAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEDgsqnc8OJg6Ln3AAAAAElFTkSuQmCC" | base64 -d > /tmp/input.png
    curl -sS -X POST "http://${IP}/v1/images/edits" \
      -F "image=@/tmp/input.png" -F "prompt=turn the red square into a blue circle" \
      -F "model=${MODEL}" -F "size=512x512" -F "n=1" \
    | jq -r "if .data[0].b64_json then (.data[0].b64_json | (if startswith(\"iVBORw0KGgo\") then \"PNG\" elif startswith(\"/9j/\") then \"JPEG\" else \"unexpected\" end) + \" image, \(length) base64 characters\") else . end"'

Expect PNG image, … base64 characters from vLLM-Omni or JPEG image, … from SGLang. A 512×512 edit takes about 15 seconds on an H100, at about 48 GB of GPU memory.

Synthesize speech. Unlike the JSON image endpoints, a successful response body is the binary audio itself:

# only when TASK=text-to-speech:
# Text in, speech out: the response body is the WAV file itself;
# a valid one starts with "RIFF"
kubectl run tts-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -o /tmp/speech.wav -w "HTTP %{http_code}, %{size_download} bytes, %{content_type}\n" \
      -X POST "http://${IP}/v1/audio/speech" -H "Content-Type: application/json" \
      -d "{\"model\": \"${MODEL}\", \"input\": \"llm-d routes speech requests to the least busy pod.\", \"voice\": \"vivian\", \"response_format\": \"wav\"}"
    echo "header: $(head -c 4 /tmp/speech.wav)"'

Expect HTTP 200 with a non-zero size and an audio content type, and header: RIFF, the start of a valid WAV file.

The endpoint also streams through the router: "stream_format": "sse" returns speech.audio.delta events with base64 audio chunks and a final speech.audio.done event carrying token usage, and "stream_format": "audio" returns raw PCM chunks as they are generated, with time-to-first-byte in the tens of milliseconds. vLLM-Omni v0.28.0 and later also report usage for non-streamed responses in x-vllm-omni-input-tokens, x-vllm-omni-output-tokens, and x-vllm-omni-total-tokens headers (see the speech API response format); the overlays pin v0.30.0, which reports them.

3. Check where the requests landed

Read the model server logs to confirm that the router scheduled each request onto a pod of the pool:

# The model servers' logs: every request above, by endpoint path,
# on the pods of the pool (each line prefixed with its pod name)
kubectl logs -n ${NAMESPACE} -l llm-d.ai/guide=${GUIDE_NAME} -c modelserver --tail=-1 --prefix \
  | grep -E 'POST /v1/(chat/completions|images/generations|images/edits|audio/speech)' || true

Every request above appears on a pod of the pool (each line prefixed with its pod name) under its endpoint path: the router parsed the request body and scheduled it onto the pool, whatever the endpoint. Send several requests concurrently and they spread across the pods by load. The logs are the evidence that works for every engine; SGLang's diffusion backend publishes no metrics to read the same from.

Cleanup

To remove the deployed components, delete your task's model server:

Omni (Default)
Text-to-Image
Image-to-Image
Text-to-Speech

# only when TASK=omni:
kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/omni/${INFRA_PROVIDER}

If you applied the exact-path rules in Gateway Mode, also delete them:

kubectl delete -n ${NAMESPACE} -f ${REPO_ROOT}/guides/${GUIDE_NAME}/httproutes.yaml --ignore-not-found=true

# only when TASK=text-to-image:
kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/text-to-image/${INFRA_PROVIDER}

# only when TASK=image-to-image:
kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/image-to-image/${INFRA_PROVIDER}

# only when TASK=text-to-speech:
kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/text-to-speech/${INFRA_PROVIDER}

Then remove the router, the monitoring resources, and the namespace:

helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring --ignore-not-found=true

kubectl delete namespace ${NAMESPACE}

Overview​

Why vLLM-Omni​

Architecture​

Supported Accelerators and Model Servers​

Prerequisites​

Get the guide​

Configure the environment​

Installation Instructions​

1. Deploy the llm-d Router​

2. Deploy the Model Server​

3. Enable Monitoring (optional)​

Split endpoints across pools (Gateway Mode)​

Verification​

1. Get the IP of the Proxy​

2. Send Test Requests​

3. Check where the requests landed​

Cleanup​

Overview

Why vLLM-Omni

Architecture

Supported Accelerators and Model Servers

Prerequisites

Get the guide

Configure the environment

Installation Instructions

1. Deploy the llm-d Router

2. Deploy the Model Server

3. Enable Monitoring (optional)

Split endpoints across pools (Gateway Mode)

Verification

1. Get the IP of the Proxy

2. Send Test Requests

3. Check where the requests landed

Cleanup