Version: dev

Accelerator

Provider

Engine

validated

Serve the Stateful OpenAI Responses API (Experimental)

Overview

This guide serves the OpenAI-compatible Responses API (/v1/responses) with the state it implies: multi-turn conversations continued by previous_response_id, webhook tool loops, and WebSocket streaming. It is the well-lit path for agentic clients and coding harnesses that speak the Responses API, such as the OpenAI Codex CLI.

The llm-d Router and vLLM are stateless: every request has to carry the whole conversation. vLLM Agentic API (vllm/agentic-api) sits in front of the router, persists each response in PostgreSQL, and rehydrates the conversation when a request names a previous_response_id, so the client sends only the new turn. Each inference round it sends on to the router is a stateless request that the router places with prefix-cache and load-aware scoring, as in the Optimized Baseline.

The reference deployment reuses the Optimized Baseline router configuration and model servers (on NVIDIA GPU, two Qwen/Qwen3-32B replicas with tensor parallelism 2, started with vLLM's tool-calling flags) and adds:

Added	What it is
`Deployment/agentic-api` + `Service/agentic-api:9000`	`vllm/agentic-api`, configured to use the router as its inference backend
`Deployment/agentic-api-postgres` + PVC	PostgreSQL 17, the state store behind `previous_response_id` continuation

The router runs in Standalone Mode: the router Service speaks plain HTTP on :80, and agentic-api dials it by DNS. To layer agentic-api onto a router another guide already deployed, see Add agentic-api to Another Guide.

Supported Accelerators and Model Servers

This guide includes configurations for the following accelerator and model server combinations (set ACCELERATOR_TYPE and MODEL_SERVER accordingly):

Accelerator	`ACCELERATOR_TYPE`	Served model	vLLM	Notes
NVIDIA GPU	`gpu`	`Qwen/Qwen3-32B`	✅ validated	H100 80 GB reference · the Optimized Baseline model servers, 2 replicas × TP=2 (4 GPUs) · `INFRA_PROVIDER`: `base`, `gke`

✅ validated: covered by a nightly E2E workflow · 🟡 community: maintained by the hardware vendor or community, not covered by nightly E2E · ❌ not supported: tracked in the linked issue · — no configuration.

agentic-api owns conversation state and delegates each inference round to vLLM's own stateless Responses API, so the model server must expose /v1/responses (ARCHITECTURE.md, ADR-01 §1.1). SGLang and TensorRT-LLM model servers are therefore out of scope. Tool calls also need vLLM started with --enable-auto-tool-choice and the tool-call and reasoning parsers for its model; the Optimized Baseline NVIDIA GPU vLLM overlays are the ones that set them.

Prerequisites

Have the proper client tools installed on your local system to use this guide, plus envsubst, openssl and Python 3 (standard library only).
Ensure your cluster has enough accelerators for your configuration (default NVIDIA GPU configuration: 2 replicas with tensor parallelism 2, 4 GPUs in total).
Ensure your cluster has a default StorageClass for the PostgreSQL PVC (1 GiB, ReadWriteOnce).
Create a HuggingFace token and export it as HF_TOKEN in your shell.

Get the guide

Every command below runs from a local clone of the llm-d repository: the manifests, Helm values, and Kustomize overlays it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Configure the environment

Set the guide-specific environment variables:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=agentic-api
export NAMESPACE=llm-d-agentic-api
export ACCELERATOR_TYPE=gpu # options: gpu
export MODEL_SERVER=vllm # options: vllm
export INFRA_PROVIDER=base # options: base, gke
export MODEL=Qwen/Qwen3-32B # the model the overlays serve (table above)
export LLM_API_BASE=${GUIDE_NAME}-epp.${NAMESPACE}.svc.cluster.local:80
source ${REPO_ROOT}/guides/env.sh # defines GAIE_VERSION, ROUTER_CHART_VERSION, router chart URLs, and CURL_TEST_IMAGE

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models:

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Create the PostgreSQL credentials. The password is generated locally and only ever stored in the Secret; reruns reuse the existing Secret so it stays in sync with an already-initialized PVC:

# The password is generated here and never stored outside the Secret. The
# database-url uses the Service's short name, so the Secret is not tied to
# any particular namespace. Reuse the existing Secret on reruns so its
# password stays in sync with an already-initialized PostgreSQL PVC.
if ! kubectl get secret agentic-api-postgres -n ${NAMESPACE} >/dev/null 2>&1; then
  PGPASS=$(openssl rand -hex 16)
  kubectl create secret generic agentic-api-postgres -n ${NAMESPACE} \
    --from-literal=password="${PGPASS}" \
    --from-literal=database-url="postgres://postgres:${PGPASS}@agentic-api-postgres:5432/agentic_api"
fi

Installation Instructions

1. Deploy the llm-d Router

Prepare the paths to the helm values files for the llm-d router. This guide uses the Optimized Baseline router configuration, which selects model server pods labeled llm-d.ai/guide=optimized-baseline:

# Paths to values files: the Optimized Baseline router configuration,
# which selects the model servers deployed below
export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/optimized-baseline/router/optimized-baseline.values.yaml"

Deploy the router in Standalone Mode, with an Envoy sidecar in front of the router. Its helm release name, ${GUIDE_NAME}, names the Service/agentic-api-epp that LLM_API_BASE points at:

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  -f ${ROUTER_VALUES} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

2. Deploy the Model Server

For model sources, caching, and startup optimization, see the Model Loading and Startup Acceleration operations guide.

Apply the Kustomize overlay for your infrastructure provider. Each overlay is the Optimized Baseline model server, which starts vLLM with --enable-auto-tool-choice, --tool-call-parser=hermes and --reasoning-parser=qwen3:

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}/

3. Deploy PostgreSQL

kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/${GUIDE_NAME}/manifests/postgres.yaml
kubectl rollout status -n ${NAMESPACE} deployment/agentic-api-postgres --timeout=120s

4. Deploy agentic-api

The manifest carries a ${LLM_API_BASE} placeholder for the router address, so it is rendered with kubectl kustomize and piped through envsubst:

kubectl kustomize ${REPO_ROOT}/guides/${GUIDE_NAME}/manifests/base \
  | envsubst '${LLM_API_BASE}' \
  | kubectl apply -n ${NAMESPACE} -f -
kubectl rollout status -n ${NAMESPACE} deployment/agentic-api --timeout=120s

5. Wait for the Model Servers

# The model servers become Available once vLLM has loaded the model
kubectl wait -n ${NAMESPACE} --for=condition=Available deployment \
  -l llm-d.ai/guide=optimized-baseline --timeout=30m

Verification

1. Reach agentic-api

kubectl port-forward --address 127.0.0.1 -n ${NAMESPACE} svc/agentic-api 9000:9000 &
PF_PID=$!
sleep 3
kill -0 ${PF_PID}
export AGENTIC_API_BASE_URL=http://127.0.0.1:9000

2. Send a Test Request

curl -sS -X POST ${AGENTIC_API_BASE_URL}/v1/responses \
  -H "Content-Type: application/json" \
  -d "{\"model\": \"${MODEL}\", \"input\": \"How are you today?\"}"

The response is a response object with an id starting with resp_; pass that id as previous_response_id on the next request to continue the conversation without resending it.

3. Verify the Mechanism

A served request only proves the stack is up. verify.py checks that agentic-api keeps state and drives tool loops:

python3 ${REPO_ROOT}/guides/${GUIDE_NAME}/verify.py --base-url ${AGENTIC_API_BASE_URL}
kill ${PF_PID:-} 2>/dev/null || true

It needs only the Python 3 standard library, discovers the served model from /v1/models, and runs four end-to-end tests. All four are entirely client-side, including the webhook receiver in [3/4], so a port-forward is sufficient: nothing in the cluster dials back to your machine.

Health & model discovery [1/4]: /health, /ready and GET /v1/models.
Stateful HTTP /v1/responses [2/4]: stores a secret code with store: true, then sends a follow-up carrying only previous_response_id and no client-side history, verifying that agentic-api rehydrated the prior turn from PostgreSQL.
Webhook mode & stateful tool loop [3/4]: starts a local webhook listener, sends a request with a function tool, dispatches the resulting function_call to the listener, and returns the function_call_output via previous_response_id. This is the test that fails if the model server lacks the tool-calling flags, and it says so.
WebSocket mode [4/4]: upgrades an RFC 6455 connection to ws://<endpoint>/v1/responses, sends a response.create frame and verifies streaming completion (pass --skip-websocket to explicitly skip this check when testing through an HTTP-only proxy).

These steps run nightly on GKE with one model server replica: nightly-e2e-agentic-api-gke-acc-gpu-vllm-x.yaml deploys this guide from its guide.yaml with scripts/nightly-deploy-gke.sh and then runs this section.

Add agentic-api to Another Guide

agentic-api needs only a router Service to send inference to, so it also layers onto a guide you already run, such as P/D Disaggregation or Wide Expert Parallelism, provided its model server is vLLM. The other guide stays as its own README leaves it.

After configuring the environment, point this guide at the other guide's namespace, router and model. The router Service is named after that guide's helm release, its GUIDE_NAME:

export NAMESPACE=llm-d-pd-disaggregation
export LLM_API_BASE=pd-disaggregation-epp.${NAMESPACE}.svc.cluster.local:80
export MODEL=openai/gpt-oss-120b

Then create the PostgreSQL credentials and run installation steps 3 and 4 and the verification; skip steps 1, 2 and 5, which the other guide covers. To remove it, run only the agentic-api and PostgreSQL cleanup steps.

The other guide's vLLM must be started with --enable-auto-tool-choice and the parsers for its model, or verification test [3/4] fails. These are CLI flags with no environment-variable equivalent, so add them to the model server overlay before you deploy that guide, next to the other vllm serve args. Where the overlay runs vLLM from a command: ["/bin/bash", "-c"] script, they go on the vllm serve line inside the script. These overlays already set them:

Model	Flags	Already set by
`openai/gpt-oss-120b`	`--enable-auto-tool-choice` `--tool-call-parser=openai` `--reasoning-parser=openai_gptoss`	`pd-disaggregation` (every `gpu/vllm` overlay) and `p2p-kv-cache-sharing` (`gpu/vllm`); `optimized-baseline` (NPU)
`Qwen/Qwen3-32B`	`--enable-auto-tool-choice` `--tool-call-parser=hermes` `--reasoning-parser=qwen3`	`optimized-baseline` and `precise-prefix-cache-routing` (`gpu/vllm`), and `pd-disaggregation` (`tpu/v6`). Missing on `tiered-prefix-cache`
`Qwen/Qwen3-0.6B`	`--enable-auto-tool-choice` `--tool-call-parser=hermes` `--reasoning-parser=qwen3`	`p2p-kv-cache-sharing` (`xpu/vllm`)
`Qwen/Qwen3.5-397B-A17B-FP8`	`--enable-auto-tool-choice` `--tool-call-parser=qwen3_xml` `--reasoning-parser=qwen3`	`pd-disaggregation` (`tpu/v7`). Its chat template emits XML-style tool calls, so `hermes` does not apply
`Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8`	`--enable-auto-tool-choice` `--tool-call-parser=qwen3_xml`	`pd-disaggregation` (`tpu/v7-dynamic-slice`). No reasoning parser: the model emits no reasoning blocks
`deepseek-ai/DeepSeek-R1-0528`	`--enable-auto-tool-choice` `--tool-call-parser=deepseek_v3` `--chat-template /vllm-workspace/examples/tool_chat_template_deepseekr1.jinja`	`wide-ep`. The model's bundled chat template never renders the `tools` list, so vLLM's replacement is required; no reasoning parser, since vLLM lists `deepseek_r1` reasoning as not supporting tool calling
`deepseek-ai/DeepSeek-V4-Pro`	`--enable-auto-tool-choice` `--tool-call-parser=deepseek_v4` `--reasoning-parser=deepseek_v4`	`models/deepseek-v4`, both the `base` and `base-low-latency` overlays
`RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block`	`--enable-auto-tool-choice` `--tool-call-parser=qwen3_coder` `--reasoning-parser=nemotron_v3`	`models/nemotron-3-ultra`

llm-d#2640 tracks the models and overlays that still omit them.

warning

A wrong parser is worse than none: vLLM starts, accepts the request, and then tries to parse tool calls with the wrong grammar.

Connecting Coding Harnesses

Once agentic-api is reachable at ${AGENTIC_API_BASE_URL}, you can attach coding harnesses (such as OpenAI Codex CLI or Claude Code) to the deployed endpoint. See the vllm-project/agentic-api repository and its Harness CLI Testing guide for supported harnesses, configuration, and usage instructions.

Cleanup

To remove the deployed components:

kubectl kustomize ${REPO_ROOT}/guides/${GUIDE_NAME}/manifests/base \
  | envsubst '${LLM_API_BASE}' \
  | kubectl delete -n ${NAMESPACE} -f - --ignore-not-found=true

kubectl delete -n ${NAMESPACE} -f ${REPO_ROOT}/guides/${GUIDE_NAME}/manifests/postgres.yaml --ignore-not-found=true
kubectl delete secret agentic-api-postgres -n ${NAMESPACE} --ignore-not-found=true
# The PVC is deleted by the manifest above; the PV's fate follows its
# StorageClass reclaim policy.

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}

helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete namespace ${NAMESPACE}

Overview​

Supported Accelerators and Model Servers​

Prerequisites​

Get the guide​

Configure the environment​

Installation Instructions​

1. Deploy the llm-d Router​

2. Deploy the Model Server​

3. Deploy PostgreSQL​

4. Deploy agentic-api​

5. Wait for the Model Servers​

Verification​

1. Reach agentic-api​

2. Send a Test Request​

3. Verify the Mechanism​

Add agentic-api to Another Guide​

Connecting Coding Harnesses​

Cleanup​