Version: dev

Fast Model Actuation

Overview

In Kubernetes, once a Pod has been allocated some GPUs it keeps that exclusive allocation for the rest of the Pod's lifetime. Scaling a model server up or down therefore means creating or destroying whole pods, and each new pod pays a full cold start — pull the image, initialize the runtime, load gigabytes of weights, and compile CUDA graphs — before it can serve a single request. On a shared GPU pool, swapping between model variants repeats that cold start on every swap.

Fast Model Actuation (FMA) addresses vLLM startup time using two technologies. One is vLLM sleep and wake, which can entirely avoid model loading and CUDA graph compilation in applicable scenarios. The other is running multiple vLLM processes as children of a launcher process that has already done the Python module loading. FMA can manage multiple vLLM instances bound to one GPU, with one instance awake at a time, to accomplish fast switching of which model that GPU is used for; this generalizes to multiple GPUs of one node. Together they turn model swap-in and replica scale-up from a minutes-scale cold start into a near-instant operation — without changing steady-state serving performance.

note

FMA's value is actuation speed, not inference throughput. Resident servers add only a small (~2.5%) CPU-memory overhead, and served performance matches a standard deployment. Workload Autoscaling decides when to scale; FMA addresses how fast the new capacity becomes ready. For autoscaled FMA, see the Fast Model Actuation + KEDA guide, where KEDA scales the requester pool on EPP flow-control metrics and each scale-up drives an FMA hot or warm start.

FMA uses the following dual pod technique to circumvent the exclusive, lifetime-long GPU allocation.

Server-requesting Pods reserve GPU resources via the Kubernetes Pod scheduler and the kubelet — keeping cluster capacity accounting accurate — but do not run inference themselves.
Launcher Pods (server-providing) run vLLM without requesting GPUs. They (a) gain access to all GPUs of their Node via special provisions that do not count this access as usage and (b) using CUDA_VISIBLE_DEVICES, as directed by the FMA controllers, get vLLM to use the specific GPU(s) reserved for the requesting pod.
FMA Controllers manage the lifecycle: creating/deleting launchers, binding/unbinding requesting pods to launchers and vLLM instances in them, creating/deleting those vLLM instances, and orchestrating sleep/wake.

Server-requesting pods are managed through standard Kubernetes set objects and controllers such as Deployments and autoscalers. An FMA controller watches the server-requesting pods and translates scheduler decisions into actions on launcher pods and GPUs.

Actuation paths

When a requesting pod is deleted, the controller puts the corresponding vLLM instance to sleep (model tensors move from GPU to main memory). Although the Kubernetes GPU allocation is released when the requesting pod is deleted, the vLLM instance retains the CUDA context in a small amount of GPU memory---until an FMA controller directs that launcher to delete that vLLM instance (which may happen, to limit memory usage). When capacity is needed again, the controller selects a launcher and takes the fastest available path:

Hot start — if a new requesting pod arrives, serving the same model with the same command-line parameters and assigned to the same GPU, the controller wakes the sleeping instance by moving weights from CPU memory back to the GPU over PCIe; it resumes in seconds.
Warm start — when there is no sleeping vLLM instance to wake but a launcher is available, the controller directs the launcher to create a new vLLM instance, taking advantage of the Python module loading the launcher has already done (tens of seconds).
Cold start (with launcher) — create a new launcher pod, then initialize the instance; still faster than a no-FMA cold start, which additionally pays pod scheduling and image pull.

To bound memory, the controller keeps at most a configurable number of instances sleeping per accelerator, evicting the least-recently-used instances when that budget is exceeded.

note

Hot start (vLLM /wake_up) only occurs if the Kubernetes Pod scheduler assigns the new requesting pod to the same node (and GPU) where the sleeping vLLM instance resides. In a cluster with a single GPU per node, if the scheduler picks the same node, the GPU is necessarily the same one. In a multi-node pool the scheduler may assign the pod to a different node.

For more details see the FMA repo README and the FMA repo docs.

Configuration

Parameter	Value
Model	Qwen/Qwen3-0.6B
Requesting pod replicas	2
Launcher count	1 (per matching node)
GPUs per requesting pod	1
Router	llm-d-router-standalone

Prerequisites

This guide assumes you have a Kubernetes cluster with GPU nodes and the llm-d router infrastructure available. If you are starting from an existing llm-d deployment, the Gateway API Inference Extension CRDs may already be installed and you can skip that step.

Have the proper client tools installed on your local system to use this guide.
Checkout llm-d repo:

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Set the guide specific environment variables:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=fast-model-actuation
export NAMESPACE=llm-d-fast-model-actuation
export FMA_VERSION=0.6.5
export FMA_CHART_INSTANCE_NAME=fma
export MODEL=Qwen/Qwen3-0.6B
export CURL_TEST_IMAGE=cfmanteiga/alpine-bash-curl-jq:latest
export BENCHMARK_REF=main
export HARNESS=nop
export WORKLOAD=nop.yaml
export GATEWAY_CLASS=epponly # options: epponly, gke, agentgateway, istio

Source the common guide environment variables (GAIE_VERSION, ROUTER_CHART_VERSION, ROUTER_STANDALONE_CHART, …):

source ${REPO_ROOT}/guides/env.sh

note

Some environment variables are common amongst guides. Inspect the file sourced above so the rest of the guide makes sense.

Install the Gateway API Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

Installation Instructions

At minimum, the user running these commands needs rights to create and manage CRDs, ClusterRoles, ClusterRoleBindings, and Helm releases across namespaces.

1. Apply FMA CRDs

kubectl apply --server-side \
  -f "https://raw.githubusercontent.com/llm-d-incubation/llm-d-fast-model-actuation/v${FMA_VERSION}/config/crds.yaml"
kubectl wait --for=condition=Established crd/inferenceserverconfigs.fma.llm-d.ai --timeout=120s
kubectl wait --for=condition=Established crd/launcherconfigs.fma.llm-d.ai --timeout=120s
kubectl wait --for=condition=Established crd/launcherpopulationpolicies.fma.llm-d.ai --timeout=120s

2. Grant RBAC Permissions

The FMA controllers need cluster-level access to list nodes (for the launcher-populator), and the launcher pods run as a dedicated fma-launcher ServiceAccount with namespace-level access to get and patch their own pod (so the state-change-reflector sidecar can write the serving-state annotation). This applies the fma-node-viewer ClusterRole and the namespace-scoped fma-launcher-pod-state-writer Role/RoleBinding:

# ClusterRole (cluster-scoped): the FMA controllers list nodes for the launcher-populator
kubectl apply -f ${REPO_ROOT}/guides/${GUIDE_NAME}/rbac/clusterrole.yaml

# ServiceAccount + Role (namespaced): the launcher pods run as the
# fma-launcher ServiceAccount so the state-change-reflector sidecar can
# patch its own pod (the dual-pods.llm-d.ai/vllm-instance-signature
# annotation).
kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/${GUIDE_NAME}/rbac/role.yaml

# RoleBinding (namespaced): bind the Role to the fma-launcher ServiceAccount.
# Created imperatively (not from a static manifest) so ${NAMESPACE} drives both the
# binding's namespace and the subject ServiceAccount's namespace.
kubectl create rolebinding fma-launcher-pod-state-writer \
  --role=fma-launcher-pod-state-writer \
  --serviceaccount=${NAMESPACE}:fma-launcher \
  -n ${NAMESPACE} \
  --dry-run=client -o yaml | kubectl apply -f -

note

Only the fma-node-viewer ClusterRole is created here. The matching ClusterRoleBinding is created by the FMA Helm chart in the next step, via --set global.nodeViewClusterRole=fma-node-viewer.

3. Deploy FMA Controllers via Helm

helm upgrade --install ${FMA_CHART_INSTANCE_NAME} \
  oci://ghcr.io/llm-d-incubation/llm-d-fast-model-actuation/charts/fma-controllers \
  --version ${FMA_VERSION} \
  --set global.nodeViewClusterRole=fma-node-viewer \
  -n ${NAMESPACE}

kubectl wait --for=condition=available --timeout=180s \
  deployment "${FMA_CHART_INSTANCE_NAME}-dual-pods-controller" -n ${NAMESPACE}
kubectl wait --for=condition=available --timeout=120s \
  deployment "${FMA_CHART_INSTANCE_NAME}-launcher-populator" -n ${NAMESPACE}

4. Deploy the llm-d Router

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${REPO_ROOT}/guides/recipes/router/base.values.yaml \
  -f ${REPO_ROOT}/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

5. Deploy the Model Server (Dual Pods)

Apply the FMA custom resources — InferenceServerConfig, LauncherConfig, and LauncherPopulationPolicy (which define the model to serve, the launcher pod template, and how many launchers to place) — together with the server-requesting pods that reserve GPUs and trigger model loading. The requester Deployment is included as a resource of modelserver/kustomization.yaml (via ../requester), so a single apply stands up both halves of the dual-pods design. A Deployment is used for the requesters (rather than a bare ReplicaSet) so they integrate cleanly with autoscalers such as the Workload Variant Autoscaler (WVA):

kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/

kubectl rollout status deployment/fma-requester -n ${NAMESPACE} --timeout=300s

note

This guide uses Qwen/Qwen3-0.6B which is publicly accessible and does not require a HuggingFace token. For gated models, you would need to mount the token via a different mechanism (FMA's ISC does not support secretKeyRef).

note

HF_HOME points at /tmp/hf_cache, which is ephemeral node storage. This keeps the guide self-contained, but every fresh launcher re-downloads the model weights and re-runs torch.compile. For production — or any repeated benchmarking — back HF_HOME with a shared, persistent volume (a PVC, e.g. ReadWriteMany) so model weights and compiled graphs persist across launcher pods and are not recomputed on each start.

note

The launcher pod does not request GPU resources from the Kubernetes scheduler or device plugin. Instead, the FMA controller sets CUDA_VISIBLE_DEVICES to point to the GPU reserved by the corresponding requesting pod, giving the launcher direct access to that GPU via the CUDA runtime. The runtimeClassName: nvidia is required on platforms (e.g., OpenShift) where GPU driver libraries are injected via the runtime class rather than the device plugin resource request.

note

launcherCount is per matching node. Setting launcherCount: 1 creates one launcher pod on each node that has nvidia.com/gpu.present: "true". Only launchers that get bound to a requesting pod will actually start a vLLM instance.

You should see:

2 requesting pods (fma-requester-*) in Ready state
Launcher pods in Running state (one per GPU node in your cluster)
FMA controller pods
Router/EPP pods

6. (Optional) Enable monitoring

Install the Monitoring stack.
To enable Prometheus monitoring on the llm-d router, add -f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml during the router installation step.

Verification

1. Get the IP of the Router

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send a Test Request

Open a temporary interactive shell inside the cluster and send a completion request (model-aware; set MODEL to the name you want to query):

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\"}"'

3. Demonstrate Sleep/Wake

This section demonstrates FMA's core value: fast model actuation via sleep/wake. Scaling the requesting pods to 0 triggers the FMA controller to unbind them from their launchers and tell vLLM to sleep (the model stays in GPU memory but stops serving). Scaling back up re-binds them and wakes the sleeping instances — resuming in seconds rather than cold-starting:

set -eu
IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')
for i in 1 2; do
  echo "== sleep/wake cycle ${i} =="
  kubectl scale deployment fma-requester -n ${NAMESPACE} --replicas=0
  kubectl wait --for=delete pod -l app=fma-requester -n ${NAMESPACE} --timeout=120s
  # Confirm the launcher actually went to sleep before we wake it.
  kubectl wait pod -l app.kubernetes.io/component=launcher -n ${NAMESPACE} \
    --for='jsonpath={.metadata.labels.dual-pods\.llm-d\.ai/sleeping}=true' \
    --timeout=120s
  kubectl scale deployment fma-requester -n ${NAMESPACE} --replicas=2
  kubectl rollout status deployment/fma-requester -n ${NAMESPACE} --timeout=300s
  kubectl get pods -l app.kubernetes.io/component=launcher -n ${NAMESPACE} \
    -o 'jsonpath={range .items[*]}{.metadata.name}{" sleeping="}{.metadata.labels.dual-pods\.llm-d\.ai/sleeping}{"\n"}{end}' || true
  kubectl run curl-wake-${i} --rm -i --restart=Never --attach \
    --image=${CURL_TEST_IMAGE} \
    --namespace="${NAMESPACE}" \
    --env="IP=${IP}" \
    --env="MODEL=${MODEL}" \
    -- /bin/sh -c 'set -e; resp=$(curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\", \"max_tokens\": 5}"); echo "${resp}"; echo "${resp}" | jq -e ".choices[0].text | length > 0" >/dev/null'
done

Re-run the inference request from step 2 to confirm the model is serving again.

note

Wake latency depends on the Kubernetes scheduler assigning the new requesting pod to the same node and GPU where the sleeping vLLM instance resides. If a different GPU is assigned, a new vLLM instance starts from scratch (cold start). Sleep/wake is most valuable in multi-GPU-per-node configurations where multiple models share the same GPU pool and can be swapped in and out without cold-starting.

Benchmarking

This guide uses llmdbenchmark — the supported standard CLI for llm-d performance benchmarking. It defaults to the nop harness (which stands the stack up and validates it end-to-end without driving a synthetic load); the richer, FMA-specific experimentation workflow lives in llm-d-benchmark itself.

info

The Benchmarking section below contains only the fast-model-actuation-specific commands needed to drive the stack you just deployed — for everything else (and especially when something goes wrong), start at docs/operations/validation/benchmark.md.

1. Install the CLI

curl -sSL https://raw.githubusercontent.com/llm-d/llm-d-benchmark/${BENCHMARK_REF}/install.sh | bash
cd llm-d-benchmark
source .venv/bin/activate
llmdbenchmark --version

note

Subsequent llmdbenchmark commands assume you are inside the llm-d-benchmark repo directory with the venv activated. If you open a new shell, re-run the commands above.

2. Resolve the endpoint of the stack you just deployed

export ENDPOINT_URL="http://$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')"

3. Run the benchmark

llmdbenchmark \
  --spec           guides/${GUIDE_NAME} \
  run \
  --endpoint-url   "${ENDPOINT_URL}" \
  --gateway-class  "${GATEWAY_CLASS}" \
  --model          "${MODEL}" \
  --namespace      "${NAMESPACE}" \
  --harness        "${HARNESS}" \
  --workload       "${WORKLOAD}" \
  --analyze

Cleanup

To remove all deployed components. Order matters: delete the model-server CRs and the requester Deployment first, then wait for the requester and launcher pods to drain while the dual-pods controller is still running to strip their finalizers, and only then remove the controllers. Removing the controller before the pods drain leaves finalizer-bound pods stuck in Terminating forever.

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/ --ignore-not-found=true

kubectl wait --for=delete pod -l app=fma-requester -n ${NAMESPACE} --timeout=120s

kubectl wait --for=delete pod -l app.kubernetes.io/component=launcher -n ${NAMESPACE} --timeout=120s

helm uninstall ${FMA_CHART_INSTANCE_NAME} -n ${NAMESPACE}

helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -f ${REPO_ROOT}/guides/${GUIDE_NAME}/rbac/clusterrole.yaml --ignore-not-found=true

kubectl delete -n ${NAMESPACE} -f ${REPO_ROOT}/guides/${GUIDE_NAME}/rbac/role.yaml --ignore-not-found=true

kubectl delete rolebinding fma-launcher-pod-state-writer -n ${NAMESPACE} --ignore-not-found=true

kubectl delete namespace ${NAMESPACE}

kubectl delete crd inferenceserverconfigs.fma.llm-d.ai launcherconfigs.fma.llm-d.ai launcherpopulationpolicies.fma.llm-d.ai

Fast Model Actuation + KEDA: scale-from-zero autoscaling of FMA requesters on EPP flow-control metrics.
FMA repository: controllers, CRDs and design docs.

Overview​

Actuation paths​

Configuration​

Prerequisites​

Installation Instructions​

1. Apply FMA CRDs​

2. Grant RBAC Permissions​

3. Deploy FMA Controllers via Helm​

4. Deploy the llm-d Router​

5. Deploy the Model Server (Dual Pods)​

6. (Optional) Enable monitoring​

Verification​

1. Get the IP of the Router​

2. Send a Test Request​

3. Demonstrate Sleep/Wake​

Benchmarking​

1. Install the CLI​

2. Resolve the endpoint of the stack you just deployed​

3. Run the benchmark​

Cleanup​

Related​

Overview

Actuation paths

Configuration

Prerequisites

Installation Instructions

1. Apply FMA CRDs

2. Grant RBAC Permissions

3. Deploy FMA Controllers via Helm

4. Deploy the llm-d Router

5. Deploy the Model Server (Dual Pods)

6. (Optional) Enable monitoring

Verification

1. Get the IP of the Router

2. Send a Test Request

3. Demonstrate Sleep/Wake

Benchmarking

1. Install the CLI

2. Resolve the endpoint of the stack you just deployed

3. Run the benchmark

Cleanup

Related