Version: dev

Fast Model Actuation + KEDA Autoscaling

Overview

This guide combines Fast Model Actuation (FMA) with scale-from-zero autoscaling via KEDA. A KEDA ScaledObject scales the FMA server-requesting Deployment on Endpoint Picker (EPP) flow-control metrics — all the way down to zero when the pool is idle, and back up on the first queued request. Each requesting pod reserves a GPU and drives the FMA controllers to bring a vLLM instance online via a hot or warm start; scaling to zero releases the GPU and puts the vLLM to sleep. (See the FMA guide for what hot and warm start mean.)

Configuration

Parameter	Value
Model	Qwen/Qwen3-0.6B
Requester replicas	1 at deploy, 0 once idle (KEDA floor) … 2 (KEDA ceiling)
Scale-to-zero delay	`cooldownPeriod: 30` + HPA 30s stabilization (shortened for the guide)
Launcher count	1 (per matching GPU node)
GPUs per requester pod	1
Scale metric	`llm_d_epp_flow_control_queue_size` (threshold `1`, activation `0`)
Autoscaler	KEDA `ScaledObject` → HPA on `fma-requester` (scale-from-zero)
Router	llm-d-router-standalone (EPP `flowControl` enabled)

Prerequisites

This guide assumes you have a Kubernetes cluster with GPU nodes, the llm-d router infrastructure, and a Prometheus/monitoring stack that scrapes the EPP. If you are starting from an existing llm-d deployment, the Gateway API Inference Extension CRDs may already be installed and you can skip that step.

Have the proper client tools installed on your local system to use this guide.
Monitoring stack with Prometheus over HTTPS — See autoscaling prerequisites and Prometheus Setup Guide. This includes KEDA installation.
EPP flow control enabled — The llm_d_epp_flow_control_queue_size metric KEDA scales on requires the EPP flow control feature gate; it is also what makes an incoming request enqueue (rather than fail fast) when there is no ready backend yet, which is what lets this guide scale from zero. This guide's router values enable the flowControl gate for you (applied in step 4), so no extra action is needed. See EPP Flow Control for details on flow control behavior.
Checkout llm-d repo:

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Set the guide specific environment variables:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=fast-model-actuation-keda
export NAMESPACE=llm-d-fast-model-actuation-keda
export MONITORING_NAMESPACE=llm-d-monitoring
export FMA_VERSION=0.6.5
export FMA_CHART_INSTANCE_NAME=fma
export MODEL=Qwen/Qwen3-0.6B
export CURL_TEST_IMAGE=cfmanteiga/alpine-bash-curl-jq:latest
export BENCHMARK_REF=main
export HARNESS=inference-perf
export WORKLOAD=shared_prefix_synthetic_heavy.yaml
export GATEWAY_CLASS=epponly # options: epponly, gke, agentgateway, istio

Source the common guide environment variables (GAIE_VERSION, ROUTER_CHART_VERSION, ROUTER_STANDALONE_CHART, …):

source ${REPO_ROOT}/guides/env.sh

note

Some environment variables are common amongst guides. Inspect the file sourced above so the rest of the guide makes sense.

Install the Gateway API Inference Extension CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Confirm KEDA is installed:

# KEDA (or, on OpenShift, the Custom Metrics Autoscaler Operator) must be
# installed cluster-wide before this guide runs.
kubectl get crd scaledobjects.keda.sh >/dev/null 2>&1 \
  || { echo "KEDA not installed -- see https://keda.sh/docs/latest/deploy/"; exit 1; }

Create a target namespace for the installation:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

note

Fast Model Actuation — This guide reuses the base Fast Model Actuation guide's manifests (LauncherConfig, LauncherPopulationPolicy, InferenceServerConfig and server-requesting Deployment) as a kustomize base and stands up its own FMA controllers. You do not need to deploy it separately.

Installation Instructions

At minimum, the user running these commands needs rights to create and manage CRDs, ClusterRoles, ClusterRoleBindings, KEDA ScaledObjects, and Helm releases across namespaces.

1. Apply FMA CRDs

kubectl apply --server-side \
  -f "https://raw.githubusercontent.com/llm-d-incubation/llm-d-fast-model-actuation/v${FMA_VERSION}/config/crds.yaml"
kubectl wait --for=condition=Established crd/inferenceserverconfigs.fma.llm-d.ai --timeout=120s
kubectl wait --for=condition=Established crd/launcherconfigs.fma.llm-d.ai --timeout=120s
kubectl wait --for=condition=Established crd/launcherpopulationpolicies.fma.llm-d.ai --timeout=120s

2. Grant RBAC Permissions

The FMA controllers need cluster-level access to list nodes (for the launcher-populator) and namespace-level access for launcher pods to patch their own pod state. In addition, KEDA needs a metrics-reader ServiceAccount bound to cluster-monitoring-view so OpenShift's Thanos Querier authorizes its trigger queries.

# ClusterRole (cluster-scoped): the FMA controllers list nodes for the launcher-populator.
# Reused verbatim from the base guide (identical RBAC).
kubectl apply -f ${REPO_ROOT}/guides/fast-model-actuation/rbac/clusterrole.yaml

# ServiceAccount + Role (namespaced): the launcher pods run as the
# fma-launcher ServiceAccount so the state-change-reflector sidecar can
# patch its own pod (the dual-pods.llm-d.ai/vllm-instance-signature
# annotation). Reused verbatim from the base guide.
kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/fast-model-actuation/rbac/role.yaml

# RoleBinding (namespaced): bind the Role to the fma-launcher ServiceAccount.
# Created imperatively (not from a static manifest) so ${NAMESPACE} drives both the
# binding's namespace and the subject ServiceAccount's namespace.
kubectl create rolebinding fma-launcher-pod-state-writer \
  --role=fma-launcher-pod-state-writer \
  --serviceaccount=${NAMESPACE}:fma-launcher \
  -n ${NAMESPACE} \
  --dry-run=client -o yaml | kubectl apply -f -

# ClusterRoleBinding (cluster-scoped): grant the KEDA metrics-reader
# ServiceAccount cluster-monitoring-view, the role Thanos Querier requires
# to answer KEDA's trigger queries.
kubectl create clusterrolebinding "keda-epp-metrics-reader-monitoring-view-${NAMESPACE}" \
  --clusterrole=cluster-monitoring-view \
  --serviceaccount=${NAMESPACE}:keda-epp-metrics-reader \
  --dry-run=client -o yaml | kubectl apply -f -

note

Only the fma-node-viewer ClusterRole is created here. The matching ClusterRoleBinding is created by the FMA Helm chart in the next step, via --set global.nodeViewClusterRole=fma-node-viewer.

3. Deploy FMA Controllers via Helm

helm upgrade --install ${FMA_CHART_INSTANCE_NAME} \
  oci://ghcr.io/llm-d-incubation/llm-d-fast-model-actuation/charts/fma-controllers \
  --version ${FMA_VERSION} \
  --set global.nodeViewClusterRole=fma-node-viewer \
  -n ${NAMESPACE}

kubectl wait --for=condition=available --timeout=180s \
  deployment "${FMA_CHART_INSTANCE_NAME}-dual-pods-controller" -n ${NAMESPACE}
kubectl wait --for=condition=available --timeout=120s \
  deployment "${FMA_CHART_INSTANCE_NAME}-launcher-populator" -n ${NAMESPACE}

4. Deploy the llm-d Router (EPP flow-control enabled)

The router values for this guide enable the EPP flowControl feature gate so the flow-control queue-depth metric is emitted for KEDA to scale on (and so requests enqueue when no backend is ready yet):

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${REPO_ROOT}/guides/recipes/router/base.values.yaml \
  -f ${REPO_ROOT}/guides/fast-model-actuation/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml \
  -f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

5. Deploy the Model Server (Dual Pods)

Apply the FMA custom resources — InferenceServerConfig, LauncherConfig, and LauncherPopulationPolicy — together with the server-requesting fma-requester Deployment in a single kubectl apply -k modelserver/.

This creates 1 server-requesting pod, which reserves a GPU, and the FMA controllers bind it to a launcher pod — the launcher is what actually runs the vLLM instance. KEDA takes over the replica count in step 6.

kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/fast-model-actuation/${GUIDE_NAME}/modelserver/

kubectl rollout status deployment/fma-requester -n ${NAMESPACE} --timeout=300s

6. Enable Scale-from-Zero Autoscaling (KEDA)

Apply the KEDA layer — the ScaledObject, its TriggerAuthentication, and the metrics-reader ServiceAccount + token — from the ocp overlay, then wait for the ScaledObject to reconcile Ready. KEDA then creates the HPA and owns the requester's replica count from here on. The overlay follows the keda-epp guide, pointing the queue-depth trigger at Thanos Querier; on OpenShift the service-ca operator injects the Thanos CA into the token Secret, so no prometheus-token copy is required.

No traffic has arrived yet, so the EPP flow-control queue reads 0 and KEDA scales fma-requester to zero, deleting its one server-requesting pod — releasing the GPU and putting the vLLM instance to sleep, while the launcher stays running. The second wait blocks until the replica count reaches 0, about a minute with this guide's shortened delays (cooldownPeriod 30s, then the HPA's 30s stabilization window):

kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/fast-model-actuation/${GUIDE_NAME}/keda/ocp

kubectl wait --for=condition=Ready --timeout=120s \
  scaledobject/${GUIDE_NAME}-queue -n ${NAMESPACE}

kubectl wait --for=jsonpath='{.spec.replicas}'=0 \
  deployment/fma-requester -n ${NAMESPACE} --timeout=240s

kubectl get deployment/fma-requester -n ${NAMESPACE}

note

Generic Kubernetes (non-OpenShift). The command above uses the ocp overlay, which targets OpenShift's Thanos Querier. Use the keda/k8s overlay instead, and copy the bundled Prometheus CA into the namespace first so KEDA can authenticate:

# Generic Kubernetes only (do NOT run on OpenShift):
kubectl create secret generic keda-prometheus-auth \
  --namespace ${NAMESPACE} \
  --from-literal=ca.crt="$(kubectl get configmap prometheus-web-tls-ca \
    -n ${MONITORING_NAMESPACE} -o jsonpath='{.data.ca\.crt}')" \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/fast-model-actuation/${GUIDE_NAME}/keda/k8s

Verification

1. Get the IP of the Router

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send a Test Request

Send a completion request through the EPP. KEDA scaled the pool to 0 in step 6, so this request is the demand signal that scales it back up: EPP holds it in the flow-control queue while KEDA scales 0 → 1 and FMA brings vLLM up, then serves it. A single request blocks through the whole transition, so expect it to take a few minutes before returning.

Send a completion request:

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --pod-running-timeout=180s \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c '
    set -e
    code=$(curl -sS -o /tmp/body -w "%{http_code}" --max-time 600 \
      -X POST "http://${IP}/v1/completions" \
      -H "Content-Type: application/json" \
      -d "{\"model\": \"${MODEL}\", \"prompt\": \"The capital of France is,\", \"max_tokens\": 10}")
    echo "HTTP ${code}"
    jq . /tmp/body || cat /tmp/body
    [ "${code}" = "200" ] || { echo "expected HTTP 200, got ${code}" >&2; exit 1; }
    jq -e ".choices[0].text | length > 0" /tmp/body >/dev/null \
      || { echo "no completion text in response" >&2; exit 1; }'

To watch the scale-up while it runs, from another terminal:

kubectl get deployment/fma-requester -n ${NAMESPACE} -w

note

This scale-up should trigger a hot start — the fast path, waking the vLLM instance that step 6 put to sleep. For a Qwen/Qwen3-32B it takes 4.0 s mean pod startup against 85.3 s for a warm start (benchmark results). Which path you get depends on GPU assignment; see wake latency in the FMA guide.

Benchmarking

This guide uses llmdbenchmark — the supported standard CLI for llm-d performance benchmarking. The commands below run the inference-perf harness with the shared_prefix_synthetic_heavy.yaml workload. On this small model (Qwen3-0.6B) each request drains in tens of milliseconds, so the flow-control queue does not stay backed up — the value you are exercising is the scale-from-zero transition: when the requester has scaled to 0, the benchmark's first arrivals enqueue, KEDA's activation trip on that cold-start queue transient scales the requester up (0→1), and the continuing arrivals keep it above zero for the rest of the run.

info

The Benchmarking section below contains only the guide-specific commands needed to drive the stack you just deployed — for everything else (and especially when something goes wrong), start at docs/operations/validation/benchmark.md.

1. Install the CLI

curl -sSL https://raw.githubusercontent.com/llm-d/llm-d-benchmark/${BENCHMARK_REF}/install.sh | bash
cd llm-d-benchmark
source .venv/bin/activate
llmdbenchmark --version

note

Subsequent llmdbenchmark commands assume you are inside the llm-d-benchmark repo directory with the venv activated. If you open a new shell, re-run the commands above.

2. Resolve the endpoint of the stack you just deployed

export ENDPOINT_URL="http://$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')"

3. Run the benchmark

llmdbenchmark \
  --spec           guides/fast-model-actuation-${GUIDE_NAME} \
  run \
  --endpoint-url   "${ENDPOINT_URL}" \
  --gateway-class  "${GATEWAY_CLASS}" \
  --model          "${MODEL}" \
  --namespace      "${NAMESPACE}" \
  --harness        "${HARNESS}" \
  --workload       "${WORKLOAD}" \
  --analyze

Cleanup

To remove all deployed components:

warning

Order matters. Delete the KEDA ScaledObject first (so the HPA stops scaling the requester), then the model-server CRs and the requester Deployment, then wait for pods to drain while the dual-pods controller is still running to strip their finalizers, and only then remove the controllers. Removing the controller before the pods drain leaves finalizer-bound pods stuck in Terminating forever.

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/fast-model-actuation/${GUIDE_NAME}/keda/ocp --ignore-not-found=true

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/fast-model-actuation/${GUIDE_NAME}/modelserver/ --ignore-not-found=true

kubectl wait --for=delete pod -l app=fma-requester -n ${NAMESPACE} --timeout=120s

kubectl wait --for=delete pod -l app.kubernetes.io/component=launcher -n ${NAMESPACE} --timeout=120s

helm uninstall ${FMA_CHART_INSTANCE_NAME} -n ${NAMESPACE}

helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}

kubectl delete -n ${NAMESPACE} -f ${REPO_ROOT}/guides/fast-model-actuation/rbac/role.yaml --ignore-not-found=true

kubectl delete rolebinding fma-launcher-pod-state-writer -n ${NAMESPACE} --ignore-not-found=true

kubectl delete clusterrolebinding "keda-epp-metrics-reader-monitoring-view-${NAMESPACE}" --ignore-not-found=true

kubectl delete -f ${REPO_ROOT}/guides/fast-model-actuation/rbac/clusterrole.yaml --ignore-not-found=true

kubectl delete namespace ${NAMESPACE}

kubectl delete crd inferenceserverconfigs.fma.llm-d.ai launcherconfigs.fma.llm-d.ai launcherpopulationpolicies.fma.llm-d.ai

Benchmarking Reports

Empirical benchmark reports measuring what FMA is worth under KEDA autoscaling. On identical hardware, each compares a Baseline of EPP + KEDA alone — no FMA, so every scale-up loads the model cold — against two FMA paths: Warm (new vLLM instance on a running launcher) and Hot (wake a sleeping vLLM).

Qwen/Qwen3-14B on H100 and vLLM — queue-based autoscaling trigger
Qwen/Qwen3-32B on H100 and vLLM — saturation-based and queue-based autoscaling triggers

Overview​

Configuration​

Prerequisites​

Installation Instructions​

1. Apply FMA CRDs​

2. Grant RBAC Permissions​

3. Deploy FMA Controllers via Helm​

4. Deploy the llm-d Router (EPP flow-control enabled)​

5. Deploy the Model Server (Dual Pods)​

6. Enable Scale-from-Zero Autoscaling (KEDA)​

Verification​

1. Get the IP of the Router​

2. Send a Test Request​

Benchmarking​

1. Install the CLI​

2. Resolve the endpoint of the stack you just deployed​

3. Run the benchmark​

Cleanup​

Benchmarking Reports​

Overview

Configuration

Prerequisites

Installation Instructions

1. Apply FMA CRDs

2. Grant RBAC Permissions

3. Deploy FMA Controllers via Helm

4. Deploy the llm-d Router (EPP flow-control enabled)

5. Deploy the Model Server (Dual Pods)

6. Enable Scale-from-Zero Autoscaling (KEDA)

Verification

1. Get the IP of the Router

2. Send a Test Request

Benchmarking

1. Install the CLI

2. Resolve the endpoint of the stack you just deployed

3. Run the benchmark

Cleanup

Benchmarking Reports