Autoscaling Workloads with KEDA and EPP Metrics

This guide configures KEDA to scale an llm-d model server Deployment from demand signals emitted by the Endpoint Picker (EPP). KEDA is the recommended and user-facing autoscaling path described here.

This guide scales on EPP demand signals - in-flight load, queue depth, or pool saturation - not CPU or GPU utilization, which an inference accelerator pins high at both low and high concurrency. The default path scales on in-flight load and needs no Flow Control.

For how KEDA+EPP scaling works and why demand signals beat utilization, see KEDA with EPP Metrics.

Prerequisites

1. Complete the optimized-baseline guide, including enabling monitoring. Confirm that Prometheus is scraping the EPP metrics endpoint before configuring autoscaling.

Set Environment Variables

Set the guide environment variables. SIGNAL selects the scaling signal (inflight, queue, or saturation) and ENV selects the platform (existing or ocp); together they name the overlay you apply.

The deployment-specific variables are rendered into the overlay with envsubst at apply time, so install envsubst (shipped with GNU gettext; brew install gettext on macOS, where it is not present by default). The defaults below match the stock optimized-baseline deployment, so if you followed that guide unchanged you can leave them as-is. When adapting the guide to your own deployment you typically change only two of them, NAMESPACE and MODEL; the other three follow from the Helm release name (and accelerator) you chose at install time:

• NAMESPACE - the namespace you deployed into.
• MODEL - the model you serve; must match the model_name label on the EPP metrics.
• EPP_SERVICE - the EPP service name, <release>-epp.
• INFERENCE_POOL - the InferencePool name, which equals <release>.
• TARGET_DEPLOYMENT - the decode Deployment the ScaledObject targets, <release>-<accelerator>-vllm-decode (for the stock release, optimized-baseline-nvidia-gpu-vllm-decode).

export BRANCH=main
export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export NAMESPACE=llm-d-optimized-baseline
export MONITORING_NAMESPACE=llm-d-monitoring
export KEDA_NAMESPACE=keda # options: keda, openshift-keda
export MODEL=Qwen/Qwen3-32B
export TARGET_DEPLOYMENT=optimized-baseline-nvidia-gpu-vllm-decode
export SCALEDOBJECT_NAME=optimized-baseline-keda-epp
export HPA_NAME=keda-hpa-optimized-baseline
export EPP_SERVICE=optimized-baseline-epp
export INFERENCE_POOL=optimized-baseline
export SIGNAL=inflight # options: inflight, queue, saturation
export ENV=existing # options: existing, ocp
export OVERLAY_ROOT=${REPO_ROOT}/guides/workload-autoscaling/keda-epp/optimized-baseline

Source the common guide environment variables:

source ${REPO_ROOT}/guides/env.sh

Configure Prerequisites

1. Configure observability by following the shared observability setup guide. Record the Prometheus endpoint and its TLS and authentication requirements; you will use them when reviewing the example ScaledObject.

2. Install KEDA, or the platform-provided KEDA operator, as described in Kubernetes Metrics Adapter.

3. Only for the queue and saturation signals (which require EPP Flow Control): upgrade the optimized-baseline router with the KEDA+EPP overlay, which enables Flow Control. Reapply the monitoring feature values used during optimized-baseline installation so that the EPP metrics port and its ServiceMonitor remain enabled. The default inflight signal needs no router change - it scales on a metric the stock EPP already emits, so skip this step and apply the ScaledObject directly:

# only when SIGNAL=queue or saturation:
helm upgrade optimized-baseline \
  ${ROUTER_STANDALONE_CHART} \
  -f ${REPO_ROOT}/guides/recipes/router/base.values.yaml \
  -f ${REPO_ROOT}/guides/optimized-baseline/router/optimized-baseline.values.yaml \
  -f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml \
  -f ${OVERLAY_ROOT}/router.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

Confirm the EPP ServiceMonitor exists (and, for queue/saturation, that Flow Control is enabled):

# only when SIGNAL=inflight:
kubectl get servicemonitor -n ${NAMESPACE}

# only when SIGNAL=queue or saturation:
kubectl logs deployment/optimized-baseline-epp -c epp -n ${NAMESPACE} | grep "Initializing Flow Control layer"
kubectl get servicemonitor -n ${NAMESPACE}

Validate EPP Metrics in Prometheus

First confirm that EPP exposes the metrics directly.

In terminal 1, keep the port-forward running:

kubectl port-forward -n ${NAMESPACE} \
  service/optimized-baseline-epp 9091:9090

In terminal 2, query the endpoint:

curl -s http://localhost:9091/metrics | \
  grep -E 'llm_d_epp_inflight_requests|llm_d_epp_flow_control_queue_size|llm_d_epp_flow_control_pool_saturation|llm_d_epp_request_running'

Then stop the EPP port-forward. Open the query interface for the Prometheus installation configured in the observability setup and run the query for your chosen signal:

sum(llm_d_epp_inflight_requests{namespace="llm-d-optimized-baseline",producer_name="inflight-load-producer"})

sum(llm_d_epp_flow_control_queue_size{namespace="llm-d-optimized-baseline",service="optimized-baseline-epp",model_name="Qwen/Qwen3-32B"})

max(llm_d_epp_flow_control_pool_saturation{inference_pool="optimized-baseline",namespace="llm-d-optimized-baseline"})

sum(llm_d_epp_request_running{namespace="llm-d-optimized-baseline",service="optimized-baseline-epp",model_name="Qwen/Qwen3-32B"})

Each query must return a scalar or a single-element vector. Inspect the raw series in Prometheus before continuing and update the selectors for your deployment. The inflight metric keys on producer_name (no service or model_name label), so its query sums across the namespace - the guide assumes one inference pool per namespace; in a multi-pool namespace, narrow the selector. The running-request metric does not expose inference_pool, and the pool-saturation metric keys on inference_pool (the InferencePool name, i.e. the EPP's --pool-name), not service. Scrape-time labels vary between monitoring installations; do not copy selectors without checking the live series.

The metrics may remain at zero until requests are sent. If a series is absent, check the Prometheus target first rather than treating absence as zero.

Configure Prometheus Access

KEDA reads authentication Secrets from the ScaledObject namespace. On generic Kubernetes the checked-in ScaledObject reaches the bundled kube-prometheus-stack over plain in-cluster HTTP with no client authentication, so there is no Secret to create. If your Prometheus endpoint requires a bearer token, mTLS, basic authentication, or cloud workload identity, update each trigger's serverAddress and add a TriggerAuthentication using the KEDA Prometheus authentication documentation.

info

KEDA's prometheus scaler drops a TriggerAuthentication CA unless the trigger also sets authModes; a CA-only trigger fails serving-cert verification with x509: certificate signed by unknown authority. The generic-k8s path uses plain in-cluster HTTP for this reason; the OpenShift path sets authModes: bearer.

Platform notes

The Prometheus endpoint, KEDA operator namespace, and authentication method depend on the platform. Update KEDA_NAMESPACE, each trigger's serverAddress, and any TriggerAuthentication before applying the example.

Bundled llm-d observability stack

The checked-in ScaledObject targets the bundled Prometheus installation documented in the observability setup guide, reached over plain in-cluster HTTP at its service address - nothing to configure and no Secret to create.

To open its Prometheus query UI, keep this command running in a terminal and open http://localhost:9090:

kubectl port-forward -n ${MONITORING_NAMESPACE} \
  service/llmd-kube-prometheus-stack-prometheus 9090:9090

OpenShift

OpenShift environments use the Custom Metrics Autoscaler Operator (KEDA) and cluster monitoring through Thanos Querier. The OpenShift leaf overlays (overlays/ocp/inflight, overlays/ocp/queue, and overlays/ocp/saturation) configure this for you - apply the one for your signal instead of the overlays/k8s/* leaves:

• Points each trigger at thanos-querier.openshift-monitoring.svc.cluster.local:9091 and enables authModes: bearer. Without it Thanos rejects the query (401 when unauthenticated, 403 when the ServiceAccount lacks cluster-monitoring-view), the trigger errors, and the ScaledObject goes Ready=False with KEDAScalerFailed events. No fallback is configured, so autoscaling fails visibly rather than looking healthy while doing nothing.
• Provisions a dedicated keda-epp-metrics-reader ServiceAccount granted the cluster-monitoring-view ClusterRole, and adds a keda-prometheus-auth TriggerAuthentication pointing at that SA's token Secret. On OpenShift the service-ca operator injects service-ca.crt (the CA that signs Thanos's serving certificate) into the token Secret automatically, so no CA copy is required.

The overlays carry ${...} placeholders for the namespace, model, target deployment, EPP service, and inference pool, so you set those as environment variables and the apply step renders them with envsubst - no manual YAML edits. The OCP leaf also renders the metrics-reader ClusterRoleBinding subject namespace from ${NAMESPACE}, so the binding follows your deployment namespace. Because the ClusterRoleBinding is cluster-scoped, the OCP leaf also renders its name as keda-epp-metrics-reader-monitoring-view-${NAMESPACE}, so deploying to multiple namespaces on a shared cluster does not make the bindings collide - no manual rename is needed.

Choose Your Path

Pick one signal, set the matching environment variables, and apply the leaf overlay below. Do not apply two ScaledObjects to one Deployment.

SIGNAL	Leaf overlay	Use when
inflight (default, recommended)	overlays/{k8s,ocp}/inflight	Simplest path; no Flow Control; nightly-covered; scales on EPP in-flight requests, emitted by the stock optimized-baseline EPP.
queue	overlays/{k8s,ocp}/queue	Requires Flow Control; mature; portable startup smoothing.
saturation (experimental)	overlays/{k8s,ocp}/saturation	Requires Flow Control; scale before requests queue; validate thresholds first.

Signal background and the saturation-detector choice: Scaling Signals, Saturation Detector, and Flow Control On vs. Off. The queue and saturation signals require EPP Flow Control; inflight does not. The inflight and queue signals have nightly end-to-end coverage; saturation does not, so validate it against your own load before production use.

note

This guide is validated with vLLM model servers. The EPP demand signals are engine-agnostic, but the default thresholds are tuned for vLLM; validate them before relying on the guide with another engine.

Configuration

The checked-in ScaledObject provides the following default configuration for this guide:

Parameter	Value
Target Deployment	optimized-baseline-nvidia-gpu-vllm-decode
Minimum replicas	1
Maximum replicas	8 (inflight / queue) / 10 (saturation)
In-flight-request threshold	16 (inflight)
Queue-size threshold	1 (queue)
Pool-saturation threshold	0.7 (saturation)
Running-request threshold	16 (queue)
Polling interval	15s
Cooldown period	300s
Scale-up stabilization window	300s
Scale-down stabilization window	300s

For tuning guidance on each value, see KEDA with EPP Metrics.

For how thresholds are interpreted (per-replica AverageValue targets) and how to validate them, see Dual-Metric Strategy.

For how flow-control on/off changes which trigger sees demand, see Flow Control On vs. Off.

Overshoot While Pods Start

New replicas take minutes to load a model, so scale-up can overshoot: the HPA keeps seeing demand that in-flight capacity will soon absorb. The overlays smooth this with HPA stabilization windows (300s scale-up and scale-down). For how this works, see Overshoot and Startup-Time Mitigation.

Apply the KEDA ScaledObject

Review the ScaledObject for your platform and signal before applying. Each overlay carries a full scaledobject.yaml (the link points at the generic-Kubernetes queue overlay). The namespace, target deployment, and the PromQL label selectors are rendered from the environment variables in the export block above by envsubst at apply time, so the fields to review and adjust directly in the YAML are:

• Prometheus serverAddress (the bundled kube-prometheus-stack on generic Kubernetes; the OCP overlays point it at Thanos Querier)
• The trigger thresholds (per-replica AverageValue targets)

This walkthrough intentionally begins with one target replica so that a 1-to-N scale-up is observable. Scale the target Deployment down before creating the ScaledObject, then wait for it to become available:

kubectl scale deployment ${TARGET_DEPLOYMENT} -n ${NAMESPACE} --replicas=1
kubectl rollout status deployment/${TARGET_DEPLOYMENT} -n ${NAMESPACE} --timeout=15m

Apply the leaf overlay for your SIGNAL and ENV. Each apply builds the leaf, renders its ${...} placeholders with envsubst, and pipes the result to kubectl apply.

Platform specifics

The apply command differs by platform: the overlay path (overlays/k8s/* vs overlays/ocp/*) and the Prometheus auth are not the same on a generic cluster and on OpenShift. Pick the block for your ENV.

Generic Kubernetes

On a generic Kubernetes cluster with the bundled kube-prometheus-stack (plain in-cluster HTTP, no auth secret), use the overlays/k8s/* leaf.

In-flight-load signal (default, recommended; no Flow Control):

# only when SIGNAL=inflight and ENV=existing:
kubectl kustomize ${OVERLAY_ROOT}/overlays/k8s/inflight | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl apply -f -

Queue signal (requires Flow Control):

# only when SIGNAL=queue and ENV=existing:
kubectl kustomize ${OVERLAY_ROOT}/overlays/k8s/queue | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl apply -f -

Saturation signal (experimental):

# only when SIGNAL=saturation and ENV=existing:
kubectl kustomize ${OVERLAY_ROOT}/overlays/k8s/saturation | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl apply -f -

OpenShift

On OpenShift, use the overlays/ocp/<signal> leaf instead (see the OpenShift note - it points the trigger at Thanos Querier and bearer-authenticates via a dedicated ServiceAccount; no CA copy is needed).

In-flight-load signal (default, recommended; no Flow Control):

# only when SIGNAL=inflight and ENV=ocp:
kubectl kustomize ${OVERLAY_ROOT}/overlays/ocp/inflight | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl apply -f -

Queue signal (requires Flow Control):

# only when SIGNAL=queue and ENV=ocp:
kubectl kustomize ${OVERLAY_ROOT}/overlays/ocp/queue | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl apply -f -

Saturation signal (experimental):

# only when SIGNAL=saturation and ENV=ocp:
kubectl kustomize ${OVERLAY_ROOT}/overlays/ocp/saturation | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl apply -f -

Verify KEDA Metric Evaluation

Check the ScaledObject status and events:

kubectl get scaledobject ${SCALEDOBJECT_NAME} -n ${NAMESPACE}
kubectl wait --for=condition=Ready scaledobject/${SCALEDOBJECT_NAME} -n ${NAMESPACE} --timeout=120s

Ready=True confirms that the scaler configuration is valid. Because this example has minReplicaCount: 1, the Active condition is not the best signal for 1-to-N scaling. Inspect the generated HPA's current metrics and the target Deployment's replica count instead. Active becomes relevant to zero-to-one activation in the optional scale-to-zero configuration below.

KEDA creates the HPA named in horizontalPodAutoscalerConfig:

kubectl get hpa ${HPA_NAME} -n ${NAMESPACE}
kubectl get hpa ${HPA_NAME} -n ${NAMESPACE} -o jsonpath='{.status.currentMetrics}' | jq

A non-empty currentMetrics list shows that the generated HPA is receiving the metrics exposed by KEDA. It can take several polling intervals for the first values to appear.

Generate Bounded Load

note

These load commands target the EPP service directly (${EPP_SERVICE}:80). That path is valid only in standalone/epponly router mode, where the EPP pod carries the envoy-proxy sidecar (the topology the nightly exercises). In gateway mode (agentgateway/istio/gke), send inference through llm-d-inference-gateway instead; the EPP service :80 returns 404.

Run a temporary curl pod in the workload namespace:

kubectl run curl-load --rm -it \
  --image=curlimages/curl \
  --restart=Never \
  --namespace=${NAMESPACE} \
  --env=MODEL=${MODEL} \
  --env=EPP_SERVICE=${EPP_SERVICE} -- sh

From inside the pod, send a bounded set of concurrent requests:

cat > /tmp/request.json <<EOF
{
  "model": "${MODEL}",
  "prompt": "Write a detailed explanation of how continuous batching works.",
  "max_tokens": 256
}
EOF

seq 1 100 | xargs -P 16 -I{} \
  curl -sS --max-time 180 -o /dev/null -w '%{http_code}\n' \
    -X POST http://${EPP_SERVICE}/v1/completions \
    -H 'Content-Type: application/json' \
    --data-binary @/tmp/request.json

Adjust concurrency only if the reference load does not cross the configured threshold. Keep request counts and timeouts bounded while tuning.

For the default inflight signal, the reference load above drives scale-up directly: in-flight requests rise with concurrency and cross the per-replica target (16). The queue and saturation signals (Flow Control) behave differently - they stay near zero until a replica is genuinely saturated. To exercise the queue trigger specifically - where demand comes from a backlog rather than from concurrency - raise -P (concurrency) and max_tokens until llm_d_epp_flow_control_queue_size goes non-zero. The queue only forms once load exceeds a replica's serving capacity, and the level needed depends on the model, accelerator, tensor-parallel degree, and max-model-len, so climb from a few hundred concurrent requests rather than assuming a fixed value.

Verify Scale-Up

While the load is running, watch the ScaledObject, generated HPA, and target Deployment:

kubectl get scaledobject,hpa -n ${NAMESPACE} -w

kubectl get deployment optimized-baseline-nvidia-gpu-vllm-decode \
  -n ${NAMESPACE} -w

An increased desired replica count confirms that the HPA made a scale-up decision. A new replica can take substantially longer to become Ready while the model is loading.

After the additional replica is Ready, repeat a normal inference request and confirm it succeeds.

Troubleshooting

ScaledObject is not Ready

kubectl describe scaledobject optimized-baseline-keda-epp -n ${NAMESPACE}
kubectl get events -n ${NAMESPACE} --sort-by='.lastTimestamp'
kubectl logs -n ${KEDA_NAMESPACE} \
  -l app.kubernetes.io/name=keda-operator --all-containers

Common causes are an unreachable serverAddress, missing authentication (on platforms that require it), a TriggerAuthentication CA that KEDA drops because the trigger sets no authModes, or a PromQL query that returns more than one element.

Generated HPA shows unknown metrics

Re-run the exact query in Prometheus, verify its labels, and inspect the generated HPA:

kubectl describe hpa keda-hpa-optimized-baseline -n ${NAMESPACE}

Do not create a second HPA to work around this condition. Fix the ScaledObject query or Prometheus connectivity instead.

Metrics are missing

kubectl get servicemonitor -n ${NAMESPACE} -o yaml
kubectl get endpoints optimized-baseline-epp -n ${NAMESPACE}
kubectl logs deployment/optimized-baseline-epp -n ${NAMESPACE}

Confirm the Prometheus target is UP, Flow Control is enabled, and the live metric labels match the selectors in the ScaledObject.

By default, the KEDA Prometheus scaler ignores an empty Prometheus result (ignoreNullValues defaults to true). If a scaler remains inactive unexpectedly, verify that the PromQL query returns a value rather than relying only on status conditions.

Desired replicas increase but new replicas are not Ready

If the generated HPA raises the desired replica count but the Deployment's Ready replica count does not increase, the scaler has already made its decision. Inspect pod events, scheduling status, image or model download progress, and model-server logs. Model startup delay is distinct from a Prometheus or HPA metric failure.

Deployment does not scale

Check whether another HPA or controller targets the same Deployment. This can happen when a manually created HPA remains alongside KEDA or another autoscaling controller manages the workload.

If autoscaling is managed exclusively by KEDA and there is exactly one ScaledObject for the target Deployment, KEDA owns the generated HPA and this duplicate-HPA scenario should not occur.

Also check that the HPA calculates a desired count above the current replica count, maxReplicaCount is greater than the current count, metrics are available, and the generated HPA has no scaling-limited conditions.

Cleanup

# only when SIGNAL=inflight and ENV=existing:
kubectl kustomize ${OVERLAY_ROOT}/overlays/k8s/inflight | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl delete --ignore-not-found=true -f -

# only when SIGNAL=inflight and ENV=ocp:
kubectl kustomize ${OVERLAY_ROOT}/overlays/ocp/inflight | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl delete --ignore-not-found=true -f -

# only when SIGNAL=queue and ENV=existing:
kubectl kustomize ${OVERLAY_ROOT}/overlays/k8s/queue | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl delete --ignore-not-found=true -f -

# only when SIGNAL=saturation and ENV=existing:
kubectl kustomize ${OVERLAY_ROOT}/overlays/k8s/saturation | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl delete --ignore-not-found=true -f -

# only when SIGNAL=queue and ENV=ocp:
kubectl kustomize ${OVERLAY_ROOT}/overlays/ocp/queue | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl delete --ignore-not-found=true -f -

# only when SIGNAL=saturation and ENV=ocp:
kubectl kustomize ${OVERLAY_ROOT}/overlays/ocp/saturation | envsubst '$NAMESPACE $MODEL $TARGET_DEPLOYMENT $EPP_SERVICE $INFERENCE_POOL' | kubectl delete --ignore-not-found=true -f -

Deleting the ScaledObject also removes the HPA managed by KEDA. It does not delete the target Deployment and can leave that Deployment at its current replica count. Scale the Deployment explicitly if a different post-cleanup count is required.

Optional: Scale to Zero

KEDA supports scale-to-zero without the Kubernetes HPAScaleToZero feature gate. Set minReplicaCount: 0 only after validating scale-up from one replica. When the Deployment is at zero, the Flow Control queue-size metric is the activation signal: EPP holds incoming requests until a model server becomes Ready.

For how zero-to-one activation, cooldownPeriod, and cold-start latency behave, see Scale to Zero.

Legacy Prometheus Adapter Path

Existing direct-HPA deployments can refer to the Prometheus Adapter notes while migrating. New EPP autoscaling deployments should use KEDA and should not install Prometheus Adapter solely for this guide.