Workload Autoscaling

warning

The Workload Variant Autoscaler (WVA) is deprecated. The WVA path is no longer developed and receives no further releases; v0.9.0 is the final version, and the guide assets in wva/ are pinned to it. WVA-based features, including Replica Rebalancing (since removed), are deprecated along with it. New deployments should use one of the KEDA + EPP paths below; existing WVA deployments should migrate to the KEDA + EPP Metrics guide and select its saturation signal overlay, the closest replacement for WVA's saturation signal.

Traditional autoscaling indicators like resource utilization metrics (CPU/GPU) are often lagging indicators — they only reflect saturation after it has already occurred, by which point latency has spiked and requests may be failing. For LLM inference, this problem is compounded by the fact that GPU utilization is often pegged near 100% during active batching regardless of actual load, making it an entirely unreliable signal.

Effective LLM autoscaling requires proactive, SLO-aware signals that reflect the true state of the inference system — queue depth, in-flight request counts, and KV cache pressure — so that capacity can be added before end-user latency is impacted.

This guide covers the autoscaling strategies available in llm-d. The EPP paths use KEDA as the scaling primitive, while the WVA path publishes a desired-replica metric for an HPA or KEDA. The paths differ in the use cases they target, the metrics that drive them, and the operational complexity they require.

Prerequisites

Before choosing an autoscaling path, you must have a monitoring stack. KEDA's Prometheus scaler queries Prometheus directly; deployments using the legacy direct-HPA path or the current WVA example may also require Prometheus Adapter.

Prometheus

You must have a Prometheus instance running in your cluster. See Prometheus Setup Guide for guidance on setting up Prometheus. Make sure to enable TLS as WVA requires it to securely access the Prometheus API.

Kubernetes Metrics Adapter

Installing KEDA (Recommended)

Follow the Install KEDA guide. KEDA evaluates scaler values and exposes them through its metrics server to the Kubernetes External Metrics API. The HPA generated by KEDA consumes those external metrics. KEDA also supports scaling a target to zero.

note

On OpenShift, follow the Custom Metrics Autoscaler Operator documentation.

Installing Prometheus Adapter (Deprecated)

See the Prometheus Adapter guide for installation instructions. Note that the Prometheus Adapter is planned for deprecation, and it is recommended to use KEDA instead for autoscaling needs.

Paths

KEDA + EPP Metrics (Recommended)

In-flight, Queue, and Saturation Autoscaling

The KEDA + EPP Metrics path uses KEDA's Prometheus scaler with signals emitted directly by the Endpoint Picker (EPP). KEDA creates and owns the HPA that scales the model server Deployment. It ships three scaling signals as selectable overlays; the guide covers choosing between them:

• In-flight load (default, recommended) scales on EPP in-flight requests, a signal the stock optimized-baseline EPP already emits - no Flow Control required. The simplest path, and nightly-covered.
• Queue depth (requires Flow Control) scales on requests waiting in EPP Flow Control plus running-request count. Reflects unmet demand once a replica saturates.
• Pool saturation (requires Flow Control, experimental) scales on the InferencePool-scoped saturation level (0.0-1.0+, a normalized measure of how loaded the pool is) plus running-request count, reacting before requests queue.

This path requires KEDA and Prometheus; it does not require Prometheus Adapter.

Token-Aware Autoscaling

The Token-Aware Autoscaling path scales on tokens of outstanding work rather than a count of requests. An 8192-token prompt is 16x the prefill work of a 512-token one, and a request or queue-depth counter rates them the same, so the same request rate can mean a third of a replica or three replicas of prefill work.

This path measures each phase in its own unit: prefill divides the EPP's in-flight token backlog by a calibrated peakPrefillThroughput to get seconds of queue wait, comparable directly against a share of your TTFT SLO, while decode reads per-pod KV cache occupancy. Overlays are provided for both topologies: P+D co-located (both phases in one pod) and P/D-disaggregated (split across pods). Best when prompt sizes vary widely, or when you want the prefill threshold derived from a latency target rather than tuned by trial. Requires one calibration run per hardware/model combination.

SLO-Aware Autoscaling (KEDA + estimated latency)

A specialization of this path drives the HPA from the pool's latency rather than its queue depth. The SLO-Aware Autoscaling sub-path scales directly against latency SLOs, using the EPP's estimated TTFT/TPOT as the signal — either its ML-predicted latency (from the online-trained predictor) or the actual measured latency aggregated in real time when the predictor isn't enabled.

A Prometheus recording rule turns that estimate into a single saturation ratio (latency ÷ SLO), and a KEDA ScaledObject with an expr-lang formula computes the desired replica count and drives a standard HPA — no custom controller. With the ML predictor, capacity is added as pressure builds rather than after the queue has already formed. Best when clients express per-request latency SLOs and you want scaling driven by the objective itself rather than a proxy metric.

KEDA + WVA Metrics (Deprecated)

warning

This path is deprecated and pinned to WVA v0.9.0. See the note at the top of this guide.

The Workload Variant Autoscaler (WVA) path integrates KEDA with the aggregated signal emitted by WVA: wva_desired_replicas.

WVA is designed for operators running multiple variants of the same model across different GPU hardware types (A100s, H100s, L4s), each with different cost and performance characteristics. WVA continuously monitors KV cache utilization, queue depth, and performance budgets to determine optimal replica counts across variants. Rather than scaling all variants equally, WVA preferentially adds capacity on the cheapest available variant and removes it from the most expensive — optimizing infrastructure cost without violating latency SLOs.

Choosing a Scaling Signal

	In-flight / Queue Autoscaling	Saturation-based Autoscaling	Token-Aware Autoscaling	SLO-Aware Autoscaling	KEDA + WVA Metrics
Best for	Homogeneous deployments; scale on in-flight load (default) or absolute queue depth / running requests (per-deployment thresholds, no over-provisioning)	Homogeneous deployments; scale on a normalized saturation ratio (leading, more portable thresholds, can potentially over-provision)	Deployments with heterogeneous prompt sizes; scale on token backlog, with the prefill threshold derived from a TTFT SLO	Single-pool deployments with per-request latency SLOs, scaled on predicted latency	Multi-variant deployments with cost-aware placement across heterogeneous hardware
Scaling signal	EPP in-flight requests (default, no Flow Control), queue depth, or running request count	normalized pool saturation level (0.0-1.0+) and running request count	EPP in-flight tokens ÷ calibrated prefill throughput (seconds of backlog), and per-pod KV cache occupancy	EPP estimated TTFT/TPOT (ML-predicted, or measured) vs the SLO	KV cache utilization, queue depth, performance budgets
Cost optimization	None — scales based on load signals only	None — scales based on load signals only	None — scales based on load signals only	Minimizes replicas needed to meet the SLO	Optimizes across variants by preferring lower-cost hardware
Additional components	KEDA and Prometheus only	KEDA and Prometheus only	KEDA and Prometheus, plus one peakPrefillThroughput calibration run	KEDA + one Prometheus recording rule	KEDA and WVA controller
Scale to zero	Supported	Supported (Needs Validation)	Supported (Needs Validation)	Supported (via KEDA)	Supported

Observability & Troubleshooting

Autoscaling is a closed loop, so the useful view is not a single metric but the composed signal across the model servers, the autoscaler, and the resulting pool. Watch the loop end to end: a demand signal rises, the autoscaler raises a replica target, new replicas become ready, and the demand signal comes back down. When scaling misbehaves, the break is almost always at one of those handoffs.

Full definitions for the model-server and EPP signals live in the metric reference and PromQL reference; WVA's own metrics are defined in the WVA Prometheus reference.

Key metrics for this path

Stage in the loop	Signal	Why it matters
Demand	vllm:num_requests_waiting, vllm:kv_cache_usage_perc (PromQL → Basic Model Serving)	The leading indicators autoscaling exists to react to. Queue depth and KV pressure rise before latency does
Demand (EPP view)	llm_d_epp_request_running (Metrics → EPP)	On the KEDA + EPP path this is the scaling signal itself. Compare it against replica count to see whether scaling is tracking demand
Decision (WVA)	wva_desired_replicas, wva_current_replicas, wva_saturation_utilization	WVA's replica target vs. what is running. A gap that persists means the decision is made but not being actuated
Input health (WVA)	wva_metrics_freshness_status	WVA scales on scraped Prometheus data. Stale inputs produce stale decisions, so a bad-freshness signal explains "why isn't it scaling" before you touch thresholds
Convergence	llm_d_epp_ready_endpoints (Metrics → EPP)	The pool the router can actually route to. If desired replicas climb but ready endpoints do not follow, new pods are not becoming Ready
Outcome	vllm:time_to_first_token_seconds, vllm:num_requests_waiting (Metrics → vLLM)	Confirms the loop closed: after capacity lands, queue depth and TTFT should recover. If they do not, you scaled on the wrong signal

Common failure modes

• Demand signal saturated but replicas flat — the autoscaler is not reacting. On the WVA path check wva_metrics_freshness_status and wva_desired_replicas for the variant; on the KEDA + EPP path check that the ScaledObject is Ready and the external metric is resolving. See the metric-plumbing steps in the KEDA + EPP troubleshooting section.
• Replica target climbs but the pool does not grow — wva_desired_replicas (or the HPA desired count) rises while wva_current_replicas / llm_d_epp_ready_endpoints lag. New pods are not scheduling or not passing readiness, usually GPU quota or image pull. See Desired replicas increase but new replicas are not Ready.
• Replicas flapping — the target oscillates and pods churn. Thresholds are too tight for the demand pattern; widen the stabilization window or the scaling band. See Choosing Scaling Thresholds.

For alert rules covering the model-server and EPP signals, see Alerting.

Kueue-Based Replica Rebalancing (Experimental)

When several models share one GPU budget, their HPAs scale independently and none of them knows what the others are consuming. The Kueue-Based Replica Rebalancing guide enforces that shared budget with Kueue. Each model gets a ClusterQueue with a guaranteed GPU floor in a shared cohort, and replica pods are admitted only when quota is free — so an idle model's GPUs are lent to a busy one and reclaimed by preemption. Enforcement happens below the HPA, so KEDA-generated HPAs work unmodified, but over-budget demand shows up as Pending pods rather than a lowered maxReplicas.