KEDA with EPP Metrics

The Endpoint Picker (EPP) and KEDA integration scales model server replicas using demand-side signals from EPP. Rather than relying on coarse resource utilization, KEDA queries EPP metrics from Prometheus, exposes evaluated scaler values through its metrics server, and creates an HPA that consumes those external metrics. This approach is well-suited for homogeneous deployments where each target model-server pool can be isolated by metrics and scaled independently.

This is the recommended autoscaling path in llm-d. The WVA path is deprecated.

Functionality

The KEDA+EPP integration provides:

• Demand-driven scaling: Replicas are added or removed based on EPP signals — queue depth, active request count, pool saturation, in-flight tokens, or estimated latency — which reflect inference demand rather than accelerator utilization.
• Standard Kubernetes actuation: KEDA owns an autoscaling/v2 HPA, so HPA status and behavior remain available through standard Kubernetes APIs.
• Optional scale-to-zero: KEDA can remove all model server replicas when traffic is idle and reactivate the Deployment from an EPP Flow Control queue.
• No custom controller and no Prometheus Adapter: the whole loop is KEDA, Prometheus, and a ScaledObject.

Design

The LLM Autoscaling Problem

During active token generation, GPU utilization can remain high whether a model server is lightly loaded or saturated. A utilization-based autoscaler cannot reliably distinguish spare serving capacity from an overloaded batch.

EPP Flow Control moves excess demand to the gateway, where it can be measured before end-user latency degrades. EPP exposes that demand as metrics that map directly onto scaling decisions — reactive saturation signals (queued requests), proactive concurrency signals (running requests, in-flight tokens), and objective-relative signals (pool saturation, estimated latency against an SLO).

Architecture

Scaling Pipeline

The scaling pipeline is:

1. Metric emission: EPP exposes its metrics (for example llm_d_epp_flow_control_queue_size and llm_d_epp_request_running) on its metrics endpoint.
2. Metric collection: Prometheus scrapes EPP through the router's ServiceMonitor.
3. Metric evaluation: KEDA's Prometheus scaler evaluates one scalar or single-element PromQL result per trigger. Optionally, spec.advanced.scalingModifiers combines several triggers into one formula.
4. Metric publication: KEDA exposes evaluated scaler values through its metrics server to the Kubernetes External Metrics API.
5. Scaling decision: The HPA generated by KEDA computes the desired replica count and reconciles the target Deployment.

The user creates a ScaledObject, not a second HPA. Multiple HPAs targeting the same Deployment conflict with one another.

Prometheus Adapter is not part of this pipeline because the KEDA Prometheus scaler queries Prometheus directly. The Prometheus Adapter path is deprecated.

Scaling Signals

The pipeline above is signal-agnostic: what changes between the deployable paths is which EPP metrics the triggers read and how thresholds are derived.

Signal	EPP metrics	Threshold is	Guide
In-flight load (default)	llm_d_epp_inflight_requests	An absolute per-replica target (concurrency), tuned per deployment; no Flow Control required	keda-epp (inflight signal)
Queue depth	llm_d_epp_flow_control_queue_size, llm_d_epp_request_running	An absolute per-replica target, tuned per deployment	keda-epp (queue signal)
Pool saturation	llm_d_epp_flow_control_pool_saturation, llm_d_epp_request_running	A normalized ratio (0.0–1.0+), more portable across hardware	keda-epp (saturation signal)
Token backlog	llm_d_epp_inflight_tokens, per-pod KV cache occupancy	Seconds of prefill queue wait, derived from a share of the TTFT SLO and a calibrated peakPrefillThroughput	keda-epp-token-aware
Estimated latency	EPP predicted/actual TTFT and TPOT histograms	Latency ÷ SLO, with a hysteresis band	slo-aware

Queue depth and pool saturation are lagging-to-leading counts of unmet demand. Token backlog exists because request counts rate an 8192-token prompt the same as a 512-token one, while the prefill work differs by 16×. Estimated latency scales against the objective itself rather than a proxy for it; its control law is derived in SLO-Aware Autoscaling with KEDA.

Dual-Metric Strategy

Most of these paths configure two triggers rather than one, so that unmet demand and active concurrency are both represented. The queue-based path is the canonical example:

Metric	Signal	KEDA metric type	Interpretation
EPP Flow Control queue size	Unfulfilled demand	AverageValue	Target aggregate queued requests per model server replica
Running requests	Active concurrency	AverageValue	Target in-flight requests per model server replica

Each AverageValue threshold is a per-replica target. For each metric, the HPA calculates a desired count from the aggregate value and target; roughly, the aggregate must exceed threshold × current replicas to scale beyond the current count. When all configured metrics are available, Kubernetes uses the largest desired replica count.

The SLO-aware path is the exception: it collapses its triggers into a single scalingModifiers formula whose output is the desired replica count, against a pass-through target of 1. Use that shape when the triggers must be combined arithmetically rather than raced against each other.

Tuning each value. For an AverageValue trigger, the threshold is the per-replica load you are willing to carry before adding capacity: lower it to react earlier and hold more headroom (more replicas, higher cost), raise it to pack each replica harder before scaling (fewer replicas, more queueing risk). The replica bounds frame that range: minReplicaCount is the warm floor that keeps a keep-warm replica up and damps flap, and maxReplicaCount is the ceiling the HPA silently caps at, so set it high enough to absorb peak demand. The stabilization windows and cooldownPeriod are covered in Overshoot and Startup-Time Mitigation and Ownership and Scaling Behavior.

Validating the thresholds. These values are not universal. Validate them against representative load for your deployment: drive load up and watch queue growth, pool saturation, and running-request concurrency against your latency objectives, and find the point where adding a replica stops relieving demand - that is where the threshold belongs. Do not assume a value validated for one model, accelerator, or tensor-parallel degree transfers to another; a larger model or a smaller GPU saturates at lower concurrency.

Saturation Detector

The saturation detector estimates how loaded the inference pool is. It gates dispatch under flow control and feeds the llm_d_epp_flow_control_pool_saturation gauge, so it shapes both the queue and saturation signals the keda-epp guide can scale on. EPP ships two detectors that define "loaded" differently:

• utilization-detector (default, recommended) - a closed-loop detector that reacts to real-time telemetry (queue depth and KV-cache pressure), so it reflects actual memory pressure rather than request counts. It is the EPP default, so the guide's router.values.yaml leaves flowControl.saturationDetector unset. It has known limitations under sudden bursts and in heterogeneous pools.
• concurrency-detector - an open-loop detector based on in-flight request accounting. It reacts instantly but treats load as a raw request count, so high concurrency need not mean the pool is full (with prefix caching many concurrent requests can be cheap cache hits), and it is blind to KV-cache pressure. Prefer the default unless you have a specific reason to pin it.

See Saturation Detectors for the full comparison and each detector's tuning knobs.

Flow Control On vs. Off

The keda-epp guide's default path (inflight) scales without EPP flow control; the queue and saturation signals require it. Flow control changes where unmet demand accumulates, which changes what each trigger can see:

• Flow control on (the queue and saturation signals). Flow control is work-conserving: it dispatches whenever the pool has spare capacity and pauses dispatch only once pool saturation reaches the band ceiling (full saturation under the default policy), buffering further requests in its priority queues. Which trigger drives scale-up therefore depends on the load regime. Below the saturation ceiling nothing queues, llm_d_epp_flow_control_queue_size stays at zero, and a single replica can run far more concurrent requests than the running-request threshold (16), so the running-request trigger is the signal that moves and can drive scale-up on its own. Only once saturation reaches the ceiling does unmet demand surface as queue size, where the queue-size trigger and its low default threshold (1) react promptly. The 16 is a scaling threshold, not a per-endpoint admission ceiling; flow control does not cap concurrency per replica. Exercise the path at a concurrency high enough to actually saturate a replica, since moderate load is dispatched with the queue at zero.
• Flow control off (the guide's default inflight path). The queue and saturation signals (llm_d_epp_flow_control_queue_size and llm_d_epp_flow_control_pool_saturation) are emitted only when the flowControl feature gate is on; with it off those series are not produced and the queue and saturation triggers resolve to no data. The stock optimized-baseline EPP instead emits in-flight-load signals from its inflight-load-producer plugin - llm_d_epp_inflight_requests (in-flight requests, summed across the pool's endpoints) and llm_d_epp_inflight_tokens (token-weighted) - which need no flow control and rise directly with concurrency. The guide's default inflight signal scales on llm_d_epp_inflight_requests with a per-replica AverageValue target, the simplest path and no extra EPP components. (llm_d_epp_request_running is also emitted without flow control and tracks concurrency similarly; the guide uses the inflight-load-producer signals so in-flight requests and the token-weighted inflight_tokens share one producer.) The reactive backlog signals a stock EPP can expose - per_endpoint_queue_size, vLLM num_requests_waiting, and the KV-cache-utilization saturation proxy - stay at zero until a replica is genuinely saturated (at concurrency 40 with the decode running all 40, num_requests_waiting and per_endpoint_queue_size held 0 and KV utilization barely moved), so they trigger late; the in-flight-load signals rise immediately and are the better default.

Overshoot and Startup-Time Mitigation

Model-server pods take minutes to load a model and become Ready, and that startup lag distorts autoscaling two ways:

1. Overshoot. While a new replica is still coming up it serves nothing, so the demand signal it was meant to relieve stays high and the HPA keeps asking for more replicas. The HPA's external-metric math does not, by itself, credit pods that are provisioned but not yet serving.
2. Reaction-interval mismatch. For scaling between 1 and N replicas the HPA reconciles on the kube-controller-manager's sync period (--horizontal-pod-autoscaler-sync-period, about 15s by default) - not KEDA's pollingInterval (default 30s), which controls how often KEDA polls the trigger source and drives scale-to-zero activation. A replica can take minutes to become Ready - on the order of 240s for Qwen3-8B at TP1 - so a reading taken during that gap reflects a replica that has not yet begun to help.

A trigger is just a PromQL query, so it could read the target's replica counts from kube-state-metrics and credit capacity that is provisioned but not yet serving - the SLO-aware path does this with an in-flight r/n credit on the latency signal. The queue path takes the portable route instead: HPA stabilization windows.

The ScaledObject sets both scaleUp.stabilizationWindowSeconds and scaleDown.stabilizationWindowSeconds to 300s. The scale-up window holds recommendations and acts on the most conservative one, so a replica that is still loading gets time to become Ready and relieve demand before another is added; the scale-down window holds capacity across brief dips so the pool does not flap when a slow-starting replica finally absorbs a backlog. Size the scale-up window near the target Deployment's measured cold-start time (the 300s default was validated against a ~240s Qwen3-8B TP1 cold start). Windows need no extra dependencies, but they are blunt: they delay all scale-up equally, not just startup-driven overshoot.

Sharing a Contended Accelerator Budget

Each ScaledObject scales its own Deployment against its own signal, so two pools sharing one GPU budget can both scale up and collectively ask for more accelerators than exist. This is not resolved in the autoscaling loop, and it does not need to be: admission is a scheduling concern, and it is handled one layer down by Kueue.

Every replica pod becomes a Kueue Workload, held at a scheduling gate until quota is free for it. Each model gets a guaranteed floor of GPUs in a shared cohort, lends what it is not using to the others, and preempts to take its floor back when its own demand returns. The HPA is untouched and unannotated, so KEDA keeps sole ownership of the HPA it generates and the autoscaling configuration above is unchanged: over-budget replicas simply stay Pending until quota frees.

See the Kueue-based replica rebalancing guide. It supersedes the experimental replica rebalancer (removed in #2498), which enforced the same budget from above the HPA by patching maxReplicas on annotated HPAs — an approach that requires HPA annotations KEDA does not propagate. The two must never run together.

Metric Isolation

Each PromQL query aggregates to one value and must isolate the EPP/InferencePool associated with the target Deployment. Label values such as namespace, model_name, and inference_pool depend on the deployment and scrape configuration. The running-request metric does not expose inference_pool. Because multiple pools can serve the same model, selectors may also need EPP scrape labels such as service. Verify isolation against live Prometheus series.

Ownership and Scaling Behavior

KEDA owns the generated HPA for the lifetime of the ScaledObject:

• KEDA evaluates trigger activation and handles zero-to-one activation.
• The generated HPA handles scaling between one and N replicas.
• cooldownPeriod controls scale-down to zero; it does not control ordinary HPA scale-down while the minimum replica count is one or greater.
• ScaledObject.spec.advanced.horizontalPodAutoscalerConfig.behavior is copied to the generated HPA and controls stabilization and scaling policies.
• Deleting the ScaledObject removes the generated HPA but does not delete the target Deployment.

Scale to Zero

EPP Flow Control enables scale-from-zero by buffering requests while no model server endpoint is ready. A queued request raises the queue-size metric, KEDA activates the target Deployment, and EPP dispatches the request after a model server becomes Ready. Those requests buffer in EPP's in-memory queues, so a request queued while the pool is at zero is lost if EPP restarts before a model server becomes Ready.

Scale-to-zero should be enabled only after one-to-N scaling is validated. Model cold-start time becomes request latency, and clients must allow enough time for the model to load.

Limitations

• No cross-variant placement. Each ScaledObject scales one Deployment against its own metrics. Pools serving the same model on different hardware scale independently, with no cost-aware preference between them.
• No cost-aware choice of hardware. The autoscaler asks for more of the variant it is pointed at; it will not decide that demand is better served by a cheaper accelerator. Sharing a contended accelerator budget between pools is solved, but below the autoscaler — see Sharing a Contended Accelerator Budget.
• Warmup is paid at every scale-up. Model load and torch-compile time sit between the decision and the capacity. Cutting pod-ready time is usually the highest-leverage tuning available; see the warmup patch discussed in SLO-Aware Autoscaling with KEDA.

Deployment Guides

Start from the workload autoscaling guides, which cover the reusable router values, ScaledObjects, authentication notes, and verification steps for each of the signals in the table above.