Skip to main content
Version: dev

KEDA with EPP Metrics

The Endpoint Picker (EPP) and KEDA integration scales model server replicas using demand-side signals from EPP. Rather than relying on coarse resource utilization, KEDA queries EPP queue and request metrics from Prometheus, exposes evaluated scaler values through its metrics server, and creates an HPA that consumes those external metrics. This approach is well-suited for homogeneous deployments where each target model-server pool can be isolated by metrics and scaled independently.

Functionality​

The KEDA+EPP integration provides:

  • Demand-driven scaling: Replicas are added or removed based on EPP queue depth and active request count, which reflect inference demand rather than accelerator utilization.
  • Standard Kubernetes actuation: KEDA owns an autoscaling/v2 HPA, so HPA status and behavior remain available through standard Kubernetes APIs.
  • Optional scale-to-zero: KEDA can remove all model server replicas when traffic is idle and reactivate the Deployment from an EPP Flow Control queue.

Design​

The LLM Autoscaling Problem​

During active token generation, GPU utilization can remain high whether a model server is lightly loaded or saturated. A utilization-based autoscaler cannot reliably distinguish spare serving capacity from an overloaded batch.

EPP Flow Control moves excess demand to the gateway. Its llm_d_epp_flow_control_queue_size metric directly counts requests waiting for backend capacity, while llm_d_epp_request_running measures active running requests. Together they provide reactive saturation and proactive concurrency signals.

Scaling Pipeline​

The scaling pipeline is:

  1. Metric emission: EPP exposes llm_d_epp_flow_control_queue_size and llm_d_epp_request_running on its metrics endpoint.
  2. Metric collection: Prometheus scrapes EPP through the router's ServiceMonitor.
  3. Metric evaluation: KEDA's Prometheus scaler evaluates one scalar or single-element PromQL result per trigger.
  4. Metric publication: KEDA exposes evaluated scaler values through its metrics server to the Kubernetes External Metrics API.
  5. Scaling decision: The HPA generated by KEDA computes the desired replica count and reconciles the target Deployment.

The user creates a ScaledObject, not a second HPA. Multiple HPAs targeting the same Deployment conflict with one another.

Dual-Metric Strategy​

Using both metrics provides complementary scaling behavior:

MetricSignalKEDA metric typeInterpretation
EPP Flow Control queue sizeUnfulfilled demandAverageValueTarget aggregate queued requests per model server replica
Running requestsActive concurrencyAverageValueTarget in-flight requests per model server replica

Each AverageValue threshold is a per-replica target. For each metric, the HPA calculates a desired count from the aggregate value and target; roughly, the aggregate must exceed threshold × current replicas to scale beyond the current count. When all configured metrics are available, Kubernetes uses the largest desired replica count.

Each PromQL query aggregates to one value and must isolate the EPP/InferencePool associated with the target Deployment. Label values such as namespace, model_name, and inference_pool depend on the deployment and scrape configuration. The running-request metric does not expose inference_pool. Because multiple pools can serve the same model, selectors may also need EPP scrape labels such as service. Verify isolation against live Prometheus series.

Ownership and Scaling Behavior​

KEDA owns the generated HPA for the lifetime of the ScaledObject:

  • KEDA evaluates trigger activation and handles zero-to-one activation.
  • The generated HPA handles scaling between one and N replicas.
  • cooldownPeriod controls scale-down to zero; it does not control ordinary HPA scale-down while the minimum replica count is one or greater.
  • ScaledObject.spec.advanced.horizontalPodAutoscalerConfig.behavior is copied to the generated HPA and controls stabilization and scaling policies.
  • Deleting the ScaledObject removes the generated HPA but does not delete the target Deployment.

Prometheus Adapter is not part of this pipeline because the KEDA Prometheus scaler queries Prometheus directly.

Scale to Zero​

EPP Flow Control enables scale-from-zero by buffering requests while no model server endpoint is ready. A queued request raises the queue-size metric, KEDA activates the target Deployment, and EPP dispatches the request after a model server becomes Ready.

Scale-to-zero should be enabled only after one-to-N scaling is validated. Model cold-start time becomes request latency, and clients must allow enough time for the model to load.

info

EPP Flow Control queues are stored in memory. Requests in the queue are lost if EPP restarts. Plan EPP availability and client retries accordingly.

Deployment Guide​

See the KEDA + EPP workload-autoscaling guide for the reusable router values, ScaledObject, authentication notes, and verification steps.