Multi-Inference Pool Setup

This guide adds additional InferencePools to an existing optimized-baseline deployment. Each additional pool gets its own EPP and model server Deployment in the same namespace. Repeat the steps below for every pool you want to add.

Prerequisites

Complete the optimized-baseline guide. At the end of that guide you should have one Helm release (optimized-baseline), one InferencePool, one EPP, and model server pods running in the llm-d-optimized-baseline namespace.

note

InferencePools can also be deployed in separate namespaces.

Step 1: Deploy an Additional Helm Release

Install an additional Helm release in the same namespace as the optimized-baseline. Each release must use a unique matchLabels selector so its InferencePool discovers only the correct model's pods. The example below adds a pool called model-b; repeat with a different release name and values file for every additional pool.

Set the guide environment variables:

export BRANCH=main
export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export NAMESPACE=llm-d-optimized-baseline
export RELEASE=model-b
export VALUES=model-b.values.yaml

Source the common guide environment variables:

source ${REPO_ROOT}/guides/env.sh

Install the additional release:

helm install ${RELEASE} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${REPO_ROOT}/guides/recipes/router/base.values.yaml \
  -f ${REPO_ROOT}/guides/workload-autoscaling/multi-inference-pool/${VALUES} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

warning

The standalone chart creates a ConfigMap named envoy with a hardcoded name (not prefixed with the release name). Installing another release in the same namespace will fail with an ownership conflict on this ConfigMap. To work around this, reassign the ConfigMap's Helm ownership annotations to the new release before installing it:

kubectl annotate configmap envoy -n ${NAMESPACE} \
  meta.helm.sh/release-name=model-b meta.helm.sh/release-namespace=${NAMESPACE} --overwrite
kubectl label configmap envoy -n ${NAMESPACE} \
  app.kubernetes.io/managed-by=Helm --overwrite

The values file sets a unique pool selector via router.modelServers.matchLabels. See model-b.values.yaml for an example. Create a similar values file for each additional pool, ensuring every pool uses a distinct matchLabels selector so InferencePools do not cross-select each other's pods.

note

Replace model-b with your actual model identifier in the values file.

Step 2: Deploy the Model Server

Deploy the model server for the new pool the same way as the optimized-baseline, with its Kustomize overlay setting the matching llm-d.ai/model label. Ensure the Deployment's pod template labels match the matchLabels in the corresponding Helm values file. If they don't match, the InferencePool will not discover the pods and the EPP will have no endpoints to route to.

Verification

kubectl get inferencepools,svc -n ${NAMESPACE}
kubectl get pods -n ${NAMESPACE} --show-labels

Configuring Autoscaling

Once the additional pools are deployed, configure one scaler per target Deployment. Either scaling path can be used:

• KEDA + EPP Metrics: Create one KEDA ScaledObject per target Deployment. Each PromQL query must isolate the EPP/InferencePool associated with that Deployment using the labels currently exposed by EPP and by the Prometheus scrape target.

  Label availability differs by metric. The Flow Control queue-size metric may include InferencePool-level labels, while llm_d_epp_request_running is labeled by model, target model, fairness ID, and priority rather than inference_pool. When multiple EPP Services expose series for the same model, include scrape labels such as service to isolate the intended EPP deployment. Do not rely on model_name alone when multiple pools can serve the same model. This is a current EPP metric-label limitation.

• HPA + WVA Metrics: Create one HPA per model using the wva_desired_replicas metric. Each HPA must carry the WVA discovery annotations (llm-d.ai/managed, llm-d.ai/model-id, llm-d.ai/variant-cost).

Cleanup

Uninstall each additional release you added:

helm uninstall ${RELEASE} -n ${NAMESPACE}