Version: dev

GKE Gateway

This guide shows how to deploy llm-d with GKE Gateway as your inference gateway. By the end, inference requests will be forwarded by a GKE-managed Gateway to your model servers via the llm-d EPP.

note

This guide assumes familiarity with Gateway API and llm-d.

Prerequisites

The environment variables ${GUIDE_NAME}, ${MODEL} and ${NAMESPACE} should be set as part of deploying one of the well-lit path guides.

You have verified the version in and sourced in the llm-d repo wide env.sh file:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
source ${REPO_ROOT}/guides/env.sh

The following steps from the GKE Inference Gateway deployment documentation should be run:

Step 1: Install Gateway API and Gateway API Inference Extension CRDs

note

On GKE 1.34.0-gke.1626000 or later, with the Gateway API enabled as required above, GKE installs and manages the Gateway API Standard-channel CRDs and the v1 InferencePool CRD: skip to Step 2. The release GKE manages can trail the one the guides pin; see Google's GKE Inference Gateway page for what GKE manages before applying a newer InferencePool CRD yourself.

On GKE versions earlier than 1.34.0-gke.1626000, install the Inference Extension CRDs:

# GAIE_URL is automatically calculated from GAIE_VERSION at ${REPO_ROOT}/guides/env.sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Verify the APIs are available:

kubectl api-resources --api-group=gateway.networking.k8s.io
kubectl api-resources --api-group=inference.networking.k8s.io

Step 2: Deploy the Gateway

GKE supports two types of Application Load Balancers. Choose the one that matches your network requirements:

Regional External Application Load Balancer

For internet-facing workloads accessible from outside your VPC:

LLM_D_VERSION=main  # Use 'main' for latest, or a release tag like 'v0.7.0'
GATEWAY_CLASS=gke-l7-regional-external-managed
kubectl apply -n ${NAMESPACE} -k "https://github.com/llm-d/llm-d/guides/recipes/gateway/${GATEWAY_CLASS}?ref=${LLM_D_VERSION}"

Regional Internal Application Load Balancer

For private workloads accessible only within your VPC:

LLM_D_VERSION=main  # Use 'main' for latest, or a release tag like 'v0.7.0'
GATEWAY_CLASS=gke-l7-rilb
kubectl apply -n ${NAMESPACE} -k "https://github.com/llm-d/llm-d/guides/recipes/gateway/${GATEWAY_CLASS}?ref=${LLM_D_VERSION}"

Step 3: Verify the Gateway

Verify the Gateway is programmed:

kubectl get gateway -n ${NAMESPACE} llm-d-inference-gateway

Expected output:

NAME                      CLASS                              ADDRESS         PROGRAMMED   AGE
llm-d-inference-gateway   gke-l7-regional-external-managed   xx.xx.xx.xx     True         30s

Wait until PROGRAMMED shows True before proceeding.

Step 4: Deploy a Model Serving Workload

Before sending inference requests through the Gateway, you must deploy a model server (such as vLLM), an InferencePool, and an HTTPRoute connecting the Gateway to the pool. You can deploy using a standard baseline or enable High Availability.

Option A: Standard Deployment (Well-Lit Path)

Deploy one of the well-lit path guides with the GKE Gateway provider enabled:

Deploy the llm-d Router in Gateway mode:

export PROVIDER_NAME=gke
helm install ${GUIDE_NAME} \
  ${ROUTER_GATEWAY_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  --set provider.name=${PROVIDER_NAME} \
  --set httpRoute.create=true \
  --set httpRoute.inferenceGatewayName=llm-d-inference-gateway \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

Deploy the model server with the GKE overlay:

export ACCELERATOR_TYPE=gpu # options: gpu, tpu/v6, tpu/v7, cpu
export MODEL_SERVER=vllm    # options: vllm, sglang, trtllm
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/gke/

Option B: High Availability with Preferred Backends

Deploying Endpoint Picker (EPP) with GKE Preferred Backends enables an active-passive routing topology backed by Cloud Load Balancing. Steady-state ext_proc traffic routes exclusively to the primary EPP replica (epp-0) to keep KV-cache tracking and request state centralized. If epp-0 becomes unhealthy, Cloud Load Balancing shifts traffic to the warm standby replica (epp-1), with InferencePool fail-open mode providing an additional safety net against dropped requests.

Key Benefits

State Consistency: Directs 100% of steady-state ext_proc traffic to the primary EPP instance (epp-0), preserving KV-cache state and request scheduling context.
Zero-Downtime Failover: If the primary EPP pod crashes or undergoes maintenance, Cloud Load Balancer detects the failure via active gRPC health checks and immediately routes traffic to the warm standby replica (epp-1).
Fail-Open Resilience: Paired with failureMode: FailOpen on the InferencePool, transient routing blips bypass EPP and forward directly to model servers without dropping user requests.

1. Deploy the llm-d Router with Preferred Backends enabled:

note

When Preferred Backends is enabled, the chart deploys a single StatefulSet where pod ordinals map to target priority tiers and generates corresponding GCPBackendPolicy resources for each tier.

export PROVIDER_NAME=gke
helm install ${GUIDE_NAME} \
  ${ROUTER_GATEWAY_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  --set provider.name=${PROVIDER_NAME} \
  --set provider.gke.preferredBackends.enabled=true \
  --set provider.gke.preferredBackends.preferredReplicas=1 \
  --set provider.gke.preferredBackends.defaultReplicas=1 \
  --set httpRoute.create=true \
  --set httpRoute.inferenceGatewayName=llm-d-inference-gateway \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

Key configuration parameters (see values.yaml):

preferredReplicas (Default: 1): the replica count for active primary pod ordinals pinned to the PREFERRED backend preference tier (epp-0). Setting a value greater than 1 scales concurrent active routing capacity.
defaultReplicas (Default: 1): the replica count for standby pod ordinals pinned to the DEFAULT backend preference tier (epp-1). Setting a value greater than 1 scales warm standby failover capacity.
balancingMode (Default: RATE): the calculation mode used to determine load thresholds (RATE, UTILIZATION, or CONNECTION).
maxRatePerEndpoint (Default: 100): the maximum requests per second per pod instance before spilling over to standby tiers.
capacityScalerPercent (Default: 100): the effective capacity ceiling percentage (0 to 100).

2. Deploy the model server with the GKE overlay:

export ACCELERATOR_TYPE=gpu # options: gpu, tpu/v6, tpu/v7, cpu
export MODEL_SERVER=vllm    # options: vllm, sglang, trtllm
kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/gke/

3. Discover and attach the primary Preferred Network Endpoint Groups (NEGs) across zones to the generated Google Cloud `BackendService`:

note

The InferencePool resource natively targets the standby backup Service (Service/${GUIDE_NAME}-epp-backup). Anchoring declarative Gateway ownership to the standby Service ensures that GKE Gateway Controller reconciliation loops do not detach manually attached primary NEGs during pod restarts or updates.

export PROJECT="<your GCP project>"
export REGION="<your GCP region>"
export NAMESPACE="<your namespace>"
export GUIDE_NAME="<llm-d guide name>"

export BACKEND_SERVICE=$(gcloud compute backend-services list --project=${PROJECT} --format="value(name)" | grep "${GUIDE_NAME}-epp")

export PRIMARY_NEG=$(kubectl get svc ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.metadata.annotations.cloud\.google\.com/neg-status}' | jq -r '.network_endpoint_groups["9002"]')

for ZONE in $(kubectl get svc ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.metadata.annotations.cloud\.google\.com/neg-status}' | jq -r '.zones[]'); do
  gcloud compute backend-services add-backend ${BACKEND_SERVICE} \
    --network-endpoint-group=${PRIMARY_NEG} \
    --network-endpoint-group-zone=${ZONE} \
    --region=${REGION} \
    --project=${PROJECT} \
    --balancing-mode=RATE \
    --max-rate-per-endpoint=100
  sleep 15
done

info

Once zonal NEGs are attached to the Google Cloud BackendService, scaling replica counts (preferredReplicas or defaultReplicas) or restarting pods does not require running gcloud commands again. The GKE NEG Controller automatically synchronizes individual pod IP endpoints into the registered zonal NEGs.

If you delete and recreate the parent Gateway or InferencePool resource, the GKE Gateway Controller recreates the underlying Google Cloud BackendService with a new cloud identifier. In that event, execute the attachment workflow above to attach primary zonal NEGs to the newly created BackendService.

Step 5: Send a Request

Get the Gateway external address:

export IP=$(kubectl get gateway llm-d-inference-gateway -n ${NAMESPACE} -o jsonpath='{.status.addresses[0].value}')

Send an inference request via the managed Gateway:

curl -X POST http://${IP}/v1/completions \
    -H 'Content-Type: application/json' \
    -d '{
        "model": '\"${MODEL}\"',
        "prompt": "How are you today?"
    }' | jq

Monitoring

Cloud Monitoring provides the out-of-the-box llm-d Overview dashboard (Monitoring → Dashboards → Integrations → llm-d), which includes Application Load Balancer metrics alongside GPU networking, llm-d EPP, and vLLM metrics; see GKE Monitoring & Telemetry.

Cleanup

kubectl delete gateway llm-d-inference-gateway -n ${NAMESPACE}

Troubleshooting

Gateway not showing `PROGRAMMED=True`

kubectl describe gateway llm-d-inference-gateway -n ${NAMESPACE}

Verify all prerequisites were applied, especially enabling Gateway API in your cluster and configuring a proxy-only subnet. Also make sure the cluster is running a supported GKE version.

HTTPRoute not accepted

kubectl describe httproute ${GUIDE_NAME} -n ${NAMESPACE}

Verify that parentRefs matches the Gateway name and backendRefs matches the InferencePool name.

No response from Gateway IP

kubectl get gateway llm-d-inference-gateway  -n ${NAMESPACE} -o jsonpath='{.status.addresses[0].value}'

If the address is empty, your Gateway may still be waiting for a LoadBalancer service. Check that your cluster supports external load balancers.

Getting `fault filter abort` response

A couple of issues may cause this:

The request doesn't match the routing rules setup on HTTPRoute.
A misconfiguration in the gateway's backend routing. When configured correctly, the HTTPRoute status should have a condition of type Reconciled and reason ReconciliationSucceeded.

Prerequisites​

Step 1: Install Gateway API and Gateway API Inference Extension CRDs​

Step 2: Deploy the Gateway​

Regional External Application Load Balancer​

Regional Internal Application Load Balancer​

Step 3: Verify the Gateway​

Step 4: Deploy a Model Serving Workload​

Option A: Standard Deployment (Well-Lit Path)​

Option B: High Availability with Preferred Backends​

Key Benefits​

1. Deploy the llm-d Router with Preferred Backends enabled:​

2. Deploy the model server with the GKE overlay:​

3. Discover and attach the primary Preferred Network Endpoint Groups (NEGs) across zones to the generated Google Cloud BackendService:​

Step 5: Send a Request​

Monitoring​

Cleanup​

Troubleshooting​

Gateway not showing PROGRAMMED=True​

HTTPRoute not accepted​

No response from Gateway IP​

Getting fault filter abort response​

Prerequisites

Step 1: Install Gateway API and Gateway API Inference Extension CRDs

Step 2: Deploy the Gateway

Regional External Application Load Balancer

Regional Internal Application Load Balancer

Step 3: Verify the Gateway

Step 4: Deploy a Model Serving Workload

Option A: Standard Deployment (Well-Lit Path)

Option B: High Availability with Preferred Backends

Key Benefits

1. Deploy the llm-d Router with Preferred Backends enabled:

2. Deploy the model server with the GKE overlay:

3. Discover and attach the primary Preferred Network Endpoint Groups (NEGs) across zones to the generated Google Cloud `BackendService`:

Step 5: Send a Request

Monitoring

Cleanup

Troubleshooting

Gateway not showing `PROGRAMMED=True`

HTTPRoute not accepted

No response from Gateway IP

Getting `fault filter abort` response