Skip to main content
Version: dev

NGINX Gateway Fabric

This guide shows how to deploy llm-d with NGINX Gateway Fabric as your inference gateway. By the end, inference requests will flow from an NGINX-managed Gateway to your model servers via the llm-d EPP.

note

This guide assumes familiarity with Gateway API and llm-d.

Prerequisites​

  1. The environment variables ${GUIDE_NAME}, ${MODEL_NAME} and ${NAMESPACE} should be set as part of deploying one of the well-lit path guides.
  2. A Kubernetes cluster running one of the three most recent Kubernetes releases
  3. Helm
  4. jq

Step 1: Install Gateway API and Gateway API Inference Extension CRDs​

Install the required CRDs by following the CRD installation guide.

Step 2: Install NGINX Gateway Fabric​

Install NGINX Gateway Fabric with the Gateway API Inference Extension enabled:

helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
  --create-namespace \
  --namespace nginx-gateway \
  --set nginxGateway.gwAPIInferenceExtension.enable=true \
  --wait

If you prefer to pin the version of your NGINX Gateway Fabric deployment you can set the version as a variable and use it with --version during helm install:

NGF_VERSION=2.7.2

Install NGINX Gateway Fabric with the Gateway API Inference Extension enabled:

helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
  --version ${NGF_VERSION} \
  --create-namespace \
  --namespace nginx-gateway \
  --set nginxGateway.gwAPIInferenceExtension.enable=true \
  --wait

Verify the installation:

kubectl get pods -n nginx-gateway
kubectl get gatewayclass nginx

Expected output:

NAME    CONTROLLER                                   ACCEPTED   AGE
nginx   gateway.nginx.org/nginx-gateway-controller   True       30s

Step 3: Deploy the Gateway​

Set the llm-d version to match your deployment:

LLM_D_VERSION=main  # Use 'main' for latest, or a release tag like 'v0.7.0'

Deploy a Gateway using the nginx GatewayClass:

kubectl apply -k "https://github.com/llm-d/llm-d/guides/recipes/gateway/nginx-gateway-fabric?ref=${LLM_D_VERSION}" -n ${NAMESPACE}

Verify the Gateway is programmed:

kubectl get gateway llm-d-inference-gateway -n ${NAMESPACE}

Expected output:

NAME                      CLASS   ADDRESS         PROGRAMMED   AGE
llm-d-inference-gateway   nginx   10.xx.xx.xx     True         30s

Wait until PROGRAMMED shows True before proceeding.

Step 4: Send a Request​

info

Before sending requests, you must deploy a well-lit path guide. This sets up a model server deployment, an InferencePool, and an HTTPRoute to connect the Gateway to the pool.

Get the Gateway external address:

export IP=$(kubectl get gateway llm-d-inference-gateway -n ${NAMESPACE} -o jsonpath='{.status.addresses[0].value}')

Send an inference request via the managed Gateway:

curl -X POST http://${IP}/v1/completions \
    -H 'Content-Type: application/json' \
    -d '{
        "model": '\"${MODEL_NAME}\"',
        "prompt": "How are you today?"
    }' | jq

Cleanup​

kubectl delete gateway llm-d-inference-gateway -n ${NAMESPACE}
helm uninstall ngf -n nginx-gateway
kubectl delete namespace nginx-gateway
kubectl delete -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/main/deploy/crds.yaml

To uninstall the Gateway API and Gateway API Inference Extension CRDs, see the CRD installation guide.

Troubleshooting​

Gateway not showing PROGRAMMED=True​

kubectl describe gateway llm-d-inference-gateway -n ${NAMESPACE}
kubectl logs -n nginx-gateway deployment/ngf-nginx-gateway-fabric
kubectl logs -n ${NAMESPACE} deployment/llm-d-inference-gateway-nginx

Verify the `nginx` `GatewayClass` is present and accepted:

```bash
kubectl get gatewayclass nginx

HTTPRoute not accepted​

kubectl describe httproute ${GUIDE_NAME} -n ${NAMESPACE}

Verify that parentRefs matches the Gateway name and backendRefs matches the InferencePool name.

No response from Gateway IP​

kubectl get gateway llm-d-inference-gateway -n ${NAMESPACE} -o jsonpath='{.status.addresses[0].value}'

If the address is empty, your Gateway may still be waiting for a LoadBalancer service. Check that your cluster supports external load balancers.

See also​