Gateway Guides
This directory contains guides for deploying a Kubernetes Gateway as a proxy for the llm-d Router. For more information on this capability, see llm-d Router in Gateway Mode.
Why do you need a Gateway?​
The llm-d Router extends compatible Gateway providers with load balancing optimizations for LLM traffic across model server replicas.
The integration with a Gateway allows self-hosted models to be exposed in a wide variety of network topologies including:
- Internet-facing services
- Internal to your cluster
- Through a service mesh
and take advantage of key Gateway features like:
- Traffic splitting for incremental rollout of new models
- TLS encryption of queries and responses
By integrating with a Gateway -- instead of developing an llm-d specific proxy layer -- llm-d can leverage the high performance of mature proxies and take advantage of existing operational tools for managing traffic to services. Compatible Gateway implementations may use proxies like Envoy or other high-performance data planes under the hood.
Installing Gateway API CRDs​
Before deploying any Gateway provider, see Installing Gateway API CRDs to install the required Gateway API and Gateway API Inference Extension CRDs.
Supported Gateway Providers​
llm-d requires you select a Gateway implementation that supports the Gateway API Inference Extension. Your infrastructure may provide a default compatible implementation, or you may choose to deploy a gateway implementation onto your cluster.
- Istio - Istio is an open source service mesh and gateway implementation. It provides a fully compliant implementation of the Kubernetes Gateway API for cluster ingress traffic control. Official Istio docs
- GKE Gateway - GKE's implementation of the Gateway API is through the GKE Gateway controller which provisions Google Cloud Load Balancers for Pods in GKE clusters. The GKE Gateway controller supports weighted traffic splitting, mirroring, advanced routing, multi-cluster load balancing and more. Official GKE Docs.
- Agentgateway - Agentgateway is a high-performance, Rust-based AI gateway for LLM, MCP, and A2A workloads that can also serve as a Gateway API and Inference Gateway implementation. Official Agentgateway docs.
- Envoy AI Gateway - Envoy AI Gateway is an open source project for using Envoy Gateway to handle request traffic from application clients to GenAI services that can also serve as a Gateway API and Inference Gateway implementation. Official Envoy AI Gateway docs.
- NGINX Gateway Fabric - NGINX Gateway Fabric is an open source project that provides an implementation of the Kubernetes Gateway API using NGINX as the data plane. Official NGINX Gateway Fabric docs.
Gateway Guide Maintainers​
Maintainers for each gateway guide are listed below. Each guide requires a named maintainer responsible for keeping its contents up to date.
| Gateway | Guide | Maintainers |
|---|---|---|
| GKE Gateway | GKE Gateway | TBD |
| Istio | Istio | TBD |
| Agentgateway | Agentgateway | TBD |
| Envoy AI Gateway | Envoy AI Gateway | TBD |
| NGINX Gateway Fabric | NGINX Gateway Fabric | Marko Sluga (@markosluga) |
Setting up a Gateway generally requires cluster administration rights.
Other Providers​
For other compatible Gateway implementations not listed above, follow the installation instructions for your selected Gateway provider. Ensure the necessary CRDs for Gateway API and the Gateway API Inference Extension are installed.