Skip to main content

Gateway Guides

This directory contains guides for deploying a Kubernetes Gateway as a proxy for the llm-d Router. For more information on this capability, see llm-d Router in Gateway Mode.

Why do you need a Gateway?​

The llm-d Router extends compatible Gateway providers with load balancing optimizations for LLM traffic across model server replicas.

The integration with a Gateway allows self-hosted models to be exposed in a wide variety of network topologies including:

  • Internet-facing services
  • Internal to your cluster
  • Through a service mesh

and take advantage of key Gateway features like:

  • Traffic splitting for incremental rollout of new models
  • TLS encryption of queries and responses

By integrating with a Gateway -- instead of developing an llm-d specific proxy layer -- llm-d can leverage the high performance of mature proxies and take advantage of existing operational tools for managing traffic to services. Compatible Gateway implementations may use proxies like Envoy or other high-performance data planes under the hood.

Supported Gateway Providers​

llm-d requires you select a Gateway implementation that supports the Gateway API Inference Extension. Your infrastructure may provide a default compatible implementation, or you may choose to deploy a gateway implementation onto your cluster.

  • GKE Gateway - GKE's implementation of the Gateway API is through the GKE Gateway controller which provisions Google Cloud Load Balancers for Pods in GKE clusters. The GKE Gateway controller supports weighted traffic splitting, mirroring, advanced routing, multi-cluster load balancing and more. Official GKE Docs.
  • Istio - Istio is an open source service mesh and gateway implementation. It provides a fully compliant implementation of the Kubernetes Gateway API for cluster ingress traffic control. Official Istio docs
  • Agentgateway - Agentgateway is a high-performance, Rust-based AI gateway for LLM, MCP, and A2A workloads that can also serve as a Gateway API and Inference Gateway implementation. Official Agentgateway docs.
  • Envoy AI Gateway - Envoy AI Gateway is an open source project for using Envoy Gateway to handle request traffic from application clients to GenAI services that can also serve as a Gateway API and Inference Gateway implementation. Official Envoy AI Gateway docs.
note

Setting up a Gateway generally requires cluster administration rights.

Other Providers​

For other compatible Gateway implementations not listed above, follow the installation instructions for your selected Gateway provider. Ensure the necessary CRDs for Gateway API and the Gateway API Inference Extension are installed.