Skip to main content

llm-d-infra

This repository provides the Kubernetes infrastructure components, Helm charts, and operational tooling for deploying llm-d - a Kubernetes-native high-performance distributed LLM inference framework.

What This Repository Contains​

Infrastructure Components:

  • Helm charts for deploying llm-d gateway infrastructure
  • Kubernetes Gateway API configurations with support for Istio, kGateway, and GKE
  • Service mesh integration and traffic management policies

Operational Tooling:

  • Interactive benchmarking and testing utilities
  • Smoke testing scripts for deployment validation
  • Development helpers and automation tools

Quick Start​

Prerequisites​

  • Kubernetes 1.28+ cluster
  • Helm 3.10+
  • Gateway API v1.3.0+ installed
  • Gateway controller (Istio, kGateway, or GKE) deployed in your cluster

Install llm-d Infrastructure​

# Add the Helm repository
helm repo add llm-d-infra https://llm-d-incubation.github.io/llm-d-infra/
helm repo update

# Install the infrastructure components
helm install my-llm-d-infra llm-d-infra/llm-d-infra

For detailed configuration options, see the Helm chart documentation.

Documentation and Guides​

Note: All quickstart guides and user documentation have moved to the main llm-d repository:

This repository contains:

Contributing​

  1. Issues and Features: Report issues or request features in the main llm-d repository
  2. Infrastructure Changes: Submit pull requests to this repository for:
    • Helm chart improvements
    • Operational tooling enhancements
    • Infrastructure configuration updates
  3. Development Setup: See development documentation

Code Owners​

See CODEOWNERS for component-specific maintainers.

Releases​

License​

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

Content Source

This content is automatically synced from README.md in the llm-d-incubation/llm-d-infra repository.

📝 To suggest changes, please edit the source file or create an issue.