Start building with a well-lit path.
llm-d provides Well-Lit Paths—tested deployment recipes for common production patterns. They’re adaptable starting points across models, hardware, and workloads.
The fastest path to state-of-the-art LLM inference on any accelerator.
llm-d is an open-source, Kubernetes-native stack for distributed LLM inference. It runs vLLM, SGLang, and more across your cluster, turning single-node engines into production-grade serving on the hardware you already have.
3x higher output throughput
2x faster TTFT
Up to 70% higher tokens/sec
30% throughput improvement
40% reduction in TTFT and ITL in Google Vertex
Portability across heterogeneous hardware
llm-d runs production LLM inference across GPUs, TPUs, XPUs, CPUs, and emerging NPUs with consistent performance patterns, so teams can deploy once and scale anywhere with predictable cost and behavior.
GPU
Google TPU
Intel XPU
CPU
v0.8.1
Join the community
From first-time users the teams debugging complex deployments, the community is open to everyone.