Skip to main content

🎉 llm-d 0.5 is now released! Check out hierarchical KV offloading, cache-aware LoRA routing, resilient networking with UCCL, and scale-to-zero autoscaling. Read the announcement →

Architecture Guides Usage Community Blog Videos

Recent posts

2026

Native KV Cache Offloading to Any Filesystem with llm-d
llm-d 0.5: Sustaining Performance at Scale

2025

llm-d 0.4: Achieve SOTA Performance Across Accelerators
llm-d 0.3: Wider Well-Lit Paths for Scalable Inference
KV-Cache Wins You Can See: From Prefix Caching in vLLM to Distributed Scheduling with llm-d

Tags

A

Announcements4

B

blog posts3

C

Community1

H

Hello1

K

KV Cache1

L

llm-d release news7

N

News Releases2

R

Releases4

S

SIG-Benchmarking1
Storage1

U

Updates3

W

Welcome!1

Architecture

Overview
Latest Release
Inference Scheduler
KV Cache
Model Service
Benchmark Tools

Guides

Getting Started
Prerequisites
Inference Scheduling
Prefill/Decode Disaggregation
Wide Expert Parallelism

Community

Contact us
Contributing
Code of Conduct

More

Blog
Privacy Policy

Social

Join our Slack