Skip to main content
Version: dev

Core Capability Building Blocks

Core Capability Building Blocks represent the individual functional optimization, intelligent routing, and physical inference execution features of llm-d.

These guides teach single architectural capabilities that you can configure independently or compose together into comprehensive production workloads.

Intelligent Routing​

  • Optimized Baseline: Strategies for handling the unique challenges of LLM request scheduling, moving beyond traditional round-robin approaches.
  • Predicted Latency-Based Routing: Using online-trained machine learning models to predict latency and optimize scheduling.

Advanced KV-Cache Management​

  • Precise Prefix Cache Routing: Near-real-time routing based on exact cache state published by model servers.
  • Tiered Prefix Cache: Efficiently managing KV caches by offloading to CPU RAM, NVMe, or network storage to improve prefix-cache re-use.
  • Enable P2P Prefix Cache Sharing: Pulling cached prefix KV blocks directly from a peer's CPU offload tier instead of recomputing them, turning per-pod prefix caches into a fleet-wide resource.

Serving Large Models​

Multimodal and Omni Models​

  • Serve Multimodal Models: Routing image, video, and audio requests on prefix-cache affinity that covers the media as well as the text, with aggregated serving or dedicated Encode workers (E/PD, E/P/D).
  • Serve Omni Models: Serving a model that answers in text and audio from one vLLM-Omni pool behind the llm-d Router, or a text-to-image, image-to-image, or text-to-speech model on vLLM-Omni or SGLang.

Stateful APIs​

  • Serve the Stateful OpenAI Responses API (Experimental): Serving /v1/responses with conversation state kept in PostgreSQL by vLLM Agentic API in front of the llm-d Router, so agentic clients and coding harnesses continue a conversation by previous_response_id, run tool loops, and stream over WebSocket.