Architecture

High-level guide to llm-d architecture. The llm-d architecture is organized into seven core functional areas:

1. Router

The llm-d Router is the intelligent entry point for inference requests, providing LLM-aware load balancing, request queuing, and policy enforcement without reimplementing a full network proxy:

• InferencePool: The Kubernetes custom resource that bridges Gateway routing with backend model servers via label selectors and ext-proc attachment.
• Proxy: A high-performance L7 proxy conformant with the Gateway API Inference Extension (GAIE) that accepts user requests and consults the EPP via the ext-proc protocol.
• Endpoint Picker (EPP): The routing engine that scores and selects model server pods based on real-time metrics, KV-cache affinity, and configured policies.
• Request Handling: Request parsing, header extraction, and admission control.
• Flow Control: Multi-priority queuing, fair-share scheduling, and proactive overload protection.
• Request Scheduling: Extensible Filter-Score-Pick pipeline selecting optimal candidate endpoints.
• Data Layer: Real-time telemetry ingestion from model servers and Kubernetes APIs.
• EPP Configuration: Declarative YAML configuration schema and plugin setup.
• Latency Predictor: Online ML regression models predicting TTFT and TPOT to enforce latency SLOs.

See Router for full details.

2. Model Servers

The Model Servers layer consists of accelerator-native inference engines that load model weights and execute forward passes:

• vLLM: Default engine featuring PagedAttention, native tiered offloading, NixlConnector RDMA for P/D disaggregation, and wide expert parallelism.
• SGLang: High-performance engine featuring RadixAttention, HiCache hierarchical caching, prefill bootstrap room coordination, and Mooncake/Nixl transfer backends.
• TensorRT-LLM (trtllm-serve): NVIDIA's high-throughput engine with optimized In-Flight Batching and specialized kernels.

See Model Servers for telemetry protocols, metric specifications, and dynamic LoRA serving.

3. Disaggregation

The Disaggregation section covers decoupling compute-bound and memory-bound phases as well as scaling mixture-of-experts models across multi-node slices:

• Disaggregated Serving: Separating prefill (compute-bound) and decode (memory-bound) stages across specialized workers with high-speed RDMA KV transfer.
• Wide Expert Parallelism: Distributed serving for large Mixture-of-Experts (MoE) models combining data-parallel attention, expert-parallel MLP layers, and rank-aware routing.

See Disaggregation for full details.

4. KV Cache Management

llm-d provides a comprehensive ecosystem for managing and reusing the KV cache across the inference pool:

• Prefix-Cache Aware Routing: Heuristic and precise techniques to maximize cache hits.
• KV-Cache Indexing: Event-driven tracking of cache state across all model servers.
• KV Offloading: Tiered storage hierarchy (CPU, SSD) for extending cache capacity.
• P2P KV-Cache Sharing: Pulling cached prefix KV blocks from a peer's CPU tier instead of recomputing them.

See KV Cache Management for full details.

5. Model Server Orchestration

The Model Server Orchestration layer provides Kubernetes custom resources for managing distributed model groups and synchronized serving topologies:

• LeaderWorkerSet (LWS): Kubernetes SIG workload controller for deploying multi-node accelerator pod groups with leader-worker topology, gang scheduling, and all-or-nothing restart semantics.
• DisaggregatedSet: Workload controller orchestrating multi-role disaggregated serving topologies (such as prefill and decode) as synchronized, versioned slices.

See Model Server Orchestration for full details.

6. Autoscaling

llm-d supports proactive, SLO-aware autoscaling driven by metrics exported by the EPP:

• KEDA + EPP Metrics: Scaling on real-time queuing signals (queue depth, pool saturation, token backlog) using KEDA Prometheus triggers.
• SLO-Aware Autoscaling: Closed-loop control scaling replicas based on estimated latency headroom against SLO targets.
• Workload Variant Autoscaler (WVA): Global multi-variant optimization across heterogeneous hardware (deprecated).

See Autoscaling for full details.

7. Batch and Async Serving

Batch and offline inference workloads are handled by two modular components that can be deployed independently or together:

• Batch Gateway: An OpenAI-compatible Batch API (/v1/batches, /v1/files) for submitting, tracking, and managing batch inference jobs.
• Async Processor: A lightweight dispatch agent that pulls requests from message queues (Redis, Pub/Sub) and meters dispatch against flow control.

See Batch and Async Serving for full details.