2 posts tagged with "vLLM"

Running multi-trillion-parameter Mixture-of-Experts (MoE) models in production shifts the primary engineering bottleneck from model weights to distributed systems orchestration. With Moonshot AI's Kimi K3.1 Pro (moonshotai/Kimi-K3.1-Pro), infrastructure engineers must accommodate a ~2.8-trillion-parameter footprint (~104B active parameters per token across 896 routed experts), a 1M-token context window, and a hybrid 93-layer attention stack (69 Kimi Delta Attention layers paired with 24 Gated Multi-head Latent Attention layers).

At this footprint, the model weights alone surpass the High-Bandwidth Memory (HBM) capacity of any single machine, requiring multi-node Tensor and Expert Parallelism across high-bandwidth interconnects. Compounding the hardware footprint, long-horizon agentic sessions expose the limits of stateless Kubernetes load balancing: 1M-token KV caches rapidly exhaust accelerator HBM, uncoordinated routing forces replicas to recompute identical multi-turn prefixes, and compute-heavy prefill bursts stall ongoing token generation.

To eliminate these bottlenecks at launch, we worked with Inferact (vLLM) and RadixArk (SGLang) to deliver Day-0 distributed serving recipes on llm-d. Running llm-d as a unified, Kubernetes-native inference control plane, the same stack serves Kimi K3.1 Pro across both TPUs (Ironwood TPU v7x) and NVIDIA GPUs (GB300 NVL72), here on Google Cloud (A4X-Max). Because the control plane is accelerator- and cloud-independent, platform teams gain a portable serving stack that handles state-aware KV routing, hierarchical memory offloading, and prefill/decode disaggregation out of the box.

Text inference has a comfortable shape in llm-d. A request arrives as a few thousand tokens, prefill and decode run back to back on one replica, and a prefix-aware router keeps the KV cache warm. Every layer of the stack — scheduler, load balancer, autoscaler — is calibrated against that shape.

Then someone attaches an image, and it is tempting to think of the result as the same request with a picture bolted on. It is not. A multimodal request differs from a text-only one in almost every property a serving system actually reads: how big it is, how much it costs, how long it takes before the first output token, what identity its cache entries have, and how many stages it has to move through. The parts of the stack that were calibrated on text keep working, silently, on numbers that no longer mean what they used to.

This post walks those differences deliberately. First what makes media content different from text content. Then what that does to an inference request, difference by difference. Then, for each one, what llm-d actually does about it — the token estimator, the content-hash routing path, the retuned affinity gate, and the encode tier. Model server like vllm is the engine throughout; its per-replica mechanisms are the floor, not the answer.