Scaling Vision-Heavy Kimi-VL with Heterogeneous E/PD on llm-d and SGLang
Vision-language model (VLM) inference involves several stages: vision encoding, language model prefill, and language model decode. Each stage has different compute and memory characteristics, making heterogeneous disaggregation a natural fit. This gives us the flexibility to place different stages to different accelerator resources based on their compute and memory characteristics.
In this post, we evaluate a vision-heavy Kimi-VL serving workload on llm-d and SGLang. We place vision encoding stage in 4 Intel Arc Pro B60 (Intel B60) GPUs while keeping prefill and decode stage on 1 NVIDIA H200 GPU. Compared with the collocated baseline, the measured heterogeneous E/PD configuration achieved 2.4x-2.8x higher throughput and reduced mean Time To First Token (TTFT) by roughly 69%-80% under load.