Multi-Tenant and Multi-Priority Model Serving with llm-d
ยท 19 min read
You have multiple teams consuming multiple models, with use-cases of varying importance โ and one pool of expensive GPU hardware to serve them all. You need to impose quotas and priorities so that no team starves another and no batch job exhausts the cluster.
In this write-up we walk through a recommended architecture for this scenario built on the llm-d open-source stack, focusing on the llm-d Async component: an asynchronous dispatch processor that provides quota management, rate limiting, and strict priority scheduling in front of your inference gateways.