Skip to main content

One post tagged with "Multi-Tenancy"

Multi-tenant serving, quotas, and priority isolation on llm-d

View All Tags

Multi-Tenant and Multi-Priority Model Serving with llm-d

ยท 19 min read
Shimi Bandiel
Software Engineer, Google
Jacob Murry
Software Engineer, Google

You have multiple teams consuming multiple models, with use-cases of varying importance โ€” and one pool of expensive GPU hardware to serve them all. You need to impose quotas and priorities so that no team starves another and no batch job exhausts the cluster.

In this write-up we walk through a recommended architecture for this scenario built on the llm-d open-source stack, focusing on the llm-d Async component: an asynchronous dispatch processor that provides quota management, rate limiting, and strict priority scheduling in front of your inference gateways.