Manage Traffic and Tenants

Control how requests reach the inference pool when several tenants, priorities, or models share it.

Queue and Prioritize Requests (Flow Control)

Intelligent request queuing in the EPP: priority bands, per-tenant fairness, and saturation detection for multi-tenant deployments and traffic spikes.

Tune Flow Control Concurrency

Derive maxConcurrency for the saturation detector from your hardware, model, and workload with the tuning wizard.

Deploy Multiple Inference Pools

Add InferencePools, each with its own EPP and model server Deployment, to an existing deployment so several models or tenants are served side by side.

Route to Multiple Models and LoRA Adapters

Serve multiple base models and LoRA adapters behind a single endpoint using the Inference Payload Processor (IPP).