Rollouts

Rollouts are incremental deployment operations that gradually introduce new versions of inference infrastructure with minimal service disruption. Rather than replacing all running instances at once, a rollout shifts traffic progressively — allowing you to monitor behavior at each stage and roll back immediately if problems arise.

Why Rollouts Matter for LLM Inference

LLM inference workloads have properties that make all-at-once replacements especially risky:

• Long-running requests: a request mid-generation cannot be migrated. Abrupt pod replacement causes visible failures.
• Warm-up cost: new model server pods require time to load model weights, compile CUDA graphs, and fill KV caches. Traffic arriving before warm-up completes sees high latency.
• Resource intensity: GPU memory is non-trivially expensive. Rolling back a bad deployment means waiting for a full cold-start cycle.

Gradual rollouts mitigate all three by keeping the stable version fully operational while the new version is validated under real traffic.

Rollout Strategies

Rolling Update

A Rolling Update is the standard Kubernetes deployment strategy that updates pods gradually within a single InferencePool. This approach works in both standalone and llm-d router gateway modes.

How it works:

• Updates pods incrementally (e.g., 25% at a time)
• Old pods continue serving traffic until new pods are healthy
• Built into Kubernetes Deployments

Use Rolling Updates for:

• General, non-critical updates where strict traffic percentages do not matter
• Scenarios where you want to conserve compute resources
• Development and staging environments

Learn more: Kubernetes Rolling Update Tutorial

Blue-Green Update (HTTPRoute Traffic Splitting)

A Blue-Green Update creates a second complete InferencePool and uses HTTPRoute to control traffic distribution between the old (blue) and new (green) versions. This strategy requires llm-d router gateway mode. It supports node/accelerator hardware upgrades, base model rollouts, and model server framework updates.

How it works:

• Deploy a complete new InferencePool alongside the existing one
• Optionally mirror live requests to the new pool first, if your gateway supports percentage mirroring (HTTPRouteRequestPercentageMirror) to an InferencePool
• Use HTTPRoute to gradually shift traffic (e.g., 1% → 5% → 10% → 50% → 100%)
• Instant rollback by adjusting HTTPRoute weights

Use Blue-Green Updates for:

• Critical, high-risk production deployments that require gradual canary rollouts
• Scenarios requiring fast rollbacks
• Header-based routing (e.g., routing beta users to new version)
• Updates that need precise traffic control

Guide: Blue-Green Update

Disaggregated (P/D) Rollouts

Prefill/decode deployments on a DisaggregatedSet need every request's prefill and decode pods to run the same version, and prefill pods to stay up until decode pods have pulled their KV cache. The router's rollout screener and graceful drain settings cover both, with either an automatic rolling update or a step-by-step canary across two sets.

Guide: Safe Rollouts with DisaggregatedSet

LoRA Adapter Rollout

LoRA (Low-Rank Adaptation) adapter rollouts allow you to update model customizations without changing the base model or infrastructure. This works in both standalone and llm-d router gateway modes.

How it works:

• Use InferenceModelRewrite to map stable model names to specific adapter versions
• Gradually shift traffic between adapter versions
• No infrastructure changes required

Use LoRA Adapter Rollouts when:

• You need to deploy new versions of LoRA adapters without disrupting service
• You want to test adapter changes with a subset of traffic
• You need to maintain multiple adapter versions simultaneously

Guide: LoRA Adapter Rollout

Strategy Comparison

Feature	Rolling Update	Blue-Green Update
Routing Control	Random/Even across all healthy pods	Precise Percentage (e.g., exactly 1% or 10%)
Blast Radius	High (All users exposed randomly)	Low (Isolated to specified target weight)
Rollback Speed	Slow (Requires creating new pods in reverse)	Instant (Flip HTTPRoute weight back to 0)
Resource Costs	Low (Only temporary surge of pods)	High (Requires running two full environments)
Version Coexistence	Simultaneously active inside one Service	Strictly separated across two distinct Services
Deployment Mode	Standalone and Gateway	Gateway only

LoRA adapter rollouts have no infrastructure cost and work in either mode, but apply only when the change is limited to adapter weights — not base model or serving framework changes.

note

Capacity management may also play a role in choosing between these strategies.

General Rollout Pattern

All rollout guides follow a similar pattern:

1. Deploy new infrastructure - Create the new version alongside the existing one
2. Configure traffic splitting - Gradually shift traffic to the new version (e.g., 10% → 50% → 100%)
3. Monitor and validate - Verify the new version performs correctly at each stage
4. Complete rollout - Direct 100% of traffic to the new version
5. Clean up - Remove the old version once the new version is stable

Prerequisites

Before following these guides, ensure you have:

• A working llm-d deployment (see getting started guide)
• Access to kubectl and the Kubernetes cluster
• Understanding of Kubernetes Gateway API concepts (for gateway mode)
• Familiarity with your model serving infrastructure (vLLM, etc.)