Serving GLM-5.2 for Agentic Workloads on llm-d

llm-d's well-lit paths are tested, benchmarked recipes for serving large language models in production. The agentic-serving path starts with the workload and combines the capabilities it needs into a deployment for each accelerator. This post applies that pattern to GLM-5.2-FP8 on H200, using the 219 production Claude Code sessions in SemiAnalysis's Weka trace corpus.

GLM-5.2 is a 744B-parameter (39B active) mixture-of-experts model with 78 layers. Its Multi-head Latent Attention (MLA) stores one compressed latent instead of separate KV heads, so tensor parallelism cannot shard it: TP=8 keeps eight copies of the KV cache. Tensor-parallel all-reduce is also expensive across nodes: every layer performs the column-parallel, row-parallel, and all-reduce sequence twice. llm-d's Wide-EP path avoids both costs with data-parallel attention and expert-parallel MoE, making the KV budget viable for long-context agentic serving.

At a measured output rate of roughly 37-41 tokens per second per user, a 64-GPU llm-d deployment works out to $0.05 per million input tokens and $3.55 per million output tokens at an H200 rental rate of $2.71 per GPU-hour. Z.ai's public GLM-5.2 row lists $1.40 input and $4.40 output; Claude Opus 4.8 lists $5 input and $25 output. These figures assume sustained load and assign prefill rental to input and decode rental to output.

TL;DR: workload, measured gains, cost, and guide

• The workload is mostly reading. Across 219 production Claude Code sessions, the median main-agent request carries 195K input tokens and returns 317 output tokens. Among main-agent turns, 96% reuse at least 90% of their input, and more than half of all requests belong to sub-agent groups.

• Serving this workload takes six llm-d capabilities. MLA compresses KV, wide expert parallelism makes room for it, prefix-aware routing finds it again, CPU offloading keeps more of it, separate prefill and decode pools let each side optimize its own work, and optional MTP reduces decode steps. All six are open in the llm-d stack.

• More concurrent requests increase total output without materially slowing an active stream. On the 64-GPU reference with three prefill instances and one decoder, output rises 71%, from 3,679 to 6,281 tok/s, between concurrency 64 and 128, while median time between output tokens moves from 15.5 to 19.1 ms. Input makes up 98.6% of all tokens served, so prefill capacity is the limit.

• Routing avoids redundant prefill. Prefix-aware routing captures 96-97% of the reuse the workload makes available. In a separate paused-conversation run, turns with a cached prefix reach first token 2.8x faster than first turns without one.

• CPU offloading keeps active prefixes reusable after HBM fills. In matched 24-GPU runs at concurrency 32, offloading raises output 13%, cuts median time to first token 32%, and cuts median turn completion 27%, while median time between output tokens is unchanged. At concurrency 128, baseline output remains near 951 tok/s while offloading reaches 2,776 tok/s.

• Multi-Token Prediction lets the same 32 GPUs serve at least eight times as many concurrent requests without dropping below 30 output tokens per second per user. With CPU offloading enabled in both runs, MTP holds that rate through 128 concurrent requests and produces 5,647 tok/s; without MTP, it holds only through 16 requests and 561 tok/s.

• At sustained load, self-hosted GPU rental accounts for a fraction of the per-token cost of a managed API. The two 64-GPU layouts work out to $0.05-$0.08 per million input tokens and $1.92-$3.55 per million output tokens, versus $1.40 and $4.40. These figures cover GPU rental only, not the full cost of operating an API service.

• Run it. The GLM-5.2 H200 guide provides the serving pattern and composable MTP, CPU-offloading, and context-length components.

The shape of agentic traffic

A coding agent's turn is mostly reading: a repository-scale context in, a tool call or a short diff out, then the whole context again next turn, slightly longer. The Weka trace corpus published by SemiAnalysis contains 219 production Claude Code sessions and 78,099 model requests. Each session captures interactive repository work: a main agent grows and reuses a working context across tool calls, pauses while tools run, and delegates work to Task-tool sub-agents that execute concurrently. The corpus preserves request timing, input and output lengths, per-request KV-block hashes, and the original sub-agent fan-out.

Read the percentiles across each row, not down the column: the p90 input length and p90 pause do not belong to the same request or session. The wall-clock gap is the time between main-agent calls, not generation latency. Input-length growth is the increase from the previous main-agent turn, not a cache miss.

Main-agent metric	p50	p90	p99
Input tokens per request	195K	521K	849K
Output tokens per response	317	1,705	7,551
Input-length growth from prior turn	632	3,735	148K
Wall-clock gap between turns	2 s	24 s	11.4 min
Turns per session	107	354	992

note

Input and output counts use different tokenizers. The corpus uses o200k for input because those tokens determine the per-request KV block hashes; output counts come directly from Anthropic's API. The 195K-to-317 ratio therefore mixes units.

Left: input and output token distributions across 37.7K main-agent requests. Center: cumulative share of turns by prefix overlap with their best recent predecessor. Right: wall-clock gaps between consecutive main-agent requests.

Most GPU work happens before the first output token. The median main-agent request carries 195K input tokens and returns 317 output tokens, so serving is dominated by processing context. Output then arrives in short bursts between tool calls. Responsiveness depends on processing each context quickly and streaming its response steadily.

The 317-token median covers one model call. A result the user sees may require many such calls, plus tool calls and sub-agent runs, before the final response appears.

Nearly everything has been read before. A main-agent request is only 632 tokens longer than the preceding request at the median. When each request is matched to its best earlier turn, 96% reuse at least 90% of their input as a verbatim prefix; median reuse is 99.6%. A GPU cache hit avoids both recomputation and transfer. CPU or remote hits still have to move KV back to the GPU. Anthropic’s one-tenth price for cache-read input reflects the same reuse economics.

Sessions pause, then burst. Main-agent calls are two seconds apart at the median and eleven minutes apart at p99, while the median session lasts an hour. Cached state must survive those gaps to remain useful. Sub-agents add a second pressure: 40,367 of 78,099 requests arrive in groups, with seven requests per group at the median and 51 at p90. The serving layer receives that concurrency without advance notice.

One production session combines all three properties:

A continuous 151.7-minute segment from one production session: 38 streaming main-agent requests and 5,527 sub-agent requests. The main-agent context grows to 754K tokens, while sub-agent work spans the full segment.

Prefix resets, where a turn shares less than half its input with any of its previous eight main-agent turns, occur on 2.8% of turns corpus-wide. Context compactions, where a context above 500K tokens returns below 30K, occur on 0.03% of turns. These are expected production pressures alongside sub-agent divergence and cache eviction under load. In the load benchmarks, measured reuse is governed by the admitted request distribution and cache churn, so routing on a single session-level prefix is insufficient.

How llm-d serves this workload

The serving path starts with the context: fit it in memory, keep it reusable across turns, then handle context processing and token generation separately. MLA and wide expert parallelism make room in HBM. CPU offloading keeps prefixes available as HBM fills, and KV-cache-aware routing sends later turns back to them. MTP reduces decode work, while separate prefill and decode pools let long reads and token generation scale independently. The sections below follow that path, then show how to tune and scale those pools for this workload.

Topology notation

Measured prefill and decode topologies and their naming.

Topology names describe the number and width of the serving instances. xPyD means x prefill instances and y decode instances. DEP16 means TP=1, DP=16, and expert parallelism across 16 GPUs. DEP8 uses the same layout across eight GPUs. The benchmark sections state the GPU total when it matters.

What the performance results cover

The performance tests use requests derived from the production traces, not the complete corpus unchanged. Limited-input-length runs admit requests below 142,000 tokens; full-input-length runs admit longer contexts. Corpus statistics describe the full production workload, while performance results describe the requests admitted by each test. Absolute throughput is comparable only within the same test.

Deployment and benchmark details

The appendix lists the full model configuration, software stack, measured topologies, and settings for each benchmark.

Fitting repository-scale sessions in memory

Multi-Head Latent Attention (MLA) keeps the KV cache small enough for repository-scale contexts.

GLM-5.2 stores about 44 KB of KV state per token. In the measured 64-GPU 3P1D layout, that gives about 47.2M prefill-cache tokens and 17.3M decode-cache tokens beside the model's weights. It stores 576 FP8 elements per layer per token: a 512-element latent plus 64 RoPE elements. Across 78 layers, that is 0.045 GB per 1K tokens. A counterfactual with 64 KV heads and 256-dimensional keys and values would use about 57x more KV memory; eight KV heads at those dimensions would use about 7x more.

Per-token KV footprint from the architecture math (FP8, 78 layers): MLA’s 576 stored elements per layer, compared with counterfactual GQA using 8 KV heads and MHA using 64 KV heads, both with 256-dimensional keys and values.

At concurrency 128, the 64-GPU 3P1D full-input-length run carries 14.1M logical tokens in flight, indicating the live request demand the cache must serve.

KV is allocated in 64-token blocks. Full-input-length runs set gpu-memory-utilization to 93.5% on prefill and 95% on decode. Depending on topology, each DEP16 prefill rank holds about 958K-990K KV tokens; each decoder rank holds about 1.078M. Across the 64-GPU 3P1D layout, that is about 47.2M prefill-cache tokens and 17.3M decode-cache tokens. These measured allocations include the three-token MTP head.

Wide expert parallelism

Wide expert parallelism spreads model weights across GPUs, leaving more memory for KV.

In the full-input-length runs, KV uses about 40-41 GiB on each prefill GPU and 45 GiB on each decode GPU. At TP=1, each GPU holds about 65 GiB of weights.

The guide's default layouts use single-node DEP8 prefill instances and a two-node DEP16 decode pool, with LeaderWorkerSet resources and DP-aware scheduling. The measured 3P1D reference uses two-node DEP16 prefill instances. DeepEP sends tokens to the GPUs holding their experts over NVLink and InfiniBand, and deep_gemm runs the expert computation.

Width determines how much KV remains on each rank. In the limited-input-length sweep, the DEP8 prefill rank holds about 182K KV tokens and DEP16 about 990K. The narrower layout therefore reaches the CPU-backed cache sooner.

Logical tokens in active requests rise from 1.45M at c16 to 3.35M at c32, 7.09M at c64, and 14.09M at c128 on the 64-GPU 3P1D full-input-length run.

Per-GPU HBM accounting by topology. Total device memory is measured; weights and KV allocation come from the serving configuration, and the remainder groups runtime and communication allocations.

The aggregate TP=8 topology fits in one node and avoids the wide-EP communication buffers, but it gives up KV capacity.

With decode-context parallelism disabled, vLLM copies the same MLA cache onto all eight ranks: 598K logical tokens with a 25 GiB copy per rank, or roughly 200 GiB of physical KV per node. Expert weights use 89.5 GiB per GPU, versus 65.1 GiB in the wide layout.

On the same hardware, aggregate TP=8 holds 598K KV tokens per node; DEP8 holds 1.77M, and a DEP16 node-share holds about 5.2M. The aggregate layout trades that 3× to 9× session-capacity gap for a single-node scheduling unit without the NVSHMEM heap or all-to-all fabric traffic.

Routing turns back to their cache

Prefix-aware routing sends each turn back to a replica that already has its context.

With MTP and CPU offloading enabled, GPU memory supplies about 90-92% of reusable prefill prefixes across the DEP16 1P1D and 2P1D runs through concurrency 128. The llm-d Router schedules prefill and decode differently:

• Prefill placement looks at how much of a request's prefix is already in GPU or CPU memory, how many requests are queued, and how full the KV cache is. It sends the turn to the replica that offers the best balance of cache reuse and load.

• Decode placement balances GPU and CPU cache locality against the number of active requests.

The optimized-baseline configuration applies separate prefill and decode filters, and the router sees prefixes in both GPU and CPU memory. The load sweep keeps the corpus’s overlap pattern but uses shorter inputs, around 50K tokens at the median. Because each new suffix is a larger fraction of the request, at most 93.9–94.4% of the input can be reused.

Decode-local reuse is lower because NIXL moves KV from prefill to decode.

The paused-conversation run shows the latency effect: first turns without a cached prefix take 7.46 s median time to first token at about 20K input tokens; later turns with a cached prefix take 2.68 s on inputs up to four times longer. This is an observed workload result, not a matched cache-only control.

Left: in the limited-input DEP8 1P1D run at concurrency 64, the router reaches a 91.4% prefill GPU hit rate against a 93.9% reusable-prefix ceiling, capturing 97% of the available reuse. Right: in the DEP16 1P1D run, decode-local hit rate is 51.3% at concurrency 64 and 44.6% at 128 because NIXL moves KV to the decode pool.

Tiered prefix cache

CPU offloading keeps evicted prefixes reusable instead of rebuilding them.

On the 24-GPU 1P1D DEP8/DEP16 layout at concurrency 32, it raises output 13%, cuts median time to first token 32%, and cuts median turn completion 27%, while median time between output tokens is unchanged.

24-GPU 1P1D with DEP8 prefill and DEP16 decode; MTP is disabled in both runs. At concurrency 32, input distributions match within 1%. At concurrency 128, baseline output remains near 951 tok/s while CPU offloading reaches 2,776 tok/s; the fixed-duration stages complete different request slices, so the endpoint is a capacity comparison rather than a request-identical A/B. Higher output is better; lower latency is better.

Tool calls leave session KV idle, so the serving system may evict it from HBM before the next turn. CPU offloading keeps that KV in a per-rank DRAM pool and restores it when the conversation resumes. MultiConnector handles two paths: OffloadingConnector moves KV between GPU and CPU, while NIXL moves it between prefill and decode. The CPU tier evicts least-recently-used entries first. The increasing-load sweeps allocate 100 GiB per prefill rank; the paused-conversation run uses 150 GiB. Local NVMe sits behind CPU memory and receives no reads in these measurements.

The increasing-load sweep measures the capacity ceiling; the paused-conversation runs measure whether session state survives the gaps between turns.

When HBM fills

Offloading does little until HBM fills. On the compact 16-GPU DEP8 topology, with MTP disabled in both runs, output is effectively unchanged at concurrency 16. At concurrency 128, offloading raises output 27% and cuts median turn completion 36%. Each stage runs for a fixed time, so the two runs may finish different requests. This compares how much work each configuration can handle, not the same requests one by one.

The dedicated 64-GPU 3P1D run shows the other regime: CPU memory accepts roughly 5 TB of writes over 24 minutes but serves no restores because the active prefixes still fit in HBM. CPU offloading adds capacity under HBM pressure; before that pressure, stores add modest overhead.

Paused conversations

The paused-conversation test places two DEP8 prefill instances beside one DEP16 decoder, for 32 GPUs total. Each DEP8 rank has about one-third of a DEP16 rank's KV capacity—220K versus 650K tokens—because it holds twice as much expert weight.

With 100–400 ongoing conversations and production-measured pauses of 2.1 s at the median and 25.8 s at p90, live session data reaches 1.4–5.6× GPU cache capacity. CPU memory holds the excess. As conversations resume, restore traffic rises from 3.5 to 22.4 GB/s.

At 400 conversations, every reused prefix is restored from CPU memory, totaling 9.7 TB in one stage, while the time between output tokens remains 34–37 ms. Restoring saved context is faster than recomputing it: 8.4 s versus 14.8 s median across the run, and 23.0 s versus 26.2 s at 400 conversations.

At 400 ongoing conversations on the mixed-width 2P1D topology and load, median time to first token reaches 124.5 s without CPU offloading versus 22.2 s with it. The p90 approaches six minutes, while the time between output tokens stays flat in both configurations. Pauses make recomputation expensive: each eviction forces the entire prefix to be rebuilt before the conversation can resume.

With the GPU cache at its natural size, 100-400 ongoing conversations and production-measured pauses exceed GPU capacity by 1.4x to 5.6x. CPU memory supplies the missing capacity without increasing the time between output tokens.

note

At higher scales and longer pauses, disk offloading is expected to bring similar gains at different granularities.

Restore versus recompute

A controlled comparison on the mixed-width 2P1D topology isolates the cost of a cache miss. The GPU KV cache is limited to roughly 230K tokens per rank, slightly above its natural allocation, and CPU offloading is switched on or off for the same workload. At concurrency 32, nothing is evicted and the configurations match within 2%. At higher load, both GPU caches fall to single-digit hit rates; the only meaningful difference is whether a miss restores saved context or recomputes it.

At the same workload and GPU-cache limit, offloading restores evicted context while the baseline recomputes it. Client TTFT includes time waiting in the prefill queue.

With CPU offloading, a miss takes about 59 ms to restore, and median time to first token remains 8.3 s at concurrency 128 with full throughput. Without offloading, a miss requires recomputing about 45K tokens. The server still produces a first token in about 5.7 s once work starts, but the client waits 59 s because 97 requests are queued for prefill. Throughput falls 78%, and the system completes 4.5x fewer turns. The time between output tokens remains 31-36 ms in both configurations because prefill and decode run on separate GPU pools.

Queue-inclusive TTFT separates server processing from time waiting for prefill, which dominates the result without offloading.

Across the run, engine counters record 12.7 TB of KV, or 283M tokens, restored at 26 ms per GB. At concurrency 128, the CPU cache restores KV at 22.7 GB/s while the topology processes 474K prompt tokens per second.

The cache restores 222M tokens while serving 217M prompt tokens. That ratio exceeds 100% because the limited GPU cache can evict a block that was just restored, forcing it to be loaded again. The churn suggests that a retention policy better than LRU could reduce data movement.

Adding prefill capacity

The narrowest layout queues the most prefill work. At concurrency 128, 1P1D reaches 3,867 tok/s with 7.9 seconds median time to first token. Adding a second prefill instance raises output 23% to 4,770 tok/s and cuts the median wait for first token 38% to 4.9 seconds. A third prefill instance adds 3% output and brings the median wait to 4.0 seconds. All three runs use one fixed DEP16 decoder and a 100 GiB CPU tier per rank.

Concurrency 128 with one fixed DEP16 decoder. The second prefill instance removes most of the queueing pressure; the third adds less capacity with the decoder unchanged.

Adding prefill instances keeps more KV in HBM and reduces restore traffic; P/D disaggregation lets that input capacity scale independently of decode.

Speculative Decoding

Multi-Token Prediction (MTP) gets more output tokens from each decode step.

On the matched 32-GPU 1P1D DEP16 topology, it raises aggregate output 1.6-2.0x and average end-to-end output per user 1.6-1.8x through concurrency 128.

Long contexts limit how many requests fit in a decode batch, while MoE layers run best with larger batches. MTP adds useful token work to each decode step without adding more requests.

At concurrency 64, median turn completion falls 34% and p90 falls 44%. The 32-GPU 1P1D comparison keeps CPU offloading enabled in both runs, so the measured difference comes from MTP. The configuration drafts three extra tokens per decode step.

Matched 32-GPU 1P1D DEP16 deployments with CPU offloading. Each curve follows concurrency 16 through 128. MTP moves the frontier right and up: users receive tokens faster while the deployment produces more total output.

Highest measured concurrency that sustains at least 30 average tok/s/user. MTP meets the target through c128 and serves 5,647 tok/s; CPU offloading alone meets it only at c16 and serves 561 tok/s. That is at least 8x the concurrency and 10.1x the aggregate output. The target is an average output-rate floor, not a TTFT or tail-latency bound.

Each MTP step emits about 2.95 tokens, including the normal token and accepted drafts. Draft acceptance stays between 64.0% and 70.5% across both input-length sweeps; by draft position, it is about 80%, 65%, and 52%. Acceptance stays similar across runs, so throughput differences are more likely to come from topology, queueing, and request mix.

The table compares the two configurations at the same concurrency. Across c32-c128, MTP cuts median time between output tokens by about 52%. Time to first token does not improve; MTP speeds decode, not prefill. Each fixed-duration stage may complete a different request slice.

Concurrency	TTFT p50 (offloading / MTP)	ITL p50 (offloading / MTP)	Output throughput (offloading / MTP)	E2E output / user (offloading / MTP)
32	1.1 s / 1.3 s	28.6 / 13.5 ms	1,145 / 2,159 tok/s	28.7 / 51.5 tok/s/user
64	1.3 s / 1.6 s	30.7 / 14.7 ms	2,178 / 3,788 tok/s	25.9 / 45.1 tok/s/user
128	1.6 s / 2.4 s	35.1 / 17.1 ms	3,505 / 5,647 tok/s	21.6 / 35.0 tok/s/user

Left: limited-input-length runs with a 142K benchmark admission cap. Right: full-input-length runs. Each line follows one topology through concurrency 128; hollow points have median time to first token above 10 seconds. Moving right means higher user-visible output speed; moving up means more output per decode GPU. Absolute throughput is comparable only within a panel.

Splitting prefill from decode

Separate prefill and decode pools let input processing and generation tune and scale independently.

In the 64-GPU 3P1D DEP16 sweep, output rises 71% from concurrency 64 to 128 while median time to first token moves from 1.71 to 2.18 seconds.

Over the same range, median time between output tokens moves from 15.5 to 19.1 ms.

There is no matched aggregate control, so this sweep measures scaling within the P/D layout rather than a controlled P/D-versus-aggregate gain. Prefill uses deep_gemm with deepep_high_throughput; decode uses deepep_low_latency; NIXL moves KV between the pools. Both pools use wide expert parallelism over InfiniBand.

IBM users exercised GLM-5.2 tool calling through the P/D deployment. That path requires glm45, glm47, and automatic tool choice on both prefill and decode; configuring parsing only on decode breaks tool-calling requests.

Across concurrency 16 to 128, input throughput rises from 86K to 431K tok/s and output from 1,167 to 6,281 tok/s. Active prefill work grows faster proportionally than decode, so prefill capacity increasingly sets the wait for first token. Decode capacity continues to set streaming speed and full-turn completion.

The full-input-length reference sweep enables native MTP, CPU KV offloading, and disaggregated routing. It uses max-model-len=auto and keeps DEP16 across the prefill and decode pools. The 64-GPU 3P1D reference uses three two-node prefill instances and one two-node decoder across eight nodes. Between concurrency 64 and 128, median input length is about 47K-57K tokens and p99 is above 347K:

Concurrency	Requests	TTFT p50 / p90	ITL p50 / p90	E2E lat. p50	Output throughput
64	3,543	1.71 s / 6.36 s	15.5 ms / 18.9 ms	9.6 s	3,679 tok/s (57/GPU)
128	6,242	2.18 s / 15.82 s	19.1 ms / 23.1 ms	14.0 s	6,281 tok/s (98/GPU)

Across concurrency 16 to 128, median TTFT stays between 1.5 and 2.2 seconds, while p90 rises from 4.3 to 15.8 seconds. Median ITL rises from 11.7 to 19.1 ms, and p90 rises from 14.4 to 23.1 ms.

The output rate hides most of the work. At concurrency 128, each GPU serves 6,839 total tokens per second, and 98.6% are input. For this workload, the cluster is primarily buying prefill capacity.

Faster decode shifts the bottleneck toward prefill.

Sizing prefill and decode capacity

For this workload, decode replicas raise output; prefill replicas shorten the wait before generation.

At concurrency 128, one more decoder raises output 21%, while one more prefill replica cuts median time to first token 21% and restore traffic 94%. The full-input-length sweep compares DEP16 layouts with MTP and CPU offloading enabled; limited-input-length results are comparable only within that separate test.

Adding prefill or decode capacity

The full-input-length sweep keeps every instance at DEP16 and compares 1P1D, 2P2D, 3P1D, and 3P2D with native MTP and CPU KV offloading enabled.

At concurrency 128, keeping three prefill replicas fixed and adding a second decoder raises output from 6,281 to 7,598 tok/s (+21%) and end-to-end output per user from 36.6 to 44.2 tok/s/user (+21%). Median turn completion falls from 14.0 to 11.5 seconds, while median time between output tokens falls from 19.1 to 15.5 ms. Median time to first token remains near 2.1 seconds.

Prefill width changes a different part of the turn. With two decoders fixed, moving from 2P2D to 3P2D cuts median time to first token from 2.70 to 2.12 seconds and CPU-to-GPU restore traffic from 759 to 45 GiB; median time between output tokens stays near 15.4 ms. In this sweep, at the same 64-GPU budget, 2P2D produces 8% more output and completes the median turn 10% sooner, while 3P1D lowers median time to first token 19% and restore traffic 63%.

Each line follows one full-input DEP16 topology through concurrency 128. Solid lines use two decoders; dashed lines use one. Moving right means higher user-visible output speed; moving up means more work per GPU.

Each stage runs for a fixed time, so comparisons between layouts may include different completed requests.

Where the bottleneck moves

Decode and prefill width remove different queues. The limited-input-length sweep keeps native MTP and CPU KV offloading enabled while widening one side of the deployment at a time.

At concurrency 128, widening the DEP8 decoder to DEP16 raises output from 1,501 to 3,233 tok/s and cuts the decode queue from 39 to 3 requests. The narrow prefill pool still has 137 requests waiting and restores 7,229 GiB of KV from CPU memory.

Widening prefill from DEP8 to DEP16 moves the active KV set back into HBM. Per-rank prefill capacity rises from about 182K to 990K KV tokens, CPU restore traffic falls from 7,229 to 50 GiB, and median time to first token falls from 20.9 to 2.38 seconds. Output reaches 5,647 tok/s.

A second DEP16 prefill instance removes the remaining restore traffic. At concurrency 128, 2P1D reaches 6,647 output tok/s and 1.59 seconds median time to first token. The 32-GPU 1P1D topology delivers 85% of that output with one-third fewer GPUs; the 48-GPU 2P1D topology buys lower latency and more prefill headroom. Against the 48-GPU baseline with both features disabled, the optimized 32-GPU layout produces 54% more output, delivers 51% more end-to-end output per user, and completes the median turn 18% sooner while using one-third fewer GPUs; median time to first token is 2.38 versus 1.26 seconds. The fixed-duration stages do not complete the same request slice, so these are operating-point comparisons rather than request-identical A/Bs.

At concurrency 128, widening decode removes the decode queue but drives more work through a prefill pool that is still too small. Widening prefill moves the active KV set back into HBM; a second prefill replica buys the remaining headroom. Higher output is better; lower TTFT and restore traffic are better.

What it costs at an interactivity target

At sustained load, self-hosted GPU rental accounts for a fraction of the per-token cost of a managed API.

At concurrency 128, the 64-GPU 3P1D topology with MTP and CPU offloading serves 431K input and 6,281 output tokens per second, delivering 36.6 end-to-end output tokens per second per user. The 64-GPU 2P2D topology serves 489K input and 6,781 output tokens per second, delivering 40.9 end-to-end output tokens per second per user. End-to-end throughput includes queueing and prefill, so it is the user-visible rate.

Using the same assumed $2.71 per GPU-hour rental rate, 3P1D implies $0.08/M input and $1.92/M output, while 2P2D implies $0.05/M input and $3.55/M output. Both layouts rent 64 GPUs. 2P2D assigns more of them to decode, favoring streaming throughput and full-turn completion; 3P1D assigns more to prefill, favoring time to first token and lower CPU restore traffic.

Input tokens are logical tokens: cached prompt tokens count at their full prompt length, not by the bytes restored from cache.

Z.ai lists its GLM-5.2 API at $1.40 per million input tokens, $0.26 cached input, and $4.40 per million output tokens. Anthropic lists Claude Opus 4.8 at $5 input and $25 output, with cache reads at $0.50 per million tokens. At the stated rental rate, self-hosted 2P2D's $3.55/M output is about 81% of Z.ai's public GLM-5.2 rate and about one-seventh of the Opus rate. Its $0.05/M input cost is 3.5% of Z.ai's list rate and about 20% of its cached-input rate. Input accounts for 98.6% of served tokens at this operating point. Prefill accounts for half of GPU rental in 2P2D and three-quarters in 3P1D.

Cost basis and limits

These figures assume that the cluster sustains the measured load 24/7. Lower utilization raises effective cost in inverse proportion. The prices cover GPU rental only; storage, egress, engineering, support, and taxes are excluded. The input/output split assigns each P/D pool's cost to the tokens it processes; another allocation changes the two columns, not total cluster cost. API rates include managed operations, autoscaling, service commitments, and provider margin. Spot rates also carry interruption and availability risk. The comparison describes token economics at a fixed operating point, not full total cost of ownership or an API-equivalent service.

Conclusion

Agentic serving depends on input reuse as much as decode speed. On the 64-GPU 3P1D reference, increasing concurrency from 64 to 128 raises output 71%, from 3,679 to 6,281 tok/s, while median time between output tokens moves from 15.5 to 19.1 ms. Because there is no matched aggregate control, this demonstrates scaling within P/D rather than a measured P/D-versus-aggregate gain.

Each optimization removes a different bottleneck. Wide expert parallelism makes room for long contexts, while routing and CPU memory keep them reusable. Separate prefill and decode pools keep long reads from disrupting generation; MTP reduces decode steps; and additional prefill replicas add capacity where the workload queues. In this sweep, at the same 64-GPU budget, 2P2D produces more output and finishes turns sooner, while 3P1D reaches first token sooner and restores less KV from CPU memory.

At high concurrency in the measured layouts, queueing on the fixed DEP16 decoder limits how much generation speed reaches each user. Adding decode capacity is the next sizing question.

The agentic-serving cookbooks include Nemotron-3-Ultra-550B on H200, with P/D disaggregation and CPU KV offloading, and Qwen3-Coder-480B on TPU v7, with routing and CPU offloading. The GLM-5.2 H200 configuration follows the same agentic-serving pattern.

Where this goes next

For GLM-5.2, KV capacity is the main deployment constraint. The cache hierarchy can extend beyond CPU DRAM and local NVMe to distributed DRAM and network storage. Each tier adds capacity but also restore latency, so it helps only while loading saved KV remains cheaper than recomputing it. The right boundary depends on the workload: active streams are latency-sensitive, while paused agent sessions can move state to a cheaper tier and restore it before work resumes. The deployment policy should choose the cheapest tier that still beats recomputation.

The open problem is serving a complete agent program, not isolated requests. That means replaying think time, tool calls, sub-agent fan-out, and the dependency graph that connects them, then measuring how routing, prefix retention, and prefill/decode sizing affect the time a user waits for a result.

Next, compare P/D with a matched aggregate topology, test wider decode pools at high concurrency, and validate tool calls under MTP.

The Agentic Inference SIG develops this work in the open. Join #sig-agentic-inference on the llm-d Slack or a community meeting; bring traces.

Get Involved with llm-d

• Explore the llm-d Community Quickstart → Start here

• Join our Slack → Get your invite

• Explore the code → GitHub organization

• Attend meetings → all open; add our public calendar

---

Appendix: Deployment and benchmark details

Model and software stack

GLM-5.2 is a 744B-parameter mixture-of-experts model with 39B active parameters and 78 layers. Each token uses only a subset of its experts, but the full FP8 weight set still has to be distributed across GPUs. Multi-head Latent Attention reduces the KV state stored for each token, and the model's multi-token prediction head drafts up to three additional tokens per decode step.

The measured deployment runs on Kubernetes and uses CoreWeave CKS with InfiniBand:

Layer	Component	Role
Model server	vLLM	FP8 inference, MLA KV cache, agentic parsers
Routing	llm-d EPP	Prefix-cache-aware routing and P/D scheduling
KV transfer	NIXL	KV movement between prefill and decode over InfiniBand
Tiered prefix cache	CPU DRAM cache + NVMe storage evictor	CPU-backed KV capacity below HBM; NVMe is provisioned but inactive in these measurements
MoE compute	DeepEP + deep_gemm	Expert parallelism across GPUs
Attention	FlashMLA Sparse	Attention backend
Multi-node pods	LeaderWorkerSet	Manages each multi-node replica as one Kubernetes unit
Benchmarking	AIPerf	Load sweeps against the gateway

The measured benchmarks use disaggregated prefill and decode layouts. Aggregate TP=8 appears only in the memory-profile comparison.

Topology inventory

The full-input-length sweep uses DEP16 throughout and scales both pools. 1P1D uses 32 GPUs, 2P2D and 3P1D each use 64, and 3P2D uses 80. The two 64-GPU layouts spend the same hardware budget differently: 2P2D adds decode capacity, while 3P1D adds prefill capacity. Comparing 3P1D with 3P2D isolates the effect of adding a decoder.

The limited-input-length sweep changes instance width before adding another instance. A DEP8/DEP8 1P1D layout uses 16 GPUs. Widening only the decoder to DEP16 uses 24 GPUs; widening prefill too uses 32. Adding a second DEP16 prefill instance produces the 48-GPU 2P1D layout.

The paused-conversation test uses another 32-GPU 2P1D layout: two DEP8 prefill instances and one DEP16 decoder. The topology name counts instances; their widths determine the GPU total.

Benchmark setup

Each benchmark admits only requests below its admission limit. The limited-input-length runs filter the trace at 142,000 tokens; the full-input-length runs admit longer contexts. The server sets max-model-len automatically in both, with GPU-memory utilization at 93.5% for prefill and 95% for decode.

In AIPerf, concurrency is the number of simulated clients, each keeping one request active. tok/s/user is throughput per simulated client, not per person or complete agent task. End-to-end turn completion covers queueing, processing, and generation from request submission through the final output token.

The load sweep increases concurrency with completions near the corpus median of about 317 output tokens. The offloading A/B keeps matched 16- and 32-GPU topologies fixed and switches CPU offloading on or off. The paused-conversation test uses shorter completions and production-measured pauses across 100-400 ongoing conversations.

The cache-limit test isolates restore versus recompute. The offloading width sweep changes the number of DEP16 prefill instances while keeping one DEP16 decoder fixed. The limited-input-length sizing sweep widens decode and then prefill with MTP and CPU offloading enabled. The full-input-length sweep changes the number of DEP16 prefill and decode instances. Absolute throughput should be compared only within the same test.