From agent traces to replay

Say you run inference for an agentic product. Sooner or later you have to answer questions like: how will my serving stack hold up when thousands of agents are calling tools at once? What tail latency will a user hit across a long, multi-turn session? Would a different scheduler, a bigger KV cache, or another GPU move those numbers before I commit to buying it? Did last week's model upgrade quietly change how the agent behaves? You can answer each of these, but only with real agentic traffic. A synthetic prompt loop won't reproduce the tool calls, the multi-turn structure, or the load shape that make agents behave like agents.

That traffic comes from a trace dataset: a recording of what real agents did, which you can play back instead of inventing sessions. A dataset is only half the answer, though. A trace does nothing until something replays it, and the replay mechanism decides what you can do with what the dataset kept: turn it into live load, simulate it, or compare two runs. A rich trace that no tool can replay the way you need is a dead end. A capable mechanism can't rebuild detail the trace already threw away. So the question is which dataset to use with which mechanism, and the answer depends on what you're trying to learn.

Datasets differ because they were built for different goals. One built to characterize inference infrastructure load looks nothing like one built to study human-agent interaction, and neither looks like one built as a replayable behavior record. A dataset can "have agent traces" and still be unable to answer your question.

This post surveys these datasets, within a deliberate scope. It covers the trace datasets the inference-serving world actually uses: the ones a replay mechanism (Inference-Perf, AIPerf, GuideLLM, vLLM bench serve, SGLang bench_serving, NVIDIA DynoSim) can turn back into a live or simulated workload. For each, we ask what the dataset keeps, what it throws away, and which questions that leaves answerable. Side by side, the differences are more than cosmetic: each schema follows the goal it was built for.

We didn't hand-pick the list. We went to each of the six replay mechanisms above and collected every agentic-trace format it can load and re-run natively. The datasets below are exactly that set.

Two terms first. A session is the actual run: a developer's coding session, or one attempt at a benchmark task. A trace is what a dataset captured of that session and published. The two can differ a lot. The linked WEKA release has 391 traces drawn from real sessions, but it hashes the content away, so a rich session doesn't guarantee a rich trace. Each count below uses the unit the dataset's own documentation uses. Mooncake is counted in requests, because it publishes no session or trace boundaries at all.

The datasets, side by side

Dataset	Scale	Origin	Built for	What's captured
WEKA / AgentX SemiAnalysis	391 traces	Human, real usage	Inference-infra workload characterization	WEKA / AgentX nested conversation records with subagent traffic patterns and request shape. Content is hashed away, since the dataset was built to study load.
SWE-smith SWE-bench	5,017 traces	Benchmark-generated	Training data for SWE agents (SFT)	SWE-agent trajectory records with full multi-turn conversations plus the resulting code patch. Structured tool calls appear in the tool split and are flattened to prose in the others, but there are no token counts or timing. Software-engineering tasks only.
TraceLab University of Washington	4,265 sessions	Human, real usage	Agent-serving infra (batching, latency)	TraceLab session-and-round records for replayable requests and tool calls, built for serving-system analysis. Content is stripped out. Tool calls and cache-prefix structure are inferred heuristically after the fact.
Mooncake Moonshot AI · FAST'25	23,608 requests	Recorded serving traffic	KV-cache reuse & scheduling research	Timed request records sampled from one hour of a real serving system (Kimi): timing, token counts, and prefix hashes only. The requests come from conversation and tool-agent workloads, but each is published as an independent record, with no session or dependency IDs. Shared prefixes stay visible across requests, so KV-cache reuse can be studied. Session and call-graph boundaries are lost.
Baseten Trace Baseten	Not publicly stated	Recorded serving traffic	Large-scale serving / infra benchmarking	Baseten Parquet request rows grouped by session ID. Strongest here on KV-cache structure (block hashes and size) and timing. Nothing tool-shaped, since they replay /v1/completions. It ships as an AIPerf loader format, with no published corpus: the replay schema is documented, but the trace's provenance and scale aren't publicly stated.
Exgentic Agent LLM Traces v2 IBM	10,057 traces	Benchmark-generated	Cross-model / cross-scaffold agent comparison	OpenTelemetry spans following the GenAI semantic conventions, with full model request and response bodies, tool definitions, and request parameters. No prefix or cache signal, and tool calls live inside message content instead of in their own span. Covers many task domains: software engineering, web, customer care, and deep research.

Disclosure: we build one of the datasets compared here, Exgentic's Agent LLM Traces, introduced in What Agents Do: A Large-Scale Public Dataset of Agent Runs.

Each of these design choices reflects the dataset's intended use. WEKA and TraceLab hash content because they study which request patterns reach the server. Mooncake retains timing and prefix structure because KV-cache reuse is a scheduling problem whose behavior does not depend on message semantics. Exgentic fixes the task set because comparing models and scaffolds requires uniform workloads more than realistic ones.

Origin matters more than it first seems because it also determines the privacy story. Traces rarely reveal where they came from, so that information must be known in advance. WEKA/AgentX, TraceLab, and Baseten are derived from real user activity. The first two remove content by design, while Mooncake publishes production traffic without message bodies. Exgentic and SWE-smith sit at the opposite extreme: both are entirely benchmark-generated. They are replayable by construction and contain no real user data, but they also reveal little about how people actually work.

The datasets differ most clearly in the content they retain. Exgentic and SWE-smith preserve full request and response bodies, whereas WEKA, TraceLab, and Mooncake retain only token counts, timing, and prefix structure. That choice is irreversible: text can be discarded later, but text that was never collected cannot be reconstructed. While this is appropriate for the infrastructure questions those systems target, it eliminates analyses that require semantic context, such as diagnosing agent failures or evaluating prompt changes. Token counts can show that a session was expensive, but not why.

The datasets also differ in their support for controlled comparison. Because Exgentic evaluates a fixed set of tasks, it can separate the effects of the model, scaffold, and task. Real-world usage datasets cannot, since each session follows a different trajectory. This advantage is not absolute, however: failed runs left some model-task combinations underrepresented, with trace counts ranging from roughly 300 in the smallest cell to more than 2,000 in the largest. Exgentic therefore enables controlled comparisons, but not always between equally sized groups.

Exgentic alone records two further fields. Tool definitions enumerate every tool available to the model, not just the one it selected, while request parameters capture settings such as temperature and max_tokens for each call. Most datasets record only what happened. These fields also record what could have happened, enabling counterfactual analyses such as whether a different temperature or an unused tool would have changed the outcome.

What each dataset actually keeps

The table above is the summary. Underneath it, each dataset is a specific set of fields, and the fields decide whether a question can be answered at all. In the diagram below, each box is a field. Each box lists the datasets that record that field, and a faded name means partial support. Click a dataset in the legend to highlight only the fields it has.

From a trace file to a running workload: the replay mechanisms

A trace format is only half of the practical story. To benchmark a serving stack, something has to turn the recording back into requests. Here, replay does not always mean resending the original bytes. Some replay paths preserve a session's dependency graph and feed each live model response into the next turn. Others preserve only arrival times, input and output lengths, or prefix-sharing relationships, and generate substitute payloads with the same serving shape. Those are both useful experiments, but they answer different questions.

Mechanism	Execution model	Scheduling	Payload construction	Target
Inference-Perf	Builds a dependency graph for each session and releases calls as their predecessors complete	Controls session concurrency or start rate; dependency-relative waits can be capped	Uses recorded or reconstructed inputs and, where supported, substitutes live predecessor outputs. OTel replay can duplicate sessions and add cache-isolating markers.	A live inference endpoint through its chat API
AIPerf	Converts each dataset through a format-specific adapter into conversations or independent requests	Supports recorded fixed schedules or closed-loop conversations, plus delay caps and concurrency controls	Uses recorded messages when present; otherwise synthesizes prompts from token lengths and cache-prefix information. Supports filtering, model rewriting, and cache isolation.	A live OpenAI-compatible inference endpoint
GuideLLM	Converts trace rows into independent requests or multi-turn conversations through a format-specific loader	Uses trace timestamps and conversation dependencies; supports time scaling, wait caps, and minimum session concurrency	Generates prompts from token lengths and hash IDs. Where supported, it models subagents and tool calls with supplied or built-in schemas and placeholder results.	A live OpenAI-compatible inference endpoint
vLLM bench serve	Treats each timed-trace row as an independent request	Schedules requests from their timestamps; timestamp units are configurable	Deterministically expands hash IDs into shared token blocks up to the recorded input length; field names and block size are configurable	A live online-serving endpoint, commonly vLLM
SGLang bench_serving	Treats each trace row as the start of a synthetic multi-round session	Can use trace timestamps, adjusted by a slowdown factor; the configured rounds within a session are emitted as a burst	Builds prompts from hash IDs, adds synthetic user turns and placeholder assistant responses, and uses a configurable output length	A supported live serving endpoint, including SGLang and OpenAI-compatible backends
NVIDIA DynoSim	Runs trace events through simulated engine cores in virtual time	Uses trace timestamps with a configurable speedup and stopping horizon	Maps token lengths and prefix hashes into a configurable model of workers, scheduling, KV capacity, routing, and aggregated or disaggregated serving	Simulated Dynamo engine cores, without HTTP or live workers

Inference-Perf, AIPerf, GuideLLM, vLLM bench serve, and SGLang bench_serving drive live inference endpoints, but their format-specific loaders do not necessarily preserve the same parts of a trace. DynoSim instead evaluates a simulated serving topology. Compare results only where the replay target, preserved trace properties, and reported metric align.

That boundary matters when reading benchmark results. Request-level throughput from two endpoint load generators can be compared if model, hardware, concurrency, timing policy, and payload construction are aligned. It is not directly comparable to end-to-end session latency from a causal graph replay, which includes dependency and branch effects. Nor is either number directly comparable to a DynoSim prediction, where no model is decoding tokens. The mechanisms can share a source trace without measuring the same system.

Trace dataset	Replay mechanism	Dataset-specific replay	Documentation
WEKA / AgentX	Inference-Perf	Reconstructs WEKA's parent-child turns, subagent branches, live-output flow, and recorded think-time gaps, subject to a configurable cap.	Weka Trace Replay
	AIPerf	Reconstructs WEKA's parent/subagent graph, infers flattened branches from hash-prefix and timing evidence, synthesizes cache-equivalent prompts, and follows recorded timing.	Replay WEKA Agentic Coding Traces
	GuideLLM	Preserves WEKA conversations and subagent spawn/join topology; tool-call turns use supplied or synthetic tool definitions and placeholder results.	Trace Replay: WEKA
TraceLab	AIPerf	Groups rounds into conversations, synthesizes prompts and prefix-block IDs from token counts, and heuristically attaches subagents by timing containment.	Replay TraceLab Agentic Coding Traces
Mooncake	AIPerf	Replays timestamps, token lengths, and optional hash IDs; it can instead send exact preformatted messages and tool definitions when a trace provides them.	Trace Replay with Mooncake Traces
	GuideLLM	Replays each Mooncake row independently, preserving timestamp offsets, token lengths, and shared hash-ID prefixes with generated prompts.	Trace Replay: Mooncake
	vLLM bench serve	Schedules independent rows by timestamp and deterministically expands hash IDs into shared token blocks of the recorded input length.	Replay Timed Traces: Running MoonshotAI traces
	SGLang bench_serving	Starts each record at its trace timestamp, builds a hash-derived prompt, and emits the configured number of synthetic conversation rounds as a burst.	Bench Serving: Mooncake; Mooncake loader
	NVIDIA DynoSim	Runs a Mooncake-format trace's timestamps and hash-block demand through simulated Dynamo engine cores to predict routing, scheduling, latency, and prefix-cache reuse.	Run a DynoSim Simulation
SWE-smith	SGLang bench_serving	Replays pre-built multi-turn agentic traces round by round via the agentic-trace dataset, feeding the server's live reply back into the next round's history. SWE-smith / OpenHands trajectories are the documented example; the trace JSON is supplied by the user.	Bench Serving: agentic-trace
Baseten Trace	AIPerf	Streams large Baseten Parquet traces in bounded batches via the baseten_trace loader, projecting only the columns each replay option needs and replaying recorded prompts without regenerating them.	Trace Replay: Baseten, Baseten trace tutorial
Exgentic	Inference-Perf	Reconstructs Exgentic's OTel dependency graph and, by default, substitutes each live model response into downstream requests. It preserves shared-prefix and tool-call structure, using recorded tool-result messages rather than executing tools, and can duplicate sessions with cache-isolating markers.	Exgentic workload; OTel Trace Replay
	AIPerf	Sends self-contained recorded call snapshots, preserving messages, tool definitions, output limits, request controls, and start times. Fixed-schedule mode follows recorded offsets; closed-loop mode waits between calls, but neither mode feeds live responses into later requests. Recorded tool interactions are not executed.	Exgentic Agent Trace Replay

Native loaders only, generic JSON/CSV conversion paths are excluded. These entries describe dataset-specific behavior, while the mechanism table above identifies each system under test.

So which combination do you actually want?

It depends on the stage. A serving feature moves from a proof of concept on a few GPUs to sizing, a pre-production gate and production. Each stage asks a different question, so the trace and the replay mechanism change with it. The table pairs them the way we use them in llm-d and in our own disaggregated serving fleet for agent workloads.

Stage	Question	What the trace must keep	Trace	Replay
Proof of concept	Does the mechanism work, and by how much?	A controlled shape: input and output lengths, prefix sharing, concurrency.	Shared-prefix synthetic shapes; Mooncake hash-ID traces.	Inference-Perf for synthetic shapes; vLLM bench serve or AIPerf for Mooncake; DynoSim before the hardware exists.
Sizing	How much hardware, and where does throughput stop growing?	Session shape: prompts that grow turn by turn, subagent fan-out, gaps while tools run.	WEKA / AgentX.	AIPerf or Inference-Perf, swept across concurrency.
Pre-production	Does the full stack behave with real agent requests?	Content: messages, tool definitions, request parameters, and the dependency between calls.	Exgentic.	Inference-Perf's dependency-graph replay; functional checks on the release images; live agents.
Production	Is it still right, and what changed?	Live traffic, measured at the router and at the engines.	Live traffic (no trace).	Controlled load ramps through the same gateway, read against production metrics.

Each row names the pairing we reach for first at that stage.

Proof of concept. The point is to change one thing and see what it does, so everything else stays fixed. llm-d's guides run through llm-d-benchmark, which drives Inference-Perf, AIPerf, GuideLLM or vLLM bench serve. The prefix-routing guide uses an Inference-Perf shared-prefix synthetic workload, because routing only cares about which prefixes repeat. We test engine and routing changes the same way on a separate GLM-5.3 test deployment (H200, disaggregated prefill and decode): every run uses the same trace and concurrency, and a fresh random seed so it never reads another run's cache. Replaying WEKA / AgentX through AIPerf at 64 concurrent sessions, switching vLLM's decode cudagraph_mode from FULL_DECODE_ONLY to FULL_AND_PIECEWISE raised average per-request decode speed by 28% (one 15-minute run per setting).

Sizing. How much hardware you need depends on how prompts grow over a session, how subagents fan out and how long tools run, so the trace needs real sessions. We replay WEKA / AgentX through AIPerf at rising concurrency, as llm-d's agentic-serving and P2P guides do. Each session waits for its own responses, so at a given concurrency a slower system finishes fewer requests per second, and raising concurrency to recover throughput costs each user speed. That is why we compare systems on throughput per GPU against speed per user. Our production agent workloads reuse 92% of prompt tokens, and Inference-Perf's catalog measures Exgentic at about 91% (92% of its calls share a prefix with an earlier one). Sizing also depends on how the deployment is set up, so report the setup with the numbers: GPU type, how prefill and decode are split, parallelism, and where KV cache can live. With reuse this high, KV cache capacity decides how much of that reuse turns into cache hits. Offloading KV cache to CPU memory adds that memory to the cache. Without offloading, the same hit rate needs more GPU memory.

Pre-production. Now the request content matters. Tool definitions and settings like reasoning effort change how a request turns into tokens, and so whether it hits the cache. Synthetic prompts have neither. By looking at real requests we found rendering issues whose fixes raised prompt reuse from 59% to 96%. Exgentic is the only corpus here that keeps tool definitions and request parameters, and Inference-Perf replays it the way an agent runs, feeding each response into the next call and keeping the recorded gaps between calls, capped at 15 s by default.

We also ran 500 of our own agents for an hour next to interactive users, to check priority and fairness. Live agents behave like the real thing, but no two runs are the same. Exgentic sits in between: real agent runs on fixed tasks that can be replayed as often as needed.

Production. Live traffic is the benchmark here, since no trace matches it. We watch latency and throughput per tenant, compare each release with the one before, and ramp up synthetic agent sessions through the same gateway to see how much headroom is left.

The practical check before any run: understand what the trace was designed to represent and exactly how the harness replays it. Otherwise, you can run a perfectly valid benchmark that does not measure the behavior you actually care about.

Put together, benchmarking moves from cheap, controlled shapes early on to real agent content close to release, and what production shows feeds the next round. Synthetic and hash-based traces keep the early stages fast and repeatable, and agent traces with full content, like Exgentic replayed through Inference-Perf, cover the last check before a change reaches users.

Where to get them

Most are public today: Exgentic Agent LLM Traces v2, WEKA / AgentX, and SWE-smith on HuggingFace, and TraceLab and Mooncake on GitHub. Baseten Trace ships as an AIPerf loader format rather than a published corpus.

This post covered what each dataset lets you learn: which questions it can and can't answer, given what it keeps. A separate question is how the underlying sessions came to exist: generated from a benchmark, like Exgentic and SWE-smith, or recorded from real usage, like WEKA/AgentX, TraceLab, and Mooncake. That changes what a dataset can honestly claim about the behavior in its traces, however much each trace captures. It deserves its own post, coming next.