Seeing Through the Stack: End-to-End and Fine-Grained Tracing in llm-d

Production llm-d deployments usually already expose Prometheus metrics, GPU utilization, and latency histograms. Those signals answer whether the fleet is healthy. They rarely answer the question that dominates incident response and optimization work:

For this slow or failed request, which hop decided what, and where did the time actually go?

That gap is structural. llm-d’s value sits in control-plane decisions: precise prefix-cache scoring, load-aware selection, prefill/decode (P/D) disaggregation, multi-hop proxies. Fleet aggregates cannot reconstruct those decisions after the fact. Distributed tracing can.

This post walks through two complementary capabilities now available across the llm-d stack:

1. End-to-end (E2E) distributed tracing: one trace_id across Gateway → EPP → KV-cache → P/D proxy → vLLM
2. Fine-grained tracing: first-party spans at the decision points that make well-lit paths real

Together they move observability from “the cluster looks busy” to “this request chose this pod for this cache score, ran prefill here, and spent most of its time to first token waiting in the decode queue.”

Blog key takeaways

• Metrics detect; traces explain. Prometheus/Grafana remain the detection and capacity plane; OpenTelemetry traces are the request-level causal spine for RCA and optimization validation.
• E2E tracing propagates W3C Trace Context (traceparent / tracestate) so Gateway, EPP, KV-cache, P/D proxy, and vLLM share one tree, not four orphan timelines.
• Fine-grained tracing manually instruments llm-d decisions (prefix-cache scoring, KV index lookup, P/D profile pick, proxy stages) that HTTP auto-instrumentation cannot name.
• Telemetry is metadata-only: token counts, latencies, routing attributes, cache ratios, never prompts or completions.
• Production defaults: parent-based ratio sampling at 10% unless you override it, an OpenTelemetry Collector in the middle, and GenAI semantic conventions where engines emit them.

FIGURE 1: Metrics, traces, and logs answer different questions. This post focuses on traces as the request spine.

Why LLM inference breaks classical “just add APM” advice

Kubernetes-era Application Performance Monitoring (APM) habits assume relatively uniform microservices: short RPCs, similar cost per call, cheap retries. Distributed LLM inference violates those assumptions in ways that change what must be instrumented:

Classical microservice	Distributed LLM inference (llm-d)
Request cost is roughly similar	Prefill vs decode, prompt length, and cache hit rate swing cost by orders of magnitude
Stateless replicas are interchangeable	KV-cache locality makes pods not interchangeable
Latency is “service time + queue”	Latency is admission + scoring + optional P/D coordination + engine phases
One hop failure is obvious	Failure can surface far from the decision that caused it
Auto-instrumented HTTP spans are enough	The important story is inside the scheduler and cache index

llm-d already invests heavily in making those control-plane decisions correct; see Intelligent Inference Scheduling and KV-Cache Wins You Can See. Observability has to keep pace: if a precise prefix-cache win or a P/D mis-decision cannot be inspected on a real request, those paths cannot be operated or improved with confidence.

From an observability design standpoint, that means tracing is a first-class well-lit-path concern (configuration, sampling, span contracts, and backend wiring), not an optional debug switch flipped after an outage.

Design goals for llm-d tracing

The distributed tracing proposal and the operator guide Distributed Tracing target five outcomes:

1. Performance diagnostics: decompose TTFT / ITL / e2e latency across hops
2. Optimization validation: prove KV-aware routing and P/D policy behave as intended on individual requests
3. Error attribution: keep failures on the same tree as the scheduling decision that preceded them
4. Cost / usage attribution: token counts and cache effectiveness per request, not only fleet averages
5. Safe-by-default telemetry: metadata and timings without shipping user content into the backend

Those outcomes require both breadth (E2E context propagation) and depth (fine-grained decision spans). Either alone is incomplete: a connected tree of opaque HTTP spans still hides why the scheduler chose a path; rich local spans without shared context still force timestamp archaeology.

Before: the fragmented request story

Early llm-d rollouts (and many DIY inference platforms) typically show one of three incomplete pictures:

Metrics-only. Dashboards for queue depth, GPU utilization, TTFT histograms. Excellent for SLOs and capacity planning. Blind for a single request’s path through scorers, proxies, and engines.

Component-local traces. vLLM emits llm_request; the gateway logs a request ID; the P/D sidecar keeps local timers. Without shared context, engineers glue timelines together with wall-clock timestamps and hope clocks agree.

Generic HTTP spans only. Auto-instrumentation draws POST /v1/chat/completions between services. It does not record why the prefix-cache scorer preferred pod B, or why the profile handler completed the request on decode alone instead of running a prefill leg first.

FIGURE 2: The operational difference E2E + fine-grained tracing is meant to create.

In observability terms: the stack had a detection plane without a reliable explanation plane.

Use cases the instrumentation targets

These are the questions that drove span placement and attribute design.

1. Where did TTFT go?

A chat or agent turn feels slow. Metrics report TTFT p95 is high. With a full trace, latency decomposes along:

Gateway admission → EPP scoring → optional P/D prefill → decode-side first-token attributes from the engine.

Debate about “network vs GPU” becomes secondary until the waterfall has identified the dominant span on the critical path.

FIGURE 3: Recommended RCA loop: detect with metrics, explain with traces, confirm with logs, verify the fix on subsequent sampled trees.

2. Did KV-cache-aware routing actually help?

Fleet-level cache hit rate can look healthy while a slice of traffic is still routed poorly. Fine-grained spans expose candidate endpoints, score distributions, and block hit ratios for that request. Which spans appear depends on how routing is configured: scorer-based setups emit scoring with one scorer.<scorer type> child per scorer, carrying llm_d.epp.scorer.score.max / .avg and llm_d.epp.scorer.endpoints_scored as attributes. The precise prefix-cache path adds produce_precise_prefix_cache (llm_d.epp.producer.total_blocks, .max_match_blocks) over an index_lookup (llm_d.kv_cache.lookup.cache_hit, .blocks_found). Prefix affinity there is not a scorer, so it appears as prefix-cache-affinity-filter shrinking the pool under filter_endpoints, while any scorers you also configure keep emitting their own scoring subtree next to it. On a warm repeat of a shared prefix, that subtree is where the hit becomes visible: in a capture on a two-pod deployment, the producer reported max_match_blocks equal to total_blocks at 13 of 13, the lookup reported cache_hit=true, and filter_endpoints narrowed two candidates to the single pod already holding those blocks. That is how precise prefix-cache scheduling is validated in production, in the same spirit as the wins in the KV-cache blog, but request-scoped.

The affinity filter's own routing decision (sticky to the cache-affine pod, no match, overridden by load, or skipped for exploration) is captured today as the llm_d_epp_prefix_cache_affinity_filter_decisions_total metric, labeled by outcome, not yet as a span; a fine-grained span for it is tracked in llm-d-router#2539.

3. Was P/D the right call?

Disaggregation is not free. pick_disagg_profile records each stage decision on llm_d.epp.profile_handler.decision: run_decode, then run_prefill or skip_prefill, and finally complete_prefill-decode or complete_decode-only. The handler is called once per stage, so a P/D request carries several of these spans rather than one. When separation adds coordination overhead instead of TTFT benefit, the rationale is on the span, ready for threshold tuning with evidence rather than folklore.

4. Which component failed, and after which decision?

Errors set span status and remain attached to the parent tree. A decode failure still sits under the Gateway root that includes the scoring and profile decision that sent the request there. Mean time to resolution drops because causality does not have to be reconstructed from three log systems.

5. Can tokens and cache effectiveness be attributed without leaking prompts?

vLLM’s llm_request spans carry GenAI semantic convention attributes (prompt/completion token counts, latency breakdowns). Combined with llm-d routing attributes, each sampled request yields a cost shape, still without putting prompt text into Jaeger or Tempo.

Architecture: request path and telemetry path

llm-d tracing is built on four layers:

• Manual OpenTelemetry instrumentation in Gateway/EPP, KV-cache, and P/D proxy
• Upstream engine tracing (vLLM llm_request, and the same OTEL env pattern for other capable engines)
• W3C Trace Context propagation on HTTP hops
• OTLP → OpenTelemetry Collector → backend (Jaeger in the recipes; Tempo or commercial backends in production)

Sampling is parent-based ratio (parentbased_traceidratio), the only sampler these components accept, so one decision at the entrypoint covers the whole tree. For production traffic, start around 10% (OTEL_TRACES_SAMPLER_ARG=0.1) and adjust for QPS, retention cost, and whether staging needs denser samples.

FIGURE 4: The request path carries W3C context; every component exports OTLP to a Collector that forwards to the trace backend.

Component responsibilities

Layer	Span role	What to inspect
Gateway	request (SERVER): the root when the gateway's own tracing is off, otherwise a child of the gateway's span; injects traceparent / tracestate	Total e2e time; whether context reached children
EPP plugins	request_orchestration over scoring, profile pick, and pre-request spans	Why an endpoint / mode was chosen
KV-cache	index_lookup on the request path, index_add when speculative indexing records the request's own blocks, index_evict from KV-event processing; emitted inside the EPP process, so they carry the EPP service name in Jaeger; attributes keep the llm_d.kv_cache.* prefix even though the span names no longer do	Block hits, lookup cost, index churn
P/D proxy	<METHOD> <path> (otelhttp) → forward_request → conditional prefill/decode children	Stage timing, connector, skip reasons
vLLM (or peer engine)	Continued llm_request with GenAI attributes	Prefill/decode time, TTFT, token usage

Propagation and sampling details

A few mechanics matter when validating a deployment:

• Entry sampling wins. With parentbased_traceidratio, the gateway (or an upstream traced client) decides once. Downstream components should respect that parent decision so the tree stays complete; partial trees are worse than fewer trees.
• Headers are the contract. W3C traceparent / tracestate must survive Envoy, ext-proc, and sidecar hops. Dropped headers produce orphan llm_request spans that look “traced” but cannot join the gateway root.
• Collectors are not optional in production. Exporting OTLP directly from every pod to a UI backend skips batching, noise filtering (for example /metrics scrapes), and fan-out to long-term storage. The recipes put a Collector in the path for that reason.
• Service names are operational metadata. Set distinct OTEL_SERVICE_NAME values per role (vllm-prefill, vllm-decode, EPP/gateway name). Ambiguous names make multi-cluster Jaeger search painful.

Why manual instrumentation (not agents alone)

“Drop in an auto-instrumentation agent” is rejected as the sole control-plane strategy. Agents are useful for baseline HTTP/gRPC spans. They cannot emit:

• a profile-handler decision enum
• a block hit ratio from the KV index
• a score distribution across candidate pods

Those are llm-d product semantics. They belong in first-party spans with stable llm_d.* attribute names, the same way first-party PromQL surfaces in the observability docs.

Capability 1: End-to-end tracing (breadth)

E2E tracing answers: show the whole journey of this request.

The mechanism is intentionally standard:

1. Create (or continue) a root span at the gateway.
2. Propagate W3C context on every hop to the P/D proxy and model servers.
3. Export via OTLP so the Collector assembles one tree in Jaeger/Tempo.

Operationally, this removes timestamp archaeology. One trace_id surfaces gateway time, scheduling time, prefill, decode, and engine spans as parent/child relationships, even when those processes run in different pods (as long as they share the Collector/backend).

Acceptance check in Jaeger: the EPP service name, which the router chart registers as llm-d-router/epp, and the engine names you set, such as vllm-prefill / vllm-decode, appear on the same trace. If only generic GET/POST spans appear, export is often working while llm-d/engine detailed instrumentation is not; follow the checklist in the tracing guide.

Capability 2: Fine-grained tracing (depth)

E2E alone can still leave a fat opaque span inside the scheduler. Fine-grained tracing opens those boxes with spans that name llm-d decisions.

FIGURE 5: Fine-grained spans map onto the control-plane decisions that define llm-d.

Span map (representative)

Span	Insight
request_orchestration	The EPP-side envelope every decision span hangs from
filter_endpoints / pick_endpoints	Candidate set before and after filters; top endpoints and scores
scoring → scorer.<scorer type>	Endpoints scored, score max/avg, scorer name and weight
produce_precise_prefix_cache	Resident-block match for the request (total_blocks, max_match_blocks)
index_lookup / index_add / index_evict	Lookup and speculative add on the request path; eviction from KV-event processing
pick_disagg_profile	Per-stage disaggregation decision (run_prefill, complete_decode-only, …)
prepare_disaggregation	Whether prefill / encode headers were set for the proxy
<METHOD> <path> → forward_request → prefill / decode	Stage timing, connector, targets, skip reasons
llm_request (vLLM)	Engine TTFT, phase times, token usage

One naming detail is worth knowing before searching Jaeger: the scorer span uses the plugin type, not its instance name, so a profile that configures the same scorer twice (for example two endpoint-attribute-scorer instances scoring different attributes) produces two identically-named scorer.endpoint-attribute-scorer spans, distinguishable only by the llm_d.epp.scorer.name attribute.

Naming has now converged: every span name in llm-d-router is bare (request, request_orchestration, filter_endpoints, scoring, scorer.<type>, pick_disagg_profile, index_lookup, prefill, decode, the P/D proxy root <METHOD> <path>, …); component prefixes only remain on attributes (llm_d.epp.*, llm_d.kv_cache.*, llm_d.pd_proxy.*). The last holdouts, the gateway. prefix on the EPP root spans and the llm_d.epp./llm_d.pd_proxy. prefixes on the scoring and P/D-proxy stage spans, were dropped in llm-d-router#2557 and llm-d-router#2558. The tracing guide is the source of truth if that changes again.

On the KV-cache side, a wrapper such as NewTracedIndex keeps tracing orthogonal to the index implementation: business logic stays clean; span names and attributes stay consistent. That pattern, instrumentation as a decorator, is the preferred control-plane approach over copy-pasted StartSpan calls inside every algorithm.

Attribute contract (what belongs on a span)

# KV index lookup (index_lookup)
llm_d.kv_cache.lookup.cache_hit = true
llm_d.kv_cache.lookup.blocks_found = 48
llm_d.kv_cache.index.lookup.block_count = 52

# Scoring (llm_d.epp.scorer.<scorer type>)
llm_d.epp.scorer.score.max = 0.91
llm_d.epp.scorer.score.avg = 0.44
llm_d.epp.scorer.endpoints_scored = 8

# Disaggregation profile handler (pick_disagg_profile)
llm_d.epp.profile_handler.decision = run_prefill
gen_ai.request.model = meta-llama/Llama-3.1-8B-Instruct

# vLLM / GenAI semantic conventions
gen_ai.usage.prompt_tokens = 128
gen_ai.usage.completion_tokens = 512
gen_ai.latency.time_to_first_token = 0.015

In scope: timings, counts, enums, bounded identifiers (model name, request ID where cardinality is managed), cache ratios, error status.

Out of scope by design: raw prompts, completions, full token ID lists, unbounded label sets. Payload debugging belongs in controlled logging, not the trace backend.

Span status follows a minimal convention: leave success as unset/default; set Error only on failures. Traces carry flow and timing; detailed exception text stays in logs correlated by trace_id where needed.

After: one request, one tree

Here is one real request, captured end to end on a live P/D deployment and exported from Jaeger. This is the shape to recognize once E2E and fine-grained tracing are enabled:

FIGURE 6: A real 25-span trace for a single disaggregated chat completion (2,071 prompt tokens, 200 completion tokens) across four services. EPP tracing came from the router chart; the routing sidecar and both vLLM engines were wired by hand.

Three things in that capture are worth dwelling on, because they are easy to get wrong from the waterfall alone.

Routing is usually not where the time goes. The entire EPP orchestration here, three profile decisions, two scheduler profiles, filtering, scoring, and picking, took 0.295ms out of 3146ms, so those scheduling spans are microsecond-scale. The exception worth knowing is the precise prefix-cache path, where the EPP calls the engine's render endpoint to tokenize the prompt before it can hash blocks: in a separate capture that single HTTP child accounted for 7.1ms of a 7.6ms orchestration. Still small against engine time, but no longer invisible. Either way, when a request is slow the trace tells you immediately whether to look downstream.

decode really is nested under prefill. With the default nixlv2 connector the decode span is started from the prefill span's context, so Jaeger renders it as a child rather than a sibling. This capture confirms it on main, and the same context reuse is present in v0.9.0. A prefill span reporting 49.9ms while containing a 3092ms child is that quirk, not a slow prefill. Read the stage durations from the attributes the proxy puts on the decode span, llm_d.pd_proxy.prefill_duration_ms and llm_d.pd_proxy.decode_duration_ms, rather than from the nesting.

Two TTFT numbers, two questions. The P/D proxy sidecar recorded llm_d.pd_proxy.true_ttft_ms = 50, derived from the prefill leg, while the decode engine independently reported gen_ai.latency.time_to_first_token = 0.343 with gen_ai.latency.time_in_queue = 0.322. Neither is wrong; they measure different things, and only a trace that carries both makes the difference legible. Also note the prefill engine's completion_tokens = 1: the prefill leg is sent with max_tokens=1, visible as gen_ai.request.max_tokens on that engine span, so prefill reports a single token while the decode engine reports the full 200.

One deployment detail affects the root span. This capture ran the standalone epponly topology, where an Envoy sidecar fronts the EPP rather than a Kubernetes Gateway, and that proxy was not itself traced, so request is the root. Where the gateway is traced, or a payload processor runs ahead of the EPP, request becomes a child of that span instead.

Compared with Figure 2: one ID, explicit parent/child edges, and attributes that explain routing and P/D policy, not only HTTP status codes.

What this unlocks for platform teams

1. Faster bottleneck localization: decompose the critical path of a real request before changing scorer weights or P/D thresholds.
2. Optimization validation: treat precise prefix-cache and P/D as measurable per-request behaviors, not faith-based features.
3. Shared technical vocabulary: scheduling, KV, and serving work can refer to the same stable span names in design reviews and incident channels.
4. Safer default telemetry: metadata-only tracing is easier to enable in regulated environments.
5. Vendor-neutral ops: OTLP + W3C + GenAI conventions; swap Jaeger for Tempo (or a commercial backend) without re-instrumenting llm-d.

Traces do not replace metrics. PromQL still owns SLOs, Grafana still owns fleet views, and alerts still own detection. Traces are what keeps the step from alert to action out of guesswork.

Getting started (operator checklist)

Full steps live in the docs; the mental model is:

1. Install the telemetry plane: OTel Collector + Jaeger (or another backend) via the observability recipes.
2. Enable model-server tracing: for vLLM, pass --otlp-traces-endpoint, add --collect-detailed-traces all if you want the queue and model-phase latencies used above, and set OTEL_SERVICE_NAME on the pod. The engine ships no service name of its own, so without it every engine span arrives as unknown_service, which reads like broken instrumentation rather than a missing variable.
3. Enable EPP tracing: one switch, router.tracing, in the llm-d-router chart that deploys your EPP. It sets the EPP's --tracing flag along with its exporter, endpoint, and sampler, so nothing else is needed there.
4. Wire the P/D routing sidecar yourself: the EPP switch above does not reach it, and the shipped P/D model-server manifests carry no tracing configuration at all. That container needs --tracing=true, which defaults to false, and its own OTEL_TRACES_EXPORTER=otlp plus OTEL_EXPORTER_OTLP_ENDPOINT. Miss the flag and the proxy stages simply never appear; miss the exporter variable and the spans are pretty-printed to the pod's stdout instead, because the default exporter is console. Both failures look exactly like the dropped-header symptom above.
5. Send one request, then open the UI: confirm a multi-service tree, not orphan engine spans.
6. Turn sampling down for production: start near 0.1 unless actively debugging or soaking in staging. One trap here: parentbased_traceidratio is the only sampler the llm-d components accept. Any other value, including standard OpenTelemetry names like parentbased_always_on, logs an error and falls back to ratio 0.1, as does a OTEL_TRACES_SAMPLER_ARG that fails to parse as a float. A debug run you believed captured everything then quietly keeps a tenth.

Docs:

• Observability overview
• Distributed Tracing
• Metrics

Production sampling

Sampling every request (ratio 1.0) is fine for demos and small staging clusters. For production inference traffic, prefer parent-based ratio sampling and a Collector that batches and filters noise (for example /metrics scrape spans). Retention and cardinality costs climb quickly with QPS.

Roadmap: tightening the observability loop

E2E and fine-grained tracing are a foundation, not an end state. Near-term work in the observability area continues to focus on:

• Tighter metrics ↔ traces correlation (exemplars / trace links from Grafana panels into Jaeger/Tempo)
• Stable span and attribute conventions across llm-d components so dashboards and runbooks can rely on names
• Better empty-trace diagnostics when only generic HTTP spans appear
• Multi-cluster / multi-tenant service-naming guidance (OTEL_SERVICE_NAME discipline)

Gaps (missing spans on a path that matters, attributes needed for a well-lit path, Collector recipes for a specific backend) are welcome as issues and proposals in the llm-d observability channels and GitHub.

Closing

llm-d’s advanced serving features only pay off if they can be seen working on live traffic. End-to-end tracing supplies the map of the request. Fine-grained tracing labels the turns that matter: cache scores, P/D mode, proxy stages, engine phases.

That is the observability bar for operating llm-d as a distributed inference platform: not merely hosting GPUs with charts on the side, but maintaining a request spine that holds up under pressure.