Observability

Monitor and debug llm-d deployments with Prometheus metrics, Grafana dashboards, and OpenTelemetry distributed tracing.

note

Every well-lit path guide links here for observability setup. Install the stack once and reuse it across guides.

Documentation

• Set Up the Monitoring Stack — Install Prometheus and Grafana, load dashboards, and deploy tracing backends
• Collect Metrics — Scrape model server and EPP metrics into Prometheus and query them
• Use Grafana Dashboards — Access Grafana and import the llm-d dashboards
• Model Server Metrics — vLLM and SGLang metric reference
• Router (EPP) Metrics — llm-d Router Endpoint Picker metric reference
• Batch Gateway Metrics — Batch Gateway metric reference
• Track Inference Cost — Per-model cost attribution with OpenCost
• PromQL Query Reference — Ready-to-use queries for dashboards and alerting
• Configure Alerts — Apply the default EPP and Batch Gateway Prometheus alerting rules
• Trace Requests — Configure OpenTelemetry across vLLM, the routing proxy, and the EPP
• Monitor GKE TPUs — Interpret TPU hardware metrics and troubleshoot missing data

Runnable assets

Scripts, Grafana dashboard JSON, alerting rules, and tracing manifests live in guides/recipes/observability/ in the llm-d repository (not published as website pages).