Collect Metrics

This page covers how to scrape metrics from an llm-d deployment into Prometheus and query them. For monitoring stack setup, see Set Up the Monitoring Stack.

note

Commands in this page use ${NAMESPACE} for the namespace where your llm-d workload runs. Set it before following along:

export NAMESPACE=<your-llm-d-namespace>

Prerequisites

• A running llm-d deployment with an InferencePool and model servers — see the quickstart if needed
• A monitoring stack configured — see Set Up the Monitoring Stack

note

TPU hardware metrics require the GKE TPU monitoring recipe, which scrapes the GKE device-plugin exporter rather than the model server. Monitor GKE TPUs documents metric names, units, labels, and environment checks. Metric availability depends on the GKE runtime and TPU type; the vLLM image version alone does not determine that interface.

Step 1: Scrape Model Server Metrics

Model server metrics are enabled by default on port 8000 (at /metrics for vLLM and SGLang, and at /prometheus/metrics for TensorRT-LLM).

OSS Prometheus / OpenShift

Apply the Kustomize monitoring component to create PodMonitor (monitoring.coreos.com/v1) resources for /metrics (vLLM and SGLang):

kubectl apply -n ${NAMESPACE} -k guides/recipes/modelserver/components/monitoring

Or include the monitoring component directly in your kustomization.yaml:

components:
  - ../../../recipes/modelserver/components/monitoring       # decode PodMonitor
  # - ../../../recipes/modelserver/components/monitoring-pd  # add for prefill/decode disaggregation

See guides/recipes/modelserver/components/monitoring/ for details (for TensorRT-LLM, set path: /prometheus/metrics on the PodMonitor endpoint).

Verify the PodMonitor resources exist:

kubectl get podmonitors -n ${NAMESPACE}

Expected output (prefill also appears when monitoring-pd is applied):

NAME                    AGE
decode                  5m

GKE (Managed Prometheus)

When GKE automatic application monitoring is enabled (--auto-monitoring-scope=ALL; see GKE Monitoring & Telemetry), GKE automatically detects vLLM and SGLang model server workloads and provisions PodMonitoring (monitoring.googleapis.com/v1) resources labeled source: gke-auto-monitoring in your workload namespace. You do not need to manually create PodMonitor resources or apply the Kustomize monitoring component for vLLM or SGLang.

Verify the auto-generated PodMonitoring resource in your namespace:

kubectl get podmonitoring -n ${NAMESPACE} -l source=gke-auto-monitoring

Expected output:

NAME                                                                     AGE
gke-ac-9b1b9035d7cd5d78ca0d37904b667bddb67b31c156d4822687e8c3e1725f0947   1m

For TensorRT-LLM (trtllm-serve), which exposes Prometheus metrics at /prometheus/metrics rather than /metrics, apply a PodMonitoring resource targeting /prometheus/metrics:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: trtllm-decode
  namespace: ${NAMESPACE}
spec:
  selector:
    matchLabels:
      llm-d.ai/guide: optimized-baseline
      llm-d.ai/role: decode
  endpoints:
    - port: modelserver
      path: /prometheus/metrics
      interval: 15s

Step 2: Scrape EPP Metrics

The router Helm chart configures EPP metrics scraping via a monitoring values file (see Router Recipes — Enable Prometheus Monitoring):

OSS Prometheus / OpenShift

Layer guides/recipes/router/features/monitoring.values.yaml when installing or upgrading the router Helm chart to create a ServiceMonitor (monitoring.coreos.com/v1) for the EPP.

Verify the ServiceMonitor exists:

kubectl get servicemonitors -n ${NAMESPACE}

Expected output:

NAME                    AGE
epp-servicemonitor      5m

GKE (Managed Prometheus)

GKE automatic application monitoring creates PodMonitoring resources for model servers (vLLM and SGLang), not for the EPP deployment (*-epp). Layer guides/recipes/router/features/monitoring-gke.values.yaml when installing or upgrading the router Helm chart (add --set router.monitoring.provider.gmp.autopilot=true on GKE Autopilot) so the chart creates an authenticated PodMonitoring (monitoring.googleapis.com/v1) for the EPP (:9090/metrics), along with the metrics-reader ServiceAccount, token Secret, and RBAC bindings for gmp-system.

Verify the EPP PodMonitoring resource exists:

kubectl get podmonitoring -n ${NAMESPACE}

Expected output:

NAME                 AGE
optimized-baseline   1m

Step 3: Query Metrics in Prometheus

OSS Prometheus / OpenShift

Port-forward the Prometheus service:

kubectl port-forward -n llm-d-monitoring svc/llmd-kube-prometheus-stack-prometheus 9090:9090
# Open http://localhost:9090 (or https://localhost:9090 if TLS is enabled)

To visualize these metrics in Grafana, see Use Grafana Dashboards.

GKE (Managed Prometheus)

Open Metrics Explorer in the Google Cloud console (select PromQL mode to run Prometheus queries) or view the out-of-the-box Cloud Monitoring dashboards described in GKE Monitoring & Telemetry.

Metric Reference

Page	Covers
Model Server Metrics	vLLM (including NIXL KV transfer and KV offloading) and SGLang (including HiCache)
Router (EPP) Metrics	Requests and latency, inference pool, scheduler and plugins, flow control, prefix and multimodal cache, predicted latency
Batch Gateway Metrics	Batch Gateway API server, processor, and GC reconciler
Track Inference Cost	Per-model hourly cost and cost per million tokens via OpenCost

For ready-to-run queries, see the PromQL Query Reference.

Troubleshooting

Metrics not appearing in Prometheus

1. Check that scraping resources exist:

   • OSS Prometheus / OpenShift: kubectl get podmonitors,servicemonitors -n ${NAMESPACE}
   • GKE (Managed Prometheus): kubectl get podmonitoring -n ${NAMESPACE}

2. Verify the collector is scraping the targets. On the OSS Prometheus UI (http://localhost:9090/targets), check that model server and EPP targets show UP. On GKE, verify that the PodMonitoring status reports ConfigurationCreateSuccess=True.

3. Confirm pods expose metrics:

   MODEL_POD=$(kubectl get pods -n ${NAMESPACE} -l llm-d.ai/role=decode -o jsonpath='{.items[0].metadata.name}')
   kubectl port-forward -n ${NAMESPACE} ${MODEL_POD} 8000:8000
   curl http://localhost:8000/metrics | head -20

For Prometheus TLS and installation issues, see the setup troubleshooting.