Skip to main content
Version: v0.7

Quickstart Prometheus Grafana Stack

The prometheus-grafana setup script can be used to deploy a simple observability stack on Kubernetes.

Step 1: Install Monitoring Stack​

Skip if using OpenShift, GKE, or any other Kubernetes environment where Prometheus already exists and is accessible:

# Central monitoring (default - Prometheus monitors all namespaces)
./scripts/install-prometheus-grafana.sh

# Central monitoring in custom namespace
./scripts/install-prometheus-grafana.sh -n monitoring

# Individual user monitoring (isolated - Prometheus only monitors selected namespaces based on namespace labels)
./scripts/install-prometheus-grafana.sh --individual -n my-monitoring-namespace

Step 2: Enable Monitoring for Your Deployments​

Choose the approach that matches your monitoring setup:

Option A: Central Monitoring (Default)​

No additional configuration required! Central monitoring automatically discovers all ServiceMonitors and PodMonitors across all namespaces.

Option B: Individual User Monitoring​

For Prometheus to watch llm-d PodMonitors, label the namespace where llm-d is running.

# Replace 'my-monitoring-namespace' and 'my-llm-d-namespace' with your actual namespaces
kubectl label namespace my-llm-d-namespace monitoring-ns=my-monitoring-namespace

Step 3: Enable Metrics in Your Deployments​

In any llm-d helmfile example, update the modelservice values to enable monitoring:

# In ms-*/values.yaml files
decode:
  monitoring:
    podmonitor:
      enabled: true

prefill:
  monitoring:
    podmonitor:
      enabled: true

Step 4: Access Prometheus & Grafana UIs​

kubectl port-forward -n <your-monitoring-namespace> svc/llmd-kube-prometheus-stack-prometheus 9090:9090
# Visit http://localhost:9090

kubectl port-forward -n <your-monitoring-namespace> svc/llmd-grafana 3000:80
# Visit http://localhost:3000

# Grafana login: admin/admin

Cleanup​

To remove the Prometheus and Grafana stack:

./scripts/install-prometheus-grafana.sh -u -n <your-monitoring-namespace>

Distributed Tracing​

For OpenTelemetry distributed tracing across vLLM, the routing proxy, and the EPP/inference scheduler, see the Distributed Tracing guide.

To deploy an OTel Collector and Jaeger backend for viewing traces:

./scripts/install-otel-collector-jaeger.sh -n <your-namespace>