Skip to main content
Version: dev

LiteLLM for Hosting llm-d Inference Stack and External APIs

This guide demonstrates how to deploy a LiteLLM proxy (Helm chart, PostgreSQL-backed) on Kubernetes to provide a single, OpenAI-compatible entry point that fronts both an existing llm-d inference stack and external LLM provider APIs.

Overview​

By placing LiteLLM in front of your infrastructure, a single proxy entry point serves:

  1. In-cluster model endpoints: An existing llm-d inference stack serving models in your Kubernetes cluster (e.g. Qwen/Qwen3-32B).
  2. External provider APIs: Third-party LLM services (e.g. Google Gemini via a Google AI Studio API key).

End users can issue OpenAI-compatible API calls using virtual keys generated by LiteLLM to access self-hosted models or external provider APIs without managing individual provider API credentials directly.

Prerequisites​

  1. Kubernetes Cluster: A running Kubernetes cluster with kubectl configured.
  2. llm-d Inference Stack: An active llm-d deployment set up using the Optimized Baseline hosting a model (e.g., Qwen/Qwen3-32B).
    • Gateway Mode (Default): Reached via the Kubernetes Gateway IP (http://<gateway-ip>/v1).
    • Standalone Mode (Optional): Reached directly via the Endpoint Picker (EPP) Service: http://optimized-baseline-epp.llm-d-optimized-baseline.svc.cluster.local:80/v1.
  3. External Provider API Key: An API key from Google AI Studio for gemini/gemini-3.5-flash.
  4. Local Tools: kubectl, helm, openssl, and curl.
note

Validated Versions: This guide was tested and verified with the LiteLLM Helm chart 1.94.1 (oci://ghcr.io/berriai/litellm-helm), LiteLLM proxy v1.90.2 (ghcr.io/berriai/litellm-database), PostgreSQL 16, and Redis 7.4-alpine.

Set up the LiteLLM target namespace:

export NAMESPACE=litellm

kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -

Step 1: Create Secrets for LiteLLM​

Generate random keys for administrative management and database encryption, and create the Kubernetes Secrets required by LiteLLM:

export DB_PASSWORD="$(openssl rand -hex 24)"
export GEMINI_API_KEY="<your-gemini-api-key>"

# 1) Core secret: Admin master key, salt key (encrypts stored creds), and DB creds
kubectl -n "$NAMESPACE" create secret generic litellm-secrets \
  --from-literal=master-key="sk-$(openssl rand -hex 24)" \
  --from-literal=salt-key="sk-$(openssl rand -hex 24)" \
  --from-literal=db-username="litellm" \
  --from-literal=db-password="$DB_PASSWORD"

# 2) Environment secret: Provider API keys (+ salt key for the chart's env)
SALT=$(kubectl -n "$NAMESPACE" get secret litellm-secrets -o jsonpath='{.data.salt-key}' | base64 -d)
kubectl -n "$NAMESPACE" create secret generic litellm-env \
  --from-literal=LITELLM_SALT_KEY="$SALT" \
  --from-literal=GEMINI_API_KEY="$GEMINI_API_KEY"
info

Replace <your-gemini-api-key> with your actual Google AI Studio API key.

Step 2: Set Up PostgreSQL and Redis Backends​

LiteLLM requires two storage tiers:

  1. PostgreSQL (Durable State): Stores virtual keys, user spend metrics, model aliases, and audit logs. Deployed as a StatefulSet with a persistent volume.
  2. Redis (Ephemeral Coordination): Coordinates cross-pod rate limiting (RPM/TPM), budget locking, and spend counters across proxy replicas. Deployed as a lightweight Deployment using the official upstream image (redis:7.4-alpine).

Deploying both components as standalone Kubernetes resources avoids deprecated, unpinned third-party subcharts and provides full control over versions and resources.

1. Deploy PostgreSQL StatefulSet​

Create postgres.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          env:
            - name: POSTGRES_DB
              value: litellm
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: db-username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: db-password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - containerPort: 5432
          readinessProbe:
            exec:
              command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\""]
            initialDelaySeconds: 10
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432

2. Deploy Redis Coordination Store​

Create redis.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7.4-alpine
          ports:
            - containerPort: 6379
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 5
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  selector:
    app: redis
  ports:
    - port: 6379
      targetPort: 6379

Apply both manifests and wait for readiness:

kubectl apply -n "$NAMESPACE" -f postgres.yaml -f redis.yaml

kubectl -n "$NAMESPACE" rollout status statefulset/postgres
kubectl -n "$NAMESPACE" rollout status deployment/redis

The LiteLLM proxy communicates with PostgreSQL at postgres:5432 and Redis at redis:6379 within the same namespace.

Step 3: Construct LiteLLM Helm Values and Install​

Create a values.yaml configuration file for the LiteLLM Helm chart. The model_list array configures the routing for both self-hosted llm-d models and external providers.

Configuration (values.yaml)​

replicaCount: 3

image:
  repository: ghcr.io/berriai/litellm-database   # Bundled Prisma client for DB migrations
  tag: "v1.90.2"

# Run Redis as a standalone Deployment (Step 2) rather than the chart's
# optional bundled subchart, so the image and version stay under your control.
redis:
  enabled: false

# Admin master key secret reference
masterkeySecretName: litellm-secrets
masterkeySecretKey: master-key

# Database integration via in-cluster StatefulSet
db:
  useExisting: true
  deployStandalone: false
  endpoint: "postgres:5432"
  database: litellm
  secret:
    name: litellm-secrets
    usernameKey: db-username
    passwordKey: db-password

# Pass provider keys as environment variables to LiteLLM
environmentSecrets:
  - litellm-env

# Execute DB migrations as a Helm hook Job
migrationJob:
  hooks:
    helm: { enabled: true }
    argocd: { enabled: false }

# Disable runtime schema updates on proxy pods (handled by migration job)
envVars:
  DISABLE_SCHEMA_UPDATE: "True"

proxy_config:
  # Global router coordination: cross-replica rate limiting and model routing state
  router_settings:
    redis_host: redis
    redis_port: 6379

  model_list:
    # ─────────────────────────────────────────────────────────────────
    # 1) DEFAULT: llm-d Gateway Mode
    # Points to your Kubernetes Gateway IP (from `kubectl get gateway`).
    # ─────────────────────────────────────────────────────────────────
    - model_name: qwen/qwen3-32B
      litellm_params:
        model: hosted_vllm/Qwen/Qwen3-32B
        api_base: http://<gateway-ip>/v1
        api_key: "none"          # llm-d requires no API key
      # Self-hosted models have no public price. These are chargeback rates —
      # amortized GPU cost, not a vendor price. Replace with your own:
      #   tokens_per_hour = 3600 x sustained_output_tok/s x utilization
      #   output_cost_per_token = (GPU $/hr x GPUs) / tokens_per_hour
      # Without these, LiteLLM records spend=0 for llm-d models and the
      # max_budget below only ever counts Gemini.
      model_info:
        input_cost_per_token: 0.0000008   # $0.80 per 1M input tokens
        output_cost_per_token: 0.0000024  # $2.40 per 1M output tokens

    # ─────────────────────────────────────────────────────────────────
    # 2) OPTIONAL: llm-d Standalone Mode
    # Points directly to the EPP Service endpoint (bypassing Gateway API).
    # ─────────────────────────────────────────────────────────────────
    - model_name: qwen/qwen3-32b-standalone
      litellm_params:
        model: hosted_vllm/Qwen/Qwen3-32B
        api_base: http://optimized-baseline-epp.llm-d-optimized-baseline.svc.cluster.local:80/v1
        api_key: "none"
      # Self-hosted models have no public price. These are chargeback rates —
      # amortized GPU cost, not a vendor price. Replace with your own:
      #   tokens_per_hour = 3600 x sustained_output_tok/s x utilization
      #   output_cost_per_token = (GPU $/hr x GPUs) / tokens_per_hour
      # Without these, LiteLLM records spend=0 for llm-d models and the
      # max_budget below only ever counts Gemini.
      model_info:
        input_cost_per_token: 0.0000008   # $0.80 per 1M input tokens
        output_cost_per_token: 0.0000024  # $2.40 per 1M output tokens

    # ─────────────────────────────────────────────────────────────────
    # 3) EXTERNAL API: Google Gemini
    # Uses the Google AI Studio API key injected via GEMINI_API_KEY.
    # ─────────────────────────────────────────────────────────────────
    - model_name: gemini-flash
      litellm_params:
        model: gemini/gemini-3.5-flash
        api_key: os.environ/GEMINI_API_KEY
note
  • What Redis Does and Doesn't Enforce: Model-level rate limits are checked in Redis, so they hold across all replicas. Virtual key limits (rpm_limit on /key/generate) are counted in each pod's memory and synced on a delay — with 3 replicas, a key limited to 60 RPM can briefly reach ~180. Spend and budgets are written to PostgreSQL and lag similarly. For a hard per-key ceiling, run a single replica or divide rpm_limit by the replica count.
  • Self-Hosted Model Cost Tracking (model_info): Commercial models (like gemini-3.5-flash) have pre-configured pricing in LiteLLM's public pricing dictionary. Self-hosted models (hosted_vllm/*) require explicit model_info with input_cost_per_token and output_cost_per_token so LiteLLM can track usage spend and deduct from virtual key budgets accurately.

Install via Helm​

Deploy LiteLLM using the official OCI Helm chart:

helm install litellm oci://ghcr.io/berriai/litellm-helm \
  -n "$NAMESPACE" -f values.yaml

kubectl -n "$NAMESPACE" rollout status deploy/litellm

Step 4: Verify Both Models​

Retrieve the deployed LiteLLM proxy service:

kubectl -n "$NAMESPACE" get svc

Expected output:

NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
litellm    ClusterIP   <cluster-ip>   <none>        4000/TCP   12h
postgres   ClusterIP   None           <none>        5432/TCP   12h
redis      ClusterIP   <cluster-ip>   <none>        6379/TCP   12h

1. Establish Port Forwarding​

In a separate terminal, forward port 4000:

kubectl -n "$NAMESPACE" port-forward deploy/litellm 4000:4000

2. Retrieve Configured Models (/v1/models)​

Fetch the admin master key and query the /v1/models endpoint:

MK=$(kubectl -n "$NAMESPACE" get secret litellm-secrets -o jsonpath='{.data.master-key}' | base64 -d)

curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer $MK" | jq .

Expected output showing both self-hosted and external models:

{
  "object": "list",
  "data": [
    {
      "id": "qwen/qwen3-32B",
      "object": "model",
      "created": 1677610602,
      "owned_by": "openai"
    },
    {
      "id": "qwen/qwen3-32b-standalone",
      "object": "model",
      "created": 1677610602,
      "owned_by": "openai"
    },
    {
      "id": "gemini-flash",
      "object": "model",
      "created": 1677610602,
      "owned_by": "openai",
      "max_input_tokens": 1048576,
      "max_output_tokens": 65535
    }
  ]
}

3. Generate Scoped Virtual Key with Budget and Rate Limits​

Use the admin master key to provision a scoped virtual key with model access restrictions, a $10 budget cap, and a 60 RPM rate limit:

KEY_RESP=$(curl -s http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $MK" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["qwen/qwen3-32B", "gemini-flash"],
    "max_budget": 10.0,
    "rpm_limit": 60,
    "duration": "30d",
    "user_id": "team-analytics"
  }')

echo "$KEY_RESP" | jq .

# Export the generated virtual key for client inference:
export VIRTUAL_KEY=$(echo "$KEY_RESP" | jq -r .key)

Expected output:

{
  "key": "<generated-virtual-key>",
  "max_budget": 10.0,
  "spend": 0.0,
  "models": ["qwen/qwen3-32B", "gemini-flash"],
  "rpm_limit": 60,
  "duration": "30d",
  "user_id": "team-analytics"
}

4. Test Inference Requests with Virtual Key​

Client applications use their scoped virtual key ($VIRTUAL_KEY) rather than the admin master key.

A. Call Self-Hosted Model (qwen/qwen3-32B via llm-d)​

curl -s -m 45 http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3-32B",
    "messages": [{"role": "user", "content": "hi"}],
    "max_tokens": 16
  }' | jq .

Expected output (containing vllm fingerprint, confirming execution through llm-d):

{
  "id": "<response-id>",
  "object": "chat.completion",
  "created": 1783624185,
  "model": "qwen/qwen3-32B",
  "system_fingerprint": "vllm-0.23.0-tp2-f1888500",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "<think>\nOkay, the user sent \"hi\". That's pretty casual. I"
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 16,
    "total_tokens": 25
  }
}

B. Call External Model (gemini-flash via Google AI Studio API)​

curl -s -m 45 http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-flash",
    "messages": [{"role": "user", "content": "hi"}],
    "max_tokens": 16
  }' | jq .

Expected output (translated OpenAI format returned from Gemini):

{
  "id": "<response-id>",
  "object": "chat.completion",
  "created": 1783624118,
  "model": "gemini-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I help you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 2,
    "completion_tokens": 9,
    "total_tokens": 11
  }
}

5. Check Spend Tracking and Remaining Budget​

Query the database via LiteLLM's key management endpoint to verify spend auditing:

curl -s "http://127.0.0.1:4000/key/info?key=$VIRTUAL_KEY" \
  -H "Authorization: Bearer $MK" | jq .

Expected output:

{
  "key": "<generated-virtual-key>",
  "info": {
    "spend": 0.00042,
    "max_budget": 10.0,
    "models": ["qwen/qwen3-32B", "gemini-flash"],
    "rpm_limit": 60,
    "user_id": "team-analytics"
  }
}

Cleanup​

To remove the LiteLLM proxy, standalone Redis coordination store, PostgreSQL database, and all associated resources:

helm uninstall litellm -n "$NAMESPACE"
kubectl delete -n "$NAMESPACE" -f redis.yaml -f postgres.yaml
kubectl delete namespace "$NAMESPACE"