Skip to main content

Quickstart

This guide provides a simplified, end-to-end walkthrough for deploying an Optimized Baseline configuration using llm-d. This setup reduces tail latency and increases throughput through load-aware and prefix-cache aware balancing.

For this quickstart, we will use the Standalone Mode deployment, which is the easiest way to get started with llm-d.

Prerequisites​

  • Installed proper client tools (kubectl, helm).

  • Set the following environment variables:

    export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
    source ${REPO_ROOT}/guides/env.sh
    export GUIDE_NAME="quickstart"
    export NAMESPACE=llm-d-quickstart

  • Checkout llm-d repo:

      export branch="main"
      git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${branch}

  • Install the Gateway API Inference Extension CRDs:

    kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${GAIE_VERSION}/v1-manifests.yaml

  • Create a target namespace for the installation:

    kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

  • Create the llm-d-hf-token secret in your target namespace with the key HF_TOKEN matching a valid HuggingFace token to pull models.

export HF_TOKEN=<your HuggingFace token>
kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

Installation Instructions​

1. Deploy the llm-d Router (Standalone Mode)​

The llm-d Router provides the intelligent load balancing. In Standalone Mode, it includes a built-in proxy (Envoy).

helm install ${GUIDE_NAME} \
    ${ROUTER_STANDALONE_CHART} \
    -f guides/recipes/router/base.values.yaml \
    -f guides/optimized-baseline/router/optimized-baseline.values.yaml \
    -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

2. Deploy the Model Server​

Deploy the default model server (vLLM running on NVIDIA GPUs). This will deploy 8 replicas of Qwen/Qwen3-32B by default.

kubectl apply -n ${NAMESPACE} -k guides/optimized-baseline/modelserver/gpu/vllm/
tip

If you are using different hardware (AMD, Intel, TPU, or CPU), you can find alternative configurations in the guides/optimized-baseline/modelserver/ directory.

Verification​

1. Get the IP of the Proxy​

Retrieve the ClusterIP of the llm-d Router service:

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

2. Send a Test Request​

Open a temporary interactive shell inside the cluster to send a request:

kubectl run curl-debug --rm -it \
    --image=cfmanteiga/alpine-bash-curl-jq \
    --namespace="$NAMESPACE" \
    --env="IP=$IP" \
    -- /bin/bash

Inside the shell, send a completion request:

curl -X POST http://${IP}/v1/completions \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "Qwen/Qwen3-32B",
        "prompt": "How are you today?"
    }' | jq

Cleanup​

To remove all resources created in this guide:

helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}
kubectl delete namespace ${NAMESPACE}