Skip to main content
Version: dev

Asynchronous Processing with Async Processor

The Async Processor provides a way to process inference requests asynchronously using a queue-based architecture. This is ideal for latency-insensitive workloads or for filling "slack" capacity in your inference pool.

This page deploys the Async Processor for a single model. The queue backend — GCP Pub/Sub (default) or a Redis Sorted Set — is chosen in Installation, in tabs.

The Multi-tenant guide builds on it: the advanced setup of team × tier with per-team reserved/overflow quota (classifying redis-quota), tier-priority dispatch, and pool saturation back-off. It runs on either queue backend.

note

For production sizing, scaling, and container-resource guidance, see Async Processor Operations.

Overview​

This guide deploys the Async Processor (Helm chart llm-d-async) in front of an existing optimized baseline stack. It consumes requests from one message queue — GCP Pub/Sub by default, or a Redis Sorted Set, set up in Step 2 — and dispatches them to the llm-d Router, either directly to the router Service (Standalone mode) or through the Gateway (Gateway mode).

Clients enqueue work and read results from the result topic or list later instead of holding an HTTP connection open, and retries happen without touching real-time traffic. Dispatch gates, set in the Helm values, decide when queued work is released — for example only while the model servers have spare capacity.

For how the processor works — dispatch gates, worker pools, merge policies, retries and deadlines, and queue semantics — see the Async Processor Architecture.

When to Use This Path​

  • Batch Inference: Processing large datasets where completion time is measured in minutes or hours rather than milliseconds.
  • Slack Capacity Filling: Using idle GPU cycles between real-time request spikes to perform background tasks like document summarization or embedding generation.
  • Offline Evaluation: Running model evaluation pipelines without competing for production resources.

Prerequisites​

Before installing Async Processor, ensure you have:

  1. Kubernetes cluster: A running Kubernetes cluster (v1.31+).
    • For local development, you can use Kind or Minikube.
    • For production, GKE, AKS, or OpenShift are supported.
  2. Gateway control plane (Gateway mode only): if you run the optimized baseline behind a Gateway, configure and deploy your Gateway control plane (e.g., Istio) before installation. In Standalone mode the Async Processor dispatches to the llm-d Router directly and no Gateway is needed.
  3. llm-d Inference Stack: Async Processor requires an existing optimized baseline stack to dispatch requests to.

Get the guide​

Every command below runs from a local clone of the llm-d repository: the Helm values it applies live next to this guide. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Installation​

Async Processor can be installed via Helm. We recommend following the pattern used in the optimized baseline guide.

Step 1: Deploy llm-d Router​

Apply the optimized baseline guide and get the llm-d Router's IP address:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
# If using Standalone Mode:
export IP=$(kubectl get service optimized-baseline-epp -n llm-d-optimized-baseline -o jsonpath='{.spec.clusterIP}')

# If using Gateway Mode:
export IP=$(kubectl get gateway llm-d-inference-gateway -n llm-d-optimized-baseline -o jsonpath='{.status.addresses[0].value}')

Step 2: Set up the Queue Backend​

Set up one queue backend and fill in its values.yaml; Step 3 installs the chart with it.

GCP Pub/Sub is the default backend: cloud-native, scalable messaging on Google Cloud. You need a GCP project with the Pub/Sub API enabled. The chart runs the processor under a Kubernetes service account named after the Helm release (llm-d-async in the namespace you install into) and does not annotate it; grant that identity access as described below.

Topics and subscription. We recommend a topic per model and priority, i.e. per inference objective. For one model and one use case, create a single request topic, and a result topic and a dead-letter topic (DLQ):

export REQUEST_TOPIC_NAME=async-proc-requests   # request topic
export SUBSCRIPTION_NAME=async-proc-requests-sub # subscription for the request topic
export DLQ_NAME=async-proc-requests-dlq          # dead-letter topic
export RESULT_TOPIC_NAME=async-proc-results      # result topic

gcloud pubsub topics create $REQUEST_TOPIC_NAME
gcloud pubsub topics create $DLQ_NAME
gcloud pubsub topics create $RESULT_TOPIC_NAME

Give each request topic a subscription with exactly-once delivery, retries with exponential backoff, and a DLQ. Without a DLQ, every redelivery of an undeliverable message is counted again in the llm_d_async_async_request_total metric. Also subscribe to the DLQ topic so dead-lettered messages are not lost:

gcloud pubsub subscriptions create sub-$DLQ_NAME \
    --topic=$DLQ_NAME

gcloud pubsub subscriptions create $SUBSCRIPTION_NAME \
    --topic=$REQUEST_TOPIC_NAME \
    --dead-letter-topic=$DLQ_NAME \
    --max-delivery-attempts=35   \
    --enable-exactly-once-delivery

Pub/Sub forwards to the dead-letter topic as its own service agent, so that agent must be allowed to publish to the DLQ topic and to subscribe to the request subscription. gcloud only warns when these grants are missing, and undeliverable messages are then never dead-lettered:

export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format='value(projectNumber)')
export PUBSUB_SA="service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"

gcloud pubsub topics add-iam-policy-binding $DLQ_NAME \
    --member="serviceAccount:${PUBSUB_SA}" --role=roles/pubsub.publisher
gcloud pubsub subscriptions add-iam-policy-binding $SUBSCRIPTION_NAME \
    --member="serviceAccount:${PUBSUB_SA}" --role=roles/pubsub.subscriber

Grant the processor access to Pub/Sub. The processor needs the following roles:

RoleUsed for
roles/pubsub.subscriberpulling requests from $SUBSCRIPTION_NAME
roles/pubsub.publisherpublishing results to $RESULT_TOPIC_NAME
roles/pubsub.viewerthe readiness probe's GetSubscription on an idle subscription (a permission-denied answer is tolerated, but a granted viewer role gives you a real probe)
roles/monitoring.viewerthe llm_d_async_async_broker_backlog gauge, which reads the subscription backlog from Cloud Monitoring; without it the gauge is absent and llm_d_async_async_broker_backlog_source_available stays 0

With Workload Identity Federation for GKE you grant them to the Kubernetes service account's principal directly; no Google service account or annotation is needed. NAMESPACE must be the namespace you install into in Step 3:

export NAMESPACE=llm-d-async
export KSA_PRINCIPAL="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/llm-d-async"

for role in roles/pubsub.subscriber roles/pubsub.publisher roles/pubsub.viewer roles/monitoring.viewer; do
  gcloud projects add-iam-policy-binding ${PROJECT_ID} \
      --member="${KSA_PRINCIPAL}" --role="${role}" --condition=None
done

If you would rather use a Google service account, give it the same roles, bind it with roles/iam.workloadIdentityUser for ${PROJECT_ID}.svc.id.goog[${NAMESPACE}/llm-d-async], and annotate the Kubernetes service account after the Helm install in Step 3:

kubectl annotate serviceaccount llm-d-async -n ${NAMESPACE} \
    iam.gke.io/gcp-service-account=<gsa-name>@${PROJECT_ID}.iam.gserviceaccount.com

The multi-tenant guide's gcp-setup.sh scripts this service-account variant.

Values. In gcp-pubsub/values.yaml, replace REPLACE_WITH_YOUR_PROJECT in project_id, result_topic_id, and subscriber_id with your project ID (and the topic and subscription names, if you changed them). Leave igw_base_url; Step 3 sets it.

Step 3: Deploy Async Processor​

Deploy the Async Processor using the selected queue implementation's configuration:

export NAMESPACE=llm-d-async
export MQ_PROVIDER=gcp-pubsub # options are gcp-pubsub or redis
export ASYNC_VERSION=v1.0.0   # llm-d-async release

[ "$MQ_PROVIDER" = "redis" ] && TARGET_KEY="ap.transportConfig.queues[0].igw_base_url" || TARGET_KEY="ap.transportConfig.topics[0].igw_base_url"

helm install llm-d-async \
    oci://ghcr.io/llm-d/charts/llm-d-async \
    -f ${REPO_ROOT}/guides/batch-serving/asynchronous-processing/${MQ_PROVIDER}/values.yaml \
    --set ${TARGET_KEY}=http://${IP}:80 \
    -n ${NAMESPACE} --create-namespace --version ${ASYNC_VERSION}

Testing​

Wait for the Async Processor to be ready:

kubectl get pods -n ${NAMESPACE}

Then enqueue a request and read its result. In the request, deadline and created are Unix-seconds numbers, not strings — a quoted deadline fails to decode.

  1. Publish a message. On Pub/Sub the request is published on its own, not wrapped in an InternalRequest envelope; the consumer builds that itself from the Pub/Sub message.

    gcloud pubsub topics publish $REQUEST_TOPIC_NAME --message='{"id":"testmsg","created":1700000000,"deadline":1999999999,"payload":{"model":"your-model","prompt":"Hi, good morning"}}'

  2. Pull the result. Create a subscription for the result topic if you haven't already, then pull from it:

    gcloud pubsub subscriptions create async-proc-results-sub --topic=$RESULT_TOPIC_NAME
    gcloud pubsub subscriptions pull async-proc-results-sub --auto-ack --limit=1

Cleanup​

helm uninstall llm-d-async -n ${NAMESPACE}