Skip to main content
Version: dev

[Experimental] Batch Gateway

Batch Gateway provides an OpenAI-compatible Batch API for submitting, tracking, and managing large-scale batch inference jobs. It is designed to efficiently process batch workloads alongside interactive workloads on shared infrastructure.

Overview​

When to Pick This Path​

  • You have offline inference workloads (such as evaluations, embeddings, dataset processing) that don't need real-time responses.
  • You want to utilize idle accelerator capacity for batch work while protecting interactive traffic from interference.
  • Your clients expect an OpenAI-compatible Batch API (/v1/batches, /v1/files) for job submission, tracking and management.
  • You need multi-tenant isolation — each tenant's jobs, files, and results are separated.

For queue-based asynchronous inference of individual requests, see the complementary Asynchronous Processing path. For components, data flow, and the processing pipeline, see the Batch Gateway Architecture.

Key Features​

  • Process large-scale batch jobs: Process batch jobs with up to 50,000 (configurable) inference requests per job, with progress tracking and job management capabilities.
  • Provide an OpenAI-compatible Batch API: Full schema parity with OpenAI's /v1/batches and /v1/files endpoints.
  • Optimize resource utilization: Enables batch and interactive workloads to coexist on shared infrastructure. Integrates with flow control mechanisms to adjust the batch dispatch rate based on downstream metrics.
  • Multi-tenant isolation: Per-tenant data isolation with pluggable authentication and authorization.

Components​

  1. API Server — REST API server for batch job submission, management, tracking, and file management.
  2. Batch Processor — Pulls jobs from a priority queue, builds per-model execution plans, dispatches inference requests to the llm-d Router, and writes results to output files.
  3. Garbage Collector — Cleans up expired jobs and files.

Storage Layer​

Batch Gateway uses pluggable storage backends. Each function is backed by a single plug-in, chosen at deployment time:

FunctionAvailable plug-ins
Jobs and files metadataPostgreSQL, Redis/Valkey (development/test only)
Priority queue, events, status updatesRedis/Valkey
File storage (input/output)S3, Filesystem

Prerequisites​

Before installing Batch Gateway, ensure you have:

  1. Kubernetes cluster: A running Kubernetes cluster (v1.25+).
    • For local development, you can use Kind or Minikube.
    • For production, OpenShift, GKE or AKS are supported.
  2. Helm: 3.0+
  3. llm-d Inference Stack: Batch Gateway requires an existing optimized baseline stack to dispatch requests to.
  4. PostgreSQL: 12+ for metadata storage. Redis/Valkey are available as an alternative for development/test only.
  5. Redis/Valkey: Redis 6+ or Valkey 8+ for priority queue, events, and status updates.
  6. S3 or Filesystem: For batch input and output file storage.

Get the guide​

The observability steps below apply alert rules and dashboards from a local clone of the llm-d repository. Set the branch and clone the repo (if you already have a checkout, skip this and run the remaining commands from inside it):

export BRANCH=main
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}
export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))

Installation​

Step 1: Create the Namespace​

export NAMESPACE=batch-gateway
kubectl create namespace ${NAMESPACE}

Step 2: Create the Secrets​

Batch Gateway requires a Kubernetes Secret with database and storage credentials:

kubectl create secret generic batch-gateway-secrets -n ${NAMESPACE} \
  --from-literal=redis-url="redis://redis-master.redis.svc.cluster.local:6379/0" \
  --from-literal=postgresql-url="postgresql://user:password@postgresql.postgresql.svc.cluster.local:5432/batchgateway"

Note: s3-secret-access-key is only required when using the S3 file storage backend (global.fileClient.type=s3). This guide uses the filesystem backend, so it is omitted. Add --from-literal=s3-secret-access-key="<your-s3-secret-key>" if you deploy with S3.

Step 3: Configure the llm-d Router URL​

The Batch Processor needs to know where to send inference requests. This is configured via the processor.config.globalInferenceGateway.url Helm value, or per-model via processor.config.modelGateways.

Single gateway (all models route to one endpoint). With the optimized baseline in Standalone mode, that is its llm-d Router Service; in Gateway mode, use the Gateway address instead:

export INFERENCE_GW_URL="http://optimized-baseline-epp.llm-d-optimized-baseline.svc.cluster.local:80"

Per-model gateways (different endpoints per model): see the Helm chart README for modelGateways configuration.

Step 4: Create the File Storage PVC​

This guide uses the filesystem storage backend (global.fileClient.type defaults to fs). Both the API server and the processor mount the same volume, so it must support ReadWriteMany (RWX). Create the PVC referenced by the deploy step:

kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: batch-gateway-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi  # example size, tune to your workload
EOF

Note: ReadWriteMany requires an RWX-capable storage class (e.g. NFS, CephFS). If your cluster only provides ReadWriteOnce, use the S3 file storage backend instead (global.fileClient.type=s3).

Step 5: Deploy​

From OCI Registry:

helm install batch-gateway oci://ghcr.io/llm-d/charts/batch-gateway \
  -n ${NAMESPACE} \
  --set processor.config.globalInferenceGateway.url="${INFERENCE_GW_URL}" \
  --set "apiserver.config.batchAPI.passThroughHeaders={Authorization}" \
  --set global.fileClient.fs.pvcName="batch-gateway-pvc"

From Source (a checkout of llm-d-batch-gateway):

git clone https://github.com/llm-d/llm-d-batch-gateway.git
helm install batch-gateway llm-d-batch-gateway/charts/batch-gateway \
  -n ${NAMESPACE} \
  --set processor.config.globalInferenceGateway.url="${INFERENCE_GW_URL}" \
  --set "apiserver.config.batchAPI.passThroughHeaders={Authorization}" \
  --set global.fileClient.fs.pvcName="batch-gateway-pvc"

Note: passThroughHeaders should include any authentication headers (e.g., Authorization) that the llm-d Router expects. The processor forwards these headers when dispatching individual inference requests.

Detailed Deployment Guide​

For a production deployment with authentication, authorization, and TLS on Kubernetes, see the Kubernetes deployment guide (Istio + Kuadrant + cert-manager).

Verification​

  1. Check that all pods are running:

    kubectl get pods -n ${NAMESPACE}

    You should see pods for apiserver, processor, and gc, all in Running state.

  2. Verify health endpoints:

    kubectl port-forward -n ${NAMESPACE} svc/batch-gateway-apiserver 8081:8081 &
    curl http://localhost:8081/health

Usage​

The examples below talk to the API server on port 8000. Port-forward it first (the health check in the previous section uses a separate port, 8081):

kubectl port-forward -n ${NAMESPACE} svc/batch-gateway-apiserver 8000:8000 &

Upload an Input File​

Prepare a JSONL input file with one request per line (see the OpenAI Batch API format):

{"custom_id": "req-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "my-model", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}}
{"custom_id": "req-002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "my-model", "messages": [{"role": "user", "content": "What is llm-d?"}], "max_tokens": 200}}

Upload the file:

curl -X POST http://localhost:8000/v1/files \
  -F "purpose=batch" \
  -F "file=@batch_input.jsonl"

Create a Batch Job​

curl -X POST http://localhost:8000/v1/batches \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "<file-id-from-upload>",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

Monitor Job Status​

curl http://localhost:8000/v1/batches/<batch-id> | jq '{status, request_counts}'

Download Results​

Once the job status is completed, retrieve the output file:

# Get the output file ID
OUTPUT_FILE_ID=$(curl -s \
  http://localhost:8000/v1/batches/<batch-id> | jq -r '.output_file_id')

# Download the results
curl http://localhost:8000/v1/files/${OUTPUT_FILE_ID}/content > results.jsonl

Enable Monitoring​

Optional. Requires Prometheus and Grafana — see Observability Setup.

  1. Enable metrics scraping in the chart, so Prometheus discovers the three components:

    helm upgrade batch-gateway oci://ghcr.io/llm-d/charts/batch-gateway \
      -n ${NAMESPACE} --reuse-values \
      --set processor.podMonitor.enabled=true \
      --set gc.podMonitor.enabled=true \
      --set apiserver.serviceMonitor.enabled=true
note

This step still uses the component chart's own PodMonitor/ServiceMonitor templates. Per llm-d#1983 these scrape manifests should eventually live in this repo as standalone recipes; the target path is still being settled, so this guide uses the chart flags in the meantime. The alerts and dashboards below are already served from this repo.

  1. Apply the alerting rules:

    kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/recipes/observability/alerts/batch-gateway-alerting-rules.yaml

    These are scoped to namespace="batch-gateway". If you changed ${NAMESPACE}, edit the matchers first — see Alerting.

  2. Load the dashboards:

    ${REPO_ROOT}/guides/recipes/observability/load-llm-d-dashboards.sh

    This loads every llm-d dashboard, including llm-d-batch-gateway-apiserver, -processor, and -gc. Select your namespace with each dashboard's namespace variable.

  3. Verify:

    kubectl get podmonitors,servicemonitors,prometheusrules -n ${NAMESPACE}

For what the metrics mean and how to query them, see the Batch Gateway metric reference.

Cleanup​

helm uninstall batch-gateway -n ${NAMESPACE}
kubectl delete namespace ${NAMESPACE}