EPP HTTP Headers Reference
This document describes the HTTP headers that the Endpoint Picker (EPP) inspects to manage and control inference requests, specifically for flow control, performance management, and request classification.
Request Classification and Flow Control​
These headers allow the EPP to identify the request's goals, group them for fair resource allocation, and handle model-specific targeting.
| Header | Description |
|---|---|
x-llm-d-inference-objective | Specifies the name of the InferenceObjective resource associated with the request. The EPP uses this to look up the corresponding objective resource in the same namespace as the InferencePool to apply the defined priority and performance goals. |
x-llm-d-inference-fairness-id | Provides a unique identifier for grouping requests for fairness-based flow control. Requests with the same ID share capacity according to the fairness policy. If omitted, the EPP defaults to default-flow. |
x-llm-d-model-name-rewrite | Specifies the target model name to be used for the request. This is an alternative approach to model name rewriting; while the InferenceModelRewrite API provides rule-based rewriting on the server side, this header allows for an explicit, per-request override. When present, the EPP uses this value to override the model name in the request body and for recording model-specific metrics. |
Service Level Objectives (SLOs)​
These headers are used by admission control and load balancing plugins to make decisions based on latency targets.
| Header | Description |
|---|---|
x-llm-d-slo-ttft-ms | Specifies the target Time To First Token (TTFT) in milliseconds. Used by plugins to determine if a request can be admitted while meeting the latency goal. |
x-llm-d-slo-tpot-ms | Specifies the target Time Per Output Token (TPOT) in milliseconds. Used for admission control based on predicted or observed token generation latency. |
Deprecated Aliases​
llm-d Router v0.9.0 and later accept the previous EPP-managed header names as deprecated read aliases. New integrations and examples should use the canonical x-llm-d-* names. Earlier releases expect the previous names, so mixed-version fleets should coordinate the header migration during rolling upgrades. See the llm-d Router releases page for available versions.
If a request sends both a canonical header and its deprecated alias, the canonical x-llm-d-* value wins deterministically. No alias removal release or date has been announced; treat aliases as temporary compatibility support and prefer the canonical names for all new clients.
| Deprecated alias | Canonical header |
|---|---|
x-gateway-inference-fairness-id | x-llm-d-inference-fairness-id |
x-gateway-inference-objective | x-llm-d-inference-objective |
x-gateway-model-name-rewrite | x-llm-d-model-name-rewrite |
x-slo-ttft-ms | x-llm-d-slo-ttft-ms |
x-slo-tpot-ms | x-llm-d-slo-tpot-ms |
This rename applies only to llm-d/EPP-managed user and control headers. Gateway API Inference Extension (GAIE) Endpoint Picker Protocol headers, such as x-gateway-destination-endpoint*, are unchanged.
Response Headers​
These headers are set by the EPP on responses sent back to the client.
| Header | Description |
|---|---|
x-llm-d-request-dropped-reason | Indicates why a request was dropped by flow control or admission. Present on 429, 503, and 500 responses generated by the EPP (not forwarded from a model server). The value uses a two-prefix scheme: rejected-* means the request never reached an inference server, evicted means it was dispatched and then terminated. See the table below for possible values. |
Dropped Reason Values​
| Value | HTTP Status | Meaning |
|---|---|---|
rejected-saturated | 429 Too Many Requests | Pool or priority band is at capacity and cannot accept the request. |
evicted | 429 Too Many Requests | Request was terminated after dispatch to reclaim capacity for a higher-priority request. |
rejected-ttl-expired | 429 Too Many Requests | Request entered the queue but its TTL expired before dispatch while the pool still had endpoints. |
rejected-no-endpoints | 503 Service Unavailable | Pool has no serving endpoints, or a queued request's TTL expired while the pool was empty. |
rejected-context-cancelled | 503 Service Unavailable | Client disconnected while the request was queued. |
rejected-shutting-down | 503 Service Unavailable | The EPP flow controller is draining queued requests during a graceful shutdown. |
rejected-internal | 500 Internal Server Error | An internal error occurred in the flow controller or scheduler. |
The prefix convention helps consumers decide retry strategy: rejected-* means no inference work was done (cheap to retry), while evicted means GPU cycles were consumed (factor into backoff).
Implementation Notes​
- Case Sensitivity: All header lookups are case-insensitive.
- Source: These values are typically provided as standard HTTP headers in the incoming request.