EndpointPickerConfig (EPP Configuration)
EndpointPickerConfig defines the internal configuration for the Endpoint Picker (EPP). Unlike Kubernetes resources (like InferencePool), this is a configuration schema used to initialize the EPP binary, typically provided via a ConfigMap or a local file.
Group: llm-d.ai
Version: v1
note
llm-d.ai/v1alpha1 is still accepted but deprecated; the EPP converts it to v1 at startup and logs a deprecation warning. See Configuration for the field migration.
Required marks fields the EPP rejects a configuration without. The Go API types also annotate plugins and schedulingProfiles as required, but the loader accepts a configuration without them and supplies defaults.
EndpointPickerConfig​
| Field | Description |
|---|---|
featureGates | []string A set of flags to enable experimental features (e.g., flowControl). |
plugins | []PluginSpec List of plugins to be instantiated (e.g., scorers, adapters, reporters). A plugin must be listed here to be referenced from a profile. Default plugins (picker, profile handler, parsers, saturation detector) are injected when absent. |
schedulingProfiles | []SchedulingProfile Named profiles that group plugins into routing slots. If omitted, a single default profile is created with all listed filters, scorers, and pickers. |
dataLayer | DataLayerConfig Configures the DataLayer for metadata extraction and processing. |
flowControl | FlowControlConfig Configures global and per-priority admission control. Most fields are respected only if the flowControl feature gate is enabled; saturationDetector is honored regardless. |
requestHandler | RequestHandlerConfig Specifies the handling logic used by the EPP to process incoming requests. |
PluginSpec​
Defines a plugin instance and its parameters.
| Field | Description |
|---|---|
name | string Unique name for this plugin instance. If omitted, type is used. |
type | string Required The plugin type to instantiate (e.g., max-score-picker, openai-parser). |
parameters | json.RawMessage Arbitrary parameters passed to the plugin's factory function. |
SchedulingProfile​
Groups plugins to define specific routing behavior.
| Field | Description |
|---|---|
name | string Required Name of the profile. |
plugins | []SchedulingPlugin List of plugins associated with this profile. A max-score-picker is appended if the profile has no picker. |
SchedulingPlugin​
| Field | Description |
|---|---|
pluginRef | string Required Reference to a named plugin in the top-level plugins list. |
weight | float64 Weight used if the plugin is a Scorer. |
FlowControlConfig​
Configures admission control and queuing.
| Field | Description |
|---|---|
maxBytes | resource.Quantity Global maximum aggregate byte size of all active requests. |
maxRequests | resource.Quantity Global maximum number of concurrent requests. |
defaultRequestTTL | duration Fallback timeout for queued requests. |
noEndpointRequestTTL | duration Queue-wait timeout while the candidate pool has no endpoints. If omitted, follows defaultRequestTTL. |
defaultPriorityBand | PriorityBandConfig Template for priority levels not explicitly configured. |
defaultNegativePriorityBand | PriorityBandConfig Template for priority levels below zero. If omitted, negative priorities use defaultPriorityBand. |
priorityBands | []PriorityBandConfig Explicit policies for specific priority levels. |
usageLimitPolicyPluginRef | string Reference to a UsageLimitPolicy plugin for adaptive capacity management. |
bandSelectionPolicyPluginRef | string Reference to a BandSelectionPolicy plugin that orders priority bands during dispatch. |
saturationDetector | SaturationDetectorConfig Specifies which saturation detector plugin to use. Defaults to utilization-detector. |
enableEviction | bool Enables eviction of in-flight negative-priority requests to reclaim capacity when higher-priority requests are blocked. Defaults to false. |
PriorityBandConfig​
| Field | Description |
|---|---|
priority | int Integer priority level. Higher is more critical. |
maxBytes | resource.Quantity Max bytes allowed for this priority band. |
maxRequests | resource.Quantity Max concurrent requests allowed for this band. |
defaultRequestTTL | duration Queue-wait timeout for this band while the candidate pool has endpoints. If omitted, uses the global defaultRequestTTL. |
fairnessPolicyRef | string Policy governing flow selection (default: global-strict-fairness-policy). |
orderingPolicyRef | string Policy governing request selection within a flow (default: fcfs-ordering-policy). |
RequestHandlerConfig​
Configures request handling behavior.
| Field | Description |
|---|---|
parsers | []ParserConfig List of parsing plugins used to process protocol messages. If unspecified, openai-parser, anthropic-parser, and vllmhttp-parser are configured by default. |
propagatePriority | bool Injects the resolved request priority into the outbound request body for backend-native priority scheduling. Defaults to false. |
DataLayerConfig​
| Field | Description |
|---|---|
injectDefaults | bool Controls automatic injection of the default metrics source and extractor. Defaults to true. Set to false to disable all default source injection. |
sources | []DataLayerSource List of metadata sources. |
discovery | DiscoveryConfig Endpoint and peer discovery plugins. If omitted, the EPP uses Kubernetes-based endpoint discovery and disables peer discovery. |
crossReplica | CrossReplicaConfig Publishes local per-endpoint state to other EPP replicas and reads theirs back. If omitted, no cross-replica syncer is used and plugins fall back to local data. |
DiscoveryConfig​
| Field | Description |
|---|---|
endpoints | EndpointDiscoveryConfig The EndpointDiscovery plugin used to populate the endpoint datastore. When set, the EPP bypasses the Kubernetes reconcilers and relies on the plugin to enumerate endpoints, enabling operation without a Kubernetes cluster. |
peers | PeerDiscoveryConfig The PeerDiscovery plugin used to discover peer EPP replicas. If omitted, peer discovery is disabled. |
EndpointDiscoveryConfig​
| Field | Description |
|---|---|
pluginRef | string Required Reference to a plugin instance that implements EndpointDiscovery. |
PeerDiscoveryConfig​
| Field | Description |
|---|---|
pluginRef | string Required Reference to a plugin instance that implements PeerDiscovery. |
CrossReplicaConfig​
| Field | Description |
|---|---|
syncerPluginRef | string Reference to the plugin instance used as the cross-replica syncer. |
syncInterval | duration Cadence at which each replica publishes its local per-endpoint state, independent of the datalayer polling interval. A default is used if omitted. |
publishTimeout | duration Bound on one endpoint publish, including all concurrent contributor writes. A default is used if omitted. |
DataLayerSource​
| Field | Description |
|---|---|
pluginRef | string Required Reference to a plugin providing the data source. |
extractors | []DataLayerExtractor Required Plugins that extract specific attributes from the source. |
DataLayerExtractor​
| Field | Description |
|---|---|
pluginRef | string Required Reference to a plugin that extracts attributes from the source. |
SaturationDetectorConfig​
| Field | Description |
|---|---|
pluginRef | string Reference to a plugin instance for saturation detection. |
ParserConfig​
| Field | Description |
|---|---|
pluginRef | string Required Reference to a parser plugin instance. |