Router Operations Guide

This guide covers operational best practices, high availability deployment architectures, and container sizing recommendations for the llm-d Router components.

---

1. Endpoint Picker Operations

When deploying the Endpoint Picker (EPP) in either Standalone or Gateway mode, resource allocations and multi-replica scaling behaviors depend on expected query throughput, prefix cache matching complexity, and high availability (HA) requirements.

High Availability & Scaling Modes

When running multiple replicas of the Endpoint Picker (router.epp.replicas > 1), its behavior depends on the configured HA mode.

Active-Passive Mode (Default)

By default, multi-replica EPP deployments automatically enable the --ha-enable-leader-election flag. One leader replica actively serves routing decisions and coordinates lease status, while remaining replicas act as warm standbys.

• Sizing & Capacity Impact: Scaling replica count does not increase total request throughput capacity, as only the single active leader replica handles external processing requests.
• Fail Open: EPP is configured to fail open (router.proxy.failOpen=true) to the model server in the event of a leader election or pod restart. This ensures that client requests are not dropped and continue to be processed by the model server if EPP is unresponsive or unavailable.

router:
  proxy:
    failOpen: true

Active-Active Mode

To scale routing throughput concurrently across all EPP replicas, disable leader election by passing ha-enable-leader-election: false under router.epp.flags:

router:
  epp:
    replicas: 3
    flags:
      ha-enable-leader-election: false

• Near-Linear Throughput Scaling: Multiple EPP replicas share incoming request load concurrently:

  Replicas	Scaling Factor
  1	1.0x
  2	2.0x
  3	2.7x
  4	3.5x

• Warning (Plugin & Prefix Compatibility): In active-active mode, you must only use active-active compatible plugins—specifically plugins that query backend model servers dynamically for real-time metrics and state (such as queue depth or KV-cache utilization scorers). Avoid approximate prefix caching plugins in active-active mode; because replicas do not share local memory state, prefix routing partitions across replicas and degrades cache hit rates significantly.

Container Resource Sizing

CPU Allocation

• Rule of Thumb: Allocate 0.5 to 1.0 CPU cores per request/second of expected throughput for large agentic workloads (~100k input / 1k output tokens).
• Prefix Matching Overhead: Increasing maxPrefixTokensToMatch increases CPU consumption. At lower throughputs, a large prefix limit (such as 100,000 tokens / 6,250 blocks with blockSizeTokens: 16) can increase CPU consumption by over 100% compared to a small limit (4,096 tokens / 256 blocks) due to block search overhead.
• Idle Scraping Overhead: Idle CPU consumption scales with total model-serving pods due to background Prometheus scraping. In a cluster with 100 pods, EPP idle consumption reaches approximately 7.5 cores.

Memory Allocation

• Inflight Concurrency: Memory footprint scales directly with concurrent inflight requests and output decode length.
• Sizing Guidelines:
  • At 50 to 100 requests/second with 1k output tokens, EPP requires 4 to 6 GiB of memory.
  • For long-output generation (e.g., 5k+ output tokens), memory footprint can exceed 20 GiB due to concurrent request state accumulation.

Performance Reference Data

Empirical benchmark reference data for Qwen/Qwen3-8B simulation across 100 serving pods:

Throughput and Prefix Block Sizing (100k Input / 1k Output Tokens)

Configuration	Request Rate (Req/s)	maxPrefixTokensToMatch	Peak CPU (Cores)	Peak Memory (GiB)	Scheduler P50 Latency (s)
Small Prefix Match	5.0	4096	1.19	0.26	0.00010
Large Prefix Match	5.0	100000	3.82	0.65	0.00010
Small Prefix Match	98.7	4096	35.17	2.46	0.00014
Large Prefix Match	98.8	100000	46.50	3.41	0.00020

Output Length Variation (50 Req/s Constant Throughput)

Input Tokens	Output Tokens	maxPrefixTokensToMatch	Peak CPU (Cores)	Peak Memory (GiB)
100k	500	4096	15.13	2.27
100k	500	32768	17.14	3.76
100k	1000	4096	17.51	3.66
100k	1000	32768	20.28	5.23
100k	5000	16384	30.95	12.54
100k	10000	8192	32.53	12.54

---

2. Proxy Operations in Standalone Mode

The following operational guidelines and proxy scaling architectures apply exclusively to Standalone Mode (llm-d-router-standalone), where a proxy (Envoy or Agentgateway) intercepts client requests and external-processes them via EPP.

Horizontally Scalable Proxy Service (Service Mode)

By default, the standalone chart deploys the proxy as a sidecar container inside the EPP pod. To scale data plane throughput independently from control plane intelligence, deploy the proxy as a separate horizontally scalable Deployment and Service by setting router.proxy.mode=service.

In this decoupled architecture, the proxy communicates with EPP over the in-cluster EPP Service. If EPP undergoes active-passive leader failover or momentary pod restarts, the proxy fails open by default (router.proxy.failOpen=true), preserving uninterrupted client request processing.

helm install my-standalone-router ./config/charts/llm-d-router-standalone \
  --set router.modelServers.matchLabels.app=my-vllm-service \
  --set router.inferencePool.create=false \
  --set router.proxy.mode=service \
  --set router.proxy.replicas=3

High Availability with Fail-open

By default, in service mode, fail open is enabled. To disable it, set router.proxy.failOpen=false.

Empirical benchmark reference data for Qwen/Qwen2.5-1.5B-Instruct simulation across 2 model server replicas with forceful and graceful Leader EPP pod termination. Across various Envoy proxy replica counts, router.proxy.failOpen=true maintains high request availability during leader pod teardown. Residual errors occur during socket teardown at the exact moment of pod termination.

Scenario	Envoy Replicas	EPP Replicas	Total Requests	Successful Requests (Throughput)	Errors
Scenario 1 Leader election with standby pod, pod terminates immediately without graceful termination	1	2	598	598 (100%)	0
	2	2	591	584 (98.8%)	7
Scenario 2 Leader election with no standby pod, pod terminates immediately without graceful termination	1	1	598	588 (98.3%)	1
	2	1	592	589 (99.3%)	4
Scenario 3 Leader election with standby pod, pod terminates with graceful termination	1	2	593	591 (99.7%)	2
	2	2	594	583 (98.1%)	11
Scenario 4 Leader election with no standby pod, pod terminates with graceful termination	1	1	591	591 (100%)	0
	2	1	593	589 (99.3%)	4

Proxy Container Resource Sizing

When running Envoy as the standalone proxy, CPU consumption scales linearly with client request rate, while memory consumption remains stable across workloads.

CPU & Memory Guidelines

• CPU Allocation: For < 10 requests/second, 1.2 to 2.0 cores is sufficient. For 100 requests/second at 100k context lengths, allocate at least 8 cores (peak observed at 7.27 cores). For high concurrency at smaller context lengths (892 requests/second at 10k context), allocate at least 10 cores.
• Memory Footprint: Envoy memory footprint remains stable between 1.3 and 1.4 GiB across all tested throughputs and context lengths. Allocate 2 GiB baseline.

Envoy Performance Reference Data

Input Tokens	Output Tokens	Throughput (Req/s)	Peak CPU (Cores)	Peak Memory (GiB)
100k	1k	10.0	1.20	1.30
100k	1k	100.0	7.27	< 1.40
10k	1k	892.0	8.78	1.40

Helm Resource Override Example

Example resource_overrides.yaml configuring container resources for both EPP and standalone Envoy proxy containers supporting 50 requests/second for 100k/1k token workloads:

router:
  epp:
    resources:
      requests:
        cpu: "32"
        memory: "64Gi"
      limits:
        memory: "128Gi"

  proxy:
    resources:
      requests:
        cpu: "8"
        memory: "2Gi"
      limits:
        memory: "4Gi"

helm install optimize-baseline ./config/charts/llm-d-router-standalone -f resource_overrides.yaml