Request Handler

Functionality

The Request Handler manages the lifecycle of an inference request before and after the request scheduling phase within the EPP. It handles parsing the request payload, preparing and managing state for the Request Scheduler, interacting with Flow Control and processing the response from the model server.

Design

Architecture Overview

Core Components

• Parser: Responsible for parsing the incoming request to a structured internal representation consumable by the request scheduler, and parsing the response to extract usage data if reported by the model server.
• DataProducer: A pluggable extension that allows customizing request pre-processing and producing per-request state needed for scheduling, such as tokenization, prefix-cache matches, predicted processing latency etc.
• Admitter: Decides whether to admit a request based on criteria like latency SLOs. Runs after dataProducer but before request scheduling. Requests failing admission are rejected, while admitted requests proceed to the request scheduling phase.

Advanced Hooks

The framework also supports advanced, auto-resolved hooks in the request control layer. If a plugin implements these interfaces, it is automatically wired into the execution flow:

• PreRequest: Executes before the request is processed (e.g., for incrementing in-flight counts).
• ResponseHeaderProcessor: Executes when response headers are received from the backend.
• ResponseBodyProcessor: Executes during response streaming (e.g., for usage tracking on completion).

note

In practice, these interfaces are often implemented by Data Producers to maintain state or track metrics across the request lifecycle. For example, the predicted-latency-producer implements these hooks to track request latency.

---

Concrete Plugins

Parsers

Parser plugins understand the payloads of requests and responses. This is key for features like prefix-cache aware scheduling and response usage tracking.

• openai-parser: The default parser supporting the OpenAI API. Supported endpoints: /conversations, /responses, /chat/completions, /completions, /embeddings.
• vllmgrpc-parser: Handles requests for the vLLM gRPC API. Supported methods: Generate, Embed.
• passthrough-parser: Model-agnostic parser that passes content through without interpretation. Note that payload-related scheduling (e.g., prefix-cache-scorer) is not supported with this parser.

Admitter Plugins

• latency-slo-admitter: Rejects sheddable requests (priority < 0) when no endpoint can meet latency SLO constraints.

Data Producers

• predicted-latency-producer: Trains XGBoost models via a sidecar and generates per-endpoint TTFT/TPOT predictions. It calculates SLO headroom, collects training data, and tracks per-endpoint running request queues.
• inflight-load-producer: Tracks the number of in-flight requests and estimated tokens for each endpoint. It increments counts in PreRequest and decrements them in ResponseBodyProcessor on end-of-stream.
• approx-prefix-cache-producer: Prepares data for approximate prefix cache aware scheduling by hashing prompts in blocks and matching them against an indexer of cached prefixes on servers.

---

Metrics & Observability

The Request Handling subsystem exposes metrics tracking request volume, success, latency, and token usage. Unless otherwise noted, request and latency metrics carry the label set {model_name, target_model_name, fairness_id, priority}.

Request Volume & Success

Metric	Type	Description	Labels
llm_d_epp_request_total	Counter	Total request count	model_name, target_model_name, fairness_id, priority
llm_d_epp_request_error_total	Counter	Total error count	model_name, target_model_name, fairness_id, priority, error_code
llm_d_epp_request_running	Gauge	Current active running requests	model_name, target_model_name, fairness_id, priority

Latency & SLOs

Metric	Type	Description	Labels
llm_d_epp_request_duration_seconds	Histogram	End-to-end response latency	model_name, target_model_name, fairness_id, priority
llm_d_epp_request_ntpot_seconds	Histogram	Normalized Time Per Output Token (NTPOT)	model_name, target_model_name, fairness_id, priority
llm_d_epp_request_ttft_seconds	Histogram	Time to first token (TTFT)	model_name, target_model_name, fairness_id, priority, streaming
llm_d_epp_request_streaming_tpot_seconds	Histogram	Average time per output token (streaming)	model_name, target_model_name, fairness_id, priority
llm_d_epp_request_streaming_itl_seconds	Histogram	Inter-token latency (streaming)	model_name, target_model_name, fairness_id, priority
llm_d_epp_request_predicted_ttft_seconds	Histogram	Predicted TTFT	plugin_name, plugin_type, model_name, target_model_name
llm_d_epp_request_ttft_prediction_duration_seconds	Histogram	Time spent predicting TTFT	plugin_name, plugin_type, model_name, target_model_name
llm_d_epp_request_predicted_tpot_seconds	Histogram	Predicted TPOT	plugin_name, plugin_type, model_name, target_model_name
llm_d_epp_request_tpot_prediction_duration_seconds	Histogram	Time spent predicting TPOT	plugin_name, plugin_type, model_name, target_model_name
llm_d_epp_request_slo_violation_total	Counter	Total count of requests violating SLO	plugin_name, plugin_type, model_name, target_model_name, type

Payload & Token Usage

Metric	Type	Description	Labels
llm_d_epp_request_size_bytes	Histogram	Incoming request body size in bytes	model_name, target_model_name, fairness_id, priority
llm_d_epp_response_size_bytes	Histogram	Outgoing response body size in bytes	model_name, target_model_name, fairness_id, priority
llm_d_epp_request_input_tokens	Histogram	Input token count per request	model_name, target_model_name, fairness_id, priority
llm_d_epp_request_output_tokens	Histogram	Output token count per request	model_name, target_model_name, fairness_id, priority
llm_d_epp_request_cached_tokens	Histogram	Distribution of prompt tokens read from cache	model_name, target_model_name, fairness_id, priority

Note: Response-level metrics (response sizes, output tokens, NTPOT) require Envoy body mode to be set to Buffered or Streamed. For vLLM streaming responses with usage data, include stream_options: {"include_usage": true} in the request.

Processing Overhead, Model Rewrite & Request Gauges

Metric	Type	Description	Labels
llm_d_epp_request_processing_duration_seconds	Histogram	Time from request receipt until request body has been handled	None
llm_d_epp_response_processing_duration_seconds	Histogram	EPP response processing latency	None
llm_d_epp_inference_request_metric	Gauge	Consolidated gauge for request metrics	plugin_name, plugin_type, model_name, target_model_name, type
llm_d_epp_model_rewrite_decisions_total	Counter	Total number of model rewrite decisions	model_rewrite_name, model_name, target_model