Skip to main content
Version: dev

Disaggregated Serving: Operations (vLLM)

While disaggregated serving can offer superior performance, it introduces additional operational complexity, including:

  • Dynamic Connections - how to add or remove P and D instances on the fly when instances require point-to-point RDMA connections
  • Request Cancellation - how to free KV caches from the instances when requests stop in a distributed setting
  • Fault Tolerance - how to ensure crashes do not create cascading failures and that resources are cleaned up
  • Rollouts - how to roll out changes to the service, such as the version of the vLLM image
  • Known NIXL Connector Issues and Limitations - current gaps and bugs in the NIXL connector to plan around

This page documents architectural considerations that impact these common operations flows.

Dynamic Connections​

In production environments, it is common for model server replicas to be created and destroyed during the running of the service. In a disaggregated configuration, the ability to dynamically add/remove replicas from the deployment is complicated by the need to establish/destroy connections between P and D workers on the fly.

vLLM supports this functionality via NIXL's APIs, which enable dynamically adding and removing connections.

Scale-Up​

To create new connections, vLLM executes a "NIXL Handshake" between the D and P instances to setup the RDMA connection. This is a relatively expensive operation (~5s) that is done once per pair, with all subsequent requests leveraging the existing connection. llm-d uses a "dynamic lazy" roll-out strategy, avoiding the need for a centralized bootstrap server maintaining global state.

It works like this:

  • Prefill instances run a background server thread to handle requests for NIXLMetadata. When the P instances finishes processing, it constructs the KVTransferParams, which includes (among other things) remote_host=VLLM_SIDE_CHANNEL_HOST (the pod IP) and remote_port=VLLM_SIDE_CHANNEL_PORT in the response body.
  • Decode instances receive the request with the KVTransferParams; if there is not yet a connection to the remote P worker, it runs a background thread to fetch the NIXLMetadata and create the RDMA connection. This action does not block core engine execution, enabling other requests to proceed as usually.

Discovery​

Since model server instances are added to an InferencePool via standard Kubernetes selectors and labels, new prefill and decode instances discovered automatically when their pods status becomes status: Running.

As a result, new replicas can be added to a running disaggregated deployment without restarts and without need to coordinate within any specialized service discovery plane.

Scale-Down​

In Kubernetes, there is a well-defined pod termination process:

  • Termination Triggered: The pod's state is changed to Terminating.
  • InferencePool Update: The pod is removed from the list of endpoints for associated the InferencePool, preventing new traffic from being routed to it. (note: for standard Kubernetes objects, this is equivalent to removal from a Service)
  • PreStop Hook: If defined, the preStop hook executes.
  • SIGTERM Signal: Kubernetes sends a SIGTERM signal to the main process in each container.
  • Termination Grace Period: The pod is given a set amount of time (default is 30 seconds) to shut down gracefully. If it does not terminate by the end of this period, a SIGKILL is sent to force termination.

For new requests, instances are automatically removed from the InferencePool so no new traffic is routed to terminating pods.

For running requests, we can configure how vLLM handles the SIGTERM:

  • By default, vLLM immediately aborts existing requests and terminates. This fails the running requests with an error status code.
  • vLLM can be configured with a --shutdown-timeout N. When this is set, vLLM catches the SIGTERM and drains the currently running requests for N seconds. After this timeout, it aborts any running requests still in flight, returning an error code.

Scaling Down Decode Replicas​

Since prefill instances hold the KVs until the decode instances pull them, it is important to ensure that KVs are released on the prefill instance when decode instances are scaled down.

In vLLM, regardless of whether --shutdown-timeout is set, requests are aborted during the shutdown process. As part of the abort process, decode instances with not-yet-started KV transfers send a NIXL notification to the remote prefill instances to free the blocks. Thus, scaling down decode replicas always free requests on the P instance.

Scaling Down Prefill Replicas​

When scaling down prefill replicas, decode instances may attempt to pull KV blocks from terminated remote prefill instances.

warning

At current, regardless of --shutdown-timeout, there is no way to delay shutdown of a prefill instance until after all blocks have been retrieved. This functionality is work in progress in vLLM.

As a result, prefill scale down will cause KV load failure for in-progress requests on decode instances. To avoid error codes for failed KV transfers, the decode instances can be configured with kv_load_failure_policy=recompute to recompute the prefill on the decode instance.

Request Cancellation​

Given the compute intensity and duration of inference requests, model servers like vLLM support "Request Cancellation", where currently in-progress requests are freed when the client disconnects.

In a disaggregation setup, this feature is more complicated, because the resources associated with an inference request are spread across multiple servers (as the P instances holds onto the KV caches until they have been retrieved by the D instance). As a result, if the request is canceled while it is still "in-flight" on the D instance but before the KV transfer occurs, we need to ensure that the resources on the P instance are properly cleaned up.

llm-d accomplishes this functionality by building on top of vLLM's existing request cancellation infrastructure. When requests are disconnected in vLLM, it triggers the abort codepath, which cleans up running resources. When request with do_remote_prefill=True are aborted, vLLM sends a NIXL notify message, instructing the remote prefill instance to free the KV cache for the cancelled request.

warning

There is a small window in which request cancellation will not trigger KV freeing on the P instance. If the request is disconnected after it is completed on the P worker but before it reaches the D worker's scheduler (for example, if it disconnects while the request is inside Routing Proxy), the D instance never knows about the request and therefore is unable to free the remote blocks on the P worker. In this case, the KV blocks are held on the P instance until the lease expires (kv_lease_duration, default 30s), at which point they are freed automatically.

Fault Tolerance​

In llm-d's disaggregated serving design, all D instances are connected to all P instances. This creates a critical operational risk - crashes in workers have the potential for cascading failures if the system is not tolerant of failures.

Prefill Instance Failure​

Prefill instance crashes are a critical failure mode, since D instances will attempt to pull KVs from no longer running P instances without performing any liveness checks. Since every D worker is connected to every P worker, it is critical to handle such an error on the D worker.

vLLM handles Prefill instance failure by building on top of NIXL's error handling functionality. When a READ is attempted and fails, NIXL returns an error code such as NIXL_ERR_BACKEND. vLLM catches this error and handles it according to the kv_load_failure_policy:

  • fail (default, recommended): Immediately fail the request with an error when KV load fails. This prevents performance degradation by avoiding recomputation of prefill work on the decode instance.
  • recompute: Recompute failed blocks locally on the decode instance. This may cause performance jitter on decode instances as the scheduled prefill will delay and interfere with other decodes. Furthermore, decode instances configured with low-latency optimizations (such as DeepEP LL for Wide EP deployments) may suffer significant slowdowns.

Failed Prefill Worker pods are automatically moved to status: Terminated state as part of the standard Pod lifecycle. Since llm-d leverages the Kubernetes API Server for service discovery, no additional traffic will be routed to the failed worker until the pod has been restarted and returns to the status: Running state.

In this way, llm-d isolates Prefill instance failure.

Decode Instance Failure​

While D instance failures are unlikely to result in P instance crashes (since P instance never initiates RDMA operations), there is a challenge around ensuring that KV cache memory on the P instance is not stranded (since the P instance holds onto the KV cache until it has been explicitly pulled from the D instance).

vLLM addresses this with a lease-based KV block management system. When a prefill completes, P holds the KV blocks with an initial lease of kv_lease_duration (default 30s). While the request is queued or running on the D instance, D's scheduler periodically sends heartbeat notifications to P, each extending the lease by lease_extension = kv_lease_duration * 2 // 3 (default 20s). If D crashes (or becomes unresponsive), heartbeats stop, and P automatically frees the KV blocks when the last lease extension expires — at most lease_extension seconds after the last heartbeat.

This approach keeps the worst-case block hold time short without risking premature block eviction when D instances are heavily loaded — as long as D is alive, the heartbeats keep the lease active indefinitely.

The kv_lease_duration is configurable via kv_connector_extra_config:

--kv-transfer-config '{"kv_connector_extra_config": {"kv_lease_duration": 10}}'
note

Lease-based KV block TTL requires vLLM v0.22.0 or later. In earlier versions, vLLM used a fixed-timeout approach via VLLM_NIXL_ABORT_REQUEST_TIMEOUT (default 480s): the P instance would free stranded KV blocks only after that timeout elapsed, regardless of whether the D instance was still alive. This made the worst-case hold time very long, and reducing the timeout risked premature eviction when D instances were heavily loaded.

Rollouts​

In disaggregated serving, rolling out a new version of the model server (e.g. a new version of vLLM or a new configuration) requires care, since prefill-decode instance pairs communicate with each other to execute the KV transfer operation. As a motivating example, vLLM has multiple attention kernel implementations, each of which can have slightly different KV cache layouts - since NIXL pulls the KVs directly from the GPU KV cache memory of the remote instance, we need to ensure these are matching.

By default, vLLM checks for compatibility between instances during the NIXL handshake, failing the request if scheduled to incompatible pods. There is an escape hatch to disable compatibility checking:

--kv-transfer-config '{"kv_connector_extra_config": {"enforce_handshake_compat": false}}'
info

The llm-d EPP currently assumes all P and D instances within an InferencePool are compatible and will therefore schedule requests to any arbitrary pair of P and D instances. As a result, it is currently recommended to create a new InferencePool for upgrading model servers. When deploying with a Gateway, traffic can be gradually shifted to the new InferencePool by modifying the HTTPRoute.

Known NIXL Connector Issues and Limitations​

Beyond the operational flows above, the NIXL connector has a couple of current gaps worth planning deployments around.

Prefill TP > Decode TP Is Not Supported​

warning

Prefill TP > Decode TP is not supported for most model architectures. NixlConnector only supports the fan-out direction (decode TP ≥ prefill TP, e.g. prefill TP=1 → decode TP=4). This is a deliberate guard in vLLM's own source, not a bug — always keep decode TP ≥ prefill TP.

The reverse direction (prefill TP > decode TP, e.g. prefill TP=8 → decode TP=4) is explicitly unsupported for any model that is neither MLA nor Mamba/state-space based:

# num_kv_heads > tp_size with P_TP > D_TP not supported for non-mamba.
# Mamba models can have replicated FA KV with tp_ratio < 0.
# MLA models do not need to handle kv replication.
if not self.use_mla and not self._has_mamba:
   assert not (
       tp_ratio < 0 and self.transfer_topo.is_kv_replicated(remote_engine_id)
   )

Source: vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py, function _validate_remote_agent_handshake

Hitting this path crashes the decode engine with AssertionError. A related bug in vLLM's failure-cleanup handler (_handle_failed_transfer) often surfaces it as IndexError: list index out of range instead, making it look like a generic connector bug rather than an unsupported-topology guard.

Total GPU count (TP × replicas) and replica counts on each side are unaffected by this constraint — only the TP ratio direction between whichever specific prefill/decode engine pair handles a given request matters.

Stale NIXL Agent Cache After a Prefill Pod Restart​

warning

Decode-side stale NIXL agent cache after a prefill pod restart can segfault decode (vllm-project/vllm#49238, open), or leave it silently serving against a dead engine for up to an hour. Until the fixes below ship in an official image, proactively restart any decode replica that was connected to a restarted prefill pod, rather than relying on TTL expiry.

If a prefill pod restarts while decode holds a cached NIXL agent handle for it, decode can segfault, or (after the NIXL-side fix below) silently keep serving against the dead engine's stale cached agent/rkey entries. With a pinned/stable engine ID, decode does not detect the prefill engine is gone until engine_ttl expires (default up to one hour), and every request routed to that decode replica fails until then or until the decode pod is restarted.

When decode does not segfault, it instead attempts the NIXL RDMA read against the stale agent, which fails with NIXL_ERR_BACKEND — the same error path described in Prefill Instance Failure. vLLM then applies kv_load_failure_policy (source):

  • recompute: decode silently falls back to running the prefill itself, logging only an error line for the failed transfer. The client never sees a failed request, so this shows up purely as a throughput/latency regression on that decode replica, not as an error — harder to notice than an outright failure.
  • fail (default): the request fails with a 500 response, which at least surfaces the problem to the caller.

Until engine_ttl expires, every request that lands on that decode replica repeats this failed-read-then-recompute cycle, so the performance hit persists rather than clearing up after the first failure.

Fix status:

  • NIXL-level segfault fixed upstream: ai-dynamo/nixl#1986 (merged ~2026-07-29), returns a clean disconnect error instead of crashing.
  • vLLM-side fix, vllm-project/vllm#50047, invalidates decode's cached engine state as soon as the prefill peer is detected gone, instead of waiting out the TTL.

Neither fix is in any published vLLM image yet, including nightly, since vLLM pins an exact NIXL version and the NIXL fix landed after the latest published NIXL release (v1.3.2, 2026-07-24).