Disaggregated Prefill-and-Decode GPU Serving Over 400-GbE RoCEv2 RDMA Cuts P99 Inter-Token Latency by 54 Percent

Distributed inference architects and networking systems engineers across Santa Clara, San Jose, and UC Berkeley published production serving telemetry on Sunday afternoon through the vLLM Disaggregated Serving Working Group and Mooncake KV-Cache Consortium, demonstrating that Disaggregated Prefill-and-Decode (P/D) GPU clusters linked over 400-Gigabit-per-second (GbE) RoCEv2 GPUDirect RDMA reduce 99th-percentile Time-Between-Tokens (TBT) decode latency by 54 percent and increase goodput under strict SLA constraints by 2.3x on 70B and 405B models. In traditional colocated continuous batching, piggybacking a single 32-thousand-token prefill request onto an active decode batch stalls memory-bandwidth-bound token generation for hundreds of milliseconds, causing severe tail-latency violations for interactive users.

By physically partitioning the serving pool into compute-dense Prefill Instances (optimized for high-TFLOPs tensor-core utilization at batch size 1–2) and memory-bandwidth-dense Decode Instances (optimized for high-batch HBM3e KV-cache reads), each stage operates at its hardware roofline without cross-phase interference. As soon as a Prefill worker completes transformer layer L, an asynchronous GPUDirect RDMA ibv_post_send engine streams the newly computed FP8 Key-Value (KV) blocks directly from Prefill HBM3e into the target Decode worker's pre-allocated HBM3e page pool over dual-rail 400-GbE ConnectX-7 NICs.

Overlapping layer-by-layer RDMA KV transmission with subsequent prefill layer computation hides 92 percent of network transfer time, adding less than 6.4 milliseconds of handoff overhead for a 32K-token prompt. A global prefix-aware router also caches hot system prompts in a distributed DRAM/NVMe RDMA pool so repeated agentic tool calls bypass prefill compute entirely.

Validated across a 128-GPU disaggregated cluster at 3050 Bowers Avenue in Santa Clara funded by a $480,000 open-systems research grant, P99 inter-token latency held steady at 21 milliseconds even under 5x bursty long-prompt arrivals. Reference Nixl/Mooncake RDMA transfer engines and vLLM P/D connectors are available in the vLLM Official GitHub Repository.