UC Berkeley Sky Computing Lab and vLLM Contributors Release Multi-Step Speculative Decoding Engine for GPU Clusters

Distributed systems researchers at the UC Berkeley Sky Computing Lab and open-source infrastructure engineers across San Francisco and San Jose have shipped a major October 2026 performance release for the vLLM large-language-model serving engine, introducing zero-overhead multi-step speculative decoding alongside disaggregated prefill-and-decode scheduling for multi-node GPU clusters. Announced during an open-source systems engineering program symposium in Berkeley on Thursday, October 22, 2026, the update targets the memory-bandwidth bottleneck that limits autoregressive token generation in production AI agents.

In standard autoregressive LLM inference, generating each output token requires loading hundreds of gigabytes of model weights from High Bandwidth Memory (HBM) into GPU tensor cores, leaving compute units underutilized during small-batch interactive sessions. The updated vLLM speculative decoding pipeline pairs a lightweight draft head—supporting both EAGLE-style feature-level drafting and n-gram prompt lookup—with a single fused verification pass on the target model, allowing the GPU to validate four to six candidate tokens in a single HBM read cycle.

Crucially, the October 2026 release eliminates host-CPU synchronization stalls between draft steps by compiling the entire draft-verify-accept loop into persistent CUDA graphs compatible with FP8 KV-cache quantization and PagedAttention. In production benchmarks conducted across 8-GPU NVIDIA Hopper and Blackwell nodes in Silicon Valley data centers, the engine reduced median inter-token latency (ITL) by up to 2.4x on 70-billion-parameter dense models and sparse Mixture-of-Experts (MoE) reasoning architectures without altering output probability distributions.

Combined with RDMA-accelerated KV-cache transfer between prefill and decode worker pools, the updated runtime enables San Francisco and Silicon Valley AI engineering teams to meet strict sub-25-millisecond streaming SLAs for voice and coding assistants. Container images and benchmark scripts are available under the Apache 2.0 license at docs.vllm.ai.