Dedicated On-Package SRAM Draft Coprocessors Push Speculative Decoding to 4.1 Accepted Tokens per Step

Inference systems architects and custom silicon engineers across Sunnyvale, San Jose, and Stanford University published hardware benchmarks and open vLLM integration drivers on Friday afternoon through the MLCommons Inference Research Consortium committee, demonstrating a dedicated 35-watt On-Chip SRAM Speculative Draft Coprocessor that triples interactive token generation speed on 70-billion- to 405-billion-parameter frontier reasoning models. During autoregressive chain-of-thought decoding at low batch sizes, flagship High-Bandwidth Memory (HBM) GPU accelerators spend more than 85 percent of their wall-clock cycle time stalled on HBM weight bandwidth rather than arithmetic compute. While software speculative decoding uses a small draft model on the same GPU to propose candidate token trees, running the draft model still incurs kernel-launch overhead and competes for HBM bandwidth.

By pinning an entire quantized 1.5-billion-parameter Eagle-3 draft head inside 960 megabytes of distributed on-die SRAM across a low-profile PCIe 6.0 / CXL coprocessor card, the draft engine streams eight-token speculative continuation trees in just 1.1 milliseconds per pass—over 9x faster than HBM-resident drafting. The host GPU cluster then verifies the entire candidate tree in a single parallel forward pass, amortizing weight reads across multiple accepted tokens without losing exact mathematical equivalence to greedy or temperature sampling.

Evaluated on live multi-step coding and formal-math reasoning traces in San Jose datacenter racks, the heterogeneous SRAM-draft + HBM-target architecture sustained an average acceptance length of 4.12 tokens per target verification step, pushing single-stream output velocity from 48 tokens per second to 156 tokens per second on a 70B dense model while adding less than 4 percent to total server rack power draw.

Open-source C++ runtime bindings, speculative tree-attention CUDA verification kernels, and PCIe peer-to-peer DMA transport modules have been merged into the vLLM Production Inference Repository. Systems engineers can inspect the acceptance-rate histograms and power-efficiency curves via the MLCommons Research Publications Portal.