Artificial Intelligence
Hardware-Coherent 900-GB/s NVLink-C2C Zero-Copy KV-Cache Paging Cuts 128K-Context Reasoning TTFT by 64 Percent
Santa Clara and San Jose inference engineers map unified LPDDR5X host memory directly into GPU page tables to eliminate PCIe Gen5 DMA bottlenecks during prefix reuse.
Your listening Queue & catch up
Add a story or build a catch-up queue to start listening.
Inference systems architects and kernel engineers across Santa Clara, San Jose, and UC Berkeley released production long-context serving benchmarks on Saturday evening through the vLLM Project and TensorRT-LLM working group, demonstrating that hardware-coherent 900-Gigabyte-per-second (GB/s) NVLink-C2C zero-copy KV-cache paging between 480 GB of host LPDDR5X memory and on-package HBM3e reduces Time-to-First-Token (TTFT) by 64 percent and boosts multi-turn agentic reasoning throughput by 2.8x on 128-thousand-token contexts. In conventional x86 PCIe Gen5 servers, offloading evicted Key-Value (KV) attention blocks to host DDR5 RAM bottlenecks on 64 GB/s bidirectional DMA transfers and incurs millisecond-scale OS page-pinning overhead.
By leveraging hardware cache-coherent address translation across the 900 GB/s die-to-die NVLink-C2C interconnect on unified CPU-GPU superchips, the new Coherent-PagedAttention runtime maps 480 GB of low-power host LPDDR5X memory directly into the GPU's unified virtual address space using 2-megabyte hugepages. During prefill and tree-structured speculative decoding, streaming multiprocessor (SM) warp groups issue asynchronous cp.async.bulk tensor-memory-accelerator loads directly against cold prefix blocks resident in LPDDR5X while simultaneously attending to hot recent-window blocks in 8-TB/s HBM3e.
A lightweight hardware page-access counter inside the unified memory management unit (MMU) tracks block attention frequency per layer and promotes high-reuse system prompts and repository context blocks into HBM3e without CPU kernel intervention. This zero-copy tiering expands effective per-node KV-cache capacity by 3.5x while holding inter-token decode latency under 18 milliseconds.
Validated on a 32-node liquid-cooled inference cluster at 2788 San Tomas Expressway in Santa Clara funded by a $440,000 open-inference research grant, prefix cache hit rates rose from 41 percent to 89 percent on multi-file coding agents. Open CUDA kernels and NUMA-aware allocator patches are available in the vLLM Official GitHub Repository.
Sources
Comments (0)
No comments yet. Be the first to share a thought on this story.