Global Content-Addressable KV Deduplication
Implement a content-addressable memory hashing layer to automatically detect and share redundant KV blocks across arbitrary, unrelated user requests.
Source paper
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica
Generated by AI. The paper’s authors did not write these.
Implement a content-addressable memory hashing layer to automatically detect and share redundant KV blocks across arbitrary, unrelated user requests.
Develop a runtime optimization engine that dynamically adjusts KV block sizes based on model architecture and real-time request length telemetry.
Replace FCFS scheduling with a semantic-aware preemption policy that evicts KV blocks based on predicted remaining generation length and re-engagement probability.
This paper presents vLLM, a high-throughput serving system for Large Language Models (LLMs) that addresses the memory bottleneck caused by the Key-Value (KV) cache. By introducing PagedAttention, the system manages KV cache memory in a way similar to virtual memory in operating systems, allowing for near-zero fragmentation and efficient memory sharing across multiple sequences.
The core technique is PagedAttention, which partitions the KV cache of each request into fixed-size blocks stored in non-contiguous physical memory. A centralized scheduler and KV cache manager maintain a block table to map logical blocks to physical GPU memory, enabling dynamic allocation and copy-on-write sharing for complex decoding algorithms like beam search.