Global Content-Addressable KV Deduplication
Implement a content-addressable memory hashing layer to automatically detect and share redundant KV blocks across arbitrary, unrelated user requests.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In Section 4.4 and 6.4, the authors describe sharing KV cache for pre-defined prefixes or within single requests using parallel sampling. However, the system currently lacks a mechanism to automatically detect and share redundant KV blocks across arbitrary, unrelated user requests that may overlap in content (e.g., multiple users querying the model about the same viral news article).
Limitation
Memory sharing is limited to hierarchical relationships (prefixes or samples from the same prompt) and requires manual definition or specific decoding patterns.
Proposed Solution
Implement a Content-Addressable Memory (CAM) layer for KV blocks. Future work should develop a hashing mechanism (such as Locality-Sensitive Hashing or cryptographic hashing of input embeddings) to identify identical KV blocks across the entire server's request pool. If a new request's block hash matches an existing block in physical memory, the block table should map to the existing physical block with an incremented reference count.
Why This Matters
This would significantly increase the effective batch size in multi-tenant environments where users often ask similar questions, leading to a further boost in serving throughput and reduced GPU memory pressure.
How to Get Started
- Develop a hashing function for input embeddings or KV blocks using locality-sensitive hashing.
- Modify the server's block table manager to support global reference counting and physical block sharing across requests.
- Ingest public request trace datasets to simulate high-concurrency overlapping user queries.
- Benchmark memory usage and cache hit rates against standard vLLM prefix-sharing baselines.
Code linked to this paper on Hugging Face: github.com/vllm-project/vllm
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like gKSM and CachePrune explore KV cache sharing and memory deduplication techniques for LLM inference. However, none of the listed results explicitly implement a global content-addressable memory hashing layer across the entire server's request pool with the proposed LSH or embedding hashing mechanism.
- CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference · 2026 · 2 citations
- gKSM: GPU Memory Deduplication at Driver Layer for Serverless LLM Inference · 2026 · 0 citations
We searched OpenAlex for “Implement a content-addressable memory hashing layer to automatically detect and share redundant KV blocks across arbitrary, unrelated user requests.” and “Global Content-Addressable KV Deduplication” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang +6 more · 2023
This paper presents vLLM, a high-throughput serving system for Large Language Models (LLMs) that addresses the memory bottleneck caused by the Key-Value (KV) cache. By introducing PagedAttention, the system manages KV cache memory in a way similar to virtual memory in operating systems, allowing for near-zero fragmentation and efficient memory sharing across multiple sequences.
Related Ideas
- Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared AttentionMulti-Branch Transformer🟠Medium
- Robust Suffix-Aware KV Cache IndexingLMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference🟠Medium
- Granular Importance-Aware Sparse RetrievalLMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference🔴Hard
- Prompt-Relevant Historical Memory Retrieval and Entropy-Calibrated Layer-Head KV CompressionMovieRoll: A Systematic Framework for Rolling Out Long-Form Videos🟡Fairly easy
- Dynamic Network-Compute Latency ArbitratorLMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference🟡Fairly easy