Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Global Content-Addressable KV Deduplication

Implement a content-addressable memory hashing layer to automatically detect and share redundant KV blocks across arbitrary, unrelated user requests.

Time

several weeks

Compute level

Single GPU or less

Resources

Standard GPU, public dataset (request traces)

Prerequisites

LLM serving internals, systems programming, memory management, hashing algorithms

Scope

modify existing code (memory manager), add global indexing layer

The Gap

In Section 4.4 and 6.4, the authors describe sharing KV cache for pre-defined prefixes or within single requests using parallel sampling. However, the system currently lacks a mechanism to automatically detect and share redundant KV blocks across arbitrary, unrelated user requests that may overlap in content (e.g., multiple users querying the model about the same viral news article).

Limitation

Memory sharing is limited to hierarchical relationships (prefixes or samples from the same prompt) and requires manual definition or specific decoding patterns.

Proposed Solution

Implement a Content-Addressable Memory (CAM) layer for KV blocks. Future work should develop a hashing mechanism (such as Locality-Sensitive Hashing or cryptographic hashing of input embeddings) to identify identical KV blocks across the entire server's request pool. If a new request's block hash matches an existing block in physical memory, the block table should map to the existing physical block with an incremented reference count.

Why This Matters

This would significantly increase the effective batch size in multi-tenant environments where users often ask similar questions, leading to a further boost in serving throughput and reduced GPU memory pressure.

How to Get Started

  1. Develop a hashing function for input embeddings or KV blocks using locality-sensitive hashing.
  2. Modify the server's block table manager to support global reference counting and physical block sharing across requests.
  3. Ingest public request trace datasets to simulate high-concurrency overlapping user queries.
  4. Benchmark memory usage and cache hit rates against standard vLLM prefix-sharing baselines.

Code linked to this paper on Hugging Face: github.com/vllm-project/vllm

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers like gKSM and CachePrune explore KV cache sharing and memory deduplication techniques for LLM inference. However, none of the listed results explicitly implement a global content-addressable memory hashing layer across the entire server's request pool with the proposed LSH or embedding hashing mechanism.

We searched OpenAlex for “Implement a content-addressable memory hashing layer to automatically detect and share redundant KV blocks across arbitrary, unrelated user requests.” and “Global Content-Addressable KV Deduplication” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang +6 more · 2023

This paper presents vLLM, a high-throughput serving system for Large Language Models (LLMs) that addresses the memory bottleneck caused by the Key-Value (KV) cache. By introducing PagedAttention, the system manages KV cache memory in a way similar to virtual memory in operating systems, allowing for near-zero fragmentation and efficient memory sharing across multiple sequences.

Related Ideas

Is this a good idea?

Share onXLinkedIn