Skip to content

Source paper

LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

Yuhan Liu, Yihua Cheng, Jiayi Yao and 8 more

Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, Junchen Jiang

arXiv 2510.09665 · 20252 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces LMCache, a middleware layer designed to decouple Key-Value (KV) cache management from LLM inference engines like vLLM and SGLang. By implementing efficient, chunk-based data transfer protocols and a standardized connector interface, LMCache enables high-performance cross-query context sharing and cross-engine prefill-decode disaggregation, significantly reducing latency and improving throughput in enterprise settings.

How it works (AI summary)

LMCache functions as a distributed KV cache store that sits between inference engines and storage backends (memory, disk, S3). It replaces inefficient page-wise transfers with optimized 'chunk' transfers (default 256 tokens), utilizes zero-copy operations, and pipelines compute with I/O. It exposes a 'KV connector' API to inference engines for retrieving cached states and a 'Controller' for global cache orchestration.