Skip to content
AI-generated research idea🟠Medium💡Moderate Innovation

Predictive Semantic-Aware Preemption Policy

Replace FCFS scheduling with a semantic-aware preemption policy that evicts KV blocks based on predicted remaining generation length and re-engagement probability.

Time

A couple of months

Compute level

Multiple GPUs

Resources

Standard GPU setup (e.g., 1-2 A100s for testing vLLM), public conversation trace datasets (e.g., ShareGPT or LMSYS Chatbot Arena) to train the length predictor.

Prerequisites

Deep understanding of LLM inference systems (specifically vLLM and PagedAttention), Operating Systems scheduling algorithms, and basic machine learning for regression/classification.

Scope

Modify existing vLLM scheduler code; develop and integrate a lightweight auxiliary model or heuristic function; modify the eviction logic.

The Gap

In Section 4.5, vLLM implements a First-Come-First-Serve (FCFS) scheduling policy with an 'all-or-nothing' eviction strategy when GPU memory is exhausted, swapping evicted blocks to CPU RAM or recomputing them.

Limitation

FCFS does not consider the computational 'sunk cost' or the likelihood of a request being resumed. Evicting a nearly-finished long-context request in favor of a brand-new short request is computationally inefficient.

Proposed Solution

Replace the FCFS scheduler with a Predictive Semantic-Aware scheduler. Use a lightweight heuristic or a small auxiliary model to predict the remaining generation length and the probability of user re-engagement (for multi-turn chat). Implement a 'Least-Value-at-Risk' eviction policy that prioritizes keeping KV blocks of requests that are close to completion or have high re-access probability.

Why This Matters

This would reduce the aggregate latency and 'waste' caused by recomputation or high-latency CPU-GPU swaps, improving the user experience for long-form generation tasks.

How to Get Started

  1. Train a lightweight auxiliary regression model on ShareGPT and LMSYS datasets to predict remaining generation length.
  2. Define a 'Least-Value-at-Risk' scoring function incorporating sequence progress and multi-turn user re-engagement probability.
  3. Replace the default vLLM eviction scheduler with the predictive semantic-aware scheduling policy.
  4. Measure request completion rates and tail latency under heavy memory pressure benchmarks.

Code linked to this paper on Hugging Face: github.com/vllm-project/vllm

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Not checked against the literature yet.

Source Paper

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang +6 more · 2023

This paper presents vLLM, a high-throughput serving system for Large Language Models (LLMs) that addresses the memory bottleneck caused by the Key-Value (KV) cache. By introducing PagedAttention, the system manages KV cache memory in a way similar to virtual memory in operating systems, allowing for near-zero fragmentation and efficient memory sharing across multiple sequences.

Related Ideas

Is this a good idea?

Share onXLinkedIn