Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared Attention

Combine branch-wise KV sharing with cross-layer KV caching to eliminate redundant key-value memory traffic across both network depth and parallel architectural branches.

Time

several weeks

Compute level

Multiple GPUs

Resources

Multi-GPU cluster (e.g., 4 to 8 H100/A100 GPUs) and public text datasets such as FineWeb-Edu.

What the paper used

Trained across W1-W4 scales (0.8B to 2.1B total MoE parameters) on 27.8B to 78.8B tokens using Megatron-LM in BF16; evaluated on a single NVIDIA H200 GPU.

Prerequisites

Transformer architectures, KV cache management, PyTorch, Triton or CUDA kernel development.

Scope

modify existing code

The Gap

Section 2.3 and Section 6 discuss KV-cache sharing and future directions. Joint-KV shares KV states purely across parallel branches at the same depth level (intra-layer, across-branch), while keeping KV caches separate across depth. However, long-context decoding memory consumption in the shared prelude and coda remains uncompressed, and branch layers still require separate KV projections at each sequential step.

Limitation

Joint-KV restricts cache reduction strictly to the branch dimension (1/K cache reduction within the branched stack only), leaving sequential depth layers uncompressed and yielding overall model KV savings capped at 19% to 34%.

Proposed Solution

Formulate a dual-axis KV sharing framework combining Joint-KV with cross-layer attention (CLA) and decoder-decoder architectures (YOCO). Parameterize key-value projections such that shared branch representations at depth l are reused across subsequent depths (l + 1 to l + m) or retained as a single persistent cache throughout the branch region. Develop custom FlashAttention kernels that accommodate folded branch queries attending over a single cross-depth, cross-branch shared KV tensor.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Could boost KV cache memory reduction from ~30% to over 70% during long-context serving, significantly increasing maximum batch concurrency without degrading language modeling perplexity.

How to Get Started

How to test it: Pretrain a 0.8B (W1 scale) Balanced MBT with 2-layer CLA sharing across branch depths on 28B tokens of FineWeb-Edu. Compare validation loss and H200 decoding throughput against the baseline Joint-KV Balanced MBT. Terminate the project if validation perplexity degrades by more than 0.05 or decoding latency increases due to cross-layer synchronization overhead.

  1. Clone the released Multi-Branch Transformer codebase and verify reproducing the baseline W1 Balanced MBT configuration.
  2. Implement a layer-linking module that routes the shared Joint-KV cache of branch depth l to depth l+1.
  3. Update the nano-vLLM head folding engine to read cached KV blocks across paired layers.
  4. Train small proxy models on 5B tokens to verify gradient stability and initial validation loss.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers like Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers and DepthWeave-KV explore cross-layer KV cache compression and fusion techniques, while SparseX investigates segment-level KV sharing. However, none of these listed results combine Joint-KV, cross-layer attention, and custom FlashAttention kernels for folded branch queries into a unified dual-axis sharing framework.

We searched OpenAlex and arXiv for “cross layer attention joint kv cache transformer” and “Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared Attention” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Multi-Branch Transformer

Gaotang Li, Lingjie Chen, Chunyuan Deng +6 more · 2026

This paper introduces the Multi-Branch Transformer (MBT), an architectural paradigm that parallelizes the sequential execution of Transformer blocks by routing representations from shared prelude layers through multiple independently parameterized parallel branches before fusing them in shared coda layers. To optimize inference at long contexts, the authors introduce Joint-KV, which computes a unified key-value cache across concurrent branches per depth level, alongside branch grouping and query head folding within an optimized inference runtime. Experiments on 0.8B to 2.1B parameter MoE models show that MBTs consistently improve language modeling loss and zero-shot benchmark performance under matched compute, while reducing KV cache storage up to 34% and increasing decode throughput on an H200 GPU by up to 51.4%.

Related Ideas

Is this a good idea?

Share onXLinkedIn