Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationNo close match found

Single-Pass Visual Dependence Estimation to Eliminate Dual-Pass Target Scoring

Estimate token-level visual dependence from a single target forward pass via hidden-state shifts or attention statistics to eliminate the extra no-image scoring pass.

Time

several weeks

Compute level

Multiple GPUs

Resources

8 NVIDIA H100 GPUs (matching the paper's training setup) and a subset/full copy of the ~900K LLaVA-OV-1.5-Instruct-Data training corpus.

What the paper used

Training used 8 NVIDIA H100 GPUs on ~900K regenerated LLaVA-OV-1.5-Instruct-Data samples (6 epochs, batch size 2) plus 2K LLaVA-Video-178K samples (3 epochs); evaluation used a single NVIDIA H200 GPU across Qwen3.5-4B/9B, Gemma4-E4B-it, and Qwen3.8-27B.

Prerequisites

VLM internal representations, attention/logit-lens analysis, and knowledge distillation.

Scope

Implement single-pass proxy estimators in the data scoring / online training pipeline and retrain drafters.

The Gap

In Section 4.1 and Section 4.2, MMFlash computes the token-level visual dependence score d_t as the KL divergence between the target VLM's next-token distributions with and without the image (Equation 1). As noted in Section 5.1 and Appendix D.1, responses are regenerated per target model across ~900K training samples (up to 4,096 tokens), meaning the visual dependence scoring step requires running an extra full no-image forward pass of the target model over the entire dataset.

“For every token t of a training response, we compute its visual dependence dt with Equation 1, i.e., with the target model, the image I, and the prefix x<t fixed.”
From the paper, Section 4.2, page 4. Matched against the PDF text.

Limitation

Requiring a second, image-ablated forward pass of the target model over every training sample doubles target inference cost during data preparation, cannot be recycled across different target models, and becomes increasingly expensive when scaling to larger target models (e.g., 27B+), longer contexts (e.g., 40K-token videos), or online distillation pipelines.

Proposed Solution

Develop a single-pass proxy for visual dependence computed directly during the standard with-image target forward pass used for response generation or online hidden-state tapping. Specifically, investigate three single-pass estimators calibrated against ground-truth KL on a small 10K-sample seed set: (1) cross-modal attention mass directed toward visual tokens in middle-to-late target layers, (2) L2/cosine shift between target hidden states before and after visual-attending layers via logit-lens divergence, and (3) a tiny linear probe trained on tapped target hidden states to predict the quantile rank \tilde{g}_t directly.

Why This Matters

Replacing dual-pass KL precomputation with an accurate single-pass proxy is expected to cut data-preparation compute by nearly half and make visually grounded loss reweighting practical for large target models and long video corpora.

How to Get Started

How to test it: Test whether a single-pass proxy achieves Spearman rank correlation >= 0.85 with ground-truth Equation 1 KL quantiles on a held-out 10K set and yields drafter acceptance lengths and speedups matching dual-pass MMFlash (within 1-2%) when trained on Qwen3.5-4B across Table 1 benchmarks. Stop if proxy-trained drafters lose >3% speedup relative to dual-pass MMFlash.

  1. Compute ground-truth Equation 1 KL scores alongside visual attention mass and tapped hidden-state features on a 10K-sample subset of LLaVA-OV-1.5-Instruct-Data using Qwen3.5-4B.
  2. Benchmark rank correlation (Spearman rho) between true quantile \tilde{g}_t and candidate single-pass proxies (visual attention entropy/sum, layer-to-layer logit-lens KL, and a 2-layer MLP probe on the 5 tapped hidden states).
  3. Integrate the best single-pass estimator into the online training pipeline without precomputing no-image target passes.
  4. Train the Qwen3.5-4B drafter for 6 epochs and compare Table 1 acceptance lengths and speedups against the official MMFlash and DFlash baselines.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

Works like SpecOPD and focus on multimodal hallucination tracing and attention, but none explore single-pass proxies for visual dependence using cross-modal attention mass, logit-lens divergence, or linear probes on target hidden states.

We searched OpenAlex and arXiv for “visual dependence estimation single forward pass multimodal speculative decoding” and “Single-Pass Visual Dependence Estimation to Eliminate Dual-Pass Target Scoring” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

MMFlash: Faster Speculative Decoding for Vision-Language Models

Sicheng Feng, Zigeng Chen, Song Wang +2 more · 2026

MMFlash introduces a diffusion-based speculative decoding framework designed to accelerate inference in vision-language models (VLMs) while preserving exact target output distributions. After diagnosing that draft token acceptance drops sharply on visually dependent tokens even with multimodal training, the paper proposes a visually grounded loss reweighting scheme and a sparse attention pattern for long video contexts. Across seven image and three video benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58x on Qwen3.5 (and up to 4.71x on Qwen3.8-27B), outperforming autoregressive and text-focused diffusion drafters.

Related Ideas

Is this a good idea?

Share onXLinkedIn