Single-Pass Visual Dependence Estimation to Eliminate Dual-Pass Target Scoring
Estimate token-level visual dependence from a single target forward pass via hidden-state shifts or attention statistics to eliminate the extra no-image scoring pass.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 4.1 and Section 4.2, MMFlash computes the token-level visual dependence score d_t as the KL divergence between the target VLM's next-token distributions with and without the image (Equation 1). As noted in Section 5.1 and Appendix D.1, responses are regenerated per target model across ~900K training samples (up to 4,096 tokens), meaning the visual dependence scoring step requires running an extra full no-image forward pass of the target model over the entire dataset.
“For every token t of a training response, we compute its visual dependence dt with Equation 1, i.e., with the target model, the image I, and the prefix x<t fixed.”
Limitation
Requiring a second, image-ablated forward pass of the target model over every training sample doubles target inference cost during data preparation, cannot be recycled across different target models, and becomes increasingly expensive when scaling to larger target models (e.g., 27B+), longer contexts (e.g., 40K-token videos), or online distillation pipelines.
Proposed Solution
Develop a single-pass proxy for visual dependence computed directly during the standard with-image target forward pass used for response generation or online hidden-state tapping. Specifically, investigate three single-pass estimators calibrated against ground-truth KL on a small 10K-sample seed set: (1) cross-modal attention mass directed toward visual tokens in middle-to-late target layers, (2) L2/cosine shift between target hidden states before and after visual-attending layers via logit-lens divergence, and (3) a tiny linear probe trained on tapped target hidden states to predict the quantile rank \tilde{g}_t directly.
Why This Matters
Replacing dual-pass KL precomputation with an accurate single-pass proxy is expected to cut data-preparation compute by nearly half and make visually grounded loss reweighting practical for large target models and long video corpora.
How to Get Started
How to test it: Test whether a single-pass proxy achieves Spearman rank correlation >= 0.85 with ground-truth Equation 1 KL quantiles on a held-out 10K set and yields drafter acceptance lengths and speedups matching dual-pass MMFlash (within 1-2%) when trained on Qwen3.5-4B across Table 1 benchmarks. Stop if proxy-trained drafters lose >3% speedup relative to dual-pass MMFlash.
- Compute ground-truth Equation 1 KL scores alongside visual attention mass and tapped hidden-state features on a 10K-sample subset of LLaVA-OV-1.5-Instruct-Data using Qwen3.5-4B.
- Benchmark rank correlation (Spearman rho) between true quantile \tilde{g}_t and candidate single-pass proxies (visual attention entropy/sum, layer-to-layer logit-lens KL, and a 2-layer MLP probe on the 5 tapped hidden states).
- Integrate the best single-pass estimator into the online training pipeline without precomputing no-image target passes.
- Train the Qwen3.5-4B drafter for 6 epochs and compare Table 1 acceptance lengths and speedups against the official MMFlash and DFlash baselines.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like SpecOPD and focus on multimodal hallucination tracing and attention, but none explore single-pass proxies for visual dependence using cross-modal attention mass, logit-lens divergence, or linear probes on target hidden states.
- SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding · 2026 · 1 citations
- Mitigating multimodal hallucinations through visual attention tracing and origin-point regeneration · 2026 · 2 citations
- S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation · 2026 · 6 citations
We searched OpenAlex and arXiv for “visual dependence estimation single forward pass multimodal speculative decoding” and “Single-Pass Visual Dependence Estimation to Eliminate Dual-Pass Target Scoring” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
MMFlash: Faster Speculative Decoding for Vision-Language ModelsSicheng Feng, Zigeng Chen, Song Wang +2 more · 2026
MMFlash introduces a diffusion-based speculative decoding framework designed to accelerate inference in vision-language models (VLMs) while preserving exact target output distributions. After diagnosing that draft token acceptance drops sharply on visually dependent tokens even with multimodal training, the paper proposes a visually grounded loss reweighting scheme and a sparse attention pattern for long video contexts. Across seven image and three video benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58x on Qwen3.5 (and up to 4.71x on Qwen3.8-27B), outperforming autoregressive and text-focused diffusion drafters.
Related Ideas
- Token-Normalized and Convergence-Controlled Evaluation of VLM Scaling Laws in RoboticsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🟡Fairly easy
- Parameter-Efficient Action Adaptation and Multi-Layer Feature Readout for Frontier VLMsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🔴Hard
- Adaptive Token-Level Gating for Heterogeneous Modality AlignmentInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🟡Fairly easy
- Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR RecognitionV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟡Fairly easy
- Decoupled Asymmetric Modality Branching in Vision-Language TransformersMulti-Branch Transformer🔴Hard