Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR Recognition
Combine frozen V-JEPA motion representations with text-aligned SigLIP features via dual-stream spectral projection to preserve dynamic reasoning while restoring static visual knowledge.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section I (Scope and limitations), the paper highlights that V-JEPA is trained solely via latent feature prediction without text alignment, causing V-JEPA-LM to lag behind text-aligned encoders like SigLIP on static image QA, document reading, and knowledge-heavy benchmarks (e.g., NExT-QA).
“V-JEPA is trained only to predict masked regions of video in latent space, never to align with or read text, so text recognition lies outside what the encoder was built for”
Limitation
V-JEPA-LM suffers from poor single-frame image QA and optical character recognition (OCR) capabilities due to V-JEPA's lack of text-aligned pretraining data.
Proposed Solution
Build a dual-stream spectral bridge that fuses visual tokens from frozen V-JEPA (predictive encoder for dynamics) and frozen SigLIP 2 (text-aligned encoder for fine-grained semantic/text OCR). Channel V-JEPA's change subspace tokens alongside SigLIP's frame-level semantic tokens into a joint projector MLP, fine-tuning the VLM with the combined counterfactual and standard multimodal QA mixture.
Why This Matters
Eliminates the trade-off between dynamic motion understanding and static visual knowledge, allowing a single lightweight video LLM to excel at both dynamic motion QA and static/text-heavy image QA.
How to Get Started
How to test it: Test if dual-stream fusion maintains V-JEPA's high TOMATO dynamics score while closing the gap on NExT-QA and PerceptionTest. Baseline: V-JEPA-LM single encoder. Metric: Average accuracy across TOMATO and NExT-QA. Stopping result: Stop if dual-stream fusion decreases TOMATO dynamics score by >3 percentage points.
- Set up the V-JEPA-LM repository and reproduce the 2B Qwen3 baseline model.
- Add SigLIP 2 ViT-L/14 as a parallel vision backbone alongside frozen V-JEPA 2.1.
- Concatenate/project the output representations through a dual-stream spectral bridge.
- Train on the 4.53M dataset mixture and evaluate on TOMATO (dynamics) and NExT-QA (knowledge/text).
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as 'Can World Models Benefit VLMs for World Dynamics?' and 'PROSPECT' study the fusion of predictive latent representations with semantic vision-language models. None of the listed papers combine a frozen V-JEPA predictive encoder for motion dynamics with a frozen SigLIP 2 encoder via a joint change-subspace projector MLP.
- Can World Models Benefit VLMs for World Dynamics? · 2025 · 13 citations
- PROSPECT: Unified Streaming Vision-Language Navigation via Semantic–Spatial Fusion and Latent Predictive Representation · 2026 · 3 citations
We searched OpenAlex and arXiv for “dual stream vision encoder V-JEPA SigLIP video VLM” and “Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR Recognition” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned RepresentationsKeuntae Kim, Dongjun Kim, Yong Suk Choi · 2026
This paper investigates why Video Large Language Models (video LLMs) fail to read visual motion dynamics even when powered by predictive world-model encoders like V-JEPA. The authors demonstrate that standard text captioning and QA supervision allow language models to rely on single-frame static shortcuts rather than dynamic features. To address this, they propose counterfactual supervision using fourteen pixel-level video transformations that isolate change components, enabling state-of-the-art motion and temporal reasoning across eleven video benchmarks with significantly lower compute and data requirements.
Related Ideas
- Extending Decoupled Semantic-Detail Representations to Video ModalitiesCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation⛔Very hard
- Scaling Unified Multimodal Models via Large Pre-trained Vision-Language InitializationsCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation🔴Hard
- Decoupled Asymmetric Modality Branching in Vision-Language TransformersMulti-Branch Transformer🔴Hard
- Bypassing Pixel Reconstruction for Efficient OCR-Centric Latent ProcessingCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation🟠Medium
- Query-Adaptive Video Context Compression and Frame-Level Visual Dependence for Long-Video Speculative DraftingMMFlash: Faster Speculative Decoding for Vision-Language Models🟠Medium