Bypassing Pixel Reconstruction for Efficient OCR-Centric Latent Processing
A method that replaces VAE pixel reconstruction with a specialized latent-to-latent projection network, mapping VAE latents directly into the SigLIP2 feature space for efficient OCR processing.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
To preserve fine-grained textual details for OCR, the authors found it necessary to pass VAE latents through a full VAE decoder to reconstruct the pixel-space image before feeding it into the SigLIP2-ViT semantic encoder.
“To address this issue, we introduce a VAE decoder that reconstructs the latent representation back into the pixel space before semantic encoding.”
Limitation
Reconstructing full pixel-space images from latents merely to re-encode them introduces significant computational latency and memory overhead during both training and inference.
Proposed Solution
Develop a specialized latent-to-latent projection network trained with a contrastive loss against high-frequency text regions. This adapter would map VAE latents directly into the SigLIP2 feature space, optimizing specifically for OCR retention (using ChartQA and OCRBench signals) to eliminate the need for the VAE pixel decoder.
Why This Matters
Removing the pixel reconstruction bottleneck would drastically reduce inference latency and memory usage, making the model far more efficient for real-time applications.
How to Get Started
How to test it: We hypothesize that a latent-to-latent projection adapter trained with contrastive loss on high-frequency text regions can map VAE latents directly into SigLIP2 feature space, matching the pixel-reconstructing CHEERS baseline on OCR while cutting encoding latency and memory. Evaluating under the Appendix A 256x256 protocol (up to 4.6M alignment + 858K fine-tuning) and 512x512 on ChartQA, OCRBench, TextVQA, and wall-clock/VRAM throughput, we would stop if the adapter recovers less than 90% of the pixel-reconstruct baseline's OCR scores (42.1 ChartQA / 31.5 OCRBench at 256x256) or reduces end-to-end latency by less than 15%.
- Gather a dataset of chart and document images with corresponding OCR bounding boxes using ChartQA and OCRBench.
- Extract VAE latents and target SigLIP2 features from the dataset to build training pairs.
- Implement a lightweight projection network trained with a contrastive loss to map the latents directly to the semantic feature space.
- Evaluate OCR retention performance against a baseline pipeline that uses the full VAE decoder.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
While papers like UniCom and RTPrune explore unified multimodal modeling and OCR efficiency, none of the listed works describe mapping VAE latents directly into a SigLIP2 feature space using a contrastive loss to bypass pixel reconstruction specifically for OCR retention.
- UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations · 2026 · 2 citations
- RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference · 2026 · 4 citations
- Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing · 2025 · 26 citations
We searched OpenAlex for “A method that replaces VAE pixel reconstruction with a specialized latent-to-latent projection network, mapping VAE latents directly into the SigLIP2 feature space for efficient OCR processing.” and “Bypassing Pixel Reconstruction for Efficient OCR-Centric Latent Processing” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and GenerationYichen Zhang, Da Peng, Zonghao Guo +19 more · 2026
The paper introduces CHEERS, a unified multimodal model that seamlessly integrates both visual comprehension and high-fidelity image generation within a single transformer framework. Its primary contribution is a novel architecture that decouples patch-level high-frequency visual details from low-frequency semantic representations, stabilizing multimodal reasoning while enhancing image synthesis through adaptive detail injection.
Related Ideas
- Decoupled Text Representation DistillationMobile-O: Unified Multimodal Understanding and Generation on Mobile Device🟠Medium
- Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR RecognitionV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟡Fairly easy
- Latent-Space Saliency DistillationSpotEdit: Selective Region Editing in Diffusion Transformers🟠Medium
- Accelerating Video VAE Decoding via Distilled QuantizationTurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times🟡Fairly easy
- Optical KV-Cache for Long-Horizon Dialogue AgentsDeepSeek-OCR: Contexts Optical Compression🟡Fairly easy