Parameter-Efficient Adaptation of Pre-trained Vision Encoders
Reduce high-resolution vision encoder training costs by applying parameter-efficient fine-tuning to pre-trained vision models instead of training from scratch.
Source paper
Yumeng Li, Guang Yang, Hao Liu, Bowen Wang, Colin Zhang
Generated by AI. The paper’s authors did not write these.
Reduce high-resolution vision encoder training costs by applying parameter-efficient fine-tuning to pre-trained vision models instead of training from scratch.
Improve real-world document understanding robustness by penalizing representation divergence between clean and augmented unlabelled document images.
Enable flexible information extraction by conditioning document parsing models on natural language queries rather than fixed reading progressions.
This paper introduces dots.ocr, a unified Vision-Language Model designed for end-to-end document layout parsing, text recognition, and relational understanding across 126 languages. The authors achieve this by training a 1.2B parameter vision encoder from scratch on a massive, synthetically generated multilingual dataset produced by a novel three-stage data engine, demonstrating state-of-the-art performance on OmniDocBench and the newly introduced XDocParse benchmark.
The model uses a unified ViT-LLM architecture where a high-resolution Vision Encoder (1.2B parameters, processing up to 11MP images) feeds into a Language Model Decoder (Qwen2.5-1.5B). The task is formulated as an autoregressive generation of a structured sequence containing bounding boxes, category tags, and text content, ordered by reading flow. Training relies on a scalable data engine that uses a large 'Teacher' VLM to synthesize and label diverse multilingual documents.