Multi-Granularity Visual Encoding for Small-Scale Document Elements
Incorporate a dynamic cropping and tiling strategy to re-encode high-resolution patches of detected table or formula regions and cross-attend them with global page embeddings.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In Section 5.2, the authors note that table structure recognition suffers from poor quality when processing low-resolution image crops (72 dpi), specifically impacting text transcription within cells.
Limitation
Standardized input resolution (144 DPI) creates a trade-off where small text in dense tables or complex mathematical formulas becomes illegible, leading to high character error rates.
Proposed Solution
Incorporate a dynamic cropping and tiling strategy, similar to 'DocOwl 2', where high-resolution patches of detected table or formula regions are re-encoded and cross-attended to by the language model alongside the global page embedding.
Why This Matters
Improving resolution-sensitivity would allow the compact model to compete with larger VLMs on dense, professional documents where data integrity in tables is paramount.
How to Get Started
- Review DocOwl 2 and similar dynamic patch-based visual encoding strategies.
- Set up a public document analysis dataset containing high-resolution tables and mathematical formulas.
- Implement a region detection and dynamic tiling module to extract high-resolution image crops.
- Add cross-attention layers to fuse local patch embeddings with the global page embedding and evaluate cell transcription accuracy.
Code linked to this paper on Hugging Face: github.com/docling-project/docling
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed results address multi-granularity visual encoding for small-scale document elements such as tables or formulas using dynamic cropping. The provided papers cover unrelated topics like medical image segmentation, remote sensing, and activity recognition.
We searched OpenAlex for “Incorporate a dynamic cropping and tiling strategy to re-encode high-resolution patches of detected table or formula regions and cross-attend them with global page embeddings.” and “Multi-Granularity Visual Encoding for Small-Scale Document Elements” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversionAhmed Nassar, Andres Marafioti, Matteo Omenetti +10 more · 2025
SmolDocling is a 256M parameter vision-language model designed for efficient, end-to-end document conversion, mapping page images directly to a structured 'DocTags' markup format. It significantly reduces computational overhead compared to larger models while maintaining robust performance across diverse document elements like tables, charts, and equations.
Related Ideas
- Generative Vision-Language Model for Table RecognitionMinerU: An Open-Source Solution for Precise Document Content Extraction🔴Hard
- Direct Visual-to-Graph Generation for Irregular LayoutsRAG-Anything: All-in-One RAG Framework🟠Medium
- 2D-Attentional Adapters for Complex Table TopologyPaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model🔴Hard
- Hierarchical Cell-Level Parallelism for Large TablesDolphin: Document Image Parsing via Heterogeneous Anchor Prompting🟠Medium
- Dynamic Multi-Scale Layout AnalysisMinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing🟡Fairly easy