Direct Visual-to-Graph Generation for Irregular Layouts
Fine-tune a vision-language model to directly output graph adjacency matrices from raw image crops, bypassing fragile rule-based table parsers and text conversions.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Appendix A.5 (Case 2, Figure 12) highlights 'Structural Noise in Ambiguous Table Layouts.' The current methodology relies on intermediate parsers (MinerU) and rule-based extraction to build the graph. When tables have merged cells or lack clear boundaries, the 'rigid spatial processing patterns' fail to create correct row-column edges, breaking the retrieval path.
Limitation
The reliance on deterministic parsing rules for graph construction causes the system to fail on non-standard, structurally ambiguous document layouts (like complex financial tables).
Proposed Solution
Replace the rule-based table parser with a Generative Visual-Graph Module. Fine-tune a VLM to directly output the adjacency matrix (or edge list) of the graph entities from the raw image crop of the table, bypassing the text-conversion step. This allows the model to learn 'implicit' structural relationships (like visual grouping) that rule-based parsers miss.
Why This Matters
This would significantly improve robustness in professional domains (finance, law) where document layouts are complex and inconsistent, preventing the 'catastrophic failure' seen in the baselines.
How to Get Started
- Gather benchmark table layout datasets such as PubTabNet with ground-truth structural annotations.
- Format table structural annotations into explicit adjacency matrices or edge lists.
- Fine-tune a vision-language model to directly predict graph structures from raw table image crops.
- Evaluate structural extraction performance and downstream retrieval path accuracy against rule-based parsers like MinerU.
Code linked to this paper on Hugging Face: github.com/HKUDS/RAG-Anything
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed works do not cover direct visual-to-graph generation for table layouts using VLMs. Papers such as LayoutGPT and Constrained Layout Generation with Factor Graphs deal with generating layouts or using factor graphs, but none bypass text conversion to directly output adjacency matrices from raw image crops for table parsing.
We searched OpenAlex for “Fine-tune a vision-language model to directly output graph adjacency matrices from raw image crops, bypassing fragile rule-based table parsers and text conversions.” and “Direct Visual-to-Graph Generation for Irregular Layouts” and an AI compared the idea with the top 9 results. A small search can miss relevant work.
Source Paper
RAG-Anything: All-in-One RAG FrameworkZirui Guo, Xubin Ren, Lingrui Xu +2 more · 2025
This paper introduces RAG-Anything, a multimodal Retrieval-Augmented Generation framework designed to treat non-textual elements (images, tables, equations) as first-class knowledge entities alongside text. It overcomes the limitations of text-centric RAG systems by constructing a unified dual-graph (cross-modal and text-based) that preserves structural relationships and visual semantics, enabling precise reasoning over heterogeneous documents.
Related Ideas
- Generative Vision-Language Model for Table RecognitionMinerU: An Open-Source Solution for Precise Document Content Extraction🔴Hard
- Multi-Granularity Visual Encoding for Small-Scale Document ElementsSmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion🟠Medium
- Iterative Semantic Correction for Layout AnalysisPaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model🟠Medium
- Hierarchical Cell-Level Parallelism for Large TablesDolphin: Document Image Parsing via Heterogeneous Anchor Prompting🟠Medium
- 2D-Attentional Adapters for Complex Table TopologyPaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model🔴Hard