Graph-Neural-Network Based Reading Order Regression
Replace heuristic reading order algorithms with a Graph Neural Network that predicts directed edge weights between detected text blocks to reconstruct document reading sequences.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 2.3 describes the reading order algorithm as a heuristic approach: 'MinerU developed a segmentation algorithm based on the human reading order, top to bottom, left to right' after dividing the page into columns.
Limitation
Heuristic sorting fails on complex layouts common in magazines, newsletters, or textbooks with marginalia, where the reading path does not follow a strict geometric grid. This leads to fragmented sentences and incoherent context for downstream LLM training.
Proposed Solution
Replace the rule-based segmentation with a Graph Neural Network (GNN). Construct a fully connected graph where nodes are detected text blocks (from the Layout Analysis step). Train the GNN to predict edge weights representing the probability that node B follows node A, minimizing the distance to the ground truth reading sequence.
Why This Matters
Correct reading order is the single most important factor for semantic coherence when converting PDFs to Markdown for RAG (Retrieval-Augmented Generation) applications. Solving this enables the processing of highly complex non-academic documents.
How to Get Started
- Extract text block coordinates and ground-truth reading sequences from source LaTeX and Word documents.
- Construct a fully connected graph where nodes represent text bounding boxes and edges represent potential reading flows.
- Train a Graph Neural Network to predict binary edge weights minimizing sequence distance to the ground truth.
- Evaluate the GNN-based ordering against rule-based baselines using Kendall's tau distance on test documents.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers address graph neural network-based reading order regression for document layouts. The search results only provide general surveys or applications of graph neural networks in other domains.
We searched OpenAlex for “Replace heuristic reading order algorithms with a Graph Neural Network that predicts directed edge weights between detected text blocks to reconstruct document reading sequences.” and “Graph-Neural-Network Based Reading Order Regression” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
MinerU: An Open-Source Solution for Precise Document Content ExtractionBin Wang, Chao Xu, Xiaomeng Zhao +15 more · 2024
This paper introduces MinerU, an open-source document extraction tool designed to convert diverse and complex PDF documents—such as textbooks, academic papers, and exams—into high-quality machine-readable formats like Markdown and JSON. The authors address the limitations of existing tools by creating a pipeline that integrates specialized deep learning models for layout analysis, formula detection, and content recognition, validated against a newly constructed diverse evaluation dataset.
Related Ideas
- Direct Visual-to-Graph Generation for Irregular LayoutsRAG-Anything: All-in-One RAG Framework🟠Medium
- Topological Loss Integration for Geometric ParsingDeepSeek-OCR: Contexts Optical Compression🟠Medium
- Vertical and non-Manhattan Layout ParsingDolphin: Document Image Parsing via Heterogeneous Anchor Prompting🟡Fairly easy
- Lazy Graph Construction via Student-DistillationRAG-Anything: All-in-One RAG Framework🟡Fairly easy
- Query-Driven Dynamic Layout Parsingdots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model🔴Hard