Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchNo close match found

Query-Driven Dynamic Layout Parsing

Enable flexible information extraction by conditioning document parsing models on natural language queries rather than fixed reading progressions.

Time

1 week

Compute level

Multiple GPUs

Resources

4x A100 GPUs (80GB) to retrain the autoregressive parsing decoder with prompt conditioning.

What the paper used

Autoregressive generation of structured sequence S = [(B1, c1, t1), ..., (BK, cK, tK)] trained on ~300k diverse samples.

Prerequisites

transformer architectures

Scope

large-scale data collection

The Gap

Section 3.1 ('Unified Task Formulation') defines the output as a fixed sequence S aligned with 'human reading progression' (Eq. 1).

“Crucially, the sequence S is generated in an order that aligns with human reading progression. This task formulation compels the model to not only parse content but also to comprehend the document's logical flow”
From the paper, Section 3.1, page 3. Matched against the PDF text.

Limitation

Enforcing a single, deterministic 'reading order' is restrictive for non-linear documents (e.g., newspapers, websites, forms) where multiple valid traversal paths exist. The current model rigidly serializes content, which may not match user intent when specific information is needed.

Proposed Solution

Reformulate the training objective to support Query-Driven Parsing. Condition the generation on a natural language prompt (e.g., 'Extract the table in the bottom left', 'Read the sidebar headers'). This requires modifying the data engine to generate prompt-response pairs rather than just full-document sequences.

Why This Matters

Transforms the model from a static serializer into a dynamic agent capable of targeted information extraction, significantly increasing utility for complex downstream tasks.

How to Get Started

How to test it: We hypothesize that conditioning the sequence generation on natural language prompts will enable flexible query-driven extraction. We will compare against the standard reading order model using Reading Order Edit and extraction precision metrics, stopping if general parsing performance degrades.

  1. Modify the data generation engine to pair document images with specific user queries and localized target answers.
  2. Reformulate the model's training objective to take natural language prompt tokens as conditional inputs during sequence generation.
  3. Fine-tune the model on the query-driven parsing dataset and evaluate extraction accuracy against targeted prompts.

Code linked to this paper on Hugging Face: github.com/rednote-hilab/dots.ocr

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

While Intelligent Document Parsing and other works touch on document parsing or graph attention for specific parsing tasks, none of the listed papers discuss query-driven dynamic layout parsing with natural language prompts as proposed.

We searched OpenAlex for “Enable flexible information extraction by conditioning document parsing models on natural language queries rather than fixed reading progressions.” and “Query-Driven Dynamic Layout Parsing” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model

Yumeng Li, Guang Yang, Hao Liu +2 more · 2025

This paper introduces dots.ocr, a unified Vision-Language Model designed for end-to-end document layout parsing, text recognition, and relational understanding across 126 languages. The authors achieve this by training a 1.2B parameter vision encoder from scratch on a massive, synthetically generated multilingual dataset produced by a novel three-stage data engine, demonstrating state-of-the-art performance on OmniDocBench and the newly introduced XDocParse benchmark.

Related Ideas

Is this a good idea?

Share onXLinkedIn