Retrieval-Augmented Soundness Verification Against External Literature
Augment automated paper reviewers with scholarly literature retrieval so they can verify novelty, baseline coverage, and prior-work claims against published research.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
During dataset construction (Section 4.1, Step 2) and model training/evaluation, the pipeline explicitly filters out review points containing external citations and supplies only the target paper's text to both the policy model and the soundness reward model.
“In addition, we exclude review points that rely on external citations and evaluate papers primarily from the information contained in the target paper.”
Limitation
Restricting evaluation strictly to paper-internal text prevents the AI reviewer from verifying claims about prior work, detecting omitted state-of-the-art baselines, or catching contradictions with established external findings.
Proposed Solution
Integrate a time-filtered scholarly retrieval-augmented generation (RAG) module backed by open corpora (such as Semantic Scholar or arXiv up to each submission's year) into both the policy and the soundness reward model. Train the policy with multi-turn tool-use GRPO to issue targeted search queries when assessing baseline sufficiency or prior-work claims, while updating the soundness RM prompt and training mixture to ground premise verification against retrieved abstracts and quantitative results.
Why This Matters
If successful, this could enable AI reviewers to catch missing comparisons and misrepresented related work without fabricating external citations, substantially increasing real-world reviewing utility.
How to Get Started
How to test it: Hypothesis: equipping the policy and soundness RM with pre-submission-cutoff literature retrieval improves F1 on unfiltered OpenReview review points (including external-citation points) over the paper-only PaperCritic baseline. Baseline: PaperCritic-4B retrained/evaluated without retrieval on the unfiltered test split. Metric: macro precision, recall, and F1 graded by a frontier LLM judge plus citation-hallucination rate; stop if citation precision drops below 80% or F1 fails to exceed the paper-only baseline.
- Reconstruct the ICLR 2020-2023 corpus via the OpenReview API and DeepSeek-OCR following Section 4.1 and Appendix J.1 prompts, retaining the previously filtered external-citation review points as an augmented evaluation split.
- Index arXiv and Semantic Scholar metadata/abstracts with strict publication-date filtering prior to each ICLR submission deadline to prevent temporal leakage.
- Implement a lightweight retrieval tool wrapper for Qwen3-4B-Instruct-2507 and reproduce the paper-only SFT and GRPO F1 baseline on the 889/100 split.
- Extend the soundness RM distillation pipeline (Appendix E) to accept retrieved literature snippets alongside paper text, then run tool-augmented GRPO and compare F1 and citation accuracy against the baseline.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed literature covers RAG applications in various domains like healthcare, systematic reviews, and code generation, as well as citation validation and failure measurement such as in CiteGuard and Verified Misguidance. However, none of the papers combine multi-turn tool-use GRPO with retrieval-augmented soundness reward models specifically for verifying peer-review premises against external literature.
- CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented Validation · 2026 · 13 citations
- Verified Misguidance: Measuring Structural Citation Failures in Search-Augmented LLMs · 2026 · 2 citations
We searched OpenAlex and arXiv for “retrieval augmented automated peer review citation verification LLM” and “Retrieval-Augmented Soundness Verification Against External Literature” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
PaperCritic: Training AI Reviewers to Assess the Soundness of PapersHyun Ryu, Ji Yong Cho, Eunho Yang +4 more · 2026
This paper introduces paper soundness assessment as a standalone AI reviewing task that decouples objective flaw detection from subjective criteria like novelty or clarity. The authors construct PaperSound-23K, a dataset of 23,215 quality-labeled review points across 989 ICLR submissions, and train PaperCritic via reinforcement learning to jointly maximize precision (soundness and substantiveness) and recall (coverage of high-quality reference concerns).
Related Ideas
- Grounding AI Ideation: Multi-Objective RLCF for Feasible Scientific DiscoveryAI Can Learn Scientific Taste⛔Very hard
- Calibrating Automated Scientific Literature Review Evaluators Against Structural LLM BiasPaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing🟡Fairly easy
- Meta-Analytic Agents with Verification-Aware AggregationProbing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows🟡Fairly easy
- Interactive Human-in-the-Loop Refinement with Grounded Revision ConstraintsPaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing🟡Fairly easy
- Beyond the Abstract: Full-Context Scientific Judgement via Hierarchical LLM EvaluationAI Can Learn Scientific Taste🟠Medium