Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Beyond the Abstract: Full-Context Scientific Judgement via Hierarchical LLM Evaluation

A method that fine-tunes long-context language models to evaluate full-text scientific PDFs, weighing methodology and related work sections against abstracts for expert-level judgment.

Time

4-8 weeks due to large-scale PDF acquisition, structured section extraction, and substantially slower long-context GRPO rollouts.

Compute level

Large cluster

Resources

64-128 H200-equivalent GPUs (8-16 nodes x 8 GPUs) with DeepSpeed ZeRO-3 and vLLM, full-text PDF downloading and section-parsing infrastructure for hundreds of thousands of arXiv papers, and extended-context GRPO training well beyond the paper's 2,048-token max sequence length.

What the paper used

H200-equivalent GPUs with DeepSpeed ZeRO-2/ZeRO-3 and vLLM, scaled by model size: 32 GPUs (4 nodes x 8 GPUs) for 1.5B, 64 GPUs (8 nodes x 8 GPUs) for 3B-7B, and 128 GPUs (16 nodes x 8 GPUs) for 14B-32B models; trained via GRPO (batch size 128, 1 epoch, 8 generations/prompt) on 720,341 pairs (from 2.1M arXiv papers) for SciJudge and 4,000 seed papers for SciThinker.

Prerequisites

Long-context LLMs, natural language processing

Scope

large-scale data collection and modify existing code

The Gap

Scientific Judge is currently trained exclusively on paper titles and abstracts. This forces the model to judge scientific potential without access to crucial methodological details, experimental validation, or the related work section, which are essential for true expert-level evaluation.

“Incorporating richer paper context, such as related work sections, may improve scientific judgement.”
From the paper, Section 6 (Limitations and Future Work), page 13. Matched against the PDF text.

Limitation

The model evaluates scientific ideas using only surface-level metadata (titles and abstracts), ignoring the depth and rigorous context of the full manuscript.

Proposed Solution

Extend the RLCF pipeline to process full-text PDFs by employing a long-context LLM. Fine-tune the reward model to extract and weigh specific sections such as 'Methodology' and 'Related Work' against the proposed 'Abstract' to identify logical gaps and methodological soundness.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Incorporating full text will drastically improve the accuracy of the judgement capability by grounding the AI's taste in methodological rigor rather than just catchy abstracts.

How to Get Started

How to test it: Test the hypothesis that conditioning Scientific Judge on extracted Methodology and Related Work sections alongside titles and abstracts improves impact prediction over the title-and-abstract-only SciJudge-Qwen3-4B/30B baseline. Measure position-swap pairwise accuracy on SciJudgeBench (1,000 main pairs), ICLR Metric OOD (611 pairs), and embedding-based topic-control (245 pairs), stopping if full-text context yields less than a 1 percentage point gain given the added long-context cost.

  1. Compile a corpus of full-text LaTeX sources and PDFs from the arXiv dump paired with expert review scores.
  2. Build a long-context LLM pipeline capable of parsing complete research papers including methodology and related work sections.
  3. Define an extractive fine-tuning objective to identify logical gaps and methodological soundness from full-text contents.
  4. Evaluate the fine-tuned reward model against an abstract-only baseline using correlation with human expert evaluations.

Code linked to this paper on Hugging Face: github.com/tongjingqi/AI-Can-Learn-Scientific-Taste

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers like 'An auditable pipeline for fuzzy full-text screening in systematic reviews' and 'A survey on LLM-as-a-judge' touch upon processing full-text documents and LLM judgment pipelines. However, none of them specifically implement a hierarchical RLCF pipeline that contrasts full-text sections with abstracts for scientific preference fine-tuning.

We searched OpenAlex for “A method that fine-tunes long-context language models to evaluate full-text scientific PDFs, weighing methodology and related work sections against abstracts for expert-level judgment.” and “Beyond the Abstract: Full-Context Scientific Judgement via Hierarchical LLM Evaluation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

AI Can Learn Scientific Taste

Jingqi Tong, Mingzhe Li, Hangcheng Li +20 more · 2026

This paper explores whether AI models can develop 'scientific taste' to evaluate and propose high-impact research ideas. The authors propose Reinforcement Learning from Community Feedback (RLCF) to train models using large-scale citation data, successfully creating an AI that generates research proposals with higher potential impact than baseline models.

Related Ideas

Is this a good idea?

Share onXLinkedIn