Skip to content

ReGround: Grounding Reviewer Comments in Multimodal Evidence

Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.54/
Code: https://github.com/UKPLab/emnlp2026-reground
Area: Information Retrieval
Keywords: Reviewer Comment Grounding, Multimodal Retrieval, Scientific Document Understanding, Evidence Type Routing, Modality Complementarity

TL;DR

Addressing the lack of fine-grained multimodal evidence alignment benchmarks in peer review, ReGround leverages explicit author rebuttal references to construct a large-scale retrieval benchmark across 3,656 papers and 16,274 commentโ€“evidence pairs, exposing evidence-type routing bottlenecks and substantial text-visual complementarity.

Background & Motivation

Peer review serves as the cornerstone of scientific quality assurance, generating immense volumes of expert feedback each year. As submission volumes surge, major academic conferences have begun deploying automated reviewer-assistance systems to alleviate reviewer burden. A foundational capability for such systemsโ€”as well as for scientific document QA and claim verificationโ€”is grounding evaluative reviewer feedback, criticisms, and queries into concrete evidence within the reviewed manuscript, whether that evidence takes the form of an implementation detail, a statistical result table, a section-level claim, or an empirical trend in a figure. However, real-world reviewer comments are predominantly evaluative, underspecified, and implicitly connected to the manuscript, standing in stark contrast to traditional information-seeking scientific QA benchmarks such as QASPER or PeerQA.

Crucially, scientific evidence is structurally heterogeneous across both modalities and granularities, spanning individual lines, paragraphs, entire sections, visual figures, and empirical tables. Existing peer review corpora either model discourse dynamics at the coarse document level (such as DISAPERE and APE), or track revision edits manifested in camera-ready manuscripts rather than the reviewed submission (such as ARIES). Meanwhile, rare resources that target fine-grained evidence linking against original manuscripts (such as F1000RD and CLAIMCHECK) rely on laborious manual annotation, remaining small in scale and strictly confined to plain text. The absence of a large-scale, multimodal benchmark linking evaluative comments to fine-grained evidence in original anonymous submissions has substantially hindered progress in scientific document retrieval.

This paper breaks through this bottleneck via a key insight into author-reviewer communication: during author rebuttals, authors routinely and explicitly cite specific line numbers, sections, figures, and tables in their submitted paper to clarify misunderstandings, counter criticisms, or highlight overlooked details. These rebuttal citations constitute a natural, highly precise source of expert human annotations. Core idea: exploit explicit paper references in author rebuttals as high-precision weak supervision signals to automatically construct ReGround, a large-scale multimodal retrieval benchmark linking reviewer comments to paragraphs, sections, figures, and tables in original anonymous submissions.

Method

Overall Architecture

ReGround frames reviewer comment grounding as an intra-document evidence retrieval task: given a reviewer comment \(c\) stripped of trivial explicit references, the system must retrieve all supporting evidence units from a candidate pool \(\mathcal{D} = \{d_1, \dots, d_n\}\) extracted from the original anonymous submission that authors utilized to address \(c\). The construction and benchmarking pipeline comprises structured document extraction, reference detection and alignment, three-stage precision filtering, and multimodal retrieval evaluation across diverse candidate pools.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Submission PDF + Reviews + Author Rebuttals"] --> B["Document & Sentence-Level Parsing"]
    B --> C["Explicit Manuscript Reference Detection<br/>8-category regex patterns covering lines/secs/figs/tabs"]
    C --> D["Reviewerโ€“Rebuttal Sentence Alignment<br/>Long-context LLM extracts target review spans"]
    D --> E["Three-Stage Precision Filtering<br/>Pruning review quotes/explicit clues/external content"]
    E --> F["Resolution to Structured Candidate Units<br/>Constructing multimodal evidence candidate pools"]
    F --> G["Unified / Type-Aware / Multimodal Retrieval Benchmarks"]

Key Designs

1. Structured Document Extraction and Reference Detection: Multi-Granularity Candidate Parsing To guarantee high fidelity and reproducibility, the corpus is curated from NLPeer v2, aggregating original anonymous submission PDFs from EMNLP, COLING, NAACL, and ACL. At the document parsing level, textual components are parsed via pypdf and structured into section headers and numbered lines, while GROBID reconstructs robust paragraph semantic boundaries (line-level references are resolved to their containing paragraphs). Visually, PDFFigures 2.0 extracts all embedded figures and tables alongside their captions. To detect explicit evidence citations in rebuttals, an 8-category regular expression suite was iteratively developed and verified across 250 sample rebuttals, capturing line references (e.g., lines 120โ€“135), section citations (e.g., Section 3.2, ยง4, and canonical header names), figures (e.g., Figure 2), tables (e.g., Table 1), appendices, equations, pages, and footnotes.

2. LLM-Based Reviewerโ€“Rebuttal Span Alignment: Long-Context Multi-Span Disambiguation Once a reference-bearing rebuttal sentence is detected, the pipeline identifies which exact reviewer comment span it responds to. Authors often structure rebuttals using explicit markers (e.g., "W1", "Q3") or localized paraphrasing. The pipeline employs gpt-oss-120b, providing it with the complete indexed reviewer report, the full rebuttal text, and the target rebuttal sentence. The model predicts the exact identifiers of addressed reviewer spans; cases addressing multiple points retain all corresponding alignments, while ambiguous instances are discarded. Manual double-blind validation confirms a 96% alignment acceptance rate (Cohen's \(\kappa=0.83\)).

3. Three-Stage Precision Filtering Pipeline: Distilling Implicit In-Submission Grounding Raw aligned links contain three primary failure modes that would compromise retrieval benchmark integrity: - Removing Quoted Review Text (~20% discarded): Authors often quote reviewer text verbatim before answering, which inadvertently triggers false-positive reference detections; these overlapping sentences are filtered via fuzzy string matching; - Removing Trivial Explicit References (~18% discarded): To establish a realistic implicit grounding benchmark, pairs where the reviewer comment itself mentions the target evidence (e.g., "Table 2 results are unconvincing") are purged, preventing trivial lexical matching solutions; - LLM Filtering of Out-of-Submission Content (~23% discarded): gpt-oss-120b inspects each pair under strict binary criteria to eliminate promised camera-ready modifications, new experiments introduced solely during the rebuttal period, and external paper citations, ensuring all remaining 16,274 evidence units reside strictly within the original submission.

4. Pointwise Scoring Calibration and Multimodal Retrieval Protocols: Decoupled Modality Benchmarking The benchmark evaluates sparse lexical encoders (BM25, SPLADEv3), dense dual-encoders (BGE-M3, EmbeddingGemma, Qwen-3 Embedding), cross-encoders (BGE-M3-Reranker), and LLM pointwise rankers. For LLM evaluators, given comment \(c\) and candidate \(d\), the unnormalized decision log-probabilities \(\ell_A\) (for relevant) and \(\ell_B\) (for irrelevant) yield a calibrated continuous relevance score: $\(p(A \mid c, d) = \frac{\exp(\ell_A)}{\exp(\ell_A) + \exp(\ell_B)}\)$ For visual evidence (tables and figures), the benchmark evaluates both Image-only and Image+Caption inputs across SigLIP2, OpenCLIP, Jina Embeddings v4, ColQwen2.5, and Qwen-3 VL models, establishing quantitative baselines for cross-modal late fusion.

Key Experimental Results

Main Results

The primary benchmark evaluates unified retrieval over a single heterogeneous textual pool containing paragraphs, sections, and table/figure captions, where systems must simultaneously perform content matching and evidence-type selection.

Model Family Model MRR Recall@1 (%) Recall@2 (%) Recall@10 (%)
Sparse Encoders BM25 7.62 3.57 5.65 14.90
Sparse Encoders SPLADEv3 7.39 3.04 5.38 15.17
Dense Encoders all-mpnet 9.08 4.08 6.78 19.12
Dense Encoders BGE-M3 9.41 4.45 7.14 19.05
Dense Encoders Qwen-3 Embedding 4B 6.92 3.26 5.03 14.59
Dense Encoders EmbeddingGemma 8.78 4.17 6.64 17.68
Cross Encoders BGE-M3-Reranker 7.85 3.42 5.67 16.20
Cross Encoders MiniLM-L12 7.92 3.62 5.82 15.81
LLM Rankers Gemma-3 27B 10.12 4.68 7.74 20.34
LLM Rankers Qwen-3 30B Instruct 10.87 5.12 8.29 21.15

When isolating type inference in the oracle setting (evaluating paragraphs against paragraphs, tables against tables), performance jumps substantially: Qwen-3 30B Instruct reaches 32.97 MRR / 53.64% R@10 on paragraphs and 56.17 MRR on table captions, proving that evidence-type ambiguity is the primary bottleneck in unified retrieval.

Ablation & Modality Analysis: Visual Evidence Retrieval

On visual targets, models were evaluated comparing Image-only against Image+Caption configurations.

Target Type Model Image+Caption MRR Image+Caption R@1 (%) Image+Caption R@2 (%) Image-only MRR Image-only R@1 (%) Image-only R@2 (%)
Table Retrieval SigLIP2 49.60 30.04 48.94 44.54 24.40 42.56
Table Retrieval Jina Embeddings v4 55.67 37.19 55.92 52.80 34.46 51.70
Table Retrieval ColQwen2.5 v0.2 57.95 40.48 58.33 53.73 35.65 52.16
Table Retrieval Qwen-3 VL 32B 59.53 42.39 60.34 56.79 39.22 56.97
Figure Retrieval SigLIP2 47.84 27.93 46.31 44.23 24.02 42.53
Figure Retrieval Jina Embeddings v4 52.11 32.88 52.87 46.65 26.80 45.57
Figure Retrieval ColQwen2.5 v0.2 52.53 32.82 53.79 46.54 26.10 46.14
Figure Retrieval Qwen-3 VL 32B 56.39 37.66 58.47 50.85 30.67 51.13

Key Findings

  • Unified text retrieval is remarkably challenging: Even the strongest LLM-based pointwise ranker achieves only 21.15% Recall@10 and 10.87 MRR across the unified pool. Cross-encoders suffer severe degradation on long section-level spans (MRR collapsing from 44.2 to 0.2 at Q10 length quantile), causing them to underperform bi-encoders;
  • Evidence-type routing is critical but must not be hard-filtered: While oracle type knowledge doubles or triples retrieval scores, deploying hard type classification before retrieval drops Qwen-3 30B's R@10 from 21.15% to 6.11%, because classification errors on figures/tables permanently discard gold targets. Type prediction is strictly a soft reranking feature;
  • Multi-evidence comments suffer severe retrieval breakdown: For the 25.77% of queries grounded in multiple non-redundant evidence units, even under oracle type conditions, Qwen-3 30B retrieves only 12.69% of complete sets at R@1 and 23.53% at R@2, showing that multi-evidence aggregation remains an open challenge;
  • Textual and visual evidence streams exhibit high complementarity: In table grounding, image retrieval significantly outperforms caption text alone (R@2 60.34% vs 53.29%, McNemar \(p=0.0156\)), and the modalities disagree on 29.7% of instances (both fail on 24.8%). Systems should adopt late fusion rather than relying exclusively on a single modality.

Highlights & Insights

  • Turning author rebuttals into natural weak supervision: Instead of spending hundreds of annotator hours or synthesizing artificial QA pairs, the authors cleverly harvest authentic author citations from rebuttals, generating tens of thousands of expert-verified grounding links at negligible cost.
  • Exposing the structural pitfalls of heterogeneous retrieval pools: By systematically examining candidates of wildly varying lengths (captions vs. full sections), the study reveals why standard cross-encoders fail in scientific documents and why evidence routing cannot be treated as a simple pre-filter.
  • Establishing a stress-test benchmark for scientific AI assistants: Provides an indispensable empirical evaluation platform for grounding automated peer review assistants, factual verification tools, and deep scientific research agents against source documents.

Limitations & Future Work

  • Author citation bias: The dataset only captures evidence that authors explicitly chose to mention in rebuttals; valid evidence that authors overlooked or avoided answering remains unannotated.
  • Domain limitation and coarse-grained citations: The corpus is currently restricted to NLP/AI venues due to peer review availability, and approximately 1.58% of evidence links cite entire sections where a specific sub-clause would suffice.
  • Separate candidate pools for text and vision: Multimodal retrieval is currently evaluated in partitioned pools (text vs. visual units) rather than a fully unified end-to-end multimodal index.
  • vs QASPER / PeerQA: QASPER and PeerQA formulate scientific document queries as factual, information-seeking questions with clear text answer spans. ReGround tackles authentic evaluative reviewer remarks that are underspecified and require navigating heterogeneous multimodal evidence.
  • vs ARIES / RE3: Revision datasets examine how reviewer feedback changes subsequent paper versions (camera-ready edits), whereas ReGround evaluates retrieval over the original submission as reviewed.
  • vs CLAIMCHECK / F1000RD: Prior manuscript-linking datasets are small-scale (40โ€“170 papers), manually annotated, and strictly text-only, while ReGround scales to 3,656 papers and incorporates tables and figures.

Rating

  • Novelty: โญโญโญโญโญ [Pioneering framing of author rebuttals as a scalable, high-precision supervision source for implicit document grounding]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive benchmarking across 10+ retrieval architectures, dissecting length, routing, multi-evidence, and visual modalities]
  • Writing Quality: โญโญโญโญโญ [Exemplary clarity, rigorous error analysis, and transparent documentation of limitations]
  • Value: โญโญโญโญโญ [Highly actionable for scientific document RAG, automated review assistance, and multimodal evaluation]