ReGround: Grounding Reviewer Comments in Multimodal Evidence¶
Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.54/
Code: https://github.com/UKPLab/emnlp2026-reground
Area: Multimodal VLM
Keywords: evidence grounding, multimodal retrieval, peer review, scientific document understanding, multimodal fusion
TL;DR¶
ReGround leverages explicit paper references in author rebuttals as high-precision expert supervision to automatically construct the first large-scale benchmark (10,267 comments linked to 16,274 multimodal evidence units across 3,656 papers) for grounding reviewer comments in original submissions, revealing that unified retrieval across heterogeneous evidence pools struggles severely (peaking at 21% Recall@10) and demonstrating strong visual-textual complementarity.
Background & Motivation¶
The exponential surge in academic paper submissions has driven major AI conferences to deploy large language model reviewer-assistance systems into official editorial workflows. A fundamental bottleneck for these systemsโand for scientific document question answering more broadlyโis grounding natural, evaluative reviewer comments into the precise underlying evidence within the submitted manuscript. Such evidence is highly heterogeneous, spanning hyper-specific methodological hyperparameters, quantitative comparison tables, section-level conceptual discussions, or nuanced visual patterns across figures.
Existing scientific QA benchmarks, such as QASPER and PEERQA, predominantly focus on explicit, information-seeking queries targeting clean textual answer spans in camera-ready manuscripts. Revision-tracking benchmarks like ARIES connect review comments to subsequent textual edits rather than to the evidence originally evaluated during peer review. Meanwhile, fine-grained linking datasets such as F1000RD and CLAIMCHECK rely on small-scale manual annotations and completely omit non-textual modalities. In reality, peer review comments are inherently evaluative, underspecified, and frequently target information that the reviewer overlooked. This produces an acute semantic divergence between the query and the manuscript evidence, rendering conventional lexical and semantic matching largely ineffective.
The authors identify an elegant insight: during the rebuttal phase, authors routinely cite specific parts of the reviewed manuscriptโsuch as "see Section 3.2" or "as shown in the last column of Table 2"โto directly address criticisms, clarify ambiguities, or point out overlooked results. These rebuttal citations provide naturally occurring, high-precision expert annotations that directly bind reviewer remarks to the original anonymous submission. Core idea: construct an automated extraction, alignment, and filtering pipeline using explicit manuscript citations in author rebuttals as expert supervision, establishing ReGround as a large-scale multimodal retrieval benchmark linking implicit reviewer comments to fine-grained paragraphs, sections, tables, and figures in original submissions.
Method¶
Overall Architecture¶
The grounding problem is formalized as an information retrieval task: given an evaluative reviewer comment \(c\) stripped of any explicit location indicators (e.g., without mentions like "in Section 4"), a retrieval system must identify the gold evidence units from a candidate pool \(\mathcal{D} = \{d_1, \dots, d_n\}\) extracted from the same original submission. The candidate pool includes textual units (paragraphs, full sections, table/figure captions) and visual units (cropped table and figure images).
The dataset construction pipeline sources submissions, reviews, and rebuttals across major ACL venues from NLPeer v2. The pipeline operates in four coordinated phases: (1) scientific sentence segmentation preserving reference boundaries; (2) structured multimodal PDF extraction; (3) LLM-driven reviewโrebuttal alignment; and (4) a three-stage precision filtering cascade that enforces an implicit retrieval setting.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Corpus<br/>NLPeer v2 Submissions, Reviews, Rebuttals"] --> B["Multimodal PDF Parsing<br/>GROBID paragraphs, PDF sections, PDFFigures 2.0"]
A --> C["Protected Segmentation & Regex Matching<br/>Detect explicit line, section, table, and figure citations"]
B --> D["ReviewโRebuttal Alignment<br/>Contextual LLM maps citation sentences to reviewer comments"]
C --> D
D --> E["Three-Stage Precision Filtering<br/>Discard review echoes, explicit matches, and external claims"]
E --> F["ReGround Benchmark<br/>10,267 Comments grounded to 16,274 Multimodal Evidence"]
Key Designs¶
1. Protected Boundary Segmentation and Manuscript Citation Parsing
Off-the-shelf sentence splitters frequently fail on scientific prose due to Latin abbreviations (e.g., 'e.g.', 'et al.'), explicit citation spans, and complex list enumerations. The pipeline introduces a rule-based segmenter that defines protected text spans where internal punctuation is shielded from splitting. Forced boundaries are enforced at structural markers including list items, blockquotes, and headings. Regex patterns developed across 250 manually inspected rebuttals capture eight distinct citation categories: line numbers and ranges, section numbers and canonical section titles, figure indices, table indices, appendices, equations, page numbers, and footnotes.
2. Multi-Granularity Document Parsing and Reference Resolution
To assemble the candidate retrieval pool, original anonymous submission PDFs are deterministically parsed with pypdf for text and section headers. Because PDF bounding boxes cannot reliably reconstruct multi-column scientific paragraph flows, GROBID is integrated to parse paragraph-level semantic blocks. PDFFigures 2.0 extracts figure and table bounding boxes, rasterizing them into standalone image units while separating their corresponding captions. Detected rebuttal references are dynamically resolved against this structured hierarchy: line references are mapped to their enclosing GROBID paragraphs (discarding superficial line artifacts), while section, figure, and table references are linked to their respective text spans, captions, and visual images.
3. Context-Aware Review Alignment and Three-Stage Implicit Filtering
Rebuttal sentences containing explicit citations are paired with the corresponding reviewer comments using gpt-oss-120b. The model receives indexed review segments alongside the complete rebuttal text to identify addressed comment spans. To isolate genuine implicit grounding challenges and eliminate spurious shortcuts, three cascading filters are applied: - Discarding Quoted Review Text: Authors frequently quote reviewer critiques verbatim prior to answering, which can trigger false reference matches. Fuzzy string matching discards rebuttal sentences overlapping heavily with review text (~20% discarded). - Removing Trivial Explicit Matches: If the reviewer comment already names the exact target evidence (e.g., "Table 1 lacks standard deviations"), the instance can be resolved via surface string matching. All such pairs are removed (~18% discarded), ensuring the dataset strictly tests implicit semantic linking. - Excluding Non-Submission Content: A binary LLM classifier discards rebuttal sentences referring to promised future changes in the camera-ready version, newly run rebuttal experiments, or external citations (~23% discarded). Manual validation confirms 97% precision and 93% recall for this filtering stage.
Loss & Training¶
The benchmark evaluates retrieval across multiple paradigms without modifying retriever weights. For LLM-based pointwise rankers, the model evaluates candidate evidence \(d\) given comment \(c\) and outputs a discrete binary decision token ("Yes" or "No"). To yield a continuous relevance score for ranking, the normalized softmax probability over the binary decision token log-probabilities is computed:
where \(\ell_{\text{Yes}}\) and \(\ell_{\text{No}}\) denote the model-assigned log-probabilities at the decision token position. For dense dual-encoders and multimodal late-interaction architectures (such as ColQwen2.5 and SigLIP2), pairwise cosine similarity or token-level maximum similarity is used to compute retrieval scores over candidate representations.
Key Experimental Results¶
Main Results¶
The primary benchmark evaluates sparse encoders (BM25, SPLADEv3), dense bi-encoders (all-mpnet, BGE-M3, EmbeddingGemma), cross-encoders, and LLM pointwise rankers on the unified textual retrieval pool (combining paragraphs, full sections, and captions) without prior knowledge of the target evidence type.
| Model | Encoder Family | MRR | Recall@1 | Recall@2 | Recall@10 |
|---|---|---|---|---|---|
| BM25 | Sparse Lexical | 7.62 | 3.57 | 5.65 | 14.90 |
| SPLADEv3 | Sparse Neural | 7.39 | 3.04 | 5.38 | 15.17 |
| all-mpnet | Dense Bi-Encoder | 9.08 | 4.08 | 6.78 | 19.12 |
| BGE-M3 | Dense Multi-Granularity | 9.41 | 4.45 | 7.14 | 19.05 |
| Qwen-3 Embedding 4B | Dense Bi-Encoder | 6.92 | 3.26 | 5.03 | 14.59 |
| EmbeddingGemma | Dense Bi-Encoder | 8.78 | 4.17 | 6.64 | 17.68 |
| MiniLM-L12 | Cross-Encoder | 7.92 | 3.62 | 5.82 | 15.81 |
| BGE-M3-Reranker | Cross-Encoder | 7.85 | 3.42 | 5.67 | 16.20 |
| Gemma-3 27B | LLM Pointwise Ranker | 10.12 | 4.68 | 7.74 | 20.34 |
| Qwen-3 30B Instruct | LLM Pointwise Ranker | 10.87 | 5.12 | 8.29 | 21.15 |
Ablation Study¶
To dissect the core bottlenecks, the authors isolate evidence-type inference via an oracle setting, examine multi-evidence queries, and evaluate multimodal figure and table retrieval across visual and textual modalities.
| Setup / Model | Target Evaluation | MRR | Recall@1 | Recall@2 | Note |
|---|---|---|---|---|---|
| Type-Aware Oracle (Qwen-3 30B) | Paragraph Only | 32.97 | 19.19 | 27.69 | Removing cross-type interference yields a massive gain |
| Type-Aware Oracle (GPT-5.1) | Paragraph Only | 36.26 | 21.62 | 31.34 | Top-performing frontier model (evaluated on 50% subset) |
| Type-Aware Oracle (Qwen-3 30B) | Section Only | 33.88 | 18.16 | 30.46 | Coarse document-level section retrieval |
| Type-Aware Oracle (Qwen-3 30B) | Table Caption Only | 56.17 | 33.31 | 53.29 | Smaller candidate pool yields higher early precision |
| Type-Aware Oracle (Qwen-3 30B) | Figure Caption Only | 55.40 | 34.37 | 55.16 | Clean text captions perform strongly |
| Joint Multi-Evidence (Qwen-3 30B) | Multi-Evidence (\(\ge 2\)) | 43.29 | 12.69 | 23.53 | Sharp drop on queries with multiple non-redundant targets |
| Table Visual (ColQwen2.5) | Table Image Only | 53.73 | 35.65 | 52.16 | Structural table images alone carry strong signal |
| Table Visual (ColQwen2.5) | Table Image + Caption | 57.95 | 40.48 | 58.33 | Visual layout and textual caption complement each other |
| Figure Multimodal (Qwen-3 VL 32B) | Figure Image + Caption | 56.39 | 37.66 | 58.47 | Outperforms caption text alone (55.16%) |
Key Findings¶
- Unified Retrieval Fails Across Architectures: Across all tested models, unified retrieval over an unconstrained heterogeneous pool remains under 11 MRR and peaks at 21.15% Recall@10. Cross-encoders fail to beat bi-encoders due to severe length degradation: on section-length quantile Q10, BGE-M3-Reranker MRR collapses from 44.2 (at Q5) down to 0.2, whereas LLMs sustain performance up to Q8.
- Evidence-Type Routing is the Dominant Bottleneck: Providing oracle type information boosts paragraph retrieval MRR from ~9 to ~33. However, attempting two-stage hard filtering using a predicted evidence type causes Qwen-3 30B Recall@10 to collapse from 21.15% to 6.11% because false-negative errors in figure/table classification permanently remove the gold evidence from the pool.
- Multimodal Visual and Textual Signals are Orthogonal: Augmenting figure and table images with captions consistently boosts retrieval (e.g., reaching 60.34% Recall@2 on tables with Qwen-3 VL 32B). Image-based and caption-based retrievers exhibit a 29.7% disagreement rate on tables, indicating that visual patterns and text descriptions capture distinct, complementary facets of reviewer inquiries.
Highlights & Insights¶
- Rebuttal Citations as Free Expert Alignment: Leveraging author rebuttals solves the data scarcity bottleneck in scientific document understanding, providing 16k+ natural, expert-verified evidence links without subjective manual annotation.
- Quantifying the Severe Query-Document Semantic Gap: The mean cosine similarity between reviewer comments and gold evidence is only 0.377 (compared to 0.431 for rebuttal sentences), formally proving that reviewer feedback is evaluative and underspecified rather than lexically aligned.
- Strong Case for Late Multimodal Fusion: The 24โ30% error orthogonality between image-only and caption-only retrievers provides direct empirical justification for late-fusion scoring architectures in scientific document processing systems.
Limitations & Future Work¶
- Author-Selected Evidence Bias: Ground truth relies on evidence authors selected to cite in rebuttals, which may occasionally be coarse (e.g., citing a full section rather than a paragraph in 1.58% of cases) or omit other valid evidence units.
- Disjoint Multimodal Evaluation Pools: In current benchmarks, text candidates and image candidates are retrieved from segregated pools; creating fully unified cross-modal candidate scoring remains an open challenge.
- Subfield Boundary: The dataset is derived exclusively from NLP conferences; generalizing to biomedical, chemical, and physical sciences with specialized notation remains an important future milestone.
Related Work & Insights¶
- vs. QASPER & PEERQA: While QASPER and PEERQA address explicit fact-seeking queries on camera-ready papers, ReGround tackles evaluative critique grounding on original unrevised submissions with high table/figure involvement.
- vs. ARIES & RE3: ARIES aligns reviewer suggestions with downstream revision diffs, whereas ReGround grounds reviewer comments directly in the evidence existing at the moment of review.
- vs. F1000RD & CLAIMCHECK: Prior fine-grained grounding datasets were limited to small, manually curated text-only corpora (40โ170 papers); ReGround scales to 3,656 papers and incorporates full multimodal assets.
Rating¶
- Novelty: โญโญโญโญโญ Elegant reformulation of author rebuttals into expert grounding supervision for scientific document understanding.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous benchmarking spanning sparse, dense, cross-encoder, LLM, and vision-language architectures with extensive error analysis.
- Writing Quality: โญโญโญโญโญ Clear exposition, self-consistent metrics, and thorough documentation of pipeline design.
- Value: โญโญโญโญโญ Essential benchmark for automated peer review, scientific fact-checking, and multimodal long-document retrieval.