Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/UKPLab/emnlp2026-reviewfeedbackagent
Area: LLM Reasoning
Keywords: Peer Review, Lazy Thinking, Specificity, Evolutionary Algorithm, Inference-Time Generation
TL;DR¶
Addressing pervasive lazy thinking and non-specific critiques in scientific peer review, this paper introduces an end-to-end framework integrating sentence-level BIO segmentation, neuro-symbolic multi-label violation detection, and template-grounded evolutionary feedback generation, reducing guideline violations in human rewrites by up to 92.4%.
Background & Motivation¶
Peer review serves as the cornerstone of quality control in scientific publishing, but the explosive growth in submissions—particularly in machine learning and NLP, where submissions increased more than tenfold over the past decade—has placed an unsustainable global burden on the scientific community exceeding 130 million hours annually. Under extreme reviewing fatigue, review quality has noticeably deteriorated into two prevalent failure modes explicitly cataloged by the ACL Rolling Review (ARR) guidelines: lazy thinking (heuristic shortcuts such as demanding arbitrary "nice-to-have" experiments or requiring closed-source model comparisons, which alone accounted for 24.3% of author disputes at ACL 2023) and lack of specificity (vague, unsubstantiated assertions like "method X is unclear" lacking technical justification).
Prior attempts to remedy these peer-review issues suffer from significant technical limitations and practical misalignment. On one hand, diagnostic studies have framed lazy thinking detection strictly as a single-label sentence classification problem restricted solely to the "weaknesses" section, stopping short of providing verbal, actionable feedback to guide reviewer revision; psychological and pedagogical evidence demonstrates that categorical labels alone fail to motivate substantive revisions. On the other hand, naive zero-shot generation using off-the-shelf LLMs fails dramatically due to severe distribution shifts: pilot experiments reveal that over 41% of zero-shot generations exhibit "review extension" (rewriting or fabricating additional reviewer critiques) and 23% exhibit "role separation" failures (erroneously lecturing the paper authors instead of advising the reviewer). Furthermore, open datasets covering co-occurring, multi-label guideline violations across both weaknesses and comments/suggestions sections have been entirely missing.
To resolve these challenges, the authors propose that assisting peer reviewers requires an inference-time search strategy combining structural constraints, external factual grounding, and explicit optimization objectives. Rather than unconstrained text generation, the system grounds feedback in policy-specific structural templates and uses an evolutionary search algorithm driven by intrinsic, verifiable text rewards. This approach guarantees that open-weight LLMs can reliably generate targeted, constructive, and guideline-compliant feedback without drifting off-task or confusing conversational roles. Core idea: decompose the peer review feedback pipeline into sentence-level argumentative unit segmentation, abstract feature question-based multi-label violation detection, and an evolutionary reranking feedback generator governed by composite verifiable rewards, eliminating role confusion and unconstrained extension at inference time while strictly enforcing conference policy compliance.
Method¶
Overall Architecture¶
The proposed system operates through three sequential pipeline stages: (1) Argumentative Segment Identification, where unformatted review text is split into sentences and tagged with B-I-O labels to delineate distinct argumentative segments; (2) Precision-Oriented Issue Detection, where segments are projected into discrete semantic vectors via LLM-based abstract feature questions and classified by a multi-label tree ensemble to detect co-occurring ARR guideline violations; and (3) Evolutionary Feedback Generation, where detected labels trigger issue-specific templates enriched by a three-source knowledge planner and refined using a customized genetic algorithm with verifiable composite rewards to produce final feedback for the reviewer.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Raw Review Text R<br/>(Weaknesses & Suggestions Sections)"] --> B["Stage 1: Standalone Contextual B-I-O Segmentation<br/>Full-context sentence tagging into segments s_i"]
B --> C["Stage 2: Abstract Feature Q&A and Precision Classification<br/>LLM feature vectorization + Extra Trees issue labeling L_i"]
C --> D["Stage 3: Template-Guided Knowledge Planning & Population Init<br/>25 ARR templates + paper abstract/strengths planner"]
D --> E["Stage 4: Composite Verifiable Rewards & Boltzmann Crossover<br/>Multi-objective fitness evaluation + evolutionary crossover"]
E --> F["Output: Targeted Reviewer Feedback F_i<br/>Constructive, role-aligned, guideline-compliant advice"]
Key Designs¶
1. Standalone Contextual B-I-O Segmentation: preventing cascading errors and boundary over-expansion in argumentative parsing
Prompting LLMs to directly output chunked text spans frequently produces oversized segments spanning multiple conflicting argumentative claims. Conversely, autoregressive sequential tagging that conditions on preceding label predictions suffers from compounding error propagation. To overcome these dual failure modes, this design treats segmentation as sentence-level sequence labeling where each sentence is tagged as Beginning (B), Inside (I), or Outside (O) of an argumentative unit while feeding the entire review as context. Evaluating each sentence in full global context without conditioning on prior predicted labels eliminates cascading error propagation and spurious sequential correlations, allowing open-weight models like Phi-4 to achieve an F1 score of 0.79 across variable segment lengths ranging from 1 to 20 sentences.
2. Abstract Feature Q&A and Precision Classification: converting low-resource implicit reasoning into structured pattern matching
Determining whether a review critique represents a legitimate flaw or an unjustified heuristic (e.g., whether a requested baseline is truly essential or merely an unreasonable hurdle) requires deep pragmatic nuance, causing zero-shot and directly fine-tuned LLMs to miss over 45% of frequent issues. This design decouples reasoning from classification via a two-stage neuro-symbolic mechanism: first, for each issue category \(\ell \in \mathcal{L}\), an LLM formulates a fixed battery of abstract Yes/No feature questions \(Q_\ell = \{q_1, \dots, q_{F_c}\}\) derived from contrasting canonical exemplars. The LLM then answers these questions for each target segment \(s_i\) (mapped to \(+1\) for Yes, \(-1\) for No, and \(0\) for Not Relevant), producing a compact discrete feature representation \(\mathbf{v}_i \in \{-1, 0, 1\}^{|\mathcal{L}| \times F_c}\). Finally, an Extra Trees classifier is trained on these structured feature vectors with hyperparameters optimized specifically for the precision-weighted metric \(F_{0.5}\). This approach curbs false alarms, enhances reviewer trust, and improves multi-label classification by 0.10 \(F_{0.5}\) over RoBERTa.
3. Template-Guided Knowledge Planning & Population Initialization: eliminating review extension and role separation via structural grounding
Unconstrained generation prompts often lead models to usurp the reviewer's role—writing new paper criticisms rather than offering meta-review advice—or misaddress the authors directly. To enforce rigorous boundaries, 25 specialized feedback templates \(T = \{t_1, \dots, t_{25}\}\) were hand-crafted to align with individual ARR guidelines. In parallel, a knowledge planner ingests the review segment, the detected issue, the paper abstract, and the reviewer's own summary and documented strengths to produce an explicit grounding plan \(\mathcal{P}(s_i, \ell_i) = \{(k_i, e_i)\}\). When generating the initial candidate population \(\mathcal{M}_0\) of \(n=10\) candidate responses, the model is bound to the template skeleton and planner directives, ensuring diversity while preventing off-task conversational drift.
4. Composite Verifiable Rewards & Boltzmann Crossover: reference-free evolutionary search for inference-time feedback optimization
Because no gold-standard reference feedback exists for peer reviews, the framework introduces an intrinsic, multi-objective fitness evaluation function: $\(\mathrm{fit}(fb) = - s_{\text{len}} + s_{\text{temp}} + s_{\text{read}} - \mathrm{pen}_{\text{forb}}\)$ where \(s_{\text{len}} = \min(n_{\text{sent}}, n_{\text{max}})/n_{\text{max}}\) rewards conciseness; \(s_{\text{temp}} = |n\text{-grams}_{fb} \cap n\text{-grams}_{\text{temp}}| / |n\text{-grams}_{\text{temp}}|\) penalizes deviation from guideline policy; \(s_{\text{read}} = \text{Flesch Score}/100\) promotes readability; and \(\mathrm{pen}_{\text{forb}} = n_{\text{forbidden}} / n_{\text{words}}\) penalizes unprofessional greetings or casualisms (e.g., "Dear Author", "Haha", "Cheers"). During iterative search, random mutations are explicitly omitted because semantic mutation prompts were found to trigger a 31% resurgence in role separation. Instead, parent candidates are sampled via Boltzmann tournament selection probabilities: $\(P(fb_i) = \frac{\exp(\mathrm{fit}(fb_i)/\tau)}{\sum_{j} \exp(\mathrm{fit}(fb_j)/\tau)}\)$ with temperature \(\tau=0.1\). The LLM executes guided crossover by fusing complementary structural and factual strengths from top-ranking parents across \(n_{\text{gen}}=3\) generations, isolating the globally optimal candidate \(fb^*\).
Loss & Training¶
The segmentation and feedback generation components operate purely at inference time without gradient updates. For the issue detection stage, classifier training optimizes the macro-averaged \(F_{0.5}\) score to emphasize precision over recall, ensuring reviewers are not alienated by inaccurate defect flags: $\(F_{\beta} = (1 + \beta^2) \frac{\text{Precision} \cdot \text{Recall}}{(\beta^2 \cdot \text{Precision}) + \text{Recall}}, \quad \text{where } \beta = 0.5\)$ To prevent cross-segment data leakage, the 100 expert-annotated reviews of LAZYREVIEWPLUS (1,309 sentences) are split via a distribution-aware review-level 5-fold cross-validation scheme (90% training, 10% validation). This ensures that sentences from the same review remain strictly partitioned within a single fold while maintaining a minimal label distribution distance of 0.014 across splits.
Key Experimental Results¶
Main Results¶
Feedback quality is evaluated across four dimensions using Prometheus-V2 (validated against expert human judgments): Constructiveness (Const.), Relevance (Relev.), Specificity (Spec.), and Conciseness (Conc.) on a 1–5 scale. The table below presents automated evaluation metrics across diverse model architectures and baselines (extracted from Table 2 and Table 15):
| Model | Method | Const. | Relev. | Spec. | Conc. |
|---|---|---|---|---|---|
| Phi-4 (14B) | 1-pass (Zero-shot) | 2.1 | 2.2 | 2.0 | 2.1 |
| Phi-4 (14B) | Best-of-N (BoN) | 2.2 | 2.3 | 2.1 | 2.2 |
| Phi-4 (14B) | Self-Refinement | 2.4 | 2.6 | 2.8 | 2.9 |
| Phi-4 (14B) | Template-guided (Temp) | 3.1 | 3.2 | 3.0 | 3.3 |
| Phi-4 (14B) | Plan-then-generate (Plan) | 3.3 | 3.4 | 3.2 | 3.5 |
| Phi-4 (14B) | Temp + Plan (BoN) | 3.4 | 3.5 | 3.4 | 3.6 |
| Phi-4 (14B) | Ours (Full Evolutionary) | 4.3 | 4.3 | 4.2 | 4.3 |
| GPT-OSS (20B) | 1-pass (Zero-shot) | 1.9 | 2.0 | 1.8 | 1.9 |
| GPT-OSS (20B) | Temp + Plan (BoN) | 3.2 | 3.4 | 3.2 | 3.2 |
| GPT-OSS (20B) | Ours (Full Evolutionary) | 4.0 | 4.1 | 4.0 | 3.8 |
| Yi-1.5 (9B) | Ours (Full Evolutionary) | 3.9 | 3.8 | 3.8 | 3.8 |
| Qwen-2.5 (7B) | Ours (Full Evolutionary) | 3.8 | 3.8 | 3.7 | 3.7 |
| DeepSeek (7B) | Ours (Full Evolutionary) | 3.5 | 3.6 | 3.6 | 3.5 |
Human evaluation conducted by experienced NLP PhD reviewers on Phi-4 outputs confirms these gains (Table 3), with our approach achieving 4.1 in Constructiveness, 4.2 in Relevance, 4.0 in Specificity, and 4.3 in Conciseness, displaying strong agreement with Prometheus-V2 ratings (Spearman \(\rho = 0.85\)).
In the downstream controlled rewriting study involving 50 real peer reviews (Table 6), blinded senior evaluators compared original reviews against rewrites guided by issue labels alone versus rewrites assisted by our full feedback system:
| Comparison Setting | Constructiveness (W/T/L %) | Justification (W/T/L %) | Adherence (W/T/L %) | Issue Reduction \(\Delta\)Issue (↑) |
|---|---|---|---|---|
| Issue Detection vs. Original | 85 / 5 / 10 | 80 / 5 / 15 | 85 / 5 / 10 | 85.2% |
| Issue + Feedback vs. Original | 95 / 5 / 0 | 95 / 5 / 0 | 90 / 5 / 5 | 92.4% |
| Issue + Feedback vs. Issue Detection | 85 / 10 / 5 | 90 / 5 / 5 | 85 / 10 / 5 | 78.5% |
Ablation Study¶
Ablations on system components and individual fitness rewards using Phi-4 highlight the indispensable nature of each module (compiled from Table 4 and Table 5):
| Variant / Configuration | Constructiveness | Relevance | Description / Effect |
|---|---|---|---|
| Full Algorithm (Ours) | 4.3 | 4.3 | Highest overall performance |
| w/o Template Construction | 3.6 | 3.5 | Severe off-task drift and unconstrained extension (-0.7) |
| w/o Fitness Evaluation | 3.7 | 3.7 | Loss of reward-guided candidate filtering (-0.6) |
| w/o Parent Selection & Crossover | 3.9 | 3.8 | Search space restricted to internal baseline distribution (-0.4) |
| w/o Plan Generation | 3.8 | 3.8 | Feedback lacks context from paper abstract and strengths (-0.5) |
| w/o Population Initialization | 3.9 | 3.9 | Degraded diversity in candidate exploration (-0.4) |
| w/o Final Candidate Selection | 3.8 | 3.7 | Arbitrary selection exposes suboptimal candidates (-0.5) |
| Fitness Reward Components | |||
| w/o Template Adherence (\(s_{\text{temp}}\)) | 3.7 | 4.0 | Drift from ARR guideline phrasing; constructiveness drops by 0.6 |
| w/o Length Score (\(s_{\text{len}}\)) | 3.8 | 3.7 | Unchecked length inflation; conciseness drops by 0.6 |
| w/o Readability (\(s_{\text{read}}\)) | 4.1 | 3.6 | Decreased sentence fluency and syntactic clarity |
| w/o Forbidden Term Penalty (\(\mathrm{pen}_{\text{forb}}\)) | 4.2 | 4.2 | Infiltration of informal greetings and chat colloquialisms |
Key Findings¶
- Templates form the primary safeguard against hallucinated role shifts: Removing templates causes the largest drop in constructiveness (-0.7) and triggers unconstrained review extension in 41% of cases and role confusion in 23% of zero-shot outputs, demonstrating that explicit policy templates are essential.
- Crossover outperforms mutation in constrained generation tasks: Adding semantic mutation prompts caused 31% of outputs to confuse the reviewer with the paper author. Eliminating mutation while retaining Boltzmann-weighted parent crossover reliably recombined candidate virtues without triggering semantic drift.
- Reviewers actively adopt generated feedback without losing original critiques: In the controlled human study, reviewers incorporated 96% of the generated feedback items, leading to a 92.4% reduction in guideline violations while retaining 96% of their original substantive concerns (with 18% properly reorganized into suggestions).
Highlights & Insights¶
- Neuro-symbolic question-answering abstraction: Bypassing low-resource end-to-end classification by having the LLM answer structured Yes/No feature questions and letting tree ensembles classify the resulting vectors is a powerful design pattern for delicate policy-compliance tasks.
- Mutation-free evolutionary search under composite verifiable rewards: Combining length, template overlap, readability, and forbidden-lexicon penalties with tournament crossover demonstrates how inference-time optimization can replace expensive alignment fine-tuning when gold training targets are absent.
- The LAZYREVIEWPLUS benchmark: By offering 1,309 expert-annotated sentences covering 18 lazy thinking categories and 7 specificity issue types across both weaknesses and suggestion sections, the dataset fills a major empirical void in computational peer review research.
Limitations & Future Work¶
- Upstream issue detection recall ceiling: While the Extra Trees detector achieves 0.65 precision, its recall in real-world human trials is approximately 60.0%, meaning that 40% of genuine review flaws remain unflagged for feedback generation.
- Stylistic standardization and fatigue: Long-term deployment of template-grounded feedback risks habituating reviewers to standardized rhetorical patterns, warranting future research into stylistic diversification.
- Generalization beyond ARR policies: The current dataset and templates reflect ACL Rolling Review guidelines. Adapting the system to computer vision (e.g., CVPR) or broader scientific venues will require authoring venue-specific issue taxonomies and templates.
Related Work & Insights¶
- vs. LazyReview (Purkayastha et al., 2025): LazyReview first formulated lazy thinking as single-label classification on review weaknesses. The present work expands the taxonomy to multi-label spans across both weaknesses and suggestions, and critically advances from passive diagnostic classification to active, generative reviewer feedback.
- vs. ICLR Review Assistance (Thakkar et al., 2026): Thakkar et al. utilized closed-source models (Claude) to improve review politeness and specificity without aligning with formal conference policies. In contrast, this work enforces formal ARR guideline compliance using open-weight models, preserving reviewer confidentiality and academic governance.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Pioneering integration of multi-label review segmentation with evolutionary inference-time feedback generation]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-model evaluation, thorough ablations, judge-alignment analyses, and a human-in-the-loop double-blind rewriting study]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity, mathematically disciplined reward definitions, and transparent empirical disclosure]
- Value: ⭐⭐⭐⭐⭐ [Provides a practical, privacy-preserving technical solution to mitigate peer review degradation and reviewer burnout]