Skip to content

Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/UKPLab/emnlp2026-reviewfeedbackagent
Area: LLM/NLP
Keywords: Peer Review, Reviewer Feedback Generation, Guideline Compliance, Cognitive Bias Detection, Genetic Algorithm

TL;DR

To combat rampant lazy thinking and vague critiques in academic peer review, this paper introduces the first guideline-aligned feedback generation framework combining neurosymbolic issue detection with an evolutionary reranking algorithm using verifiable composite rewards, reducing review guideline violations by up to 92.4% in a controlled rewriting study.

Background & Motivation

Peer review serves as the indispensable pillar of scientific quality control, yet the exponential surge in paper submissionsโ€”such as NeurIPS expanding 10.4-fold from 1,678 submissions in 2014 to 17,491 in 2024โ€”has overburdened the global academic community with over 130 million hours of reviewing labor each year. Under this acute workload, review quality has suffered noticeable degradation across two prominent failure modes flagged by conference standards like the ACL Rolling Review (ARR) guidelines: "lazy thinking", where reviewers rely on oversimplified heuristic shortcuts to dismiss manuscripts (accounting for 24.3% of author-reported grievances at ACL 2023, such as demanding non-essential extra experiments or rejecting papers solely for not surpassing the latest SOTA), and "specificity issues", where non-constructive, vague phrases like "X is unclear" replace actionable diagnostic critiques.

Prior computational peer review initiatives fail to resolve these challenges for two critical reasons. First, previous efforts such as LAZYREVIEW are confined to single-label classification within the weaknesses section and halt strictly at detection, providing no actionable verbal feedback to guide reviewer revision. Second, recent pilot applications at conferences like ICLR rely heavily on closed-source models and lack alignment with venue-specific reviewer guidelines. When open-weight LLMs are naively prompted in a zero-shot manner to generate review feedback, they encounter severe inference-time drift: 41% of generations trigger "review extension" by taking over and rewriting the entire review, while 23% suffer from "role separation" by directly addressing the paper authors rather than advising the reviewer.

Furthermore, empirical research in this domain is bottlenecked by the absence of multi-label evaluation corpora that cover both the summary of weaknesses and the comments/suggestions sections simultaneously. To transform heuristic guideline definitions into tangible quality improvements, a robust framework must accurately identify fine-grained argumentative boundaries, detect nuanced guideline violations without drowning reviewers in false alarms, and synthesize targeted, constructive feedback. Core idea: decompose peer review revision into sentence-level argumentative unit segmentation, abstract question-based neurosymbolic issue classification, and template-anchored evolutionary reranking governed by verifiable composite text rewards to generate privacy-preserving, guideline-compliant reviewer feedback.

Method

Overall Architecture

The framework operates as a three-stage inference pipeline designed to transform raw peer review text into structured, actionable feedback without model retraining: 1. Segment Identification: The review is parsed at the sentence level and mapped into argumentative segments via full-context B-I-O sequence labeling; 2. Issue Detection: Each segment is transformed into a discrete symbolic vector by answering a curated suite of abstract feature questions, followed by precision-oriented tree classification optimizing \(F_{0.5}\); 3. Targeted Feedback Generation: Detected issue labels trigger corresponding guideline templates and a context-aware planner, after which an evolutionary reranking algorithm refines candidate responses using verifiable intrinsic rewards and Boltzmann tournament selection.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Peer Review Text & Paper Context"] --> B["Argumentative Segment Identification<br/>Sentence-level B-I-O tagging under full context"]
    B --> C["Abstract Question-Driven Issue Detection<br/>LLM QA feature vector + Extra Trees F0.5 classifier"]
    C --> D["Guideline Scaffolds & Multi-Source Knowledge Planner<br/>Integrate templates with abstract, summary, & strengths"]
    D --> E["Evolutionary Reranking & Self-Verifiable Feedback Optimization<br/>Composite rewards evaluation + Boltzmann crossover"]
    E --> F["Actionable, Guideline-Compliant Reviewer Feedback"]

Key Designs

1. Argumentative Segment Identification: Resolving segment over-expansion and argumentative unit entanglement Directly prompting an LLM to output segmented review blocks often generates oversized passages that conflate multiple distinct argumentative assertions. The framework reframes review segmentation as a sentence-level B-I-O sequence labeling task over review sentence sequence \(R = (\text{sent}_1, \dots, \text{sent}_n)\), where each sentence is assigned \(y_i \in \{B, I, O\}\). Comparing sequential decoding (which conditions on prior tag predictions) against standalone contextual tagging reveals that sequential prediction suffers from severe error propagation and spurious correlation cascades. Consequently, the standalone formulationโ€”providing the full review text as context for each sentenceโ€”achieves superior boundary precision, isolating cohesive argumentative units for downstream diagnosis.

2. Abstract Question-Driven Issue Detection: Bridging the low-resource gap and curbing reviewer-alienating false alarms Direct zero-shot or fine-tuned LLM multi-class classification misses over 45% of frequent guideline issues due to the intricate cognitive reasoning required to discern necessary requirements from optional nice-to-haves. To circumvent this, the method introduces a two-stage neurosymbolic formulation: first, during abstract feature extraction, an LLM derives \(F_c\) objective Yes/No questions \(Q_\ell = \{q_1, \dots, q_{F_c}\}\) per issue class \(\ell\) from representative training clusters (e.g., "Does the reviewer suggest further tests without justifying why they are necessary?"). Answering these questions converts any segment into a structured vector \(v_i \in \{-1, 0, 1\}^{|\mathcal{L}| \times F_c}\). Second, an Extra Trees classifier is trained on these structured vectors. Because false accusations severely damage reviewer trust, the classifier threshold is explicitly tuned to optimize the precision-oriented \(F_{0.5}\) metric, outperforming both fine-tuned LLMs and dense sentence embeddings.

3. Guideline Scaffolds & Multi-Source Knowledge Planner: Mitigating role separation and review extension drift Unconstrained LLM prompting frequently degenerates into either writing a full paper review (review extension) or addressing authors directly (role separation). To enforce strict behavioral boundaries, the authors crafted 25 issue-specific templates \(\mathcal{T} = \{t_1, \dots, t_{25}\}\) aligned with the ARR guideline taxonomy. To ensure that the feedback remains contextually grounded rather than generic, an LLM-driven planner dynamically ingests three distinct sources: the paper abstract, the reviewer's summary, and the reviewer's acknowledged strengths. The planner generates an explicit knowledge-justification blueprint \(P(s_i, \ell_i) = \{(k_j, e_j)\}\), instructing the generation module how to frame the guideline violation around the paper's actual scope and the reviewer's own prior positive concessions.

4. Evolutionary Reranking & Self-Verifiable Feedback Optimization: Ensuring diverse, high-constructiveness synthesis without gold references Because reference-grounded feedback targets do not exist at inference time, standard greedy generation easily falls victim to verbosity and repetitive phrasing. The framework implements an evolutionary reranking algorithm inspired by genetic algorithms. An initial candidate pool \(F_0 = \{\text{fb}_1, \dots, \text{fb}_n\}\) is generated by prompting an open-weight LLM with the segment, template, and plan. The population is evaluated using a verifiable composite reward function derived from intrinsic text properties: $\(\text{fit}(\text{fb}) = -s_{\text{len}} + s_{\text{temp}} + s_{\text{read}} - \text{pen}_{\text{forb}}\)$ where \(s_{\text{len}} = \frac{\min(n_{\text{sent}}, n_{\text{max}})}{n_{\text{max}}}\) penalizes verbosity; \(s_{\text{temp}} = \frac{|n\text{-grams}_{\text{fb}} \cap n\text{-grams}_{\text{temp}}|}{|n\text{-grams}_{\text{temp}}|}\) enforces template compliance; \(s_{\text{read}} = \frac{\text{Flesch Score}}{100}\) optimizes readability; and \(\text{pen}_{\text{forb}} = \frac{\#\text{forbidden terms}}{n_{\text{words}}}\) penalizes unprofessional colloquialisms and greetings. Parents are selected via Boltzmann tournament selection \(P(\text{fb}_i) \propto \exp(\text{fit}(\text{fb}_i)/\tau)\) for multi-parent crossover synthesis. Crucially, semantic mutation is intentionally excluded because empirical ablation revealed that mutating prompts caused a 62% surge in review extension and a 31% surge in role separation.

Benchmark & Evaluation Protocol

The LAZYREVIEWPLUS Dataset

To evaluate multi-label guideline violations across review sections, the authors created LAZYREVIEWPLUS from NLPEER data, sampling 100 complete reviews (50 from ARR 2022 and 50 from EMNLP 2024 across 88 papers). Four senior NLP PhD students annotated 1,309 sentences (spanning 518 segments) with B-I-O spans and multi-label classifications across 18 lazy thinking issues and 7 specificity issues, covering both the summary of weaknesses and comments/suggestions sections. Krippendorff's \(\alpha\) reached 0.78 for segment identification and 0.53 for issue detection, reflecting rigorous expert agreement.

Evaluation Suite & Metrics

  • Evaluated Models: Phi-4 14B, Qwen 2.5 7B Instruct, Yi 1.5 9B Chat, DeepSeek LLM 7B Chat, and GPT-OSS 20B.
  • Compared Baselines: 1-pass (zero-shot generation), Best-of-N (BoN), Self-Refine (iterative self-correction), Temp (template prompting), Plan (planner generation), and Temp+Plan (combined scaffold).
  • Automated & Human Metrics: Graded on 1โ€“5 Likert scales across Constructiveness (Const.), Relevance (Relev.), Specificity (Spec.), and Conciseness (Conc.) using Prometheus-V2 (validated with Spearman \(\rho = 0.85\) against human ratings). Human evaluation was conducted by three experienced PhD reviewers on 100 randomly sampled cases per issue category.
  • Downstream Rewriting RCT: A blind controlled study on 50 real reviews where reviewers revised manuscripts using either "Issue Detection Only" or "Issue Detection + Generated Feedback", judged blindly on Win/Tie/Loss rates and net issue reduction (\(\Delta \text{Issue}\)).

Key Experimental Results

Main Results: Feedback Generation Quality & Controlled Human Rewriting

Table 1 presents automated Prometheus-V2 evaluations across feedback generation paradigms for open-weight models:

Model Method Constructiveness (Const.) Relevance (Relev.) Specificity (Spec.) Conciseness (Conc.)
Yi 1.5 9B 1-pass (Zero-shot) 1.9 2.0 1.8 1.9
Yi 1.5 9B Self-Refine 2.3 2.4 2.4 2.5
Yi 1.5 9B Temp + Plan 3.6 3.5 3.1 3.1
Yi 1.5 9B Ours (Full Algorithm) 3.9 3.8 3.8 3.8
Phi-4 14B 1-pass (Zero-shot) 2.1 2.2 2.0 2.1
Phi-4 14B Best-of-N (BoN) 2.2 2.3 2.1 2.2
Phi-4 14B Self-Refine 2.4 2.6 2.8 2.9
Phi-4 14B Temp 3.1 3.2 3.0 3.3
Phi-4 14B Plan 3.3 3.4 3.2 3.5
Phi-4 14B Temp + Plan 3.4 3.5 3.4 3.6
Phi-4 14B Ours (Full Algorithm) 4.3 4.3 4.2 4.3
Qwen 2.5 7B Ours (Full Algorithm) 3.8 3.8 3.7 3.7
DeepSeek 7B Ours (Full Algorithm) 3.5 3.6 3.6 3.5
GPT-OSS 20B Ours (Full Algorithm) 4.0 4.1 4.0 3.8

In blind human expert evaluation on Phi-4 (Table 3), the rankings completely mirror automated metrics: 1-pass scored 1.8 / 1.9 / 2.2 / 2.2, Temp+Plan achieved 3.4 / 3.4 / 3.6 / 3.6, while our full evolutionary approach reached 4.1 / 4.2 / 4.0 / 4.3, outperforming all baselines with inter-annotator agreement \(\alpha \in [0.65, 0.72]\).

In the controlled human rewriting study (Table 6), the real-world utility of the generated feedback was definitively validated:

Comparison (Model A vs. Model B) (Win / Tie / Loss) % Constructiveness (Win / Tie / Loss) % Justification (Win / Tie / Loss) % Adherence Net Issue Reduction \(\Delta \text{Issue} (\uparrow)\)
Issue det. only vs. Original Review 85 / 5 / 10 80 / 5 / 15 85 / 5 / 10 85.2%
Issue det. w/ Feedback vs. Original Review 95 / 5 / 0 95 / 5 / 0 90 / 5 / 5 92.4%
Issue det. w/ Feedback vs. Issue det. only 85 / 10 / 5 90 / 5 / 5 85 / 10 / 5 78.5%

Ablation Study

Table 2 highlights systematic component and reward ablations conducted on the primary Phi-4 model:

Model Variant / Reward Component Constructiveness (Const.) Relevance (Relev.) Specificity (Spec.) Conciseness (Conc.) Note / Diagnostic Impact
Full Algorithm (Ours) 4.3 4.3 4.2 4.3 Complete 6-stage evolutionary optimization
w/o Template Construction 3.6 3.5 3.4 3.5 Severe drift into review extension & loss of structure (-0.7 Const.)
w/o Plan Generation 3.8 3.8 3.7 3.8 Lacks paper-grounded context and scope references
w/o Population Initialization 3.9 3.9 3.7 3.8 Single-candidate bottleneck hurts generation breadth
w/o Fitness Evaluation 3.7 3.7 3.7 3.7 Vulnerable to length bias and ungrounded verbosity
w/o Parent Selection & Crossover 3.9 3.8 3.7 3.7 Disables synthesis of complementary parent strengths
w/o Final Candidate Selection 3.8 3.7 3.6 3.7 Arbitrary final pick exhibits score variance
Reward Function Decomposition:
- w/o Length Score (\(s_{\text{len}}\)) 3.8 3.7 3.8 3.6 Unchecked length inflation lowers conciseness by 0.7
- w/o Template Adherence (\(s_{\text{temp}}\)) 3.7 3.6 3.5 3.7 Deviates from ARR rhetorical framing (-0.6 Const.)
- w/o Readability Score (\(s_{\text{read}}\)) 4.1 4.0 4.0 4.1 Outputs contain convoluted syntactic structures
- w/o Forbidden Term Penalty (\(\text{pen}_{\text{forb}}\)) 4.2 4.2 4.1 4.3 Increased leakage of conversational pleasantries

In the planner input ablation (Table 27), removing both the abstract and reviewer-written strengths reduced Phi-4's constructiveness from 4.1 down to 3.7, confirming that grounding in both paper scope and reviewer concessions is critical for context-sensitive issues like "extra experiments".

Key Findings

  1. Templates serve as the primary safeguard against role collapse: Ablation proves that template scaffolding drives the largest single performance leap (+1.0 in Const.), eliminating both review extension (generating full reviews) and role separation (addressing authors).
  2. Neurosymbolic QA feature representations conquer data scarcity: In issue detection, pairing LLM-extracted QA feature vectors with an Extra Trees classifier achieved a macro \(F_{0.5}\) of 0.68 and precision of 0.65 across 5-fold CV, exceeding task-specific fine-tuned RoBERTa by 0.10 \(F_{0.5}\) and surpassing all end-to-end zero-shot LLMs (which regularly failed on over 44% of classes).
  3. Targeted feedback demonstrates overwhelming actionable adoption: Reviewers provided with targeted feedback incorporated 96% of the suggestions during blind rewriting, slashing verified guideline violations by 92.4% and achieving a 95% win rate over original reviews in pairwise constructive evaluation.

Highlights & Insights

  • Neurosymbolic decomposition for subjective meta-reasoning: Converting complex, ambiguous guideline criteria into objective Yes/No abstract questions decouples linguistic reasoning from classification calibration, enabling sample-efficient, high-precision detection with tree models under severe class imbalance.
  • Crossover-only evolutionary search for LLM text refinement: While stochastic mutation in prompt spaces triggers catastrophic role drift, restricting search to fitness-guided multi-parent crossover allows models to merge concise phrasing with specific evidence cleanly without hallucinating full reviews.
  • Production-ready privacy and latency profile: By utilizing open-weight models (Phi-4, Yi, Qwen) with vLLM batching, single-review processing requires only 2.15โ€“4.15 seconds without leaking unpublished manuscripts to proprietary third-party APIs. A standard conference workload of 5,000 reviews can be processed in under 1.5 hours across 4 GPUs.

Limitations & Future Work

  • Upstream recall ceiling: The downstream feedback generation is inherently bounded by upstream issue detection recall (which captured 60.0% of true human-annotated issues in the controlled study); optimizing operational thresholds for higher recall remains an open objective.
  • Taxonomy curation overhead across new venues: While curating the 25 templates for ARR took only 250 minutes, porting the system to non-ACL venues (e.g., CVPR, NeurIPS) or other disciplines requires formalizing conference-specific reviewing guidelines into comparable question and template taxonomies.
  • Long-term reviewer habituation and synthetic fatigue: Repeated exposure to template-guided feedback could lead to stylistic fatigue or strategic gaming by reviewers over multiple conference cycles, necessitating future research into dynamic prompt variations and interactive delivery mechanisms.
  • vs. Purkayastha et al. (2025) [LAZYREVIEW]: Pioneered single-label lazy thinking detection on 500 weakness spans but stopped entirely at label classification; this work expands scope to multi-label diagnosis across weaknesses and suggestions via LAZYREVIEWPLUS (1,309 sentences) and provides the first end-to-end verbal feedback generation mechanism.
  • vs. Thakkar et al. (2026) [ICLR Feedback Trial]: Conducted large-scale feedback trials using proprietary Claude models without grounding in conference reviewer guidelines; this paper enforces strict ARR guideline compliance using open-weight models, eliminating privacy concerns and offering transparent, controllable generation.

Rating

  • Novelty: โญโญโญโญโ˜† [Novel combination of neurosymbolic QA feature extraction and crossover-driven genetic reranking for peer review assistance]
  • Experimental Thoroughness: โญโญโญโญโญ [Exemplary evaluation combining cross-validation, Prometheus alignment, human blind scoring, multi-GPU latency profiling, and a full controlled rewriting study]
  • Writing Quality: โญโญโญโญโญ [Extremely thorough, transparently documented failure modes, complete prompts, and rigorous mathematical framing]
  • Value: โญโญโญโญโญ [Delivers an open-source, privacy-compliant, actionable tool directly addressing the escalating crisis in scientific peer review]