Skip to content

What Drives Paper Acceptance? A Process-Centric Analysis of Modern Peer Review

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Peer Review Analysis, Rebuttal Dynamics, Interaction Logs, Reviewer Disagreement, Acceptance Determinants

TL;DR

Drawing upon 28,358 submissions from ICLR 2017–2025 on OpenReview, this paper establishes a four-dimensional, process-centric framework spanning reviewer disagreement, interaction logs, external metadata, and linguistic patterns, revealing how non-content dynamics and optimal interaction windows shape final acceptance decisions.

Background & Motivation

Peer review serves as the cornerstone of scientific evaluation, yet historically it operated within an opaque, one-way decision chamber where only final verdicts were revealed. Prior empirical literature investigating academic acceptance and citation forecasting has largely conformed to an "outcome-centric" paradigm, relying on static post-hoc indicators such as acceptance decisions, future citation counts, or isolated surface-level keyword statistics. These approaches inherently treat the review process as an unobservable black box, unable to capture the nuanced multi-agent dynamics, information exchanges, and subjective negotiations that occur between authors, reviewers, and area chairs prior to the final verdict.

The advent and widespread adoption of open peer review platforms such as OpenReview have fundamentally transformed conference evaluation in machine learning into a transparent, interactive ecosystem. Detailed logs containing reviewer scores, explicit confidence ratings, author rebuttals, threaded discussions, and meta-review rationales are now systematically recorded. Nevertheless, existing studies on modern peer review remain fragmented: they frequently restrict their scope to a single conference edition, examine limited sub-samples, or analyze isolated dimensions—such as evaluating readability indices without accounting for author rebuttal activity, or inspecting score changes while ignoring external disclosures like preprints and code releases. Crucially, the field lacks a unified framework that simultaneously models the interaction between reviewer disagreement, author responsiveness, strategic submission practices, and evolving linguistic styles across heterogeneous conference scales.

To bridge this fundamental gap, this work conceptualizes modern peer review as a dynamic signaling game and an interactive belief-updating process. Utilizing the complete longitudinal corpus of ICLR from 2017 to 2025—comprising over 28,000 manuscripts and spanning an approximately 17-fold growth in submission volume—the authors construct an integrated analytical framework. Core idea: reframe peer review as an interactive signaling and belief-updating game, establishing a four-dimensional process-centric framework spanning disagreement mediation, rebuttal interaction logs, external submission signals, and paper structural/linguistic traits to systematically uncover the non-content determinants and optimal interaction windows of paper acceptance.

Method

The study develops an end-to-end, multi-source empirical pipeline that synthesizes heterogeneous conference data into a unified, process-centric schema. Starting from raw OpenReview API responses across nine consecutive years (2017–2025), the authors extract review ratings, confidence scores, multi-turn dialogue trees, and decision statements. They augment these records with synchronized arXiv metadata (timestamp differentials and version histories) and PDF-derived structural features (tables, figures, equations, readability metrics, and citation distributions). Using these rich features, the authors conduct non-parametric correlation tests, robust Welch's two-sample \(t\)-tests, and multivariate logistic regressions with extensive observable controls across four core dimensions: Reviewer Disagreement & Mediation (D), Review Interaction Logs (I), Submission Strategies & External Metadata (S), and Paper Structure & Language Quality (P).

Overall Architecture

The analytical pipeline ingests the complete corpus of 28,358 ICLR submissions, executing a multi-stage feature extraction and statistical modeling workflow. At the foundation, raw OpenReview logs and parsed manuscript PDFs are processed into four distinct feature spaces: disagreement and sentiment representations, author-reviewer interaction dynamics, external publication and submission timing metadata, and structural/linguistic metrics. These features are subsequently evaluated through group-stratified statistical tests and controlled multivariate logistic regression models to yield concrete guidelines for authors and program chairs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["ICLR 2017–2025 OpenReview Corpus<br/>complete logs of 28,358 submissions"] --> B["Disagreement & Sentiment Modeling<br/>score variance / confidence / mBERT sentiment"]
    B --> C["Interaction Log Dynamics<br/>rebuttal latency / length / depth / response count"]
    C --> D["Submission Strategy & External Metadata<br/>lead time / synchronized arXiv / reproducibility"]
    D --> E["Paper Structure & Linguistic Profiling<br/>visual density / pre-vs-post LLM style / citation recency"]
    E --> F["Multivariate Regression & Causal Decision Inference<br/>observable controls / stratified hypothesis tests / guidelines"]

Key Designs

1. Disagreement & Sentiment Modeling: score variance divergence and review sentiment gating Conventional intuition assumes that high reviewer disagreement is universally detrimental to paper acceptance. By formulating score variance \(\sigma_i^2 = \frac{1}{K_i}\sum_{k=1}^{K_i}(r_i^k - \bar{r}_i)^2\) across stratified score cohorts (Low-Score \(\bar{r}_i < 5\), Borderline \(5 \le \bar{r}_i \le 6\), High-Score \(\bar{r}_i > 6\)), the authors reveal a critical asymmetry. In the Low-Score cohort, reviewer variance exhibits a statistically significant positive correlation with acceptance (Spearman \(\rho = 0.086\), \(p = 2.52 \times 10^{-13}\), Logit OR = 1.228), demonstrating that for struggling submissions, a split decision with a passionate advocate offers a vital path to rescue. Conversely, in the Borderline and High-Score cohorts, variance acts as a strong negative predictor (High-Score OR = 0.789), where a single dissenting reviewer frequently triggers rejection. Crucially, in borderline deliberations, self-reported reviewer confidence (\(\bar{c}_i \approx 3.60\)) exhibits negligible predictive association with acceptance, whereas reviewer comment sentiment—extracted via a multilingual BERT classifier—functions as a powerful gatekeeper: borderline papers dominated by negative sentiment suffer a dismal 18.8% acceptance rate, compared to 35.4% for positive sentiment and an overall baseline of 32.7%.

2. Interaction Log Dynamics: responsiveness rescue effects and optimal dialogue depth Analyzing 20,666 submissions with complete rebuttal records within the standard 7–14 day window (\(\le 336\) hours), the authors quantify mean response latency \(\bar{T}_i\), mean response length \(\bar{L}_i\), total author response count \(N_i\), and maximum discussion tree depth \(\text{Depth}_i\). In Low-Score and Borderline groups, faster response speed (\(\bar{T}_i \downarrow\)) and longer, substantive explanations (\(\bar{L}_i \uparrow\)) significantly improve acceptance odds (Low-Score length Logit OR = 2.686, \(p = 1.1 \times 10^{-54}\); Borderline OR = 1.449, \(p = 5.6 \times 10^{-52}\)), whereas High-Score papers derive no measurable benefit from these dimensions. Furthermore, the analysis uncovers an "optimal interaction window": sustaining a dialogue depth of 2–3 turns yields the highest marginal acceptance gains. Prolonged or overly argumentative discussions beyond this threshold cause acceptance rates to plateau or sharply decline, particularly for high-scoring papers where excessive debate introduces unnecessary scrutiny or irritates reviewers.

3. Submission Strategy & External Metadata: early submission dividends and synchronized preprint readiness Although double-blind review aims to mask author identity, external submission behaviors broadcast strong operational signals of manuscript maturity. Tracking submission lead time relative to the deadline reveals that early submissions (\(\ge 7\) days before the deadline) achieve an acceptance rate of 44.0%, notably surpassing both the overall conference average of 38.9% and last-minute submissions (~43%), while eliciting higher reviewer ratings and lower reviewer disagreement (Gini index). Evaluating 14,012 submissions matched to arXiv preprints (BLEU \(\ge 0.8\) and author overlap \(> 50\%\)), the authors find that papers released on arXiv substantially outperform non-posted papers. Crucially, preprints coordinated to appear simultaneously with or immediately after conference submission achieve approximately 6.4 percentage points higher acceptance than premature preprints, signaling completed submission readiness. In parallel, disclosing Reproducibility-Supporting Resources (RS, including code and data) provides a massive boost, driving acceptance rates from 32.54% to 43.96% (Welch's \(t = 19.762\), Logit OR = 1.626).

4. Paper Structure & Linguistic Profiling: empirical validation via visual elements and post-LLM stylistic reversal By parsing manuscript PDFs, the study extracts counts of tables, figures, and equations, standard readability indices (Flesch Reading Ease, Flesch-Kincaid Grade, Gunning Fog Index), and temporal citation statistics. Visual density exhibits a functional dichotomy: while tables and equations show slight negative or neutral correlations with technical novelty, they strongly boost empirical novelty and contribution scores (table contribution OR = 1.052, equation contribution OR = 1.092). The most striking empirical discovery is the fundamental stylistic reversal triggered by the emergence of LLMs (partitioning data into Pre-LLM 2017–2023 and Post-LLM 2024–2025 around ChatGPT's launch). In the Pre-LLM era, fluent, easily readable prose with longer sentence structures enjoyed a modest acceptance advantage. In the Post-LLM era, this dynamic completely inverted: as LLMs homogenized baseline fluency, reviewers penalized verbose, rambling sentences (average words per sentence correlation with acceptance plunged to \(r = -0.156\), Welch's \(t = -21.491\)), while heavily favoring dense, professional, and concise prose with shorter sentences and elevated technical complexity (Gunning Fog index correlation shifted from negative to strongly positive, \(t = +17.415\)). Finally, citation recency analysis demonstrates that a higher ratio of recent references (published within 2 years) dramatically elevates empirical novelty ratings (OR = 2.590), confirming that active engagement with the immediate frontier is essential for perceived contribution.

Benchmark & Evaluation Protocol

The empirical evaluation encompasses 28,358 ICLR submissions across 2017–2025. Subsets with verified data integrity are allocated to specific research questions: 23,444 papers with complete ratings and confidence scores for disagreement analysis, 20,666 submissions with verified rebuttal threads for interaction dynamics, and 14,012 matched entries for arXiv analysis. Statistical validation incorporates Spearman's rank correlation (\(\rho\)), Pearson's linear correlation (\(r\)), Mann–Whitney U tests, Welch's unequal variances \(t\)-tests, and multivariate logistic regressions incorporating year, baseline ratings, submission timing, and rebuttal controls.

Key Experimental Results

Main Results: Statistical Determinants of Acceptance Across Disagreement and External Metadata

The table below details the statistical significance and effect sizes across reviewer disagreement cohorts, submission timing, and reproducibility disclosures:

Variable & Evaluation Cohort Sample Size \(N\) Spearman \(\rho\) Pearson \(r\) Welch's \(t\) Logit \(\beta\) / OR Key Empirical Implication
Overall Score Variance (RQ-D1 Overall) 23,444 -0.074 (\(p=7.1\text{e-}30\)) -0.067 (\(p=1.8\text{e-}24\)) -10.417 -0.080 / 0.923 Globally, reviewer disagreement dampens acceptance prospects
Low-Score Variance (Low-Score \(\bar{r}_i < 5\)) 7,255 +0.086 (\(p=2.5\text{e-}13\)) +0.070 (\(p=1.9\text{e-}09\)) +5.557 +0.206 / 1.228 Score divergence enables borderline rescue via champion reviewers
Borderline Variance (Borderline \(5 \le \bar{r}_i \le 6\)) 9,549 -0.105 (\(p=1.3\text{e-}24\)) -0.033 (\(p=1.1\text{e-}03\)) -3.217 -0.036 / 0.965 Consensus agreement is critical for tipping borderline decisions
High-Score Variance (High-Score \(\bar{r}_i > 6\)) 6,640 -0.141 (\(p=5.2\text{e-}31\)) -0.131 (\(p=6.3\text{e-}27\)) -8.301 -0.237 / 0.789 A single dissenting review introduces severe veto risk
Borderline Reviewer Confidence (RQ-D2) 9,549 -0.022 (\(p=0.031\)) -0.025 (\(p=0.016\)) -2.406 -0.111 / 0.895 Self-reported confidence has negligible predictive utility
Submission Timing Lead (RQ-S1 0–1d vs 1–14d) 28,086 +0.050 (\(p=1.2\text{e-}15\)) +0.058 (\(p=4.3\text{e-}20\)) -8.580 - Early submission yields higher scores and lower disagreement
Reproducibility Resources (RQ-S3 With RS) 28,086 +0.116 (\(p=2.8\text{e-}85\)) - +19.762 +0.486 / 1.626 Releasing code/data drives acceptance from 32.54% to 43.96%

Stylistic Reversal: Pre-LLM vs. Post-LLM Era Linguistic Ablation

The table below contrasts the impact of linguistic and readability metrics on acceptance rates between the Pre-LLM era (ICLR 2017–2023, 14,008 papers) and Post-LLM era (ICLR 2024–2025, 14,319 papers):

Linguistic Metric Pre-LLM (\(N \approx 14,008\)) Spearman \(\rho\) Pre-LLM (\(N\)) Welch's \(t\) Post-LLM (\(N \approx 14,319\)) Spearman \(\rho\) Post-LLM (\(N\)) Welch's \(t\) Paradigm Shift & Stylistic Insight
Average Sentence Length +0.046 (\(p=3.6\text{e-}08\)) +2.524 (\(p=0.012\)) -0.130 (\(p=1.2\text{e-}54\)) -14.400 (\(p=9.9\text{e-}46\)) Reversal: long sentences shift from mild asset to major penalty
Average Words per Sentence -0.005 (\(p=0.533\), n.s.) -1.465 (\(p=0.143\)) -0.166 (\(p=1.2\text{e-}88\)) -21.491 (\(p=1.7\text{e-}96\)) Verbosity strongly penalizes acceptance in the post-LLM era
Flesch Reading Ease +0.020 (\(p=0.020\)) +0.547 (\(p=0.584\)) -0.092 (\(p=1.5\text{e-}28\)) -10.750 (\(p=1.8\text{e-}26\)) Superficial readability no longer provides a competitive edge
Gunning Fog Index -0.034 (\(p=5.6\text{e-}05\)) -3.573 (\(p=3.5\text{e-}04\)) +0.158 (\(p=1.4\text{e-}80\)) +17.415 (\(p=3.4\text{e-}65\)) Reversal: professional, technically demanding syntax is favored
Flesch-Kincaid Grade -0.021 (\(p=0.015\)) -0.555 (\(p=0.579\)) +0.102 (\(p=1.4\text{e-}34\)) +11.617 (\(p=1.5\text{e-}30\)) Academic rigor and lexical density outweigh simplified phrasing

Key Findings

  • Reviewer disagreement functions as an asymmetric double-edged sword: it provides a rescue mechanism for low-scoring manuscripts (\(\bar{r}_i < 5\), OR = 1.228) by virtue of strong champion reviews, but serves as a fatal veto trigger for high-scoring papers (\(\bar{r}_i > 6\), OR = 0.789).
  • In borderline decisions (\(5 \le \bar{r}_i \le 6\)), reviewer self-confidence is an uninformative predictor, whereas comment sentiment operates as a strict threshold filter, suppressing acceptance to 18.8% for negative sentiment while boosting it to 35.4% for positive sentiment.
  • Rebuttal interaction adheres to an "optimal interaction window": prompt and detailed replies are indispensable for rescuing low and borderline papers (length OR = 2.686 for low scores), but discussions exceeding 2–3 turns exhibit diminishing returns or even negative effects on high-scoring submissions.
  • Submission lead time and transparency significantly elevate acceptance: submitting \(\ge 7\) days prior to the deadline yields a 44.0% acceptance rate (against a 38.9% baseline), synchronized arXiv posting adds a ~6.4% advantage over premature preprints, and open-source code/data releases boost acceptance from 32.54% to 43.96% (OR = 1.626).
  • The LLM revolution overturned peer review linguistic preferences: in the Post-LLM era, reviewers strongly penalize verbose sentence structures (words per sentence \(t = -21.491\)) and actively reward dense, concise, and professionally rigorous writing ("Professional but Concise", Gunning Fog \(t = +17.415\)).

Highlights & Insights

  • This study delivers the first unified, four-dimensional longitudinal analysis across 28,358 OpenReview submissions over nine years (ICLR 2017–2025), transforming peer review evaluation from outcome-centric observation into dynamic process reconstruction.
  • The paper quantitatively captures the macro-level impact of Generative AI on peer evaluation: by leveling base-level text fluency, LLMs shifted reviewer cognitive tolerance, creating an explicit preference for high information density and succinct phrasing.
  • It formulates actionable, data-driven guidelines for authors: advising low-scoring candidates to respond rapidly and thoroughly within a 2–3 turn window, while counseling high-scoring authors to minimize debate and consolidate support through reproducibility artifacts.

Limitations & Future Work

  • The empirical scope is restricted to ICLR on OpenReview; while representative of machine learning conferences, structural differences in reviewer assignment and meta-reviewer authority relative to ARR/ACL or journal review limit immediate external generalizability.
  • As an observational study, despite extensive multivariate controls, the reported relationships remain correlational; confounding influences from author prestige, institutional reputation, or unobserved intrinsic quality cannot be entirely ruled out.
  • Future investigations should deploy agent-based counterfactual simulations and fine-grained causal discourse analysis to trace how specific argument-evidence structures trigger cognitive updates in reviewers during rebuttal.
  • vs Outcome-Centric Peer Review Studies: Prior works predominantly relied on post-publication citations or final binary decisions, treating review as a black box; this study decodes multi-turn interaction logs, score variances, and sentiment dynamics, revealing the micro-mechanisms governing decision-making.
  • vs Single-Aspect Linguistic / Rebuttal Studies: Earlier literature examined author reply length or readability in isolation; this work integrates process, external signals, and text features into a unified model, uncovering the critical reversal of linguistic traits in the post-LLM era and the 2–3 turn optimal interaction ceiling.
  • Institutional Policy Implications: Meta-reviewers should implement variance-checking mechanisms to buffer against outlier vetoes, discount uncalibrated confidence scores in favor of textual evidence, and formally incentivize reproducibility artifacts.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes a pioneering, four-dimensional process-centric framework across nearly 30k submissions over nine years.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous statistical testing integrating non-parametric correlations, Welch's \(t\)-tests, multivariate logistic regressions, and pre/post-LLM split analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative progression addressing specific research questions (RQ-D/I/S/P) with detailed quantitative backing.
  • Value: ⭐⭐⭐⭐⭐ Offers indispensable strategic guidance for submitting authors and foundational insights for conference organizers optimizing peer review fairness.