Skip to content

Exposía: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/4978
Area: Multimodal Vision-Language Models
Keywords: Academic Writing Benchmark, Peer Feedback Assessment, Educational NLP, LLM-as-a-Judge, Formative Assessment

TL;DR

UKP Lab presents Exposía, the first educational benchmark connecting student research project proposal drafts, inline feedback, free-text reviews, and revisions across 60 fine-grained criteria, demonstrating that combined prompting significantly outperforms single-criterion prompting for LLM-based writing and feedback assessment.

Background & Motivation

Developing research project proposals (exposés) and delivering rigorous peer feedback are indispensable foundational skills for junior scholars and researchers across academic disciplines. However, university students consistently struggle with formulating core research questions, grounding methodology in theoretical frameworks, and defining concrete experimental schedules. Concurrently, while peer feedback represents a well-established pedagogical mechanism for fostering critical analytical thinking in both authors and reviewers, instructional curricula rarely teach students how to systematically conduct constructive, actionable evaluations. Existing Automated Essay Scoring (AES) benchmarks remain heavily dominated by general-domain persuasive essays scored on holistic scales or coarse language metrics, completely failing to address the structural complexity and argumentative rigour demanded by academic proposal writing.

A fundamental bottleneck lies in the fragmentation of resources bridging academic writing, feedback interactions, and document revisions. While prior scholarly NLP corpora such as PeerRead and NLPeer have collected peer reviews from scientific conferences, they operate in high-stakes professional publishing environments where reviews are coarse-grained, summative, and geared toward binary accept/reject decisions. In contrast, authentic academic learning environments require formative assessment: an iterative, multi-stage trajectory consisting of initial drafting, span-level inline annotations, comprehensive free-text critique, actionable revision, and final resubmission. Prior educational datasets either omit text-level human feedback, lack fine-grained pedagogical grading criteria, or isolate feedback from resulting textual revisions.

To resolve this critical empirical gap, the authors developed Exposía, a comprehensive open benchmark gathered across consecutive cohorts of a university Computer Science course entitled "Introduction to Scientific Work". Core idea: create the first multi-stage academic benchmark linking student research proposal drafts, span-anchored inline feedback, free-text reviews, and revisions with 36 exposé and 24 review pedagogical criteria, establishing zero-shot LLM scoring baselines evaluated via binned Jensen–Shannon divergence.

Method

Overall Architecture

The Exposía pipeline formalizes the formative loop of academic research proposal preparation and peer review in a higher education setting. The lifecycle encompasses student topic selection, structured proposal drafting, double-blind peer and instructor reviewing, and iterative revision into a final exposé. Instructional staff assess both proposals and reviews using fine-grained, topic-independent rubrics grounded in educational measurement theory. For automated assessment, modern LLMs evaluate submissions under zero-shot prompting strategies, with human-machine alignment quantified via binned distribution divergences.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Topic Assignment & Student Proposal Draft"] --> B["Multi-Stage Writing and Dual-Source Feedback Closed Loop"]
    B --> C["Pedagogically Grounded Dual Assessment Rubrics"]
    C --> D["Global Combined Prompting Evaluation Mechanism"]
    D --> E["Binned JSD Human-Machine Alignment Protocol"]
    E --> F["Criterion-Level Automated Scores & Revision Insights"]

Key Designs

1. Multi-Stage Writing and Dual-Source Feedback Closed Loop: Fragmented Static Writing Datasets -> Capturing Draft, Inline Highlights, Free-Text Reviews, and Revisions -> Substantive Gains in High-Order Argumentative Dimensions Traditional writing datasets present isolated final drafts, discarding the formative trajectory of critical feedback and textual revisions. Exposía captures a complete authentic lifecycle: each student authors a draft exposé following a standardized academic structure (Motivation, Approach, Schedule, Bibliography) across core AI domains such as Holographic AI Assistants, Interactive LLM Teaching Agents, and Generative AI in Healthcare. Each draft receives double-blind peer feedback from two peers and expert feedback from an instructor. Feedback is decoupled into two tiers: fine-grained span-level inline comments tagged with a categorical taxonomy \(Y = \{\text{Strength}, \text{Weakness}, \text{Highlight}, \text{Other}\}\), and holistic document-level free-text reviews synthesizing these comments. Students subsequently revise their drafts into final submissions. Empirical text analysis confirms that this feedback loop drives substantial development: documents expand from 528 words to 649 words on average, with the largest score gains concentrated in conceptual and argumentative dimensions such as Approach (\(+1.49\)) and Motivation (\(+1.16\)), demonstrating genuine pedagogical efficacy.

2. Pedagogically Grounded Dual Assessment Rubrics: Coarse Impressionistic Holistic Scores -> 36 Exposé and 24 Review Topic-Independent Criteria -> Fine-Grained Diagnostic Capability Transferable Across Research Disciplines Subjective holistic grading offers minimal diagnostic guidance to learners and poor interpretability for automated systems. Exposía constructs two comprehensive assessment schemes derived from established educational frameworks. The exposé rubric operationalizes 36 criteria grouped into 9 rubrics: Language/Quality (2), Metadata (2), Form/Structure (2), Motivation (7), Approach (8), Methodology (7), Schedule (4), Bibliography (2), and instructor-specific Additional points (2), scored on ordinal scales \([0,1]\), \([0,1,2]\), or \([0,-1,-2]\). These criteria were synthesized from 18 university thesis guidance documents across Germany and Austria and refined over 100 departmental proposals. Complementarily, the review rubric operationalizes 24 criteria across 8 rubrics grounded in the Hattie & Timperley feedback model and Narciss feedback components: Structure/Clarity (2), Language (2), Feed Up (2), Feed Back (6), Feed Forward (2), Content Quality (3), Errors (5), and Additional points (2). Because criteria evaluate transferable scholarly reasoning rather than domain-specific factual trivia, the rubric applies uniformly across diverse research themes.

3. Global Combined Prompting Evaluation Mechanism: High Inference Latency and Disjoint Contexts in Per-Criterion Calls -> All-in-One Multi-Criterion JSON Prompting -> Superior Alignment with Human Distributions and 75%+ Reduction in Wall-Clock Time When utilizing LLMs as automated evaluators, deploying separate independent calls for each individual criterion (Single-criterion Prompting, Ind) incurs prohibitive computational overhead due to repeated full-text encoding, while depriving the model of cross-criterion holistic context. Exposía introduces a combined prompting strategy (All), supplying the entire rubric schema, valid ordinal ranges, and textual score descriptions within a unified system prompt. The model processes the target text in a single inference call and outputs a structured JSON array containing the exact criterion name, integer score, and concise justification:

[
  {
    "criterion_name": "Teaser (RQ)",
    "assigned_score": 1,
    "justification": "The proposal outlines broad research aims but lacks an explicit research question."
  }
]
Benchmarking demonstrates that combined prompting universally outperforms single-criterion prompting across all models, simultaneously collapsing overall submission scoring latency from over 1,100 seconds down to ~250 seconds for 120B models and drastically curtailing processed token volume by over 90%, meeting the practical constraints of deadline-driven classroom grading.

4. Binned JSD Human-Machine Alignment Protocol: Artificial Inflated Agreement Under Heavy Class Imbalance -> Human-Median Binned Jensen-Shannon Divergence -> Rigorous Distributional Fidelity Measurement Free from Prevalence Bias Criterion-based educational assessments frequently exhibit extreme prevalence skew: for mechanical criteria (e.g., template compliance, bibliography existence), the dominant score category represents over 90% of instances. In this skewed regime, standard inter-rater reliability metrics such as Pearson's \(r\), Krippendorff's \(\alpha\), and quadratic weighted agreement (QWA) produce deceptive results, rewarding degenerate models that indiscriminately predict the majority class. Exposía addresses this by adopting a binned Jensen–Shannon divergence protocol. For each criterion \(c\), test instances are partitioned into agreement and disagreement bins based on the human median score. Within each bin, the divergence between the empirical human score distribution \(P_c\) and the model predicted distribution \(Q_c\) is calculated: $\(JSD(P_c \parallel Q_c) = \frac{1}{2} D_{KL}(P_c \parallel M) + \frac{1}{2} D_{KL}(Q_c \parallel M), \quad M = \frac{1}{2}(P_c + Q_c)\)$ The weighted mean binned JSD (lower is better) aggregates divergences across bins and criteria, providing an uncorrupted, distribution-level evaluation of model calibration that successfully prevents false-positive claims of superhuman machine agreement.

Loss & Training

The study establishes zero-shot baselines without fine-tuning, reflecting realistic educational settings where freshly proposed topics or novel rubrics possess zero pre-existing annotated training samples. Seven representative foundation models are evaluated: proprietary frontier models (GPT-5.2, Claude Opus 4.6, Gemini 3.1 Pro) and open-weight models (Llama 3.3 70B, Qwen 3 80B, GPT OSS 120B, GPT OSS 20B). All systems operate under constrained deterministic decoding with structured JSON output enforcement.

Key Experimental Results

Main Results

The primary benchmark evaluates all 7 LLMs on criterion-based exposé scoring (31 valid criteria) and student review scoring (22 valid criteria), measuring weighted-mean binned JSD (lower is better) alongside overall quadratic weighted agreement (QWA).

Model Class Model Name Exposé JSD (All) ↓ Exposé JSD (Ind) ↓ Exposé QWA (All) ↑ Review JSD (All) ↓ Review JSD (Ind) ↓ Review QWA (All) ↑
Closed-Source GPT-5.2 0.07 0.10 0.80 0.10 0.14 0.87
Closed-Source Claude Opus 4.6 0.07 0.08 0.80 0.05 0.09 0.90
Closed-Source Gemini 3.1 Pro 0.13 0.12 0.78 0.06 0.06 0.90
Open-Weight GPT OSS (120B) 0.09 0.11 0.77 0.10 0.14 0.78
Open-Weight GPT OSS (20B) 0.10 0.13 0.77 0.13 0.15 0.74
Open-Weight Llama 3.3 (70B) 0.17 0.22 0.76 0.08 0.10 0.90
Open-Weight Qwen 3 (80B) 0.18 0.19 0.72 0.11 0.12 0.89
Baseline Ref Human Raters IAA — — 0.76 — — 0.88

Ablation Study

Ablations examine inference efficiency across open-weight models under Combined (All) versus Single-Criterion (Ind) prompting modes, alongside the empirical score trajectory from draft to final exposé across rubrics (Cohort 1, \(N=55\)).

Model Name Mode Total Latency (s/sub) ↓ Prompt Latency (s/call) Tokens per Prompt Input Tokens Output Tokens
Llama 3.3 (70B) Combined (All) 182.10 (±12.79) 182.10 (±12.79) 7,181.78 5,606.69 1,575.09
Llama 3.3 (70B) Single-Crit (Ind) 284.30 (±26.54) 9.17 (±1.37) 3,228.31 3,162.76 65.55
Qwen 3 (80B) Combined (All) 430.37 (±15.13) 430.37 (±15.13) 7,473.75 5,719.84 1,753.91
Qwen 3 (80B) Single-Crit (Ind) 862.02 (±36.97) 27.81 (±3.81) 3,359.56 3,275.90 83.66
GPT OSS (120B) Combined (All) 258.01 (±13.74) 258.01 (±13.74) 8,227.11 5,679.85 2,547.25
GPT OSS (120B) Single-Crit (Ind) 1,133.41 (±89.76) 34.35 (±6.90) 3,383.58 3,160.07 223.51

Rubric-level score progression from draft to final exposé (\(N=55\)): - Approach: Draft \(7.91 \pm 2.25 \rightarrow\) Final \(9.40 \pm 2.06\), Gain \(+1.49\) (Max 11.00, remaining gap 1.60) - Motivation: Draft \(6.93 \pm 1.46 \rightarrow\) Final \(8.09 \pm 1.01\), Gain \(+1.16\) (Max 9.00, remaining gap 0.91) - Language/Quality: Draft \(2.69 \pm 1.02 \rightarrow\) Final \(3.45 \pm 0.78\), Gain \(+0.76\) (Max 4.00, remaining gap 0.55) - Methodology: Draft \(4.07 \pm 1.32 \rightarrow\) Final \(4.76 \pm 0.57\), Gain \(+0.69\) (Max 5.00, remaining gap 0.24) - Schedule: Draft \(4.15 \pm 0.84 \rightarrow\) Final \(4.65 \pm 0.55\), Gain \(+0.51\) (Max 5.00, remaining gap 0.35) - Form/Structure: Draft \(1.96 \pm 0.19 \rightarrow\) Final \(1.96 \pm 0.19\), Gain \(+0.00\) (Ceiling reached)

Key Findings

  • Combined Prompting Dominance: Across all tested architectures and tasks, Combined Prompting consistently minimizes distribution divergence from human instructors compared to independent single-criterion calls. Presenting criteria simultaneously provides mutual contextual conditioning while yielding dramatic wall-clock and token efficiency improvements.
  • Closed vs. Open Performance Disparity: Commercial proprietary models (GPT-5.2 and Claude Opus 4.6) consistently exhibit tighter alignment with human instructors (JSD 0.05-0.07) than open-weight alternatives. Because educational privacy frameworks frequently prohibit transmitting student coursework to public third-party APIs, there is an urgent imperative to bridge this performance gap on local open-weight models.
  • Task-Specific Model Specialization: Performance across writing assessment and review assessment is decoupled. GPT-5.2 demonstrates top alignment on proposal scoring (JSD 0.07) but degrades on review scoring (JSD 0.10). In contrast, Llama 3.3 70B excels on review scoring (JSD 0.08, matching frontier systems) but performs poorly on proposal scoring (JSD 0.17). Claude Opus 4.6 remains the only system exhibiting uniformly top-tier calibration across both evaluation domains.
  • Deceptive Correlation under Prevalence Skew: Evaluating solely via raw QWA creates an artificial illusion of superior agreement (often exceeding human inter-annotator agreement of 0.76/0.88), as models exploit skewed category distributions by predicting the mode. Binned JSD serves as an essential corrective diagnostic to evaluate genuine distributional fidelity.

Highlights & Insights

  • Authentic Pedagogical Ecology: Unlike synthetic or crowd-annotated NLP benchmarks, Exposía is rooted in an active university computer science course, capturing the authentic synergy between span-level inline annotations, peer reviews, and substantive document revisions.
  • Methodological Vindication of Binned JSD: Exposes the vulnerability of standard correlation metrics in evaluating LLM-as-a-judge pipelines under label imbalance, establishing a principled distributional evaluation methodology for subjective educational scoring.
  • Architectural Blueprint for Multi-Agent Peer Review: The decoupled schema separating span annotations from document-level critique provides a structured foundation for developing automated instructional writing tutors and automated peer-review auditing systems.

Limitations & Future Work

  • Consent-Induced Sampling Skew: Voluntary participation protocols lead to orphaned reviews where corresponding student drafts cannot be publicly released, reducing the number of complete end-to-end triplets.
  • Disciplinary Domain Scope: While the rubrics are designed to assess generic, transferable scientific communication skills, the empirical corpus is currently situated within Computer Science and AI topics, requiring further cross-disciplinary validation.
  • Evaluative Focus over Generative Feedback: Current baselines evaluate discriminative scoring; responsibly testing automatic feedback generation requires rigorous future studies examining factual correctness, actionability, and safeguards against pedagogical misdirection.
  • vs ASAP / ICLE++ (Automated Essay Scoring): Traditional AES corpora focus on secondary-school persuasive essays scored with coarse holistic grades; Exposía targets tertiary-level academic research proposals evaluated across 60 fine-grained pedagogical criteria.
  • vs PeerRead / NLPeer (Scholarly Peer Review): Existing scientific review datasets capture post-hoc summative screening for conference publication; Exposía tracks iterative formative assessment and revision trajectories within an educational developmental environment.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first multi-stage academic writing benchmark integrating drafts, span-level comments, reviews, and revisions with fine-grained pedagogical rubrics.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluates 7 leading open and closed LLMs across multiple prompting modes using binned distributional divergence metrics.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured, supported by extensive educational literature, comprehensive appendix analyses, and clear diagnostic case studies.
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable empirical foundation for advancing automated writing support, peer review instruction, and trustworthy educational AI agents.