How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Learner Simulation, Educational Feedback Evaluation, LLM Evaluation, Personalized Preference, Role-playing
TL;DR¶
Using a real-world evaluation dataset from 321 Japanese high school students across multiple biology questions, this study systematically benchmarks how well six representative LLMs simulate subjective educational feedback ratings at both group and individual levels, uncovering a critical bias where models overestimate verbose, symbol-heavy feedback while failing to capture nuanced learner preferences.
Background & Motivation¶
In modern AI-assisted educational environments, generating automated and personalized feedback via Large Language Models (LLMs) has emerged as a promising approach to alleviate instructor workload and scaffold student reflection. Crucially, the pedagogical efficacy of educational feedback inherently hinges on learners' subjective perceptions, cognitive capabilities, and learning attitudes. Differences in prior biological knowledge, learning confidence, and Big Five personality traits inevitably cause individual learners to judge the clarity, actionability, and usefulness of the exact same feedback text in radically diverse ways. However, executing large-scale empirical user studies with real learners is notoriously expensive, time-consuming, and strictly constrained by institutional ethical guidelines. Consequently, recent educational NLP efforts have aggressively pursued LLM-based simulated learners as digital stand-ins to evaluate pedagogical feedback and iteratively optimize educational agents.
Despite this surging enthusiasm, existing learner simulation and role-playing paradigms have operated almost exclusively on rule-based prompts or unverified LLM-based evaluations, without ever being rigorously grounded against real learner evaluations (RLEs). It remains fundamentally unverified whether LLMs actually possess the intrinsic capacity to reflect real learner evaluation distributions. When learner-specific profiles are completely absent, can base LLMs mirror the macro-level preferences of student cohorts? When real student profiles—such as detailed 70-item Big Five personality inventories, subject confidence ratings, and historical evaluation exemplars—are supplied as context, does this supplementary information genuinely calibrate simulated ratings or introduce destabilizing noise? Most critically, when severe evaluation discrepancies emerge between LLMs and real learners, what specific linguistic features and feedback structures trigger LLM overestimation and hallucinated evaluative justifications?
To systematically answer these foundational questions, this work investigates the fidelity of LLM-simulated subjective feedback evaluations using a dataset collected from 321 first-year students at a selective Japanese Super Science High School (SSH). The dataset spans 4,478 evaluations across seven biology questions and six distinct prompt-guided feedback types, totaling 26,868 multi-criteria judgments. Core idea: benchmark the fidelity of LLM learner simulation across group and individual levels under conditions with and without learner profiles and in-context evaluation examples, while qualitatively uncovering the feedback properties and evaluative hallucinations that drive machine-human disagreements.
Method¶
Overall Architecture¶
The evaluation framework is organized around four driving research questions (RQ1–RQ4) across two primary experimental axes: simulation without real learner evaluation examples (Exp. 1) and simulation with real learner evaluation data (Exp. 2). In Exp. 1, models evaluate 222 feedback instances under a zero-shot unconditioned baseline and a population-sampled Big Five personality condition (+BF) to assess macro group-level ranking consistency (Spearman's \(\rho\)) and absolute calibration error (MAE). In Exp. 2, the setup incorporates real learner profile information (textual Big Five responses, quantified Big Five scores, and pre-questionnaire background attributes) alongside 6-shot in-context learning (ICL) demonstrations to assess both group-level aggregation and individual-level paired agreement. Finally, the framework analyzes how answer correctness skews model behavior and isolates extreme disagreement cases (\(|\text{LLM} - \text{Human}| \ge 1.0\)) to dissect linguistic biases and hallucinated evaluation grounds.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Real Learner Evaluation Dataset<br/>321 Students·7 Biology Questions·6 Feedback Types"] --> B["Experiment 1: Simulation Without Examples<br/>Zero-shot Baseline vs Big Five Trait Sampling"]
B --> C["Experiment 2: Simulation With Profiles & Examples<br/>Profile Representation Ablations + 6-shot Cross-Question ICL"]
C --> D["Individual Simulation & Correctness Attribution<br/>Confusion Matrices for Correct/Incorrect + Paired Rank/MAE"]
D --> E["Qualitative Disagreement Case Analysis<br/>Coverage & Arrow-Notation Overestimation + Hallucinated Grounding"]
E --> F["Empirical Limits & Guidelines for Learner Modeling"]
Key Designs¶
1. Baseline Simulation vs. Population-Conditioned Personality Profiling: Decoupling Default Priors from Trait Guidance
To address RQ1 and RQ2 regarding how LLMs simulate group-level subjective tendencies without individual rating histories, the method contrasts a zero-shot baseline against a Monte Carlo trait-conditioned framework (+BF). In the baseline, models adopt the persona of a Japanese high school student evaluating feedback across six predefined criteria without persona attributes, averaging scores across three repeated inference runs. For +BF, recognizing that student populations possess diverse personality distributions rather than a monolithic profile, 20 real student Big Five response profiles (70 binary yes/no items) are randomly sampled per feedback instance, yielding 4,440 conditioned evaluations. Aggregating these evaluations isolates whether explicit psychological profiling enhances group-level rank preservation or simply shifts mean absolute scores.
2. Hierarchical Profile Representations and In-Context Exemplars: Dissecting Persona Formats vs. Behavioral Grounding
To tackle RQ3 concerning whether privileged learner information improves simulation fidelity, the method compares six profile configurations under both zero-shot and few-shot in-context learning. Profile representations span raw 70-item Big Five responses (+BF), normalized 0–12 scalar Big Five traits (+BF(Score)), and pre-questionnaire responses (+Pre-Q) capturing self-reported biology competence and AI trust, as well as joint combinations (+BF+Pre-Q and +BF(Score)+Pre-Q). Under the few-shot regime, models receive 6-shot examples comprising the target student's real ratings and justifications on six other questions. Crucially, strict JSON schema output constraints mandate that each evaluation contains exactly one positive and one negative sentence of reasoning, ensuring uniform syntactic structure for downstream qualitative auditing.
3. Correctness-Partitioned Confusion Analysis and Extreme Disagreement Auditing: Exposing Cognitive Biases and Hallucinated Rationale
To address RQ4 and unpack the behavioral anomalies driving LLM-learner divergence, the framework bifurcates evaluations by student answer correctness and filters extreme discrepancy instances where \(|\text{LLM} - \text{Human}| \ge 1.0\). Cross-tabulating human and model rating distributions into \(3 \times 3\) confusion matrices reveals whether models evaluate feedback based on objective instructional merit or merely mirror question outcome status. Furthermore, by isolating the 92 instances where at least two models significantly over- or under-rated feedback, qualitative thematic analysis identifies specific surface artifacts—such as non-natural arrow notation (→) and exhaustive knowledge enumerations—that cause LLMs to generate hallucinated justifications citing non-existent teacher lectures or textbook authority.
Benchmark & Evaluation Protocol¶
- Dataset Specification: Furuhashi et al. (2026), comprising 321 first-year students from an academically selective Japanese Super Science High School (SSH) answering 7 multiple-choice biology questions (37 answer options total, including visual diagram items). Students rated LLM-generated feedback across six criteria on a 3-point scale (1: poor/disagree, 2: neutral, 3: good/agree), yielding 4,478 feedback evaluation sessions and 26,868 individual criterion ratings. Feedback types: Normal (indirect hints), Keywords (concept highlighting), Actionability (concrete solution steps), Novelty (advanced college-level facts), Coverage (comprehensive topic exposition), and Positivity (encouraging affective framing).
- Evaluated Model Suite: 6 prominent foundation models spanning diverse architectures and providers: GPT-5 (gpt-5-2025-08-07), Gemini-3.5-flash, Gemini-2.5-flash, Gemma-4-31B-it, Qwen3-VL-32B-Instruct, and Qwen3-VL-8B-Instruct.
- Metrics:
- Spearman's Rank Correlation Coefficient (\(\rho\)): Evaluates relative ranking consistency across feedback types and criteria.
- Mean Absolute Error (MAE): Measures numerical calibration distance from human ratings.
Key Experimental Results¶
Main Results: Group-Level Simulation Performance Without Real Evaluation Examples¶
Agreement between LLMs and real learner cohorts under the Zero-shot Baseline and +Big Five (+BF) trait sampling settings (computed over 36 criterion-feedback combinations):
| Model | Spearman \(\rho\) (Base) ↑ | Spearman \(\rho\) (+BF) ↑ | MAE (Base) ↓ | MAE (+BF) ↓ |
|---|---|---|---|---|
| GPT-5 | 0.638 | 0.625 | 0.393 | 0.294 |
| Gemini-3.5-flash | 0.434 | 0.706 | 0.355 | 0.177 |
| Gemini-2.5-flash | 0.532 | 0.098 | 0.456 | 0.382 |
| Gemma-4-31B-it | 0.563 | 0.687 | 0.459 | 0.440 |
| Qwen3-VL-32B-it | 0.353 | 0.265 | 0.274 | 0.154 |
| Qwen3-VL-8B-it | 0.057 | 0.158 | 0.242 | 0.221 |
Subgroup & Partition Analysis: Impact of Student Answer Correctness (Zero-shot Baseline)¶
Stratifying evaluation metrics by whether the learner's submitted answer was correct or incorrect demonstrates pronounced divergence in model behavior:
| Model | Overall \(\rho\) ↑ | Correct \(\rho\) ↑ | Incorrect \(\rho\) ↑ | Overall MAE ↓ | Correct MAE ↓ | Incorrect MAE ↓ |
|---|---|---|---|---|---|---|
| GPT-5 | 0.712 | 0.567 | 0.651 | 0.427 | 0.468 | 0.392 |
| Gemini-3.5-flash | 0.516 | 0.328 | 0.498 | 0.359 | 0.406 | 0.337 |
| Gemini-2.5-flash | 0.563 | 0.445 | 0.468 | 0.473 | 0.489 | 0.462 |
| Gemma-4-31B-it | 0.533 | 0.566 | 0.445 | 0.482 | 0.501 | 0.468 |
| Qwen3-VL-32B-it | 0.271 | 0.375 | 0.289 | 0.347 | 0.530 | 0.240 |
| Qwen3-VL-8B-it | 0.041 | 0.081 | 0.003 | 0.220 | 0.223 | 0.273 |
Distribution of Extreme Disagreement Cases (\(|\text{LLM} - \text{Human}| \ge 1.0\))¶
Frequency of cases where models assign substantially higher ratings (High) or lower ratings (Low) than human learners across feedback types:
| Model | Total High | Total Low | Normal (H/L) | Keywords (H/L) | Actionability (H/L) | Novelty (H/L) | Coverage (H/L) | Positivity (H/L) |
|---|---|---|---|---|---|---|---|---|
| GPT-5 | 65 | 0 | 7 / 0 | 3 / 0 | 11 / 0 | 6 / 0 | 29 / 0 | 9 / 0 |
| Gemini-3.5-flash | 61 | 7 | 8 / 0 | 2 / 0 | 10 / 0 | 4 / 2 | 26 / 0 | 11 / 5 |
| Gemini-2.5-flash | 89 | 5 | 13 / 1 | 4 / 0 | 18 / 0 | 8 / 2 | 32 / 0 | 14 / 2 |
| Gemma-4-31B-it | 83 | 1 | 12 / 0 | 2 / 1 | 15 / 0 | 7 / 0 | 32 / 0 | 15 / 0 |
| Qwen3-VL-32B-it | 54 | 1 | 6 / 0 | 3 / 1 | 7 / 0 | 4 / 0 | 23 / 0 | 11 / 0 |
| Qwen3-VL-8B-it | 20 | 1 | 2 / 0 | 5 / 0 | 0 / 0 | 0 / 0 | 11 / 0 | 2 / 1 |
Key Findings¶
- Personality Prompts Aid Score Calibration but Destabilize Ranking Consistency: Conditioning on Big Five traits consistently lowers MAE across nearly all architectures, successfully shifting synthetic ratings into realistic human score bounds. However, its effect on relative ranking (\(\rho\)) is highly erratic: while Gemini-3.5-flash jumps from 0.434 to 0.706, Gemini-2.5-flash collapses from 0.532 to 0.098. Adding lengthy personality prompts can easily disrupt the internal evaluation dynamics of certain models.
- Dichotomy Between Group Baseline and Individual Few-Shot Simulation: Frontier models (GPT-5, Gemini-3.5) exhibit strong inherent macro representations of an "average learner" under the zero-shot baseline, where few-shot prompts often degrade group-level ranking consistency. Conversely, at the individual level, zero-shot correlations hover near zero (\(\rho \approx 0.1\)), indicating that LLMs completely fail to extrapolate individual idiosyncrasies from text profiles alone. Here, 6-shot in-context rating exemplars act as a vital anchor, substantially lifting individual correlations to 0.35–0.48.
- Answer Correctness Induces Asymmetric Rating Distortions: For correct answers, frontier models display severe sycophancy, collapsing predictions onto the maximum rating (3) and sharply degrading ranking correlation with real students. When answers are incorrect, large models adopt a more discerning stance, improving correlation. Conversely, smaller models (e.g., Qwen3-VL-8B) defensively collapse toward middle-range scores (2) across the board, producing deceptively low MAE while achieving a near-zero ranking correlation (\(\rho=0.041\)).
- The Detailed Feedback Illusion and Hallucinated Justifications: In extreme disagreement cases, models overwhelmingly overestimate human ratings (over 98% High). Coverage feedback alone accounts for over 35% of all extreme divergence instances (34 out of 92). LLMs systematically conflate exhaustive information coverage and formal arrow notations (
→) with pedagogical quality. In reality, high school students find encyclopedic dumps overwhelming. Furthermore, qualitative analysis reveals that models frequently fabricate nonexistent classroom context—such as claiming feedback was "consistent with the teacher's lesson" or "grounded in textbook chapters"—to rationalize inflated scores.
Highlights & Insights¶
- Exposing the "Pedagogical Volume Bias" in LLM Judges: The paper uncovers a fundamental mismatch between LLM objective metrics and human cognitive reception: LLMs treat encyclopedic completeness as synonymous with feedback excellence, whereas human learners perceive excessive verbosity as unhelpful cognitive overload.
- Controlled Two-Sentence Justification Protocol: The methodological design enforcing exactly one positive sentence and one negative sentence within JSON output successfully eliminates prompt verbosity leakage while creating a transparent textual audit trail for detecting evaluative hallucinations.
Limitations & Future Work¶
- Single-Domain Academic Homogeneity: Evaluations are restricted to multiple-choice high school biology questions completed by students from an academically elite Super Science High School in Japan, limiting immediate generalizability to humanities, open-ended problem solving, or broader demographic cohorts.
- Static Single-Point Evaluation: The study treats cross-sectional feedback ratings as static ground-truth, without capturing dynamic multi-turn interactions where learner understanding and evaluative criteria evolve longitudinally over time.
- Future Directions: Developing specialized fine-tuning (SFT) and preference-alignment (RLHF) objectives specifically tailored for student simulation, while integrating objective behavioral traces (response latency, answer revision success, confidence ratings) alongside self-reported scales.
Related Work & Insights¶
- vs. SimBench (Hu et al., 2026): While SimBench evaluated 45 LLMs across general social science tasks and found human group simulation unreliable, this paper specifically interrogates educational feedback and reveals that while large models encode decent "average student" priors, they fail to track fine-grained individual preferences without behavioral few-shot data.
- vs. FeedEval (Chu et al., 2026) / Dean of LLM Tutors (Qian et al., 2026): Whereas prior works deploy LLM evaluators to judge AI tutors based on internal heuristics, this study benchmarks LLM evaluators directly against ground-truth student perceptions, illustrating the grave risk of automated feedback evaluators overvaluing dense, unhelpful feedback.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ First extensive empirical study validating LLM learner simulation against fine-grained real student feedback ratings.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across six leading vision-language models, group and individual tiers, profile ablations, and qualitative audits.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual framing, clear empirical figures, and incisive qualitative case analysis.
- Value: ⭐⭐⭐⭐⭐ Critical cautionary benchmark for the deployment of LLM-as-a-Judge and simulated students in educational technologies.