title: >- [Paper Note] GradingBench: Evaluating End-to-End Compositional Reasoning of MLLMs for Automated Exam Grading description: >- [ECCV 2026][LLM Reasoning][Automated Exam Grading] Introduces GradingBench, the first end-to-end benchmark for automated exam grading spanning 300 full-page sheets and 3,284 questions, revealing localization as the primary bottleneck and highlighting coordination failures between perception and reasoning. tags: - ECCV 2026 - LLM Reasoning - Automated Exam Grading - Multimodal VLM - Compositional Reasoning date: 2026-09-19 content_hash: 7846205b60a8003c

GradingBench: Evaluating End-to-End Compositional Reasoning of MLLMs for Automated Exam Grading¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/ERRORSEMI/GradingBench
Area: LLM Reasoning
Keywords: Multimodal VLM, automated exam grading, compositional reasoning, visual localization, benchmark

TL;DR¶

Presents GradingBench, the first comprehensive benchmark evaluating end-to-end automated exam grading across 300 full-page K–12 exam sheets and 3,284 annotated sub-questions, demonstrating that visual localization is the primary bottleneck and that MLLMs suffer from severe coordination failures when integrating perception and reasoning.

Background & Motivation¶

Multimodal Large Language Models (MLLMs) have achieved impressive milestones on isolated vision-language tasks such as visual question answering, text recognition, and isolated mathematical reasoning. However, when deployed in complex real-world workflows that require tight synergy across visual perception and multi-step cognitive reasoning, their reliability degrades sharply. Automated exam grading exemplifies this challenge: a practical grading system must concurrently perform answer region detection, messy handwritten text recognition, and domain-knowledge-driven correctness evaluation within a single unified workflow.

Existing vision-language benchmarks decouple these competencies by testing isolated sub-skills. Most datasets provide pre-cropped clean question snippets or gold-standard transcriptions, deliberately bypassing the messy layout parsing and spatial localization intrinsic to full-page documents. Consequently, strong performance on decoupled benchmarks fails to translate into real-world automated grading, where a failure at any single perceptual or cognitive stage cascades and ruins the entire evaluation.

To close this divide, the field requires an integrated evaluation paradigm that tests whether MLLMs can execute perception, transcription, and reasoning end-to-end on uncurated documents. The core idea is to establish GradingBench, the first comprehensive benchmark for end-to-end automated exam grading on real K–12 exam papers, structuring evaluation across three task levels (single-question, specified-question, full-page) under both answer-based and answer-free settings to rigorously diagnose perception-cognition coordination bottlenecks.

Method¶

Overall Architecture¶

GradingBench measures an MLLM's ability to execute the complete grading pipeline end-to-end in a single forward pass. Given a full-page scanned or photographed exam sheet, the model is prompted to detect each handwritten answer region, transcribe student handwriting into text, infer the standard solution (in answer-free mode) or consult reference solutions (in answer-based mode), and output a structured JSON list containing bounding boxes, recognized text, reasoning traces, and final binary correctness verdicts.

The benchmark spans three progressive spatial levels: L1 (Single-question Grading) evaluates isolated performance on cropped question images; L2 (Specified-question Grading) feeds a full-page image along with gold bounding box coordinates of target questions to probe spatial instruction grounding amidst distractor regions; L3 (Full-page Grading) presents raw, uncropped exam sheets to assess fully autonomous grading. Each level is tested under Answer-Based (with reference keys) and Answer-Free (autonomous solution derivation) conditions, yielding six distinct evaluation scenarios.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Exam Paper Input<br/>Scanned / mobile-captured sheets"] --> B["Stage 1: Multi-Dimensional Benchmark Construction<br/>4 filtration criteria + 4-tier expert metadata"]
    B --> C["Stage 2: Hierarchical Task & Dual-Setting Space<br/>L1 Single / L2 Specified / L3 Full-Page × Free/Based"]
    C --> D["Stage 3: Center-Point Spatial Matching<br/>Greedy nearest-neighbor pairing robust to coordinate shift"]
    D --> E["Stage 4: Three-Stage Gated Accuracy Evaluation<br/>Spatial matching → CER filtering → Derivation & Verdict"]
    E --> F["Diagnostic Evaluation Output<br/>End-to-end Accuracy Acc & Localization mAP"]

Key Designs¶

1. Multi-Dimensional Benchmark Construction: capturing authentic educational diversity and real-world artifacts

To ensure real-world fidelity, the benchmark collects 300 full-page exam papers from Chinese K–12 education across elementary, junior high, and senior high schools. The collection explicitly preserves unconstrained imaging conditions such as uneven lighting and paper folds. Data selection enforces four strict criteria: at least 5 handwritten answers per page to guarantee density; an error rate of $\ge 10\%$ per page to maintain class balance and supply sufficient negative samples; the complete absence of pre-existing red-pen teacher grading marks to avoid label leakage; and high diversity in layout (single-column, double-column, mixed text-table-figure) and handwriting styles (neat to cursive). The dataset covers 3,284 sub-questions across Science, Mathematics, Chinese, English, and Humanities.

2. Hierarchical Task & Dual-Setting Space: probing compound spatial and reasoning cognitive load

The benchmark structures tasks across L1, L2, and L3, combined orthogonally with Answer-Based and Answer-Free settings. In Answer-Based grading, reference answers are provided in the prompt, focusing model capacity on perceptual detection, OCR transcription, and solution verification. In Answer-Free grading, the model must solve the problem independently before evaluating the student. This configuration uncovered a counter-intuitive phenomenon dubbed the L2 anomaly: under the Answer-Free setting, providing explicit question bounding box coordinates yields substantially lower accuracy than autonomous L3 grading (e.g., Doubao-seed-2.0-Pro achieves 46.30% on L2 vs. 68.14% on L3), demonstrating that simultaneously parsing textual instructions and grounding explicit numerical coordinates creates severe cognitive interference.

3. Center-Point Spatial Matching: decoupling systematic coordinate shifts from semantic localization

Standard evaluation metrics relying on strict IoU thresholds (such as $\text{IoU} \ge 0.5$) often penalize systematic coordinate offsets—where a model correctly identifies the semantic answer region but outputs a bounding box shifted slightly upward or downward. Under rigid IoU, such slight spatial discrepancies yield an overlap of zero, unfairly zeroing out correct downstream OCR and reasoning. GradingBench introduces a greedy top-to-bottom center-point matching algorithm based on Euclidean distance between predicted and ground-truth box centroids. Empirical analysis confirms this nearest-neighbor heuristic successfully matches over 97% of semantically correct detections, providing an objective bridge between visual localization and semantic evaluation.

4. Three-Stage Gated Accuracy Evaluation: preventing spurious guesses and quantifying error cascades

To ensure that final accuracy reflects genuine end-to-end competency rather than lucky guesses, GradingBench applies a rigorous multi-stage sequential filter to each question: - Spatial Correspondence Check: If no predicted box successfully pairs with the ground-truth answer region, the question is marked as failed. - Text Recognition Check: For matched boxes, Character Error Rate (CER) is computed against expert transcriptions. If $\text{CER} \ge 0.7$, the sample is rejected due to illegible or failed OCR. - Answer Derivation Check: In the Answer-Free setting, Qwen3-32B validates whether the model's derived reference answer semantically matches ground truth (verified to achieve 98.7% human expert agreement). If derivation fails, the question is marked incorrect to prevent credit for contradictory or coincidental reasoning. - Judgment Evaluation: Questions surviving all prior checks have their binary grading verdicts compared against ground-truth labels. The overall grading accuracy is calculated as: $$\text{Acc} = \frac{N_{\text{correct}}}{N_{\text{total}}}$$ In parallel, fine-grained localization precision is independently monitored via mean Average Precision over IoU thresholds from 0.2 to 0.5: $$\text{mAP}@[0.2:0.5] = \frac{1}{7}\sum_{k=1}^{7}\text{AP}(0.2 + 0.05(k-1))$$

Key Experimental Results¶

Main Results¶

The authors evaluated 18 prominent MLLMs across all six evaluation dimensions. The table below summarizes representative closed-source and open-weight models across L1–L3 under both Answer-Free (Acc-F) and Answer-Based (Acc-B) settings, along with localization mAP and the nine-metric overall score (from original Paper Table 2):

Model	L1 mAP	L1 Acc-F	L1 Acc-B	L2 mAP	L2 Acc-F	L2 Acc-B	L3 mAP	L3 Acc-F	L3 Acc-B	Overall Avg
Gemini-3.1-Pro	63.03	79.60	89.74	47.28	72.00	83.96	35.65	70.06	81.78	69.23
Doubao-seed-2.0-Pro	81.09	75.58	85.93	30.76	46.30	71.48	64.15	68.14	79.95	67.04
Kimi K2.5	20.18	63.15	74.39	16.58	32.81	58.91	25.24	59.77	67.92	46.55
Gemini-2.5-Pro	29.34	67.33	78.11	16.82	38.03	67.08	14.09	50.94	56.09	46.43
Qwen3.5-Plus	22.78	60.63	71.19	14.79	30.73	53.66	36.03	56.09	65.59	45.72
Claude-Opus-4.6	22.30	64.61	75.78	10.40	40.05	67.49	4.59	43.91	57.77	42.99
Qwen3-VL-235B-A22B	20.98	51.92	66.96	7.67	17.85	40.53	35.42	47.42	63.68	39.16
GPT-5.2	22.53	41.23	63.64	4.21	9.54	28.24	4.00	18.10	33.05	24.95
Qwen2.5-VL-72B	6.73	50.37	59.50	1.87	10.96	23.39	1.40	23.33	32.83	23.38
Qwen2.5-VL-7B	0.93	32.67	36.75	1.30	5.18	10.32	0.74	8.38	11.91	12.02

Ablation & Diagnostic Studies¶

To distinguish whether performance bottlenecks originate from raw perceptual failure or cross-task coordination interference, the authors conducted sub-capability isolation on L1 (Paper Table 3) and compared single-round versus multi-round execution on L3 (Paper Table 4):

Sub-Capability Isolation (L1 Single-Question Mode)

Model	Isolated Localization mAP	Isolated OCR (CER < 0.7)	Isolated Reasoning Acc
Gemini-3.1-Pro	67.72	92.87%	87.56%
Gemini-2.5-Pro	40.74	90.16%	80.48%
Qwen3-VL-235B-A22B	41.49	93.11%	80.60%
Doubao-Seed-1.6	13.05	85.54%	80.01%
Qwen2.5-VL-72B	7.07	84.90%	77.38%
InternVL3-78B	1.16	79.84%	68.14%

Single-Round vs. Multi-Round Execution (L3 Answer-Free Setting)

Model	Single-Round mAP	Single-Round Acc-F	Multi-Round mAP	Multi-Round Acc-F
Gemini-3.1-Pro	35.65	70.06%	39.66	74.38%
Gemini-2.5-Pro	14.09	50.94%	21.31	58.40%
Qwen3-VL-235B-A22B	35.42	47.42%	45.49	58.48%
Doubao-Seed-1.6	15.43	50.79%	22.97	58.25%

Key Findings¶

Visual localization is the hardest ceiling: In isolated capability testing, text recognition ($\sim 80\%-93\%$) and standalone reasoning ($\sim 68\%-88\%$) remained relatively strong across all models. Conversely, localization mAP showed a massive 66.56% spread (67.72% for Gemini-3.1-Pro down to 1.16% for InternVL3-78B), proving that visual grounding failure is the catastrophic root of grading failure cascades.
Multi-task instruction overhead triggers coordination collapse: When models transition from isolated localization to compound end-to-end grading, localization accuracy degrades noticeably (e.g., Gemini-2.5-Pro drops from 40.74% to 29.34% mAP), showing that simultaneous coordinate regression and semantic reasoning overload model attention.
The L2 anomaly highlights spatial grounding fragility: In the Answer-Free setting, providing explicit question bounding boxes in L2 consistently yielded lower accuracy than autonomous L3 grading (e.g., Doubao-seed-2.0-Pro scored 46.30% in L2 vs. 68.14% in L3), exposing models' inability to bind textual question instructions with geometric coordinates.
SFT remedies perception while reasoning gains remain modest: Diagnostic supervised fine-tuning of Qwen2.5-VL-7B on $\sim 10,000$ examples boosted its overall average from 8.20% to 57.47% (with L1 mAP reaching 84.81%), substantially closing the perceptual gap with leading frontier models while highlighting that complex multi-step reasoning requires deeper architectural and pre-training advances.

Highlights & Insights¶

Realistic diagnostic evaluation pipeline: Replaces brittle IoU matching with Euclidean center-point pairing coupled to a three-stage gated accuracy metric, successfully filtering lucky guesses while accommodating harmless visual coordinate shifts.
Empirical exposure of the L2 instruction anomaly: Uncovers a critical blind spot in contemporary MLLMs regarding spatial coordinate grounding under zero-shot text-prompting conditions.
Pragmatic task decomposition benefits: Demonstrates that multi-round interaction decomposes cognitive overhead and consistently improves full-page grading accuracy by 4% to 11% across models without architectural retraining.

Limitations & Future Work¶

Restricted question diversity: Currently concentrates on objective questions and short open-ended questions with binary right/wrong criteria, leaving subjective tasks (e.g., essay grading, step-by-step partial credit) unaddressed.
Single-language concentration: The dataset focuses on Chinese K–12 exam formats; cross-cultural generalization to multi-lingual scripts and diverse global exam layouts remains to be explored.
Static execution pipeline: The evaluated multi-round interaction follows a predetermined three-step schedule without dynamic error self-correction or agentic zoom-in inspection.

vs. TextVQA / DocVQA: While document VQA benchmarks focus on isolated reading of clean or lightly annotated text, GradingBench demands end-to-end integration of dense layout detection, handwriting OCR, and domain reasoning.
vs. MathVista / ScienceQA: Existing scientific reasoning benchmarks utilize clean, isolated problem images; GradingBench tests reasoning under the visual noise, clutter, and localization demands of full-page exams.
vs. Exams-v / EDU-CIRCUIT-HW: Prior educational benchmarks evaluate isolated subjects (e.g., electrical circuits) or cropped single questions, lacking end-to-end full-page automated grading pipelines.

Rating¶

Novelty: ⭐⭐⭐⭐ [First comprehensive end-to-end evaluation benchmark directly tackling the real-world automated grading workflow]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking across 18 models, 3 task levels, 2 answer settings, with isolated capability analysis, multi-round comparisons, and SFT diagnostics]
Writing Quality: ⭐⭐⭐⭐⭐ [Well-structured and clearly argued, offering rich diagnostic metrics and in-depth error taxonomy]
Value: ⭐⭐⭐⭐⭐ [Provides a crucial benchmark and diagnostic data for applying multimodal reasoning to practical educational tools]