LiveK12Bench: Have Large Multimodal Models Truly Conquered High School-level Examinations?

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: High-school examinations, multidisciplinary reasoning, full-page exam understanding, process evaluation, reasoning efficiency

TL;DR

LiveK12Bench combines continuously refreshed authentic high-school papers with evaluation of answers, reasoning processes, efficiency, and full-page visual inputs, showing that strong question-answering models are not necessarily reliable exam takers: its complex-layout table reports GPT-5 accuracy falling from 82.7% with standard inputs to 50.4% with full-page inputs, although the paired-sample basis of that table requires clarification.

Background & Motivation

High model scores on mathematical benchmarks such as MATH and AIME can easily be interpreted as mastery of high-school learning and examinations. These scores, however, commonly depend on favorable conditions: questions have been converted into clean text, necessary images have been cropped, and evaluators inspect only the final answer. Authentic papers do not provide this interface: students must locate a target question among text, diagrams, and options, sometimes with separated figures or content spanning pages. When a person preparing the input has already selected the relevant information, the test mainly measures reasoning from correctly supplied premises rather than problem solving starting from the paper. Multiple-choice questions also permit elimination, guessing, and accidental success, so a correct answer does not guarantee an explanation suitable for teaching. An explanation that misreads a condition but selects the right option may pass automated answer checking while propagating a misconception as teaching material. The paper therefore asks not only whether a model produces an answer, but whether it reads the question correctly, derives the answer validly, and finishes within a reasonable budget.

Existing educational benchmarks have expanded beyond mathematics into multiple subjects and languages, but static collections remain vulnerable to inclusion in subsequent models' training data after publication. Adding more old questions cannot resolve this risk, because dataset size and temporal independence are different properties. LiveK12Bench instead continually ingests authentic papers newly authored by teachers, rather than relying primarily on model-generated questions, so evaluation content can track examination releases. Its current material comes from Chinese high-school papers published in 2026, covers Mathematics, Physics, Chemistry, and Biology, and provides Chinese and English versions. The authors describe this sourcing as having a low leakage probability, but provide no model-by-model training-data audit; recent publication should not be equated with proven absence of contamination. Authentic papers also supply point values, reference solutions, and curriculum knowledge points, enabling evaluation beyond mean accuracy toward weighted exam performance. This introduces an engineering challenge: formulas, layouts, and sub-question structures vary across subjects, while parsed questions and reference solutions must remain trustworthy together.

The paper's approach is to design dataset maintenance and evaluation conditions jointly, rather than simply adding harder questions. The dynamic construction pipeline preserves traceable questions, images, reference answers, and derivations; the testing protocol then varies input assistance and scoring constraints. This makes it possible to examine question content, visual layout, reasoning processes, and output budgets separately instead of labeling every failure as a generic reasoning deficit. Full-page mode removes assistance in preparing test-time inputs, not expert verification during dataset construction. Core Idea: build an updateable multidisciplinary collection from fresh authentic exam papers, then evaluate full-page understanding, valid derivation, and efficient completion to test whether high scores on clean inputs transfer to more realistic examination conditions.

Method

Overall Architecture

LiveK12Bench is a dataset and evaluation protocol, not a newly trained problem-solving model. On the construction side, raw PDFs or scanned papers become structured questions with reference answers, step-by-step solutions, question types, point values, subjects, and knowledge points. On the evaluation side, these resources are organized into different input formats, models produce reasoning and final answers, and separate measures assess outcomes, processes, efficiency, and exam scores. The intermediate structured database connects original papers with grading references; reference answers are not additional context supplied to the tested model. The authors use 200 papers, with the main text and Table 1 reporting 2,114 manually verified questions. Questions are split by release time into 26-03 and 26-05; the main results in Table 3 explicitly identify 26-03, so complete reporting on both splits cannot be assumed. The total row of Table 1 lists 772 Mathematics, 396 Physics, 532 Chemistry, and 414 Biology questions. Formats include 1,473 multiple-choice, 164 fill-in-the-blank, and 477 question-answering items, meaning answer verification cannot universally reduce to option-string matching. The framework also provides Complex Layout, Rigorous Process, and Long-Horizon Reasoning subsets targeting input localization, derivation reliability, and budget sensitivity. Each subset selects 50 questions per subject, with the paper describing 600 questions across the three; this does not imply independence from the main collection. As a benchmark design, its mechanisms are explained below through construction, input conditions, process scoring, and efficiency constraints rather than a neural-network diagram.

Key Designs

1. Dynamic authentic questions: turn fresh papers into verifiable evaluation records

Construction begins with recently released authentic examinations, with professional educators annotating reference answers and step-by-step derivations so the reference is not merely an unauditable option. Structural document extraction uses MinerU to detect text boxes and embedded figures, recognize text, mathematical and chemical formulas, and tables, and crop illustrations. The authors also fine-tune detection and OCR modules on an internal examination dataset to handle special symbols and dense formulas in Chinese papers. The output is Markdown and image resources, not yet a reliably separated collection of individual questions. The next stage must therefore identify associations among stems, options, reference solutions, and figures, especially avoiding attachment of a neighboring question's image. Variable-template parsing supplies an LLM with question-type definitions, target JSON field constraints, source-layout descriptions, and the extracted Markdown. Template parameters can be adjusted to the paper structure, specifying which fields to locate, their formats, and how answers are arranged in the document. This accommodates cross-subject layouts better than fixed regular expressions and allows semantic correction of minor OCR errors, but also risks unsupported repairs by the parser.

To keep such errors out of the benchmark, structured text is rendered back into HTML for manual comparison with the original paper's crucial formulas and problem-solving clues. The check concerns faithful, solvable questions rather than merely readable JSON. Question-stem text similarity is then used for deduplication, preventing the same item from receiving excessive weight when it appears in multiple papers. The authors extract a hierarchical knowledge tree from the official curriculum and train a Qwen3-VL-4B-based classifier to assign knowledge tags to questions lacking manual annotations. The framework figure lists 2,725 knowledge points, but the text does not sufficiently detail coverage or per-category annotation quality; this count should not be treated as a count of independent reasoning abilities. Advanced LLMs also preselect challenging-subset candidates using features such as layout, knowledge points, or high point values, followed by expert verification. Automation therefore means that much of ingestion and organization is reusable, not that annotation and ongoing maintenance are cost-free. Training in this study concerns these auxiliary construction components; no scheme is reported for fine-tuning the evaluated problem-solving models on LiveK12Bench.

2. Input assistance: distinguish clean questions from original-paper understanding

Text-Only (TO) supplies the textual stem and options, primarily probing linguistic, symbolic, and knowledge-based reasoning. Text-Image (TI) supplements text with cropped images required to solve the question, such as geometric diagrams, circuit schematics, or biological structures. Both standard formats have already performed part of document parsing for the model, allowing it to reason directly about the target question. Image-Only (IO, Exam mode) instead supplies an uncropped full-page examination image and a target question index, requiring autonomous localization, extraction, and interpretation. Image-only does not mean the complete absence of textual instructions: the target index is still externally specified, and the model does not autonomously schedule its question order across a whole paper. The crucial change is that associations among stems, figures, and options are no longer explicitly supplied by preprocessing and must be recovered from page layout. Even without any change in subsequent algebraic reasoning ability, one missed condition or incorrectly associated figure can invalidate the answer. IO thus measures combined document understanding and disciplinary reasoning, not an intervention that changes only reasoning difficulty.

The Complex Layout subset includes cross-page questions, spatial separation between stems and illustrations, and tightly interleaved text and images to expose this interface difference. The Rigorous Process subset selects questions with multiple knowledge points, redundant premises, and opportunities to guess the correct option, testing whether correct answers have valid derivations. The Long-Horizon Reasoning subset emphasizes complex objectives, high point values, and questions that induce repeated deliberation, testing completion under limited budgets. These subsets do not repeat a single notion of difficulty: one emphasizes reading the input, another derivation validity, and the third completion cost. Selection combines LLM pre-annotation and expert review, but remains a targeted stress test; subset failure rates should not be interpreted as averages over all high-school questions. English versions of the Chinese questions broaden access, but the main results do not provide language-by-language comparisons, so version availability does not establish equivalent cross-language performance.

3. Joint outcome and process scoring: prevent lucky answers from hiding flawed derivations

All tested models receive a common prompt requesting step-by-step reasoning followed by a final answer enclosed in \boxed{} for extraction. The outcome dimension uses Pass@1; proof questions require extracting and evaluating the proof holistically rather than matching a final numeric value. Outcome exam scoring also accounts for sub-questions, assigning credit proportionally within the original point value of the question. Process evaluation checks three root-cause categories: Condition Interpretation Error (CIE), Logical Assumption Error (LAE), and Deductive Reasoning Error (DRE). CIE covers omitted, misread, or invented conditions; LAE covers assumptions unsupported by the premises and valid knowledge. DRE covers invalid derivations or computations: even with correctly interpreted premises, a preceding step may not legitimately entail the next. These categories assess reasoning validity without requiring the model to reproduce the reference solution's exact sequence. An answer can be judged correct while losing process credit for these errors, which is precisely the separation the protocol seeks to expose.

Page 14 repeats a readable process-penalty rule, restated here with conventional typesetting:

\[ P_i = V_i - \tau \sum_{k=1}^{3} x_{i,k}. \]

Here \(V_i\) is the question's original point value, \(x_{i,k}\) indicates the presence of the \(k\)-th process-error category, and the experiments use \(\tau=3\). Because indicators operate by category, this does not mean repeatedly deducting points for every erroneous step, nor is it step-level accuracy. The paper does not clearly specify whether scores are clipped at zero after deductions; reproductions should not silently add this implementation detail. Overall exam scoring then combines process credit, outcome credit, and the proportion of correct sub-questions before normalization by total available points to a 100-point scale. This incorporates question importance and partial correctness rather than treating a simple multiple-choice item and a multi-part problem as equally weighted. Process credit is also conditioned on the correct-sub-question proportion, reflecting the requirement for corresponding correct outcomes. The paper sets the process weight to 0.5, but the cached composite equation is damaged, so the complete ES formula is not reconstructed here. PES and OCS are separately normalized for experimental presentation and should not simply be added to obtain the tabulated OES.

Semantic judgments beyond rule-based checking use multiple evaluators; the main-experiment pool comprises GPT-4o, Gemini-3-flash, DeepSeek-V3, and Qwen3-30B. Three distinct models are selected for each judgment, avoiding self-evaluation and averaging their assessments to reduce individual-judge bias. For process-evaluation reliability, two PhD students with high-school STEM backgrounds independently annotate errors as a human reference. Table 5 reports 89.6% accuracy for multi-model arbitration, above 82.0% for a single GPT-5 judge but below 93.2% human-human agreement accuracy. Table 6 also probes repeated runs, alternative judges, and ranking stability across data partitions, so the evaluation goes beyond validating only the final leaderboard. This supports the usefulness of process scoring, not the correctness of every judgment; multiple judges may still share misunderstandings of a reference solution or question.

4. Efficiency and budget constraints: separate lengthy deliberation from reliable completion

ARL adjusts accuracy using response length, with generated token counts serving as a measure of reasoning cost. The paper describes a logarithmic length factor adjusting the contribution of correct answers: responses near the reference length retain baseline accuracy, while shorter correct responses receive better efficiency credit. The experiments use an efficiency weight of 0.15 and a reference length of 4,096 tokens selected retrospectively from response statistics. This does not mark every long answer incorrect; it distinguishes verbose and concise reasoning at comparable correctness levels. The cached ARL equation has damaged summation, multiplier, and subscript notation, so only its text-supported meaning is given here rather than presenting a guessed formula as the authors' definition. Interpretation depends on the reference length and weight, and different choices could change the relative efficiency ranking.

A separate metric, \(\operatorname{Acc}_{\le r}\), imposes a maximum generation length equal to a fraction \(r\) of the full window and measures accuracy under that constraint. Counts include intermediate thinking tokens to reduce hardware-throughput differences, rather than measuring wall-clock time. On the long-horizon subset, the paper reduces the budget to 10% of the default length, approximately 3.2k tokens, and observes failures to finish. This differs from ARL: ARL weights already generated responses for efficiency, whereas a hard cap changes whether responses can finish and become correct. A model may allocate reasoning effectively with ample budget yet lose its final answer more often under strict truncation; these findings are not contradictory. The protocol does not train a new budget controller or introduce an objective for shortening chain-of-thought; it supplies a test for detecting these differences. Figure 1 progressively combines process, efficiency, and full-page constraints, so its overall score decline should not be attributed entirely to any single factor.

Key Experimental Results

Main Results

Table 3 on page 11 reports disciplinary results for the 26-03 split; the following extracts OES on a 100-point scale, not accuracy percentages. It characterizes disciplinary exam performance rather than an overall ranking with every model under IO and hard-budget conditions.

Model	Mathematics OES	Physics OES	Chemistry OES	Biology OES
Gemini-3-pro	90.3	87.9	76.7	78.6
Claude-opus-4.6	90.0	83.7	71.1	74.1
GPT-5	85.5	72.7	44.2	53.7
Kimi-k2.5	87.8	85.4	71.8	73.2
Qwen3-VL-235B	81.3	73.8	62.8	65.2
GPT-4o	24.2	21.0	11.4	24.9

GPT-5 reaches 85.5 OES in Mathematics but only 44.2 in Chemistry, showing why mathematical performance should not represent all four subjects. Kimi-k2.5 reaches 85.4 in Physics, illustrating that open-source versus proprietary gaps depend on the subject and model rather than category alone.

Ablation Study

The paper has no model-component ablation; input-condition comparisons and process stress tests instead help explain score losses. Table 4(a) on page 12 compares standard TO/TI with full-page IO; Acc is in %, while OCS is outcome credit on a 100-point scale.

Model	Standard Acc	IO Acc	Standard OCS	IO OCS
Gemini-3-pro	88.3	60.0	92.6	53.7
Claude-opus-4.6	87.2	55.0	92.8	50.6
GPT-5	82.7	50.4	88.1	44.1
Kimi-k2.5	85.0	59.0	90.2	51.8

Subtracting the tabulated values gives GPT-5 a 32.3-percentage-point Acc decline and a 44.0-point OCS decline; the former is not a 32.3% relative reduction. Comparison caveat: the standard Acc column in Table 4(a) exactly repeats the Mathematics Acc column of Table 3, without an explanation in the cache for using these values for the complex-layout subset. The differences are therefore reported as the authors' tabulated results, not as an independently confirmed same-sample causal ablation.

Table 4(b) on page 12 reports outcome and process scores on the Rigorous Process subset; CIE, LAE, and DRE are error counts, not percentages.

Model	OCS	PES	OES	CIE	LAE	DRE
Gemini-3-pro	90.9	76.9	81.7	17	7	10
Claude-opus-4.6	89.0	68.7	78.0	36	7	11
GPT-5	83.0	56.0	61.0	61	13	30
Kimi-k2.5	87.5	71.9	77.3	25	7	16

Key Findings

• GPT-5 retains 83.0 OCS on the Rigorous Process subset but has 56.0 PES; strong outcome scores can coexist with flawed reasoning, and these are distinct measures.
• Figure 5 and the text on page 13 report Gemini-3-pro at 74.8 ARL on the long-horizon subset, down 8.2 from the overall collection; efficiency with ample budget does not guarantee robustness under hard truncation.
• The same page describes most models' accuracy at the 10% budget as approximately half their unrestricted accuracy; exact plot coordinates were not reliably extracted, so no per-model values are invented.

Highlights & Insights

• Removing manual cropping and question preparation exposes document-understanding problems hidden by clean inputs. Educational products should evaluate input parsing and disciplinary problem solving separately rather than simply reuse existing answer leaderboards.
• Recording root-cause process errors is less inclined to reward verbose explanations than counting generated steps. Its value lies in detecting mistakes masked by correct final options, not in treating every non-reference solution as invalid.
• Authentic point values and sub-question structures supply a natural basis for partial credit. The design could transfer to professional examinations with explicit rubrics, but subject weights and process penalties would need recalibration.

Limitations & Future Work

• Dataset statistics conflict: the text and Table 1 report 2,114 questions, while Figure 4 and page 14 mention 2,124; Table 1 lists 220 Physics items under IO but 396 in the Physics total. This note retains the main reported count without claiming to resolve the discrepancy.
• The cached ARL, ES, and normalization equations are damaged, and some tabulated score combinations cannot be explained by simple averaging. Reproduction requires checking the original equations and implementation rather than assembling an evaluator solely from this note.
• Recent papers reduce contamination risk without guaranteeing no training overlap; internal OCR fine-tuning data, parsing accuracy, and refresh costs are also insufficiently quantified. These are reader reservations about reproducibility and long-term maintenance.
• Source material is concentrated in four Chinese high-school science subjects, and an English version does not establish validity across curricula. Full-page images with specified question indices also do not fully cover autonomous scheduling, interactive tutoring, or student learning outcomes.
• Judges remain short of human agreement, and token budgets are not actual latency. Future work could recheck conclusions with fixed temporal splits, paired inputs for identical questions, measured runtime, and independent human grading.

Related Work & Insights

• Compared with MathVista and MathVerse: these benchmarks focus on visual mathematical understanding, whereas this paper extends the visual interface to original examination layouts and adds multiple subjects and exam scoring; the advantage is task realism, not a stronger visual encoder.
• Compared with K12Vista and MDK12-Bench: this paper emphasizes continual ingestion of authentic fresh questions, full-page inputs, and efficiency constraints. Dynamically synthesized and authentic exam questions have different distribution biases, so a dynamic label alone does not establish reliability.
• Compared with M3Exam and Exams-v: those works emphasize multilingual and multidisciplinary examination resources, while this paper focuses on how input assistance, process evaluation, and budgets jointly affect scores. A useful extension is to hold questions and rubrics fixed and identify failures caused primarily by visual localization rather than missing knowledge.

Rating

• Novelty: 4/5. Combining authentic-question refresh with multidimensional exam constraints is distinctive, while individual technical components mostly reuse existing tools.
• Experimental Thoroughness: 4/5. Covers 12 models, subjects, challenging subsets, and judge reliability, but some statistics and paired-comparison details require clarification.
• Writing Quality: 3/5. The problem framing is clear, while conflicting dataset counts and damaged equations in the current text extraction complicate verification.
• Value: 4/5. Helps distinguish high answer scores from educational reliability, but should not directly be treated as evidence of effective tutoring.