Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation¶
Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/SEE-AI-Lab/crystal
Area: Multimodal Reasoning
Keywords: Reasoning process evaluation, reference-step consensus, semantic matching, order consistency, curriculum reinforcement learning
TL;DR¶
CRYSTAL adds checkable reference reasoning chains to 6,372 visual questions, exposes incomplete reasoning behind correct answers through content coverage and relative-order evaluation, and raises Qwen2.5-VL-3B's Match F1 from 0.480 to 0.633 and accuracy from 39.85% to 47.52% with CPR-Curriculum.
Background & Motivation¶
A correct visual answer does not establish that a model understood the image correctly. In multiple-choice settings, answer distributions, language priors, or a few visual cues can lead to the right option even when the accompanying explanation contradicts the conclusion. The paper opens with exactly this failure: LLaVA-v1.6-7B selects the middle gaming console as the smallest while describing it as larger than the others. Answer evaluation awards full credit, whereas its process Match F1 is only 0.15.
Asking a model to reason step by step does not automatically make its reasoning measurable. A long explanation may contain necessary visual evidence, but it may also repeat the question or add plausible-sounding descriptions. Existing multimodal chain-of-thought and process evaluation work provides an important foundation, yet a repeatable evaluation still needs to distinguish whether statements align with evidence, whether necessary steps are missing, and whether their order is consistent. CRYSTAL adds references to existing visual tasks instead of creating another question collection scored only by final answers.
This shifts the difficulty to the reference chains themselves. One model's explanations may contain stylistic biases or errors, and a problem can admit more than one valid solution. The authors therefore aggregate independently generated steps, review them, and then extend the resulting evaluation signal to reinforcement learning. Core idea: use multi-model consensus and human quality checks to construct reference steps, measure both content coverage and relative order, and condition answer-related rewards on step alignment to reduce answer-only training shortcuts.
Method¶
Overall Architecture¶
CRYSTAL connects two processes with different purposes: offline construction of reference reasoning chains and online comparison of a tested model's output against those chains. Reference construction has access to the image, question, and correct answer; evaluation instead asks the model to generate its own steps and answer, without supplying the reference answer as input. References also support training, but the benchmark and the separately generated training data must remain distinct.
The benchmark draws from MathVision, ScienceQA-IMG, RealWorldQA, MMVP, and PlotQA, with 3,039, 2,017, 765, 299, and 252 questions respectively. References average 11.6 steps and range from 3–42. Step count, question length, and linguistic markers jointly support difficulty stratification; these describe benchmark complexity rather than proving that every step is causally necessary. The diagram connects reference construction, evaluation, and optional training, after which the trained model can be assessed through the same process-evaluation interface.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Image, question,<br/>and correct answer"] --> B["Multi-model<br/>consensus references"]
B --> C["Dual quality gates"]
C -->|Rejected| B
C -->|Accepted| D["Content and<br/>order scoring"]
P["Tested model's<br/>steps and answer"] --> D
D --> O["Diagnostic report"]
D -->|For training| E["Causal process reward<br/>and curriculum"]
Key Designs¶
1. Multi-model consensus references: avoid treating one model's wording as the reasoning standard
During reference construction, Qwen2.5-VL-72B, InternVL3-76B, Gemma3-27B, and Llama-4-Maverick independently receive the image, question, and correct answer and generate candidate steps with different random seeds. Providing the correct answer serves to construct an explanation leading to a known target; it does not measure these generators' independent question-answering ability. Multiple model families diversify candidate explanations, but do not guarantee independent errors.
Candidate steps are embedded as sentence vectors. Pairs above a cosine-similarity threshold receive undirected edges, and connected components become semantic clusters. Each cluster contributes the actual candidate with the smallest average dissimilarity to the other members, reducing duplication that would result from simply concatenating chains. Representatives are ordered according to question logic in the first round; later rounds stabilize the sequence by minimizing edit distance to the preceding round. This aggregates consensus at the step level rather than voting over final answers. A connected component also does not require every pair of its members to exceed the similarity threshold.
2. Dual quality gates: agreement still needs visual and logical verification
A fifth model, Molmo-72B, checks logical validity, sequence coherence, grounding in the image, and consistency with the answer. Rejected examples return to independent generation with fresh seeds, followed by clustering and review, rather than merely receiving stylistic edits until the validator accepts them. After automated validation, a trained human annotator checks whether perceptual claims are visible, transitions are justified, and following the chain yields the correct answer. Human rejection also triggers regeneration; the paper reports that fewer than 5% of examples require this re-iteration.
The arrangement separates two reliability questions often conflated in synthetic annotation: repeated agreement across models supports consistency, whereas human inspection against the image further tests the factual basis. The human gate is not equivalent to having people write every step from scratch. Consequently, the paper's claim of learning without manual step annotation should be read as avoiding manually authored reference chains, not as eliminating human involvement from data construction.
3. Content and order scoring: diagnose omitted steps separately from reordered steps
The evaluator embeds predicted and reference steps with all-distilroberta-v1, computes pairwise cosine similarities, and performs greedy one-to-one matching over pairs satisfying \(\tau=0.35\) or higher. A predicted step can contribute at most one match, and near-duplicate predictions cannot repeatedly claim credit for the same reference step. Matches are true positives, unmatched predictions are false positives, and uncovered reference steps are false negatives. Precision measures the fraction of predictions aligned with references; recall measures reference coverage. Their harmonic mean gives per-example F1, and Match F1 is the macro-average of those per-example values, not an F1 computed after pooling all steps.
High precision with low recall therefore characterizes outputs containing only a few easy-to-align statements, while adding irrelevant explanations can reduce precision. Correctness here means semantic alignment with the reference, not formal logical truth. The paper assigns per-example F1 of 1 when both prediction and reference sets are empty and 0 when both are nonempty but no matches exist. These cases also clarify that the metric evaluates structured step outputs rather than the overall impression of free-form prose.
Match F1 ignores order. Ordered Match F1 sorts matched pairs by reference index, extracts the corresponding prediction indices, and measures the fraction belonging to a longest increasing subsequence, the LIS ratio. One-to-one matching prevents repeated indices. A high LIS indicates a large order-preserving subset of the matched steps, but matching only a few steps can also produce a deceptively high ratio. The authors therefore use it to penalize F1 for ordering errors, with \(\alpha=0.3\); \(\alpha=0\) reduces to ordinary Match F1. Operators in the cached extraction of Equation (6) are damaged, so this note does not invent the complete expression or infer a per-example formula from macro-averaged table values.
Answer accuracy is computed separately: numerical answers use tolerance-based comparison, categorical answers use exact matching, and free-form answers use a model judge. Thus, reproducible content matching does not mean the entire evaluation avoids model-based judging. Content, ordering, and answer scores must be reported together to distinguish complete chains with wrong answers, short chains with correct answers, and aligned steps in inconsistent order.
4. Causal process reward and curriculum: prevent answers from monopolizing the learning signal
The standard Composite reward adds format, answer, and reasoning terms, allowing each to supply reward independently. CPR instead conditions the process bonus on answer correctness: a correct answer receives the answer bonus \(a_w\) plus the step bonus weighted by \(s_w\), while a wrong answer receives only the step bonus discounted by \(\lambda=0.3\). The step bonus is the step weight multiplied by actual step F1, so full reward requires both a correct answer and aligned reasoning. A correct guess with no matched process still receives \(a_w\), not zero reward. Default weights are \(a_w=0.65\) and \(s_w=0.35\). The cached layout of Equation (7) is damaged; this description follows its adjacent prose without reconstructing the broken piecewise equation.
CPR-Curriculum also changes when the model receives process supervision. Its first phase uses only format and accuracy rewards to stabilize answer generation. The second phase starts from that checkpoint, introduces full CPR at a lower learning rate, and progresses from examples with fewer reference steps to longer chains. The authors additionally report using PCGrad in both CPR variants to project conflicting gradient components from answer and reasoning objectives. The curriculum neither explicitly teaches causal dependencies between steps nor directly optimizes Ordered Match F1; its main purpose is to avoid imposing difficult process requirements before structured answer generation is stable.
Loss & Training¶
Training uses GRPO on Qwen2.5-VL-3B-Instruct and InternVL3.5-4B, with 4 A100 GPUs and DeepSpeed ZeRO-3. The separately constructed training pool contains 30,312 examples: 6,218 from ScienceQA-IMG and 24,094 from TextVQA. Reward-weight sensitivity uses a 15% held-out validation split containing 4,546 examples. The training pool size is not the CRYSTAL test-set size.
The main text reports consistent reward weights and protocols across the two architectures, but refers prompts, complete hyperparameters, and implementation details to supplementary material. The available cache contains the main paper and references only, so it does not justify reporting numerical learning rates, phase-transition points, or a specific PCGrad implementation. The reward-strategy table also explicitly selects each strategy's best checkpoint on the CRYSTAL test set, a model-selection detail that should qualify generalization claims.
Key Experimental Results¶
Main Results¶
The original Table 2 evaluates 20 models. The subset below illustrates the separation between answers, coverage, and ordering. Scores other than accuracy lie on a 0–1 scale; steps denote average output length. Commercial models absent from reference construction also score highly, supporting cross-model applicability without proving that the references are entirely unbiased.
| Model | Accuracy (%) | Match F1 | Precision | Recall | Steps | LIS | Ordered F1 |
|---|---|---|---|---|---|---|---|
| GPT-5 | 57.99 | 0.612 | 0.925 | 0.479 | 5.29 | 0.636 | 0.539 |
| GPT-5-mini | 55.59 | 0.773 | 0.978 | 0.669 | 7.57 | 0.560 | 0.670 |
| Gemini 2.5 Flash | 53.95 | 0.673 | 0.701 | 0.765 | 17.10 | 0.584 | 0.579 |
| Qwen3-VL-8B | 57.66 | 0.659 | 0.827 | 0.590 | 7.37 | 0.624 | 0.572 |
| Qwen3-VL-32B | 49.22 | 0.718 | 0.819 | 0.704 | 10.56 | 0.581 | 0.617 |
| MiniCPM-v2.6-8B | 25.54 | 0.215 | 0.709 | 0.134 | 1.31 | 0.854 | 0.186 |
Precision exceeds recall for 19/20 models, with Gemini as the exception; the abstract's description of universal cherry-picking should therefore not become a claim without exceptions. GPT-5 leads answer accuracy but trails GPT-5-mini in process F1. MiniCPM's high LIS alongside very low reference coverage also shows why ordering ratio alone cannot rank reasoning ability.
Ablation Study¶
The reward-strategy comparison in the original Table 5 uses Qwen2.5-VL-3B. The final two rows below come from Table 4 and test a different architecture; they are not same-backbone ablations.
| Backbone / Reward Strategy | Accuracy (%) | Match F1 | Recall | LIS | Ordered F1 |
|---|---|---|---|---|---|
| Qwen / Baseline | 39.85 | 0.480 | 0.347 | 0.723 | 0.434 |
| Qwen / Composite | 44.92 | 0.426 | 0.284 | 0.743 | 0.392 |
| Qwen / Answer-Only | 44.30 | 0.429 | 0.308 | 0.700 | 0.380 |
| Qwen / CPR | 41.40 | 0.633 | 0.489 | 0.631 | 0.560 |
| Qwen / CPR-Curriculum | 47.52 | 0.633 | 0.493 | 0.626 | 0.560 |
| InternVL3.5-4B / Baseline | 37.61 | 0.432 | 0.325 | 0.775 | 0.387 |
| InternVL3.5-4B / CPR-Curriculum | 45.76 | 0.833 | 0.811 | 0.562 | 0.719 |
Relative to CPR, the curriculum primarily raises accuracy from 41.40% to 47.52%, rather than further increasing Match F1, which remains 0.633. Relative to the untrained baseline, Qwen gains 0.153 F1, approximately a 32% relative improvement, not 32 percentage points. InternVL gains 8.15 percentage points in accuracy and 0.401 F1. Both models lose raw LIS, so the curriculum cannot be described as improving raw order preservation as well.
The evaluator itself also needs validation. The blinded annotation results from the original Table 3 are reproduced below. Pairs come from 3 models, with borderline similarities deliberately overrepresented.
| Cosine Similarity Band | Step Pairs | Agreement with Human Judgment |
|---|---|---|
| < 0.20 | 33 | 100.0% |
| [0.20, 0.35) | 14 | 100.0% |
| [0.35, 0.50) | 19 | 68.4% |
| [0.50, 0.70) | 25 | 64.0% |
| ≥ 0.70 | 9 | 88.9% |
| Overall | 100 | 84.0% |
Key Findings¶
- Encoder ablation covers 4 encoders, 5 thresholds, and 5 baselines, totaling 100 experiments. DistilRoBERTa-v1 achieves average F1 of 0.520 versus 0.471–0.479 for alternatives. Rankings reportedly remain stable across the 20 encoder-threshold combinations, but a higher score is not itself proof of more accurate semantic judgments.
- Human agreement is 84.0%, with Cohen's \(\kappa=0.534\); all 16 disagreements are false positives. Agreement on all 47 below-threshold pairs supports those negative decisions, not the stronger claim that the matcher never produces false matches.
- In the weight ablation, \(a_w=0.65\) and \(s_w=0.35\) yield validation accuracy of 82.70%, F1 of 0.728, and maximum KL of 0.372. These are held-out results from the training distribution, not interchangeable with CRYSTAL test accuracy of 47.52%.
Highlights & Insights¶
- Separate content, coverage, and ordering. One matched statement does not establish a complete chain, and a complete chain does not establish consistent order. The decomposition reveals omission, expansion, and ordering tendencies that a single accuracy number hides.
- A process benchmark can guide training. The same references provide diagnostic targets and process-reward supervision. The curriculum results further show that the timing of process supervision affects answer quality, beyond whether a reward contains a reasoning term at all.
- Precision and recall retain information lost in F1. Gemini's long, high-recall outputs contrast with the short, high-precision outputs of most models. These profiles can inform output-length control, reference-coverage analysis, and human review.
Limitations & Future Work¶
- References do not exhaust valid solutions. The authors acknowledge alternative valid paths; providing answers to reference generators can also encourage post-hoc rationalization. An unmatched step is not necessarily wrong. Multiple valid references or partial-order constraints could reduce this problem.
- Semantic similarity is not logical equivalence. Human validation finds 16 false positives and uses only one annotator and 100 pairs. Negation, numerical differences, and visual factuality need more direct checks. The authors expect higher agreement under the natural distribution, but do not provide that measurement here.
- Order proxies are not causal dependencies. LIS penalizes deviations from a fixed reference order and may penalize interchangeable independent steps. Likewise, the CPR name does not establish causal identification or prove that an explanation faithfully represents the model's internal computation.
- Training evidence remains limited. CPR is tested on two models at the 3–4B scale. PCGrad, curriculum, and learning-rate changes are not isolated by component-wise ablations, and reward strategies are compared using test-selected best checkpoints. Independent validation-based selection, multiple seeds, and cross-benchmark tests are needed before generalizing stability claims.
Related Work & Insights¶
- Versus MathVision and RealWorldQA. These supply visual questions and answers; CRYSTAL adds process references. The main addition is evaluation granularity, not evidence that the underlying tasks cover all visual reasoning capabilities.
- Versus Multimodal-CoT, Visual CoT, and MME-CoT. These advance explanation generation, visual reasoning chains, and process-quality evaluation. CRYSTAL combines multi-model reference construction, one-to-one matching, and order diagnosis; it should not receive credit for originating the entire process-evaluation direction.
- Versus answer-based GRPO rewards. Answer-conditioned step bonuses encourage coverage, while curriculum introduces supervision more gradually. A useful next test is external evaluation that does not reuse the same semantic encoder, checking that gains extend beyond fitting the grader.
Resources: Dataset · Implementation. Links are taken from the cached paper and were not checked online.
Rating¶
- Novelty: 4/5. Connecting consensus references, process coverage, order diagnosis, and curriculum rewards is useful, although the component ideas have precedents.
- Experimental Thoroughness: 3/5. The 20-model evaluation, evaluator ablation, and two-backbone training are substantial, but human validation, test-set checkpoint selection, and component attribution remain limited.
- Writing Quality: 4/5. Motivation and failure examples are clear; claims about universality, lack of bias, and causality require qualification against the tables and definitions.
- Value: 4/5. Useful for diagnosing coverage in multimodal explanations and constructing process rewards, but not a substitute for visual factuality checks or causal-faithfulness evaluation.