Benchmarking MLLMs on Mistake Recognition and Explanation in Single-Step Components of Cooking

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/Shun-Takashige/cook-mre-benchmark
Area: Multimodal VLM / Video Understanding
Keywords: Single-step cooking, instruction deviation, mistake explanation, visual discrimination, model evaluation

TL;DR

Cook-MRE keeps cooking videos unchanged while replacing one instruction element, using 1,289 samples to test whether models can detect and explain deviations; the best default Full Pass Accuracy among 13 open-source MLLMs is only 57.0%, showing that flagging a mistake is far from seeing and explaining it correctly.

Background & Motivation

Procedural video research increasingly targets long workflows, multiple steps, and diverse mistakes: models must detect anomalies, locate skipped or reordered steps, and even explain whether measurements or heating meet requirements. Yet failure on a complex task need not originate in long-term memory. A model may already struggle to distinguish slicing from dicing, or chopsticks from a spoon, within a short clip. Combining everything into a final score can conceal a more basic visual discrimination bottleneck.

Existing datasets often arrange real mistakes during recording. This preserves procedural context but requires dedicated video collection, while difficulty labels and explanation references add annotation costs. Cook-MRE takes a narrower approach: an action can be perfectly normal yet count as a mistake if it deviates from the supplied instruction. A video of diagonally slicing burdock root paired with an instruction to dice it is neither video anomaly detection nor recipe-order verification. The model must understand the instruction, observe the actual action, and explain the discrepancy.

The authors therefore reuse COM Kitchens' step boundaries and descriptions to construct controlled single-element deviations, then separately assess mistake presence, type, and actual observations. Core idea: separate whether a model notices an inconsistency from whether it actually perceives and explains that inconsistency, using controlled single-step problems to diagnose visual explanation weaknesses.

Method

Overall Architecture

Cook-MRE is a dataset and evaluation protocol, not a new video network. Construction begins with single-element sample construction and human visibility review; evaluation then feeds the video and instruction to a model for unhinted response generation, followed by three-criterion semantic scoring of detection and explanation. The diagram represents the construction-to-evaluation workflow, not modules inside the evaluated model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    source["COM Kitchens<br/>Step videos and descriptions"] --> construct["Single-element<br/>sample construction"]
    construct --> review["Human visibility review"]
    review --> answer["Unhinted response generation"]
    answer --> judge["Three-criterion<br/>semantic scoring"]
    judge --> report["Type and difficulty analysis<br/>Hint ablation"]

The model receives a single-step clip and an instruction specifying the intended action, and produces a free-form judgment, observation, and rationale. The original description used as the correction reference is available to the judge, not supplied as an additional hint to the tested model. These two texts serve different purposes: the synthesized instruction specifies what should happen, whereas the original description records what actually happens in the video.

Key Designs

1. Single-element sample construction: turn normal videos into attributable instruction deviations

COM Kitchens provides fixed third-person overhead videos, step start and end frames, and textual descriptions, with source data spanning more than 70 kitchen environments. The authors interpret each description through four elements: verb, ingredient, tool, and location. Verbs also include modifiers such as cutting style. Keeping the original description produces a correct sample; replacing one element produces a mistake sample with a uniquely attributable type. A case where both the tool and ingredient are wrong is therefore not folded into a single-type test.

Preprocessing excludes steps involving liquid seasonings, powdered condiments, or mixed ingredients that cannot be reliably distinguished visually. Clips are generally restricted to approximately 120 seconds, and descriptions containing only one operation element are discarded. Around 350 candidate steps are selected per element category, balancing verb and ingredient diversity while limiting excessive source-video overlap. These are construction-stage selection counts, not the final number of independent videos; a single step can yield multiple instruction samples.

Qwen2.5-7B receives the step description and surrounding recipe context, replaces one element at a time, and adjusts nearby wording for naturalness. For example, changing slicing to washing requires dropping the modifier about cutting diagonally. The intended source of difficulty is the video-instruction comparison, not an instruction that is obviously implausible from text alone. These are virtual instruction deviations rather than newly recorded real execution failures, so the distinction between low-cost construction and real-world validity matters.

2. Human visibility review: ensure deviations are observable and difficulty labels are grounded

A single annotator reviews every generated candidate. The review examines actual video frames as well as text, removing cases where the action is occluded or outside the frame and cannot be judged. It also removes effectively synonymous replacements such as chop versus cut, which do not establish a clear mistake, and checks that the same replacement pair receives consistent difficulty labels across samples. This filtering is central to label credibility, but single-annotator review is not equivalent to inter-annotator agreement testing.

Easy/hard difficulty is explicitly defined only for verbs and ingredients. Hard verb cases include closely related operations such as slicing versus dicing or tearing versus folding; easy cases involve clearly distinct actions. Hard ingredient cases involve similar-looking or same-category items, such as cabbage versus lettuce, while easy cases involve obvious differences. For tools and locations, implausibly distant replacements would already reveal a problem from language alone. After such language-biased candidates are removed, the remaining range is insufficient for the same difficulty split. It would therefore be incorrect to claim independently validated easy/hard labels for all four types.

The final dataset contains 989 mistake samples and 300 correct samples. Mistakes comprise 318 verb, 339 ingredient, 175 tool, and 157 location cases. The paper also reports 515 easy and 474 hard cases, which sum to all 989 mistakes despite its statement that only verbs and ingredients receive difficulty splits. This leaves the reporting convention unclear; this note does not infer how tools and locations enter those totals. Most clips are at most 120 seconds long, with two explicitly noted exceptions.

3. Unhinted response generation: prevent explanation from becoming answer-choice selection

The default protocol supplies neither the mistake type nor answer options. It asks whether the cook follows the instruction and requests an observation and reason. The model must consider all four elements, identifying not just an inconsistency but the action or object actually visible. For a video of diagonally slicing burdock root, repeating that the cook should dice it is insufficient; the observation must agree with the actual diagonal slicing.

This also clarifies what correction means here. The main check is whether the model's description of actual video content matches the original step description, not whether a newly proposed remedial action succeeds in the real world. A plausible cooking recommendation still fails if the model misses the actual cutting style. Separating factual observation from the intended action makes visual mistakes, instruction repetition, and genuine explanation distinguishable.

The hint experiment instead specifies which element to inspect, testing whether a narrower focus improves observation. This is not an equivalent reduction in difficulty for every sample: correct samples must be judged mistake-free under every applicable type-specific prompt, creating up to four opportunities for a false positive. Changes in Full Pass therefore combine the benefit of focused attention with the cost of additional judgments, and cannot be attributed solely to better explanation.

4. Three-criterion semantic scoring: explanation passes only when detection, type, and observation all succeed

GPT-5-mini reads the model response and corresponding ground truth, using a dedicated prompt for each binary judgment: \(r_i\) indicates correct mistake-presence judgment, \(t_i\) indicates the correct mistake type, and \(e_i\) indicates an observation semantically equivalent to the original description. The latter two apply only to mistake samples. The judge accepts cooking-domain synonyms and reasonable generalizations, such as lettuce for red leaf lettuce, rather than requiring exact reference wording.

Let \(\mathcal{M}\) denote mistake samples, \(\mathcal{C}\) correct samples, and \(N=1,289\). The following equations are reconstructed from the explicit scoring definitions in the prose. Equation extraction in the cache is damaged, so these are not claimed to reproduce the original typesetting verbatim; refer to the PDF for the original layout:

\[ \mathrm{TypeAcc}=\frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}}r_it_i, \qquad \mathrm{CorrAcc}=\frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}}r_it_ie_i. \]

\[ \mathrm{FullPass}=\frac{\sum_{i\in\mathcal{M}}r_it_ie_i+\sum_{i\in\mathcal{C}}r_i}{N}. \]

The gated products mean that accidentally naming the right type while declaring no mistake does not pass Type, and identifying presence and type while misrecognizing the ingredient does not pass Correction. Full Pass includes correct samples, so it need not rank models identically to Correction, which covers only mistake samples. MPC F1 treats mistakes as positive and is defined as \(2\mathrm{TP}/(2\mathrm{TP}+\mathrm{FP}+\mathrm{FN})\); Macro F1 averages the two class-specific F1 scores to expose class bias under the 989:300 imbalance.

The authors report more than 94% agreement with humans for each of the three judging criteria. A separate post-hoc failure categorization task achieves 89.2% agreement; these reliability results should not be conflated. Instruction Bias means an instruction-driven response that is not grounded in the video; Misrecognition means attending to the video but identifying something incorrectly; Reasoning Failure means drawing the wrong conclusion from a correct observation; Underspecification means a description is insufficiently precise, with remaining cases assigned to Others. These are response failure modes, not the dataset's four operation-element mistake types.

Loss & Training

There is no new training objective or fine-tuning. All 13 open-source models are evaluated zero-shot on all 1,289 samples, using 16 uniformly sampled frames per clip on a single NVIDIA V100 or A100. The paper states both greedy decoding and temperature 0.1, without complete decoding parameters in this cache to resolve the ambiguity. This note preserves that reproducibility caveat rather than inferring whether sampling was enabled.

Using 16 frames standardizes the input budget but does not establish saturation at 16 frames on this task. COM Kitchens' license prohibits sending videos to third parties, so tested models run locally and proprietary video MLLMs are excluded. GPT-5-mini's text-based judging should not be confused with uploading the videos for proprietary-model testing. A frame-shuffling ablation is mentioned, but its numerical results are in supplementary material absent from this cache; no conclusion that temporal order is irrelevant is justified here.

Key Experimental Results

Main Results

The table below reproduces the default, no-type-hint columns of the paper's Table 2, in %. MPC F1 and Macro F1 measure detection; Type and Correction cover mistake samples only; Full Pass covers the entire dataset.

Model	MPC F1	Macro F1	Type	Correction	Full Pass
InternVL3.5-38B	87.6	76.3	77.9	50.2	55.5
InternVL3.5-8B	80.4	68.9	65.0	32.9	43.4
InternVL3-38B	85.9	75.6	75.6	49.7	57.0
InternVL3-8B	71.4	62.1	53.3	34.1	46.2
Qwen2.5-VL-32B	77.8	65.4	62.6	31.6	41.3
Qwen2.5-VL-7B	72.5	62.5	52.9	28.9	41.3
Qwen3-VL-32B	76.7	64.4	60.1	30.5	40.5
Qwen3-VL-8B	80.0	64.6	64.3	23.7	32.0
Tarsier-34B	75.6	63.6	55.9	29.1	39.6
Tarsier-7B	82.3	48.0	63.8	26.1	22.3
ST-LLM-7B	80.3	51.7	46.3	7.8	10.7
LLaVA-Video-7B	26.0	31.1	9.8	2.4	21.3
VTimeLLM-7B	22.4	30.2	8.7	1.9	22.7

InternVL3-38B, not InternVL3.5-38B, has the highest default Full Pass at 57.0%; the latter leads MPC F1 and Correction. ST-LLM reaches 80.3% MPC F1 but only 51.7% Macro F1 and 7.8% Correction, illustrating how mistake-class F1 alone can overstate practical capability. F1 and accuracy have different meanings and denominators, so their numerical gap is not a conditional failure probability.

Ablation Study

These values also come from Table 2. Each arrow runs from Default to Hint, and Full Pass changes are percentage points. This is a task-prompt ablation, not a training ablation removing network modules.

Model	MPC F1: default → hint	Correction: default → hint	Full Pass: default → hint	Full Pass change
InternVL3.5-38B	87.6 → 87.3	50.2 → 57.4	55.5 → 62.1	+6.6
InternVL3-38B	85.9 → 86.4	49.7 → 56.5	57.0 → 61.0	+4.0
Qwen2.5-VL-7B	72.5 → 71.6	28.9 → 31.3	41.3 → 38.7	-2.6
Tarsier-34B	75.6 → 81.0	29.1 → 40.2	39.6 → 36.3	-3.3
ST-LLM-7B	80.3 → 86.8	7.8 → 2.6	10.7 → 2.0	-8.7
VTimeLLM-7B	22.4 → 58.8	1.9 → 7.8	22.7 → 10.6	-12.1

Tarsier-34B improves mistake-sample Correction by 11.1 points while overall Full Pass falls by 3.3 points, demonstrating why hint-induced false positives on correct samples cannot be ignored. Figure 6 also shows both effects for InternVL3-38B: a hint helps distinguish slicing from dicing, but can make the model hallucinate a wooden spoon in a clip where chopsticks are correctly used.

Key Findings

• Scaling is not universally effective: both Qwen2.5-VL sizes score 41.3% default Full Pass. Scaling InternVL3 from 8B to 38B improves verb easy/hard and ingredient easy/hard by 35.5, 12.4, 14.0, and 17.4 points, respectively; these precise deltas are stated in Section 5.2 of the paper.
• InternVL scales its vision encoder alongside its language model, whereas the Qwen and Tarsier comparisons keep vision-encoder size fixed. This supports a family-level correlation between visual capacity and gains, not a causal result controlling for other training and architectural differences.
• Figure 5 includes No Error in its percentage breakdown, while the prose describes associated percentages as shares of errors, leaving a denominator inconsistency. This note retains the qualitative finding that Instruction Bias is a major failure mode, without treating chart percentages as error-subset shares or recomputing them as such.

Highlights & Insights

• Replacing a single element makes answers attributable as well as judgeable. Reusing normal videos enables relatively inexpensive action and ingredient contrasts, but visibility review remains necessary rather than relying on language-model negative generation alone.
• Gated scoring is closer to usable explanation than a single detection metric. It explicitly separates raising an alarm from supplying a correct observation, a useful principle for layered evaluation in instructional assistance or assembly inspection.
• Correct samples are an essential control for prompt side effects, not filler. Evaluating only mistake samples would let Tarsier-34B's hint-induced Correction gains conceal its overall regression.

Limitations & Future Work

• The authors explicitly restrict the benchmark to cooking, single steps, and four operation elements, excluding measurement, duration, temperature, skipped steps, and ordering errors. Fixed overhead views do not establish applicability to egocentric settings.
• Synthesized mistakes preserve normal videos, and some instructions may still describe unusual cooking situations. Real execution mistakes and other procedural domains are needed to assess external validity.
• Construction uses one annotator, and despite high reported judge agreement, detailed supplementary validation is absent from this cache. Multiple samples generated from the same source clip also call for attention to dependence in future evaluation, rather than treating 1,289 samples as equally many independent videos.
• Sixteen frames may miss brief hand movements; equal frame budgets do not eliminate temporal sampling as a bottleneck. Frame-count sweeps, critical-frame coverage, and controlled vision-encoder experiments are reasonable next steps, not findings already established here.
• Difficulty totals, error denominators, and decoding terminology leave reproducibility ambiguities, and this cache lacks numerical frame-shuffling results. This note does not invent experiments or parameters to fill those gaps.

Related Work & Insights

• COM Kitchens supplies the source step videos and descriptions. Cook-MRE adds instruction deviations and explanation scoring; it does not newly record 1,289 videos of real mistakes.
• CaptainCook4D / EgoPER / EgoOops address multi-step procedural or execution errors, supporting workflow and localization questions. Cook-MRE deliberately removes multi-step context to isolate basic visual competence, making these settings complementary rather than interchangeable.
• ProMQA already provides procedural activity questions and explanations. The paper's Table 1 reclassifies 261 of its samples as single-element deviations, compared with Cook-MRE's 989 such mistakes and added difficulty analysis. This count comparison is not evidence of superior performance on a shared test set.
• Research direction: generating actual observations separately from instruction checking, while calibrating the ability to avoid false alarms on correct samples, could target the observed failures. This is a suggestion motivated by the analysis, not a new method validated in the paper.

Rating

• Novelty: 4/5. The contribution is controlled single-step mistake construction and explanation evaluation rather than a model architecture.
• Experimental Thoroughness: 4/5. Thirteen models, size comparisons, type/difficulty analysis, and hint ablations provide useful coverage, but real-error transfer and stronger causal controls remain missing.
• Writing Quality: 3/5. The task and scoring chain are clear, while ambiguous statistical conventions and decoding terminology reduce reproducibility.
• Value: 4/5. The benchmark exposes observation failures behind high detection F1 and offers a useful diagnostic for fine-grained procedural video understanding.