How Far Are Video Models from True Multimodal Reasoning?

Conference: ECCV 2026
Paper: ECCV 2026
Area: Video Generation / Multimodal VLM
Keywords: video generation evaluation, multimodal reasoning, zero-shot benchmark, automatic prompt optimization, semantic set matching

TL;DR

The paper operationalizes "true multimodal reasoning" as a set of zero-shot video generation tasks whose contexts supply only raw material and never the conclusion, builds CLVG-Bench with 6 categories, 47 subcategories and roughly 1,390 entries over four input modalities, and proposes AVE, a judge whose prompt is optimized through semantic set matching; it finds that the best video models clear 60% pass rate on conventional editing tasks but collapse to around 21% once they must supply the physical or logical causality themselves, with multi-turn interaction failing entirely at the second turn.

Background & Motivation

Video models have spent the past two years moving from instruction following toward understanding and reasoning: reference-conditioned generation has expanded from single references (first frame, depth, pose) to multi-subject and multimodal references, and the emphasis has shifted from visual fidelity to whether the model understands and respects the causal relations inside its context. Yet the community still lacks a unified formulation and evaluation of this paradigm — existing benchmarks either accept only a single input modality (VBench takes text only) or treat editing tasks as the whole story, so there is no general framework for "arbitrary combinations of multimodal context to video." The authors therefore formalize this family of tasks as Context Learning in Video Generation: given any combination of text, image, audio and video as context, generate a video that conforms to that context in a zero-shot manner.

Evaluation is stuck in the same place, across three generations of methods. Heuristic foundation metrics (CLIP-Score, FVD, LPIPS and friends) are cheap but emit a single coarse score and cannot say what went wrong. Trained reward models such as the VideoScore line produce human-aligned multi-dimensional scores at the cost of heavy expert annotation, lag behind the pace at which video tasks change, and remain vulnerable to reward hacking. The currently popular agent / LLM-as-judge frameworks do return both scores and textual feedback, but they depend on hand-written system prompts and static in-context examples, so every new task requires a rewrite and both reproducibility and generalization suffer. More importantly, they mostly police visual fidelity and hardly ever check the intrinsic logical reasoning inside a video. As a result, "how far are video models from true multimodal reasoning" has never had a reproducible, attributable answer.

This paper supplies both halves of the missing piece — what to test and how to judge. CLVG-Bench decomposes context learning into 6 categories and 47 subcategories, all built by hand so that copyright and data contamination are off the table; AVE turns open-ended evaluation into a searchable prompt-optimization problem in which the judge learns its own fine-grained rubric from its mistakes. Core idea: operationalize "true multimodal reasoning" as zero-shot generation tasks where the context supplies only raw material and never the conclusion (physical simulation / perception / logical reasoning / multi-turn interaction), then let the judge iteratively discover its own evaluation rules — the answer is that current video models only pass on literal editing.

Method

Overall Architecture

The work has two halves: the benchmark (what to test) and the evaluator (how to judge). The input is a context \(X\) over arbitrary modalities (text / reference video / image / audio), and a baseline video model \(M\) generates \(v_o=M(X)\) in a zero-shot manner. On the benchmark side, 6 categories and 47 subcategories span a gradient of reasoning load from surface editing to multi-turn interaction. On the evaluation side, a VLM judge first lists the weaknesses of \(v_o\) under a system prompt \(p\); a semantic matching function aligns that list against the human-annotated weakness set to produce an instance-level confusion matrix plus one line of textual feedback, which in turn drives a rewrite of \(p\). Once the budget is exhausted the best prompt \(p^*\) is frozen and used to score every baseline across the six categories. Finally, two controlled probes that change only the amount of information in the input or the number of interaction turns separate "actually reasoning" from "matching literal patterns."

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Four-modality context<br/>text / image / audio / video"] --> B["Four-modality context<br/>× six reasoning tasks"]
    B --> C["Contamination-free<br/>manual metadata construction"]
    C --> D["Baseline video model<br/>zero-shot generation of v_o"]
    subgraph AVE["AVE: semantic set matching + prompt optimization"]
        direction TB
        E["Semantic set matching<br/>predicted vs annotated"] --> F["Confusion matrix + text feedback"]
        F --> G["Candidate pool, Pareto sampling<br/>rewrite judge prompt"]
        G --> E
    end
    D --> E
    G -->|freeze p*, full evaluation| H["Controlled probes: instruction explicitation + multi-turn"]
    H --> I["Diagnosis: which dimensions are close to / far from true reasoning"]

Key Designs

1. Four-modality context × six reasoning tasks: decoupling "input format" from "does it require reasoning"

Existing benchmarks tie modality to task very tightly (VBench takes text conditions only, FiVE-Bench text plus video) and never stratify task difficulty by how much implicit information the model must supply itself, so a single high score cannot tell whether the model reasons or merely executes literally. CLVG-Bench starts by abstracting user input by modality format: whatever the downstream task, the input is an arbitrary combination of the four base modalities — text, image, audio, video — which makes "how many modalities can you ingest" and "how hard is the reasoning" two orthogonal axes.

It then partitions context into six categories. Element editing (E.E.), partial reference (P.R.) and script continuation (S.C.) are essentially explicit instruction following — what to do is already written in the instruction. Physical simulation (P.S.), perception (Perc.) and logical reasoning (L.R.) hide the crucial information in the context material and require the model to supply the physical causality or logical relation itself (an oil droplet floats up because it is less dense than water; a compressed ball should rebound in a particular way). The paper further introduces multi-turn interactive generation as a seventh task, asking the model to predict the next video script from user feedback instead of restating the current input instruction (Section 3.1 says this category is temporarily folded into script continuation for now). The 47 subcategories exist so that failures can be attributed to a specific capability rather than compressed into one number.

This taxonomy is the paper's operationalization of "true multimodal reasoning": genuine reasoning means the context gives only material and the conclusion must be derived; a shortcut means the conclusion is already spelled out in the instruction as surface-level editing. The gap between a single model's 61.25% on E.E. and its 21.25% on L.R. is therefore itself the measurement of "can it reason" — average the two and the information is gone.

2. Contamination-free manual metadata construction: contexts, scripts and reference material are all built by hand

Web-scraped evaluation videos bring two problems: copyright risk, and the more damaging prospect of data contamination, since the models under test may well have seen that material during pretraining and score higher by memorization. Every entry in CLVG-Bench is therefore human-created or human-verified. Contexts are first drafted with Seed 2.0 Pro in two steps — extract themes for all six categories and their subcategories, then expand each theme into a realistic, user-oriented requirement description — after which three annotators independently verify the drafts for semantic completeness and requirement accuracy according to their assigned categories, and Gemini 3 Pro adds an automated cross-check of factual statements. Any context that violates its category specification goes through collaborative review, where annotators discuss the discrepancy and produce a corrected version.

The reference videos are constructed with more care still. Following film theory (Andrew; Bordwell et al.), the authors define five fundamental elements of a video — subject, vocal characteristics of the subject, number of shots, camera movement and video genre. Each script samples its five labels at random without replacement, so the label distribution stays uniform and no single cinematic style dominates the set; Seed 2.0 Pro then writes a complete, coherent script from the sampled labels, and Seedance 2.0 generates the video from that script. Reference images come from UniVBench (already free of copyright concerns) after manually filtering out samples that would cause style drift or ambiguous references; audio comes from the Seed-VC open-source repository after filtering out noisy backgrounds and low-quality segments. Hand construction caps the scale (about 1,390 entries), and what it buys is that results cannot be explained away by "the model has seen this."

3. AVE: semantic set matching plus prompt optimization, turning open-ended evaluation into a searchable objective

Open-ended evaluation cannot be judged by exact matching: the judge emits a free-form critique while the human annotation is a set of "noticeable weaknesses," different in wording and in count — fundamentally a comparison of one set against another. The first component of AVE is the semantic matching function \(S\) designed for exactly this. It intersects the predicted weakness set \(\hat y\) with the annotated set \(y\) to obtain \(TP_{set}\), then subtracts the intersection from each side to obtain \(FP_{set}\) (what the judge invented) and \(FN_{set}\) (what it missed), and finally adds four boundary cases: both sides non-empty earns at least one TP, both empty earns a TN, an empty annotation with a non-empty prediction earns an FP, and a non-empty annotation with an empty prediction earns an FN. After normalization the four entries form an instance-level confusion matrix whose elements sum to one:

\[C=[TP,TN,FP,FN]/N_{\text{total}},\qquad N_{\text{total}}=TP+TN+FP+FN\]

The same function emits one readable line: "Perfect prediction." when everything matches, otherwise "Omissions: ⟨missed⟩; Hallucinations: ⟨invented⟩." Keeping omissions and hallucinations apart is deliberate — the optimizer does not merely learn "you were wrong" but "which kind of wrong," which is what allows it to add targeted rules to the prompt.

The second component is the prompt search built around that matrix. The annotated data is split into train, validation and test; within a budget \(B\) the loop selects a prompt from a candidate table \(T\) (preferring the best on validation, or using Pareto sampling to preserve diversity), evaluates it on a training minibatch to obtain textual feedback, has an optimizer \(O\) rewrite the prompt from that feedback, and scores the new prompt on the validation set and records it in the table. When the budget runs out, the \(p^*\) with the best \(\Phi\) on validation is frozen and reported only on the unseen test set. Unlike conventional automatic prompt optimization designed around deterministic exact matching for QA or math problems, the matching here is a semantic set operation, so it tolerates weaknesses phrased differently but meaning the same thing. What the search learns is not boilerplate but concrete clauses — on the physical task the prompt acquires rules such as "first identify at least two fixed, non-moving reference objects in the scene, then compare the target object's size and position relative to them, and only call it an anomaly when multiple fixed references confirm the change consistently."

4. Controlled probes: instruction explicitation and multi-turn feedback separate "can reason" from "generates literally"

A high pass rate does not imply reasoning; the model may simply match literal patterns in the instruction, which calls for a counterfactual that spells the implicit information out. The first probe is instruction explicitation: Seed 2.0 Pro first interprets the context and rewrites the original instruction into one that explicitly states the physical or logical conditions (the case in the paper's figure turns "the corn is burned" into "the corn turns into popcorn"), and the same video model then generates from the rewrite. If the model had been supplying the implicit causality on its own, this "help me say it clearly" step should not change much; measured, Seedance 1.5 gains +14.9 on physical simulation and Seedance 2.0 gains +27.7 on logical reasoning, and the externally assisted Seedance 1.5 overtakes the stronger Seedance 2.0 and Veo 3.1 Fast. The models were leaning on literal patterns; the implicit causality was never derived by the model itself.

The second probe is multi-turn interaction: in a designed early-education scenario the model plays a teacher while the environment issues random user feedback about its content, and the model must infer the next video script turn by turn. This turns "does it maintain a reasoning state" into an observable quantity, because a model that merely restates its current input will lose the thread as soon as feedback rewrites the goal. Neither probe retrains or modifies any model; they change only the information content of the input or the number of turns, which makes them clean counterfactual designs. Combined with the difficulty gradient across the six categories, they separate "looks like it reasons" from "actually reasons."

A Worked Example

Walk the physical-simulation prompt of the paper's Figure 5 through one AVE iteration.

The initial system prompt is a single sentence, "judge whether the video contains any physical abnormality," scoring F1 = 65.6 on the validation set. AVE evaluates it on a training minibatch: the judge returns a set of weaknesses, and after semantic matching aligns them with the human annotation two typical error types surface — it misreads the perspective scaling of an object moving along the depth axis as "the object changed size" (FP, a hallucination), and it misses persistent anomalies that only cross-frame inspection reveals (FN, an omission). The feedback therefore goes back to the optimizer in the form "Omissions: …; Hallucinations: …". The optimizer rewrites the prompt, and the new version contains concrete clauses: cross-verify a suspected anomaly across multiple consecutive frames so a transient observation error is not mistaken for an abnormality; before judging any attribute change, identify at least two fixed reference objects (supports and the like) and compare the target against them, calling it an anomaly only when multiple fixed references agree, which separates real physical change from the apparent change produced by 3D perspective projection; and for consumable objects (chalk, pencils, erasers, candles, ink) or processes that produce uniform effects across several objects, apply the corresponding special clauses. With those rules in place F1 rises from 65.6 to 79.4 (+13.8). The example shows that AVE is not fitting the surface wording of the instruction but sedimenting "what went wrong last round" into a reusable checklist.

Loss & Training

There is no gradient training here; the object being optimized is the judge's system prompt (the "parameters" living in the context). The objective \(\Phi\) is instantiated as Recall minus False Positive Rate (Rec-FPR), F1 and MCC, scored on the validation set from the confusion matrices returned by semantic matching. The search uses two automatic prompt-optimization methods, TextGrad and GEPA, as the optimizer \(O\); Seed 2.0 Pro is the prompt optimizer and Seed 2.0 Lite is the judge model. Key settings: a total optimization budget of 30 USD, temperature 0, a maximum of 32,000 tokens, and each instance judged five times with a majority vote to suppress variance. The annotated set holds 600 samples, split evenly into train, validation and test. At evaluation time the six categories are merged into three coarser training tasks — perception (Perc.), prompt following (E.E.+P.R.+S.C.), and physical plus logical reasoning (P.S.+L.R.) — while the main results (Tab. 2) still report human pass rates over all six categories separately.

Key Experimental Results

Main Results

CLVG-Bench test cases increase in difficulty from standard text-to-video generation to multi-shot, multi-subject and multi-reference-video settings. Because some baselines constrain the input format or the number of references, the authors adopt a group-wise comparison protocol: only models in the same group are compared on an identical set of test cases, so scores from different groups cannot be compared directly.

Model	Input modalities	E.E.	P.R.	S.C.	P.S.	Perc.	L.R.
Seedance 2.0*	video + audio + image	61.25	52.64	67.65	63.54	43.37	21.25
Sora 2†	video + image	46.38	49.45	52.49	48.91	45.86	38.96
Seedance 1.5‡	image only	-	35.47	-	37.50	40.00	8.75
Veo 3.1 Fast‡	image only	-	46.38	-	46.88	36.00	16.25
Wan 2.2 14B‡	image only	-	46.67	-	39.00	16.00	11.25
HunyuanVideo 13B‡	image only	-	26.67	-	26.00	14.00	5.00
UniVideo†	video + image	8.10	8.06	4.08	10.00	3.00	3.75
LTX-2†	video + image	0.72	11.29	2.04	16.00	2.00	2.50

(Metric is human-annotated pass rate in %; - means the model's group does not cover that task; * supports video/audio/image inputs, † supports video/image inputs, ‡ supports image inputs only.)

Ablation Study

The first ablation asks whether the model supplies implicit conditions on its own: Seed 2.0 Pro first interprets the context and rewrites the instruction into one that explicitly states the physical and logical conditions, and the same video model then generates from the rewrite.

Model and configuration	Physical simulation P.S.	Logical reasoning L.R.	Average
Seedance 1.5	37.5	8.8	23.2
Seedance 1.5 + Seed 2.0 Pro rewrite	52.4	23.8	38.1
Gain	+14.9	+15.0	+14.9
Seedance 2.0	63.5	21.3	42.4
Seedance 2.0 + Seed 2.0 Pro rewrite	70.3	49.0	59.7
Gain	+6.8	+27.7	+17.3

The second ablation dissects the evaluator: the base is Seed 2.0 Lite, first optimized by a prompt-optimization framework (TextGrad / GEPA) and then augmented with the semantic matching module.

Judge prompt configuration	E.E.+P.R.+S.C. MCC	P.S.+L.R. MCC	Perc. MCC	Perc. F1	Perc. Rec−FPR
Seed 2.0 Lite (original prompt)	23.9	36.8	43.3	79.0	40.3
+ TextGrad	39.4	48.5	67.1	87.2	65.0
+ TextGrad + SemanticMatch	43.8	57.7	70.7	88.6	67.7
+ GEPA	39.3	54.6	64.3	86.4	60.0
+ GEPA + SemanticMatch	47.8	51.6	70.7	88.6	67.7
Seed 2.0 Pro (original prompt)	0.00	46.2	37.9	78.7	29.2
+ transferred optimized prompt	17.8	62.0	44.0	80.0	38.1

Key Findings

• Editing is near the passing line; reasoning is far below it. Seedance 2.0 reaches the overall high of 67.65% on script continuation, 61.25% on element editing and 63.54% on physical simulation, while the same model gets only 21.25% on logical reasoning. That gap of more than 40 points says more about where the bottleneck lies than any single averaged score.
• There is a clear proprietary-versus-open-source gap, and broader input modality does not imply stronger reasoning. The best open-source entry, Wan 2.2 14B, manages 11.25% on logical reasoning, and UniVideo only 3.00% on perception; yet Veo 3.1 Fast, which accepts images only, ties Sora 2 (which takes video plus image) on partial reference at 46.38%.
• Logical reasoning is the dimension where models disagree most. The paper's prose calls 21.25% the ecosystem ceiling on logical reasoning, but the table shows Sora 2 at 38.96%, well above Seedance 2.0 (⚠️ the prose's "21.25% ceiling" is inconsistent with the table's 38.96%; trust the table), which itself shows there is no stable leader on this dimension.
• Spell out the implicit conditions and the score comes back. With VLM rewriting, Seedance 1.5 climbs from 8.8 to 23.8 on logical reasoning (+15.0), overtaking both the unaided Seedance 2.0 (21.3) and Veo 3.1 Fast (16.25). External understanding is only a partial remedy, but it also proves that the generative model itself was not doing that inference. (Figure 4(a) gives the qualitative comparison between original and rewritten instructions; the caption text in the cache is fragmentary, so the exact mapping is ⚠️ to be checked against the original figure.)
• Multi-turn interaction is the hardest bottleneck. Under the vanilla setting even the strongest video models stay below a 10% first-turn success rate and fail entirely at the second turn; with Seed 2.0 helping to organize context and predict the script, first-turn success improves by 44% and 50% respectively, and the final pass rate after several rounds reaches 26% and 34% (Figure 4(b)(c); no complete per-turn breakdown appears in the main tables).
• Semantic matching is not a universal gain — it depends on the underlying APO already pointing the right way. TextGrad alone lifts perception MCC from 43.3 to 67.1 (+23.8); stacking semantic matching on top adds a further 4.9 (MCC), 1.7 (F1) and 4.5 (Rec-FPR) on average. But on P.S.+L.R., GEPA with semantic matching scores 3.0 MCC below GEPA alone, showing the matching module amplifies whatever direction the optimizer already found — if the direction is wrong, it amplifies that too.
• Optimized prompts transfer across models for free. A prompt optimized on Seed 2.0 Lite transfers directly to Seed 2.0 Pro, gaining on average +13.2 MCC, +3.0 F1 and +12.7 Rec-FPR across the three tasks with no additional optimization budget. Kendall's τ against human judgement rises in step: for Seed 2.0 Lite, 0.433 → 0.707 on perception, 0.368 → 0.577 on physical and logical reasoning, and 0.239 → 0.480 on prompt following; the transferred prompt puts Seed 2.0 Pro at 0.620 on physical and logical reasoning.

Highlights & Insights

• A task taxonomy that turns "reasoning" into a controlled variable. Splitting the six categories into execution-type and inference-type by whether the conclusion is already written in the instruction means "can it reason" no longer needs a subjective judgement — it becomes the pass-rate gap of one model across categories. The trick transfers to any generate-and-evaluate setting: stratify tasks by how much information is left implicit, then read the gap.
• Semantic set matching replaces exact matching and makes open-ended evaluation optimizable. Turning a free-form critique into TP/FP/FN set operations yields both an optimizable numeric objective (the confusion matrix) and readable feedback, and it cleanly decouples omissions from hallucinations. The component itself ports to prompt tuning for any LLM-as-judge setup.
• Instruction explicitation is a cheap, clean counterfactual probe. No retraining, no model surgery — just have a VLM spell out the implicit conditions, and you can tell whether the model was reasoning or matching literal patterns. Any generative model claiming reasoning ability can be screened this way.
• Transferability of the judge prompt is explicitly verified. The paper does not stop at "our prompt is better"; it shows a prompt optimized on a weaker judge transfers to a stronger one and keeps improving, which lowers the deployment cost of the whole evaluation framework to a single optimization run on a cheap model.

Limitations & Future Work

• Limitations the authors admit: external VLM assistance is only a partial remedy, and improving the intrinsic reasoning ability of the generative model still has to happen during training. That is the stated motivation for their call to couple understanding and generation more tightly.
• The paper never gives "true multimodal reasoning" a formal definition; it is delimited indirectly through the task categories (P.S. / Perc. / L.R. versus the editing categories), so whether a given task counts as genuine reasoning remains the authors' judgement and is hard for others to falsify.
• The main results are human-annotated, but Tab. 2 reports no inter-annotator agreement (Kendall's τ is given only for the evaluator side), so the annotation quality of roughly 1,390 entries has no quantitative guarantee.
• The category assignment of multi-turn interaction is inconsistent: Section 3.1 says it is temporarily folded into script continuation, Tab. 1 lists Interaction as a separate seventh category, and Section 4.3 analyzes it separately. A reader cannot tell whether it is counted inside the S.C. statistics.
• Section 4.1 lists CogVideoX1.5-5B and Wan 2.7 as baselines, yet Tab. 2 reports no scores for them, so the main results table is incomplete.
• The group-wise comparison protocol makes scores across groups non-comparable, but the prose conclusion that proprietary models broadly beat open-source ones does not state that it is a within-group comparison.
• Tab. 4 heads its columns "P.E." while the prose says "Physical Simulation" (⚠️ to be checked against the original paper), and AVE is trained and tuned over three coarse tasks yet must emit six fine-grained scores, with no separate validation of judge reliability at the fine-grained level.
• Directions for improvement: give "true reasoning" a falsifiable definition (for example, require correct results under counterfactual contexts); report inter-annotator agreement; promote multi-turn interaction to a first-class category with a defined turn-level metric and full per-turn numbers.

Related Work & Insights

• vs VBench / FiVE-Bench / OpenVE-3M: these target editing and fidelity, with inputs limited to text or text plus video, and they do not test reasoning at all; this paper covers four modalities and makes reasoning tasks the main arena. At roughly 1,390 entries it is not large, trading scale for manual control.
• vs RBench-V / Video-CraftBench / VBVR-Bench: these do test reasoning, but their inputs remain text and image without audio or multi-turn interaction; VBVR-Bench is the largest (7,500 entries) with categories concentrated in physical simulation, perception and logical reasoning, and its data is web-scraped, carrying copyright and contamination risk — precisely the contrast that motivates this paper's all-manual construction.
• vs VideoScore / VideoScore2 (trained reward models): they regress human preference scores with Video-LLMs, needing heavy expert annotation and struggling to keep pace with task turnover; this paper uses only 600 annotations and reaches a strong correlation with human judgement (τ up to 0.707 on perception) through prompt optimization plus semantic matching.
• vs Video-Bench / UniVBench / VideoGen-Eval (agent-based evaluators): they return scores and textual feedback too, but depend on hand-written system prompts and static in-context examples, so every new task needs a rewrite and reproducibility and generalization suffer; AVE lets the prompt iterate on its own and additionally verifies cross-model transfer, driving the cost of adapting to a new task towards zero.

Rating

• Novelty: ⭐⭐⭐⭐ The framing (turning "true reasoning" into a task taxonomy) and the semantic-set-matching judge are both new, though the overall benchmark + LLM-as-judge skeleton is not.
• Experimental Thoroughness: ⭐⭐⭐⭐ Eight mainstream video models, two controlled probes, a full evaluator ablation and cross-model transfer make the case well; points off for missing per-turn numbers on multi-turn interaction and two listed baselines absent from the main table.
• Writing Quality: ⭐⭐⭐⭐ The problem-to-evidence chain is clear and the prompt-evolution case study is persuasive; points off for the inconsistent category bookkeeping and for statements such as the "21.25% ceiling" that contradict the table.
• Value: ⭐⭐⭐⭐⭐ It gives a reproducible, attributable answer to "can video models actually reason," and the two bottlenecks it identifies — physical causality and multi-turn state — directly inform future model design.