Skip to content

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Zefan-Cai/MMGR
Area: Multimodal VLMs
Keywords: Multimodal Generative Reasoning, Video Generation Evaluation, Embodied Navigation Generation, Process-Aware Verification, Gated Evaluation Protocol

TL;DR

MMGR is the first unified cross-modal benchmark assessing the grounded reasoning capabilities of video, image, and language generative models across 10 tasks in Abstract Reasoning, Embodied Navigation, and Physical Commonsense (1,853 instances), employing process-aware chain-of-frame criteria and deterministic gated checks to demonstrate that visual fluency does not entail genuine reasoning.

Background & Motivation

Modern multimodal generative models have attained extraordinary visual fidelity and temporal smoothness, generating photorealistic images and high-frame-rate videos directly from complex natural language instructions. However, a fundamental open question in multimodal artificial intelligence is whether this apparent visual fluency reflects genuine reasoning: when prompted to synthesize an execution trace or solution, does the model preserve the requisite physical, logical, spatial, and temporal constraints, or does it merely assemble plausible-looking pixel interpolations? Existing generative evaluation suites predominantly focus on aesthetic visual appeal, holistic text-to-media alignment, or single-frame human preference scores. Such protocols evaluate only the terminal output or final frame, leaving the validity of causal sequences, object permanence, and intermediate states entirely unchecked.

Prior evaluation efforts remain fragmented and structurally blind across multiple dimensions: image benchmarks examine static commonsense and text-guided editing, while video suites emphasize compositional binding or surface motion quality. Most existing video benchmarks rely on non-deterministic human evaluations or ungrounded vision-language model judges, and lack real-world, agent-centric embodied navigation tasks that test multi-room, multi-floor 3D spatial planning. Consequently, current evaluation paradigms fail to differentiate whether a generated video genuinely solves an underlying problem, whether it traverses valid intermediate states, or whether its apparent failures stem from geometric rendering constraints versus fundamental cognitive deficiencies.

To resolve this diagnostic impasse, evaluation must pivot decisively from asking whether generated artifacts look convincing to verifying whether they rigorously satisfy underlying task constraints through valid reasoning trajectories. This necessitates a unified diagnostic suite capable of evaluating video and image generators alongside language baselines on shared reasoning tasks with deterministic checks. Core idea: construct MMGR (Multi-Modal Generative Reasoning Benchmark and Evaluation), a cross-modal benchmark spanning Abstract Reasoning, Embodied Navigation, and Physical Commonsense that pairs process-aware chain-of-frame verification with deterministic gated metrics to strictly decouple true task-state reasoning from superficial visual simulation.

Method

Overall Architecture

MMGR organizes generative reasoning into three complementary domains comprising 10 tasks across 1,853 curated instances, probing five core reasoning capabilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal reasoning. Generative modelsโ€”encompassing state-of-the-art video generators (Google Veo-3, OpenAI Sora-2, Wan-2.2), image diffusion models (Nano-banana, Nano-banana Pro, GPT-4o-image, GPT-image-1.5, Qwen-image), and text-only baselines (Gemini-3-Flash, Gemini-3-Pro)โ€”receive problem specifications and structured prompts to produce step-by-step solution media. The evaluation executes through a hybrid verification architecture: deterministic rule-based analyzers evaluate exact symbolic and geometric constraints, while calibrated VLM evaluators score high-dimensional perceptual criteria under a unified, multi-condition gated primary metric.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Specification Input<br/>Visual Initial State + Structured Prompt"] --> B["Multi-Domain Task Grounding<br/>Abstract Reasoning + Embodied Nav + Physical Commonsense"]
    B --> C["Cross-Modal Generative Solvers<br/>Video Diffusion / Image Diffusion / LLM Baselines"]
    C --> D["Process-Aware Chain-of-Frame Tracking<br/>Trajectory Validity & Intermediate Reflection Extraction"]
    D --> E["Hybrid Verification & Metric Disentanglement<br/>Deterministic Computer Vision + Gated VLM Judges"]
    E --> F["Gated Primary Success Determination<br/>Constraint Integrity โˆง Process Validity โˆง Outcome Correctness"]

Key Designs

1. Multi-Domain Task Grounding: Systematic decoupling of five reasoning faculties across controlled difficulty tiers Addressing the narrow scope of prior benchmarks, MMGR systematically organizes generative problem-solving into three distinct domains. Domain 1 (Abstract Reasoning) targets formal symbolic manipulation through 2D Maze traversal (240 instances, scaling from \(3\times3\) to \(13\times13\) using DFS and Wilson generators), Sudoku completion (300 puzzles across \(4\times4\) and \(9\times9\) grids with controlled clue sparsity), Abstraction and Reasoning Corpus (ARC, 456 tasks across v1 and v2 testing Match and Mismatch grid restructuring), and Visual Mathematics (327 problems from GSM8K to Olympiad-tier Omni-MATH). Domain 2 (Embodied Navigation) benchmarks real-world agent behaviors across Matterport3D, HM3D, and Habitat, encompassing Panoramic View Last-Mile Navigation, Top-down View Global Planning, 3D Dollhouse Navigation, and Simultaneous Localization and Generation (SLAG), systematically stratified across 24 hard-level configurations varying environmental complexity (1-floor vs. multi-floor), view fidelity, trajectory distance, and destination specifications. Domain 3 (Physical Commonsense) probes intuitive physics across 50 atomic interactions (VideoPhy solid-solid, solid-fluid, fluid-fluid dynamics) and compositional sports activities (ballet, skiing, diving, swimming). This multi-tier architecture ensures comprehensive profiling across physical, logical, 2D spatial, 3D spatial, and temporal capabilities.

2. Process-Aware Chain-of-Frame Tracking: Enforcing valid intermediate transitions beyond terminal frames Conventional video generation benchmarks assess only the final frame or the global distributional distance (such as FVD), which fails to penalize models that reach a target through physically impossible or logically invalid shortcuts. MMGR introduces the Chain-of-Frame (CoF) criterion: every intermediate frame must represent an admissible, physically consistent transition toward the goal. In symbolic tasks, the framework tracks per-frame puzzle states to identify Action Reflection (AR)โ€”quantifying exploratory backtracking and active self-correction versus blind feedforward output. In navigation tasks, it continuously tracks agent identity and camera pose, explicitly flagging pathological behaviors such as agent duplication ("doppelgรคnger" hallucinations), clipping through walls, and scene morphing. This prevents generative models from receiving credit for ungrounded visual transitions that mimic an outcome while violating the process.

3. Hybrid Verification & Metric Disentanglement: Dual-track deterministic computation and rubrics-based VLM evaluation To eliminate the ambiguity and bias inherent in monolithic model-based evaluation, MMGR implements an objective dual-track scoring architecture. For tasks with mathematical ground truth, the pipeline uses deterministic, model-free computer vision operations: Maze solving is evaluated via pixel-level skeletonization, color segmentation, and morphological dilation to compute exact path Intersection over Union (IoU), wall-crossing ratios (Cross Wall), and structural deviations (Maze Changed); Sudoku solving relies on PaddleOCR digit extraction across frames to calculate constraint violation rates (Constraint Violation) and immutable clue alterations (Clues Changed). For open-vocabulary navigation and physical interactions, Gemini-2.5-Pro executes structured multi-criteria rubrics, complemented by Geo-Align (VGGT 3D trajectory reconstruction computing SPL, ATE, and nDTW) and View-Syn (normalized FVD/FID, SSIM, LPIPS), establishing reproducible cross-modal evaluation.

4. Gated Primary Success Determination: Zero-tolerance logic against hallucinated shortcuts and constraint corruption Generative models frequently maximize surface-level goal achievement by actively modifying input constraints (e.g., erasing a maze wall, overwriting an immutable Sudoku clue, or redrawing a navigation goal directly beneath the agent). MMGR counters this with a multi-condition Boolean gated primary metric (Overall Success = 1). An output receives a passing score if and only if all fine-grained criteria hold simultaneously: Environmental Integrity (Maze Changed = 0, Clues Changed = 0, Destination Integrity = 1), Process Adherence (Cross Wall = 0, Constraint Violation = 0, Physics Validity = 1), and Outcome Correctness (Target Achievement = 1, Completion Accuracy = 1, Success Score = 1). If any constituent constraint is breached, the entire output is assigned a score of 0, neutralizing any incentive for models to cheat by hallucinating trivial solutions.

Benchmark & Evaluation Protocol

Zero-shot generative reasoning is systematically benchmarked across leading video models (Google Veo-3, OpenAI Sora-2, Wan-2.2), image diffusion models (Nano-banana, Nano-banana Pro, GPT-4o-image, GPT-image-1.5, Qwen-image), and advanced reasoning LLMs (Gemini-3-Flash, Gemini-3-Pro) on abstract reasoning tasks. To account for sampling variance, five independent rollouts are drawn per prompt instance using default inference parameters for closed-source APIs and recommended settings for open-source systems on local NVIDIA A100 GPUs, accumulating over 22,000 generated evaluation samples. No task-specific fine-tuning or post-hoc trajectory stitching is permitted, isolating the intrinsic zero-shot reasoning capabilities of foundation generative models.

Key Experimental Results

Main Results

Zero-shot reasoning performance of state-of-the-art generative models on MMGR under the primary gated success metric (%):

Task Domain Benchmark Task Veo-3 (Video) Sora-2 (Video) Wan-2.2 (Video) Nano-banana Pro (Image) GPT-4o-image (Image) GPT-image-1.5 (Image) Gemini-3-Pro (Text)
D1: Abstract Reasoning Maze Solving (Maze) 3.67 0.00 0.00 9.17 0.00 0.97 59.33
Sudoku Solving (Sudoku) 0.00 0.00 0.00 7.00 1.33 6.78 63.75
Visual Pattern Ind. (ARC) 4.80 11.67 0.15 28.07 0.00 12.72 โ€“
Visual Math Reasoning (Math) 10.61 10.56 0.00 72.69 26.24 25.52 74.14
D2: Embodied Navigation Last-Mile Nav. (Last-Mile) 60.00 0.00 14.17 75.83 0.00 55.84 โ€“
Top-down View Nav. (Top-down) 19.49 3.39 5.09 33.05 3.39 26.27 โ€“
3D Real-World Nav. (3D R.-W.) 22.50 0.00 24.17 85.00 13.33 77.92 โ€“
Sim. Loc. & Gen. (SLAG) 11.02 12.50 0.85 37.29 16.67 31.36 โ€“
D3: Physical Commonsense Physical Concepts 41.67 76.00 26.67 โ€“ โ€“ โ€“ โ€“
Sports Scenarios 60.00 64.00 21.33 โ€“ โ€“ โ€“ โ€“

Diagnostic & Ablation Results

Fine-grained deterministic pixel-level evaluation on the 2D Maze task (Depth-First Search generator) disentangles structural preservation, constraint adherence, and path completion across difficulty tiers:

Modality Evaluated Model Difficulty Tier Maze Changed (MC) โ†“ Cross Wall (CW) โ†“ Target Achiev. (TA) โ†‘ Action Reflec. (AR) โ†‘ Gated Overall โ†‘
Video Veo-3 Easy (3ร—3โ€“5ร—5) 71.00% 76.00% 33.50% 59.50% 10.50%
Medium (6ร—6โ€“9ร—9) 100.00% 96.48% 14.07% 43.22% 0.00%
Hard (10ร—10โ€“13ร—13) 100.00% 98.50% 10.00% 22.50% 0.00%
Sora-2 Easy (3ร—3โ€“5ร—5) 100.00% 97.50% 45.00% 100.00% 0.00%
Medium (6ร—6โ€“9ร—9) 100.00% 97.50% 60.00% 100.00% 0.00%
Hard (10ร—10โ€“13ร—13) 100.00% 95.00% 52.50% 60.00% 0.00%
Wan-2.2 Easy (3ร—3โ€“5ร—5) 2.50% 90.00% 25.00% 87.50% 0.00%
Hard (10ร—10โ€“13ร—13) 0.00% 97.50% 5.00% 67.50% 0.00%
Image Nano-banana Pro Easy (3ร—3โ€“5ร—5) 7.50% 42.50% 30.00% N/A 12.50%
Medium (6ร—6โ€“9ร—9) 0.00% 30.00% 37.50% N/A 30.00%
Hard (10ร—10โ€“13ร—13) 0.00% 50.00% 7.50% N/A 0.00%
GPT-4o-image All Tiers 100.00% 62.50%โ€“77.50% 0.00% N/A 0.00%
GPT-image-1.5 Easy (3ร—3โ€“5ร—5) 96.67% 87.50% 8.33% N/A 1.67%
Hard (10ร—10โ€“13ร—13) 1.67% 90.00% 5.83% N/A 0.00%

In the deterministic OCR evaluation of Sudoku puzzles, all three video generative models (Veo-3, Sora-2, Wan-2.2) scored 0.00% across all \(4\times4\) and \(9\times9\) configurations due to severe clue corruption (80โ€“100%) and rule violations. Furthermore, on full-scale \(9\times9\) grids, every generative model across both video and image modalities failed completely (0.00% overall success), exposing a universal combinatorial bottleneck.

Key Findings

  • The Divergence Between Visual Fluency and Symbolic Correctness: Video models achieve their highest accuracy on intuitive Physical Commonsense (Sora-2 reaching 76.00% on Physical Concepts and 64.00% on Sports), but collapse completely on symbolic tasks. Every video model achieves 0.00% on Sudoku, under 12% on ARC, and under 11% on Math.
  • The "Temporal Tax" on Spatial Navigation: In Embodied Navigation, image generators consistently outperform video generators despite producing only single static frames. Nano-banana Pro leads all four navigation tasks (e.g., 85.00% in 3D Real-World Nav vs. Veo-3's 22.50%). Synthesizing a temporal sequence imposes compounding consistency drift on agent identity, layout geometry, and goal state.
  • Outcome vs. Process Disconnect (The Illusion of Reasoning): On GSM8K math problems, Veo-3 achieves a 74.00% Outcome Success Rate but only a 12.00% Process Success Rate. The model frequently arrives at the correct answer through hallucinated visual morphing, bypassing valid deductive steps.
  • Automator Failure Modes in Spatial Grounding: Reliability analysis against human annotations reveals that while deterministic checks achieve 95.56%โ€“100% agreement, VLM evaluators exhibit low reliability on complex embodied tasks (dropping to 21.67%โ€“30.00% on Nano-banana Pro runs). VLM judges routinely overlook transient wall-clipping errors during high-velocity video frames, artificially inflating perceived model competence.

Highlights & Insights

  • From Surface Aesthetics to Constraint Verification: MMGR redefines generative evaluation by measuring whether synthesized media solves verifiable problem instances rather than relying on qualitative perception or checklist scoring.
  • Strictly Coupled Gated Logic: The gated scoring mechanism eliminates benchmark gaming by ensuring that target achievement gained via environment modification or clue alteration receives a score of zero.
  • Latent Cross-View Alignment Priors: Experiments on SLAG reveal that forcing models to align 3D perspective trajectories with 2D floor plans significantly improves Scene Consistency (Veo-3 rising to 57.89%), demonstrating that multimodal structural conditioning unlocks latent world-modeling capabilities that free-form generation suppresses.

Limitations & Future Work

  • Exclusion of Reasoning-Tuned Generation Models: MMGR assesses zero-shot capabilities of foundational generators; evaluating models explicitly fine-tuned with chain-of-thought visual trajectory supervision represents an important next step.
  • Open-Ended Ecological Dynamics: The benchmark focuses on deterministic, closed-world problem formulations, leaving partially observable multi-agent games and continuous control planning for future expansions.
  • Architectural Implications: Findings highlight the urgent need for hybrid architectures that decouple latent symbolic state tracking from continuous diffusion pixel rendering, preventing temporal drift from corrupting reasoning logic.
  • vs VBench / EvalCrafter: Prior video suites benchmark visual fidelity, motion smoothness, and text alignment without testing whether an output solves an underlying problem; MMGR enforces answer-verifiable, process-aware evaluation.
  • vs WISE / Commonsense-T2I: Prior image reasoning benchmarks focus on static semantic knowledge in 2D text-to-image synthesis; MMGR bridges 2D/3D spaces, temporal frame chains, and real-world embodied navigation.
  • vs VIPER / VBVR: Concurrent process-aware video benchmarks are confined to synthetic toy domains or video-only pipelines; MMGR provides a cross-modal framework comparing video, image, and language systems across complex embodied and abstract reasoning tasks.

Rating

  • Novelty: โญโญโญโญโญ Establishes the first unified cross-modal generative reasoning benchmark with process-aware chain-of-frame verification.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluates 8 frontier generative models over 1,853 instances with 22,000+ rollouts, extensive diagnostic ablations, and human-evaluator agreement audits.
  • Writing Quality: โญโญโญโญโญ Rigorous conceptual framing, clear mathematical and diagnostic metric definitions, and insightful failure mode analyses.
  • Value: โญโญโญโญโญ Delivers critical diagnostic guidance and benchmarks for developing verifiable world models and embodied generative agents.