When Does Visual Generation Help Visual Understanding in Unified Multimodal Models?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zyb1029/VGAU-Diag
Area: Multimodal VLM
Keywords: Unified Multimodal Models, Vision Generation-Assisted Understanding, Visual Chain-of-Thought, Spatial Planning Reasoning, Evaluation Benchmark
TL;DR¶
This paper introduces VGAU-Diag, a fine-grained evaluation framework across six verifiable planning task families with oracle-assisted reference protocols, revealing that the primary bottleneck in vision generation-assisted understanding lies in visual understanding rather than visual generation, and correcting protocol-induced optimal-step prior leakage in Vision-CoT.
Background & Motivation¶
Recent advances in unified multimodal models (UMMs) have enabled a single architecture to handle both visual understanding and visual generation. This integration has fueled substantial interest in exploring whether self-generated visual aidsโvia paradigms such as Vision Chain-of-Thought (VCoT), Generate-then-Answer (GtA), or direct Vision-Reasoning (VR)โcan serve as intermediate visual reasoning processes to bolster complex visual understanding. However, existing empirical studies offer mixed and contradictory evidence regarding whether visual generation genuinely and reliably enhances understanding.
This inconsistency stems from three fundamental limitations in existing benchmarks. First, current evaluations predominantly rely on multiple-choice formats and pool instances across diverse difficulty levels into a single aggregate metric, making it impossible to disentangle true reasoning capability improvements from stochastic guessing on easy samples. Second, each task family is typically benchmarked under a single visual reasoning paradigm without controlled, apples-to-apples comparisons across alternative paradigms, confounding visual generation benefits with paradigm-specific prompt heuristics. Third, prior studies treat multimodal reasoning as a closed-loop black box where visual generation and visual understanding are tightly entangled, leaving it unclear whether failures arise from imperfect image generation fidelity or the model's inability to comprehend visual evidence.
To resolve this ambiguity, evaluation must transition from coarse task success checking to fine-grained causal diagnosis capable of isolating generation fidelity from comprehension capacity. Core idea: develop VGAU-Diag, a fine-grained evaluation framework across six verifiable visual planning tasks that incorporates difficulty stratification, unified cross-paradigm evaluation, and oracle-assisted reference protocols to decouple visual generation from understanding, pinpoint understanding-side bottlenecks, and rectify optimal-step prior leakage.
Method¶
Overall Architecture¶
VGAU-Diag benchmarks vision generation-assisted understanding on 2D visual planning tasks where spatial state transitions can be deterministically verified: Maze, Sokoban, Path, Onet, Parking, and Klotski. The pipeline comprises difficulty-aware sample stratification, cross-paradigm closed-loop execution, oracle-assisted reference diagnostics, and solver-based trajectory verification.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Task Input<br/>6 verifiable 2D spatial planning problems & goal specifications"] --> B["Difficulty Stratification & Verifiable Benchmark<br/>Rigorous Easy / Medium / Hard splits by optimal solution horizon"]
B --> C["Oracle-Assisted Reference Protocols<br/>Injecting ground-truth visual aids to decouple fidelity from understanding"]
C --> D["Cross-Paradigm Evaluation & Prior Leakage Fix<br/>Unified GtA / VCoT / VR comparison with removal of optimal-step leakage"]
D --> E["Solver-Based Execution & Verification<br/>Strict trajectory replay checking feasible and optimal solution rates"]
Key Designs¶
1. Difficulty Stratification & Verifiable Benchmark: Eliminating Multiple-Choice Guessing via Action Replay
Unlike existing benchmarks that use multiple-choice questions or step-level token overlap, VGAU-Diag requires models to generate complete, executable action sequences formatted inside an <ANSWER_JSON> block. Each output trajectory is replayed on a programmatic environment solver. Any illegal moveโsuch as running into a wall, pushing a box against an obstacle, collision, or leaving grid boundariesโimmediately flags the instance as invalid. Success is strictly split into optimal solution accuracy (\(reach\_optimal\), matching the minimal BFS step count) and feasible solution accuracy (\(reach\_feasible\), reaching the goal along any valid trajectory). Instances are stratified into Easy, Medium, and Hard sets based on optimal path length, which is shown to align with topological graph complexity measures including walkable cell counts, branch nodes, and corridor nodes.
2. Oracle-Assisted Reference Protocols: Decoupling Generation Fidelity from Visual Understanding Bottlenecks
In an end-to-end closed loop, a failed plan cannot differentiate whether the generative backbone produced degraded visual features or the visual encoder failed to interpret accurate spatial layouts. To establish rigorous causal diagnosis, the framework constructs four controlled reference conditions: text-only reasoning without auxiliary visuals (Text-CoT), providing an external ground-truth grid and coordinate overlay (Oracle-GtA), providing step-by-step ground-truth visual transitions following each action (Oracle-VCoT), and supplying grid-annotated step-by-step ground-truth transitions (Oracle-Aug-VCoT). By comparing state-of-the-art proprietary VLMs against open-source UMMs under identical oracle conditions, the framework isolates the exact contribution of visual understanding from visual generation.
3. Cross-Paradigm Evaluation & Prior Leakage Fix: Unifying Paradigms and Removing Step-Count Leakage
VGAU-Diag provides a unified implementation of three self-generation paradigms: Self-GtA (first generating an annotated grid visual, then reasoning in text), Self-VCoT (interleaved action prediction and intermediate state image generation), and Self-VR (directly drawing the solution trajectory as an image). Crucially, the authors identify a widespread evaluation flaw in prior VCoT benchmarks: previous protocols extracted the optimal step count \(K\) from the ground-truth solution and forced the model to interact for exactly \(K\) rounds. This granted the model an artificial prior on solution length, masking step-estimation failures. The proposed Self-VCoT-fixed protocol eliminates this prior by requiring the model to estimate remaining steps after each round, terminating only when the model predicts zero remaining steps.
A Worked Example¶
Consider the Parking sliding-block task: the model is presented with an 8ร8 board and must slide vehicles along their fixed horizontal/vertical orientations to let the red car R reach the right exit. Under Text-CoT, the model must mentally track all vehicle coordinates in text. Under Self-GtA, the model first generates an image with red grid lines and numerical coordinate labels, then outputs the JSON sequence. In previous flawed Self-VCoT setups, if the optimal solution took 3 steps, the test framework invoked the model for exactly 3 steps, effectively leaking the horizon. Under Self-VCoT-fixed, the model emits a move (e.g., {"car": "A", "direction": "up"}) alongside its generated state image and estimates remaining moves; if its state representation drifts, error accumulates, and it terminates prematurely or wanders indefinitely, revealing its true planning capacity.
Key Experimental Results¶
Main Results¶
Evaluating proprietary VLMs, open-source VLMs, and unified multimodal models (UMMs) on task-level optimal solution accuracy (%) on Easy subsets yields the following comparison:
| Model Category | Representative Model | Text-CoT (Maze / Parking) | Oracle-VCoT (Maze / Parking) | Oracle-GtA (Maze / Parking) | Oracle-Aug-VCoT (Maze / Parking) |
|---|---|---|---|---|---|
| Open-source VLM | Qwen2.5-VL-7B-Instruct | 0.0 / 15.0 | 10.0 / 0.0 | 0.0 / 5.0 | 15.0 / 0.0 |
| Open-source VLM | Qwen3.5-9B | 25.0 / 25.0 | 25.0 / 60.0 | 90.0 / 80.0 | 50.0 / 70.0 |
| Closed-source VLM | Qwen3.6-Plus (reasoning) | 70.0 / 70.0 | 50.0 / 75.0 | 100.0 / 100.0 | 100.0 / 90.0 |
| Closed-source VLM | Doubao-Seed-2.0-pro | 80.0 / 35.0 | 100.0 / 95.0 | 100.0 / 100.0 | 100.0 / 95.0 |
| Closed-source VLM | GPT-5.4 (reasoning) | 100.0 / 85.0 | 100.0 / 95.0 | 100.0 / 95.0 | 100.0 / 95.0 |
| Unified Model | MammothModa2 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 |
| Unified Model | STAR-7B | 0.0 / 5.0 | 10.0 / 5.0 | 10.0 / 10.0 | 10.0 / 10.0 |
| Unified Model | Janus-Pro | 0.0 / 15.0 | 0.0 / 0.0 | 5.0 / 25.0 | 0.0 / 0.0 |
| Unified Model | LLaDA2.0-Uni | 5.0 / 5.0 | 10.0 / 35.0 | 0.0 / 0.0 | 5.0 / 60.0 |
| Unified Model | Show-o2 | 5.0 / 0.0 | 15.0 / 70.0 | 11.6 / 10.0 | 10.0 / 0.0 |
| Unified Model | LongCat-Next | 10.0 / 15.0 | 13.3 / 10.0 | 60.0 / 10.0 | 5.0 / 61.6 |
| Unified Model | Nano Banana 2 | 35.0 / 10.0 | 20.0 / 0.0 | 85.0 / 70.0 | 45.0 / 35.0 |
Across 48 task-level comparisons among 8 open-source UMMs across all 6 tasks, only 12 cases fall below their Oracle-GtA upper reference, proving that providing flawless oracle visual aids does not rescue performance for most UMMs.
Ablation Study & Prior Leakage Diagnosis¶
Comparing Self-VCoT before and after removing optimal-step prior leakage reveals massive drops in reported performance across UMMs on Easy instances:
| Evaluation Setting / Model | Maze (Leaked โ Fixed) | Sokoban (Leaked โ Fixed) | Path (Leaked โ Fixed) | Parking (Leaked โ Fixed) |
|---|---|---|---|---|
| LLaDA2.0-Uni (Self-VCoT) | 10.0 โ 0.0 (-10.0) | 10.0 โ 0.0 (-10.0) | 5.0 โ 0.0 (-5.0) | 40.0 โ 0.0 (-40.0) |
| Show-o2 (Self-VCoT) | 11.6 โ 0.0 (-11.6) | 10.0 โ 0.0 (-10.0) | 6.6 โ 0.0 (-6.6) | 56.6 โ 0.0 (-56.6) |
| BAGEL (Self-VCoT) | 6.6 โ 5.0 (-1.6) | 13.3 โ 0.0 (-13.3) | 5.0 โ 0.0 (-5.0) | 0.0 โ 0.0 (+0.0) |
| LongCat-Next (Self-VCoT) | 10.0 โ 0.0 (-10.0) | 5.0 โ 5.0 (+0.0) | 1.6 โ 0.0 (-1.6) | 5.0 โ 0.0 (-5.0) |
Under controlled diagnostic visual conditions across the Qwen model scaling suite: - Stage 1: Interference: Replacing visual aids with Gaussian noise degrades Qwen3.5-9B Self-GtA from 25.0% to 8.3% (-16.7%). - Stage 2: Misleading Guidance: Providing plausible but incorrect visual aids (e.g., mismatched task images) inflicts even steeper penalties, dropping Qwen3.5-9B to 5.0% (-20.0%) and Qwen3.6-Plus from 70.0% to 15.0% (-55.0%). - Stage 3: Assistance: High-fidelity task-aligned visuals (Nano Banana 2 or Oracle) combined with strong understanding scale (27B/Plus) finally achieve robust positive gains (83.3% to 100.0%).
Key Findings¶
- Difficulty stratification is critical: Observed gains of visual generation in UMMs are concentrated almost entirely on easy, short-horizon instances and evaporate on medium/hard problems. Coarse aggregate scoring hides this failure boundary.
- Visual understanding is the dominant bottleneck: Even when provided with pristine oracle visual aids, current UMMs achieve near-zero success on complex tasks, demonstrating that the failure stems from comprehension deficits rather than visual generation fidelity.
- Three-stage non-monotonic transition: As visual generation fidelity improves, model performance does not scale monotonically; it moves from noise-induced distraction, to severe deception by plausible but incorrect visuals, before reaching helpful assistance at high fidelity.
- Targeted auxiliary generation outperforms iterative step-wise generation: After removing step-count leakage, iterative step-wise generation (VCoT) suffers catastrophically from compounding errors. In contrast, targeted single-step visual augmentation (GtA) provides far more reliable utility.
Highlights & Insights¶
- Refuting the "poor generation causes poor reasoning" assumption: Through oracle decoupling experiments, the paper convincingly demonstrates that current UMMs struggle to utilize even perfect visual aids, showing that understanding capability is the true gating factor.
- Exposing and fixing widespread evaluation leakage: Identifying optimal-step leakage in prior VCoT literature rectifies an ungrounded community belief that iterative step-by-step visual generation inherently boosts reasoning.
- Revealing the deceptive harm of plausible incorrect images: Plausible hallucinations deceive models more severely than pure noise, because vision-language decoders actively trust and ground reasoning upon false visual cues.
Limitations & Future Work¶
- Author-acknowledged limitations: Tasks are currently confined to synthetic 2D grid environments; while offering deterministic solvers and ground truth, they do not encompass continuous 3D scenes, physical dynamics, or natural visual clutter.
- Promising directions: Extending VGAU-Diag to embodied manipulation and real-world robot navigation simulators; developing adaptive GtA mechanisms where models dynamically invoke visual coordinate generation only when spatial confidence is low.
Related Work & Insights¶
- vs UniG2U-Bench & Uni-MMMU: These benchmarks claimed widespread VCoT gains based on multiple-choice or step-level metrics; this work shows that such gains were largely artifacts of optimal-step leakage and partial matching metrics.
- vs MME-Unify & RealUnify: While existing suites aggregate diverse macroscopic scores, VGAU-Diag focuses on verifiable spatial planning and introduces causal oracle intervention to dissect the generation-understanding loop.
Rating¶
- Novelty: โญโญโญโญโญ Decouples generation and understanding via clean oracle protocols and uncovers critical protocol-level leakage.
- Experimental Thoroughness: โญโญโญโญโญ Evaluates 18 models across 6 task families with difficulty splits and controlled visual fidelity levels.
- Writing Quality: โญโญโญโญโญ Rigorous methodology, crisp problem definition, and well-structured insights.
- Value: โญโญโญโญโญ Provides crucial foundational clarity for unified multimodal architecture design and fair benchmark evaluation.