Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning¶
Conference: ECCV2026
Paper: ECCV Paper
Authors: Shaofeng Yin, Jiaxin Ge, Zora Zhiruo Wang, Chenyang Wang, Xiuyu Li, Michael J. Black, Trevor Darrell, Angjoo Kanazawa, Haiwen Feng
Project: VIGA
Area: VLM Reasoning
Keywords: inverse graphics, interleaved multimodal reasoning, visual verification, semantic skills, sliding-window memory
TL;DR¶
VIGA repeatedly writes code, renders, actively inspects, and edits, using semantic skills and bounded historical memory to maintain editable scenes without training, and reports a 124.70% average improvement over one-shot generation on BlenderBench.
Background & Motivation¶
Recovering a scene from an image means more than generating similar pixels: it requires a program that a graphics engine can execute, modify, and render again. Camera extrinsics, object placement, materials, and lighting jointly determine the image. Syntactically valid code does not establish that a ball rests on a table without intersecting it, or that a light illuminates the intended region. Differentiable inverse graphics supports local optimization from pixel errors but can become trapped when the initial viewpoint is far from the target. One-shot vision-language model (VLM) code generation supplies semantic priors but lacks an output-verification loop.
The difficult step is translating a visible discrepancy into the parameter that needs changing. The target may be occluded or outside the current view, while repeated blind edits can overwrite earlier progress. An agent therefore needs both active viewpoints for gathering evidence and a record of recent attempts and their consequences. The paper borrows the motivation of local visual verification, but does not attach a differentiable renderer or gradient descent to the VLM.
Core Idea: use the graphics engine as a reasoning environment in which symbolic programs and rendered images cross-verify each other, semantic skills separate observation from modification, and sliding-window memory connects successive attempts.
Method¶
Overall Architecture¶
The input is a target specification, such as a reference image. Generation starts from an empty scene, whereas editing starts from an existing scene. The output is an executable scene program and its rendered result, not an image that can only be viewed. The code-execution and visual-feedback loop advances the task, semantic skill interfaces support controlled engine interaction, and sliding-window memory carries current evidence into the next iteration.
The same interaction pattern supports Blender scenes and, with a different execution environment, PowerPoint generation. The 4D examples additionally inspect physical motion through the timeline. Generality here means a reusable control paradigm, not the absence of task-appropriate execution tools.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Target["Target specification<br/>and initial scene"] --> Loop["Code-execution and<br/>visual-feedback loop"]
Loop --> Skills["Semantic skill interfaces<br/>Execute, render, inspect"]
Skills --> Memory["Sliding-window memory<br/>Recent attempts and evidence"]
Memory -->|Discrepancies remain| Loop
Memory -->|Stop decision next iteration| Output["Executable program<br/>and rendered result"]
Key Designs¶
1. Code-execution and visual-feedback loop: make a program's consequences inform the next decision
Each iteration in Algorithm 1 begins with a plan conditioned on the target and history. If the plan invokes end_process, the loop stops; otherwise the agent generates and executes a program. The resulting render is not merely a delivery screenshot but an observation that must be analyzed. Comparing it with the target produces actionable natural-language feedback for the next program revision. A maximum iteration count \(T\) also bounds the loop.
Inspection is not restricted to repeatedly viewing a fixed overview. In Figure 3, the agent first focuses on a character hidden among scene clutter and then zooms in before recommending a smaller belly parameter. This separates failure to locate the target from an actual shape mismatch, reducing premature edits to the wrong object. Figure 4 shows how discrepancies in mouth shape, eye opening, and lighting color correspond to specific code parameters.
Unlike ordinary coding-agent workflows, successful compilation or execution is only an intermediate condition. The agent must also inspect whether the running program produces the intended geometry. Equations (1) and (2) in the paper express conditional program and feedback generation, not a new differentiable objective. Corrections arise from VLM judgments and tool interaction rather than back-propagation through a pixel loss.
2. Semantic skill interfaces: permit repeated observation while reserving persistent edits for explicit actions
Table 1 divides the interfaces into State Observation and State Modification. Observation includes spatial navigation through set_camera and investigate, temporal inspection through set_keyframe, structural queries through get_scene_info and set_visibility, and canonical-view and target-bounding-box diagnostics through initialize_view. Under the framework's contract, these inspect without committing persistent edits to the target scene. Inspection cameras and visibility can change, so read-only should not be interpreted as an entirely unchanged display state.
Modification uses init_scene and get_better_assets to instantiate, retrieve, or generate assets, execute_code to apply programs, undo_action to revert mistakes, and end_process to finish the trajectory. The agent can still express precise parameters without repeatedly assembling brittle low-level graphics calls for every inspection. Skills reduce tool-use complexity; they do not remove dependencies on model judgments, asset sources, or engine configuration.
For 3D reconstruction, asset tools supply external meshes and the loop subsequently adjusts placement, materials, and lighting. The dynamic examples in Figure 8 obtain SAM-3D assets through init_scene or construct objects from primitives. Programs set physical properties such as mass and collision shapes. Simulation consistency comes from the engine, not an additionally trained physical-prediction network.
3. Sliding-window memory: retain evidence from recent failures without carrying the entire history
Keeping only the latest screenshot can make the model oscillate between incorrect parameters, while retaining unlimited history increases multimodal context cost and interference. Section 3.2 describes memory containing plans, feedback, programs, and rendered states, retaining the most recent \(L\) iterations. Algorithm 1 abbreviates the update as follows. Its set-style notation is inherited from the algorithm and should be understood as appending to a trajectory and retaining its tail, not an unordered set operation:
Here \(p_t\) is the current program and \(f_t\) is visual feedback. The algorithm explicitly lists only these two elements, whereas the surrounding text describes a multimodal trajectory; the shorthand does not establish that images are absent from memory. The latest complete program carries previously completed geometric edits, so dropping early reasoning need not remove scene objects. In Figure 5, the unbounded-history trajectory later loses existing objects and spatial relations, whereas the windowed trajectory continues constructing a coherent scene.
The authors call this truncation lossless, but that claim depends on the latest program fully preserving successful edits. Code can retain an object without retaining why an earlier alternative was rejected. This is not a proof that arbitrary historical information survives truncation.
A Worked Example¶
Figure 2 and Section 3.2 illustrate memory using a basketball and a table. Previous height parameters of 0.8 and 0.7 produce different visual discrepancies; the agent compares those attempts and renders before proposing 0.75. The cache does not specify physical units for these parameters, so they should not be reported as meters, nor should the example be described as guaranteed binary search.
After adjusting height, the agent can modify the ball's horizontal position to match the target. Observation tools verify spatial relations, and execution tools update parameters. The next iteration receives the modified scene and recent evidence instead of making a fresh guess. The example explains how history supports successive corrections, not guaranteed convergence within three iterations.
Loss & Training¶
VIGA is a zero-shot, training-free framework with no newly introduced training loss, fine-tuning dataset, or optimizer. PL, N-CLIP, and VLM Score are evaluation measures, not a joint objective minimized during agent training.
Iterative comparisons use best-of-\(N\): the agent produces \(N\) candidate edits per step and selects one for the next iteration. Tables 2 and 3 compare \(N=1\) and \(N=4\). This is different from selecting the best final result among four independent runs. The cached main text does not specify numerical values for window length \(L\) or iteration limit \(T\), or fully document the candidate selector and VLM scoring prompt; reproduction requires further implementation checks.
Key Experimental Results¶
Main Results¶
BlenderBench contains 27 tasks across camera adjustment, fixed-view editing, and exploratory editing. PL is Photometric Loss and N-CLIP is the measure called Negative-CLIP Score; lower is better for both. Higher VLM Score is better, with the stated goal of reducing stylistic domain-gap interference in structural evaluation. The cache does not provide complete formulas, scaling conventions, or the VLM Score range. Results below retain the original score scales without labeling them as percentages or physical units.
The following selection from Table 2 compares GPT-4o under best-of-4. The one-shot baseline itself does not use best-of-4.
| Track and metric | One-Shot | BlenderAlchemy best-of-4 | VIGA best-of-4 |
|---|---|---|---|
| Camera adjustment PL, lower is better | 48.16 | 14.50 | 5.47 |
| Camera adjustment N-CLIP, lower is better | 64.17 | 19.57 | 6.10 |
| Camera adjustment VLM Score, higher is better | 0.58 | 1.75 | 3.25 |
| Fixed-view editing PL, lower is better | 7.36 | 1.95 | 2.94 |
| Fixed-view editing VLM Score, higher is better | 2.75 | 3.53 | 3.83 |
| Exploratory editing VLM Score, higher is better | 0.25 | 0.56 | 1.61 |
The exploratory-editing score of 1.61 is a 187.50% relative improvement over BlenderAlchemy's 0.56. Fixed-view editing, however, has worse PL but better VLM Score. Superiority therefore depends on the metric, and VIGA does not win every individual comparison. The text reports average relative improvements across settings of 124.70% on BlenderBench and 35.32% on BlenderGym; neither is a success rate or a percentage-point gain.
Ablation Study¶
Figure 5 provides a qualitative sliding-window ablation without a separate numerical table. The following budget analysis comes from Table 2 and should not be mistaken for a controlled memory-removal ablation.
| Model and metric | VIGA best-of-1 | VIGA best-of-4 | Analysis |
|---|---|---|---|
| GPT-4o, camera adjustment VLM Score | 1.44 | 3.25 | More candidates substantially help viewpoint search |
| GPT-4o, fixed-view editing VLM Score | 3.58 | 3.83 | Smaller gain |
| GPT-4o, exploratory editing VLM Score | 1.53 | 1.61 | Limited gain from additional search |
| Qwen3-VL-8B, exploratory editing VLM Score | 2.25 | 2.02 | More candidates do not guarantee improvement |
For cross-task generalization, the following two clearly readable model groups are taken from Table 4 on SlideBench. Exec. retains the original fractional representation; Quality and Overall retain the table's score scales.
| Model and setting | Exec. | Quality | Overall |
|---|---|---|---|
| GPT-4o One-Shot | 0.92 | 60.7 | 55.8 |
| GPT-4o VIGA | 0.95 | 61.4 | 58.3 |
| Gemini-2.5 One-Shot | 0.61 | 67.9 | 41.4 |
| Gemini-2.5 VIGA | 0.69 | 70.9 | 48.9 |
Key Findings¶
- Execution reliability and visual quality are distinct: GPT-4o's SlideBench execution rate increases from 0.92 to 0.95, while Quality only increases from 60.7 to 61.4. Execution improvements should not all be attributed to better design ability.
- Search-budget benefits depend on the model and task. Qwen3-VL-8B's exploratory-editing results suggest candidate selection and state maintenance can remain bottlenecks.
- Table 4 labels one baseline Claude-3.5, whereas the text calls it Claude-Sonnet-4; Qwen labels also vary between Qwen3-8B and Qwen3-VL-8B. This note does not silently reconcile those versions or use the ambiguously named rows for precise model comparisons.
Highlights & Insights¶
- Code serves both as an action representation and as a scene-state carrier. It expresses precise local changes while allowing completed geometry to survive context truncation.
- Active observation addresses insufficient evidence rather than merely extending reasoning. Locating and inspecting a target before changing its parameters is more targeted than repeatedly explaining the same inadequate screenshot.
- Separating observation from persistent modification can transfer to chart, document, or CAD editing. Such transfer requires suitable rendering and rollback capabilities, not just a replacement prompt.
Limitations & Future Work¶
- The authors acknowledge limitations from the underlying VLM's spatial perception and context capacity on extremely long sequences. Iteration alone cannot repair every geometric judgment error.
- BlenderBench is limited to 27 tasks, and 4D physical interaction is primarily demonstrated qualitatively. An engine simulates configured parameters consistently; that does not establish correct recovery of real mass, collision properties, or dynamics from one image.
- The cache does not fully document scoring formulas, evaluator prompts, window and iteration settings, or runtime cost, and it lacks a quantitative isolated skill ablation. Component-level causality and computational value are consequently difficult to separate.
- Future work could verify program-state completeness, retain selected long-term constraints, and compare methods under equal tool-call budgets. These are suggestions from this note, not modules already implemented in the paper.
Related Work & Insights¶
- vs BlenderAlchemy: the paper treats it as a memory-less iterative baseline and compares under matched best-of-\(N\) settings. VIGA emphasizes cross-step state and active verification, but a full-system comparison does not isolate memory as the sole source of improvement.
- vs differentiable inverse graphics: those methods refine parameters through differentiable rendering and gradients, whereas VIGA uses visual feedback to drive discrete program edits. Both follow analysis by synthesis, but this paper does not implement a hybrid gradient optimizer.
- vs visual-programming question answering and one-shot scene generation: the former typically executes a program to obtain an answer, and the latter terminates after producing scene code. VIGA feeds renders back into subsequent iterations, enabling correction during generation.
Rating¶
- Novelty: 4/5, unifies visual verification, semantic skills, and bounded memory rather than introducing a new foundation model.
- Experimental Thoroughness: 3/5, covers multiple tasks and models but lacks some reproduction details, isolated ablations, and cost analysis.
- Writing Quality: 4/5, clearly illustrates the loop and examples, although model naming and several evaluation definitions need clarification.
- Value: 4/5, offers practical insights for editable content generation and graphics-tool agents.