AGE: Agentic Gaussian Editing in 3D Scenarios¶

Conference: ECCV 2026
Paper: Official paper page · Paper PDF
Area: Multi-Agent Systems / 3D Vision
Keywords: 3D Gaussian editing, multi-agent collaboration, spatially-aware memory, multi-turn interaction, backtracking

TL;DR¶

AGE coordinates existing 3D editing tools through planning, execution, reflection, and backtracking agents with shared spatial memory, achieving a CLIP text–image direction similarity of 0.192 versus DGE's 0.133 on a custom benchmark, although component ablations are needed to isolate the sources of improvement.

Background & Motivation¶

3D Gaussian Splatting provides an explicit scene representation that supports changes to appearance, object placement, and geometry. An editable representation, however, is not the same as an interactive editing system. Methods such as GaussianEditor and DGE usually process an instruction through a predetermined workflow, assuming that the desired change is sufficiently specified at the outset. A user who inserts an object, corrects its position, and later changes the scene's style creates a different problem: each request depends on objects, decisions, and constraints established earlier. Treating those requests independently can lose an accepted arrangement or alter unrelated content during a repair.

Spatial interpretation makes this dependence more demanding than ordinary dialogue history. Putting an object “to the right of the excavator” requires a reference object, a spatial interpretation, and an appropriate scale; “wrong side, make it half as large” additionally requires access to the preceding operation. A language model used only to parse the first instruction cannot carry those evolving constraints through every downstream tool. Giving a planner every low-level parameter is not an adequate substitute either, because it mixes task selection with tool-specific execution details. AGE therefore focuses on persistent coordination among specialized roles rather than introducing another monolithic Gaussian editing network.

The authors formulate editing as a sequential process whose inputs include the current scene, the current request, and an evolving memory bank. Its objectives encompass instruction following, rendering quality, and consistency with earlier turns, not just the plausibility of a single final image. Core idea: retain spatial constraints, execution history, and recoverable scene assets in shared memory so a planning–execution–reflection–backtracking loop can decide what to change next and where to restart, instead of repeatedly rerunning a fixed pipeline.

Method¶

Overall Architecture¶

AGE starts from an existing 3D Gaussian scene and a user instruction, optionally accompanied by a reference image or click-and-drag information. It produces an edited Gaussian scene that remains available for subsequent requests. Four agent roles coordinate this process: they are model-driven reasoning and orchestration roles, not four newly trained neural networks. Structured spatial memory supports perceive-then-plan reasoning, skill-based execution, and a reflection–backtracking loop that accepts an outcome, revises the plan, or retries a selected operation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Gaussian scene and request"] --> B["Structured spatial memory"]
    B --> C["Perceive-then-plan"]
    C --> D["Skill-based execution"]
    D --> E["Reflection–backtracking loop"]
    E -->|Accept| F["Updated scene and memory"]
    E -->|Re-plan| C
    E -->|Local retry| D
    F -->|Next turn| B

The memory node is not a fixed map computed once before editing: memory starts empty and is queried and updated by the different roles throughout execution. Reflection can also be initiated by an internal verification prompt rather than waiting for a user complaint. The state-update equations in Section 3 are visibly corrupted in the local text extraction, so this account follows the surrounding prose and Figure 2 instead of reconstructing unverifiable formulas.

Key Designs¶

1. Structured spatial memory: make earlier edits queryable and recoverable

A conversation transcript can preserve the statement “too large” without identifying the relevant asset version or its previous scale parameter. AGE separates memory into systematic spatial information, turn-level context, and a global metadata repository. Spatial information includes the planner's selected viewpoints and constraints summarized by the reflector, such as scene boundaries, relative object scales, and orientations. The executor consults these records so that a new tool invocation need not infer the coordinate relationships from scratch. Turn-level context combines summaries of complete interaction turns with fine-grained records of individual skill executions. The summaries locate earlier intentions, while the step records identify the particular operation that needs correction.

The global repository retains indexed links to Gaussian parameters and scene states across the trajectory, with assets such as rendered images and logs supporting inspection. Consequently, returning to an earlier point refers to an actual scene version rather than asking a generator to imagine the old result. This organization can preserve successful work while enabling selected erroneous steps to be re-executed. The main paper does not specify a complete storage schema, retrieval algorithm, or dependency-handling protocol for replay, however. It should therefore be understood as a state-management design, not as a formally verified transaction system.

2. Perceive-then-plan: select relevant views before selecting editing skills

The planner first performs dense scene perception and adaptively selects views relevant to the current request; the experiments retain eight views. The purpose is to reduce interference from unrelated background regions and focus reasoning on the target and its spatial references. Eight views do not, by themselves, establish complete 3D understanding. The planner then consults skill descriptions and produces an ordered sequence of atomic editing goals, such as locating a reference object before inserting another object relative to it.

A useful interface boundary is that the planner retrieves descriptions rather than all low-level skill parameters. It chooses capabilities and their order, while the executor handles coordinates, scales, and generation prompts. When a failed attempt triggers replanning, the planner also reads reflection thoughts from memory to avoid repeating the rejected strategy. Spatial grounding here relies on selected visual observations and organized geometric information; it is not evidence that the agent has learned a complete native representation of Gaussian geometry.

3. Skill-based execution: expose parameter constraints and repair knowledge together

The executor processes the plan step by step, turning each atomic instruction into parameters for a selected skill. It queries shared spatial information to maintain compatible object scales, orientations, and placement as the scene evolves. The skill library connects this reasoning layer to existing tools, including SAM for segmentation, FLUX for asset generation, and InstructPix2Pix for style transfer. AGE composes these capabilities rather than replacing their underlying editing algorithms. Each skill supplies a name and description, an argument schema and runner, parameter hints and examples, and dedicated correction guidance for the backtracker.

The Add example in Figure 3 makes the distinction concrete. Its arguments include a generation prompt, a three-dimensional placement position, and a uniform scale factor; reference-relative insertion requires the reference object to be located first. Guidance instructs the executor to preserve explicitly specified coordinates or scales rather than estimate them again. Correction guidance explains how the height axis works and how to retain an accepted position when only the object's size needs changing. A skill thus describes not only what operation is available, but also how its parameters can be repaired after failure. This improves controllability, yet the available tools still define the system's capability boundary: additional reflection cannot guarantee an edit that no underlying tool can produce.

4. Reflection–backtracking loop: identify the level of failure before restarting

The reflector compares the current scene with the user's objectives and can terminate after explicit user approval or a successful internal multimodal check. If a discrepancy remains, it distinguishes an unsuitable overall strategy from an error in a specific execution step. The former calls for replanning; the latter calls for the backtracker to select a re-entry point. The backtracker combines historical execution records with skill-specific correction guidance, produces concrete adjustments, and writes hints into memory for the next attempt. This prevents a mistaken strategy and a correct strategy with a bad placement parameter from being treated as identical reasons to restart everything.

Backtracking is valuable because it narrows the repair scope, not because it permits unlimited retries. If an inserted object has the wrong size, earlier placement and invocation records can support a targeted adjustment without arbitrarily changing the accepted arrangement. The paper illustrates such interactions but does not quantify maximum loop counts, unsuccessful termination rates, or competing backtracking policies. Its reference to an optimal re-entry point describes the agent's intended decision, not a search procedure with a demonstrated optimality guarantee.

A Worked Example¶

Figure 6 follows an editing session that uses persistent state rather than independent generations. The first request places an apple to the right of an excavator: the planner schedules Locate followed by Add, and the executor invokes them. The second request says the apple is on the wrong side and should be reduced to half its size. The reflector identifies a spatial correction, and the backtracker uses the preceding execution state to adjust insertion parameters rather than locating the entire scene again. A third request adds a bonsai to the excavator's left and applies a Van Gogh style; planning now inherits the calibrated spatial constraints while composing insertion and style editing. The next request switches to Fauvism, followed by user approval. This is a qualitative interaction trace reported in the paper, not a measurement of success probability at each turn.

Loss & Training¶

AGE does not introduce a joint loss for training the four roles, and the paper does not report end-to-end agent training. Its main contribution operates at inference time through orchestration, state management, and tool invocation. Section 4.1 assigns planning and reflection to Qwen3-VL-30B, and execution and backtracking to DeepSeek-R1-32B. It reports vLLM deployment, a generation temperature of 0.02, LangGraph orchestration, and execution phases on one NVIDIA A100 GPU. The main text does not report full interaction latency, peak memory, or per-task tool-call budgets. These model names follow the paper's implementation description; their exact release variants were not independently verified for this note.

Key Experimental Results¶

Main Results¶

The custom benchmark contains 57 scenarios: 20 common-instruction cases, 20 complex or ambiguous cases, 10 with multimodal auxiliary inputs, and seven multi-turn interactions (Section 4.1). The main tables do not separate those subsets or explain any subset-specific aggregation weights. Their overall scores should not be presented as results specifically on the multi-turn subset. The following selection from Table 1 covers semantic editing alignment and image aesthetics; higher is better for both metrics. It compares complete systems, not components under identical tool access and compute budgets.

Method	CLIP direction similarity Csim ↑	Image aesthetics IAA ↑
GaussianEditor	0.104	4.51
DGE	0.133	5.79
AGE	0.192	6.65

Csim assesses alignment between the text editing direction and the image editing direction. IAA uses an existing image-aesthetics assessment method and is not the same scale as the model-judge scores below. Table 1 additionally reports CLIP direction consistency Ccon of 0.907 for AGE and 0.892 for DGE as a complementary indication of multi-view editing consistency. The main text cites prior work for these inherited metrics rather than restating their exact calculation formulas.

Ablation Study¶

The available main paper contains no ablation removing memory, reflection, backtracking, or spatial perception, so it cannot establish which component contributes most. Instead, the table below presents a genuine complementary analysis: selected structural-preservation and instruction-completion scores from Table 2. Gemini 3.1 Pro judges rendered multi-view results on a 1–10 scale, with higher values better. Structural consistency concerns unedited regions, while completion concerns how fully the requested edits were realized. Neither dimension has the same units as CLIP similarity.

Method	Structural consistency of unedited regions ↑	Instruction completion ↑
GaussianEditor	5.24	5.70
DGE	7.39	5.23
AGE	8.90	7.17

As with Table 1, the main text does not provide subset-level results, repeated-run uncertainty, or validation of judge agreement with human raters. This analysis addresses aspects that CLIP may miss, but a model-judge score is neither an objective geometric error nor a replacement for a user study.

Key Findings¶

Relative to DGE, AGE improves Csim by 0.059 and IAA by 0.86, indicating gains in both semantic alignment and assessed image quality rather than a single metric alone.
DGE preserves structure better than GaussianEditor according to Table 2 but receives a lower completion score, illustrating that background preservation and fulfilling an edit are distinct objectives.
AGE's completion score is 7.17, not evidence of universal success; Section 4.3 identifies low-level tool limitations as a persistent bottleneck.
The results support the full system over the selected baselines, not an isolated attribution to memory, four-role specialization, or additional retries.

Highlights & Insights¶

History is an asset record, not just text. Turn summaries identify intent, while execution states and asset links support actual recovery of scene versions.
Repair knowledge belongs in the skill interface. Exposing correction rules alongside argument schemas helps convert verbal feedback into specific spatial parameter changes.
Replanning and local retries address different failures. Preserving accepted edits is particularly important in an ongoing creative session, where a completely new output may be less useful than a narrow repair.

Limitations & Future Work¶

Author-reported execution bottleneck: the backtracker may diagnose a problem correctly while the underlying generator still fails to produce an asset matching that feedback.
Author-proposed directions: improve specialized 3D skills and explore partial agent fine-tuning for a more direct understanding of Gaussian geometry rather than reliance on 2D visual proxies.
Reader assessment of causal evidence: without ablations or matched budgets, the effects of stronger tools, additional inference opportunities, and state management cannot be separated.
Reader assessment of long-horizon evidence: only seven benchmark scenarios are multi-turn, and the main paper lacks turn-by-turn degradation, recovery success rates, and long-sequence cost measurements.
Source boundary: the local cache contains the complete 19-page main paper and references, but not the appendix mentioned in Section 4.1. Detailed scenario lists and supplementary settings cannot be checked, and corrupted state equations have not been reconstructed here.

vs GaussianEditor: the evaluated baseline is Chen et al.'s Swift and Controllable 3D Editing with Gaussian Splatting (reference 7), not the other paper sharing the GaussianEditor name. AGE emphasizes persistent interaction and closed-loop repair rather than a single editing operator.
vs DGE: DGE performs direct Gaussian editing through consistent multi-view editing; AGE manages plans and repair above the tool layer, so this is not a like-for-like module replacement.
vs Vinedresser3D: the paper acknowledges prior agentic 3D editing and emphasizes memory and feedback as further distinctions. It does not provide a quantitative comparison with Vinedresser3D, so broad superiority over that work is not established.

Rating¶

Novelty: 3/5 — A task-specific integration of spatial memory, correction-aware skills, and backtracking, largely built from established agent and editing components.
Experimental Thoroughness: 2/5 — Two evaluation families and interaction examples are useful, but ablations, budget controls, human validation, and subset statistics are missing.
Writing Quality: 3/5 — Role responsibilities and memory organization are clear; implementation details and stopping policies remain underspecified.
Value: 4/5 — Useful guidance for stateful interactive 3D editing, with practical performance still constrained by tool precision and invocation cost.