ReDesign: Recovering Editable Design Structures from Raster Images via Agentic Decomposition¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Raster-to-Vector, Editable Design Reconstruction, Agentic Tree Decomposition, Graceful Verification, Figma Edit Replay Benchmark
TL;DR¶
ReDesign casts the recovery of editable design structures from raster images into a progressive layer tree expansion managed by a VLM controller over heterogeneous tools, equipped with local graceful verification to accept, prune, or retry branches, achieving state-of-the-art editability and a 7.1× parallel speedup.
Background & Motivation¶
In modern graphic and interface design workflows, practitioners routinely need to adapt a single asset across diverse media platforms, repurpose marketing layouts for new campaigns, and make targeted adjustments for accessibility. In everyday engineering handoffs and real-world asset management, however, vast quantities of legacy designs exist exclusively as flattened raster screenshots or exported bitmap files. When designers need to modify a specific text heading, tweak a theme color, or adjust the spatial flow, they are forced to manually recreate the design from scratch in tools such as Figma or Adobe Illustrator. This manual redrawing process is tedious, error-prone, and dramatically slows turnaround times. While recent diffusion-based image editing models can synthesize plausible single-step modifications, pure raster pixel editing fundamentally lacks explicit object identities, vector contours, and typography parameters, frequently causing unwanted global warping, aspect-ratio drift, and pixel-bleeding artifacts under multi-step edits.
Recovering an editable design file from a raster image represents a severe inverse problem. The underlying rasterization process is an intrinsically lossy, many-to-one mapping that collapses layered vector geometry, font styling, grouping relationships, and z-order stacking into a flat array of color values. Existing approaches typically tackle isolated sub-problems—such as optical character recognition (OCR), layered image decomposition, or closed-contour vector tracing. However, naively chaining these specialized models into an end-to-end system produces fragile serial pipelines where early segmentation or detection errors cascade down the execution trajectory. Furthermore, traditional agentic systems typically rely on terminal verification at the very end of generation; once a flaw is detected, the entire pipeline must restart from scratch, resulting in prohibitive computational waste and low success rates.
This paper tackles the challenge by aligning the reconstruction process with the intrinsic data representation of modern design systems: editable designs are naturally organized as hierarchical layer trees. By formalizing reconstruction as an iterative tree expansion, high-level semantic layouts can be disentangled from fine-grained atomic primitives, while scoping verification locally to individual parent-child proposals. Core idea: formulate raster-to-editable reconstruction as a structured tree expansion driven by a VLM controller, where each node expansion invokes specialized multimodal tools and undergoes local graceful verification (accept, prune, retry) to isolate failures and enable massively parallel, high-fidelity recovery of editable design hierarchies.
Method¶
Overall Architecture¶
The core architecture of ReDesign operates around the incremental growth of a Partial Reconstruction Tree. Initialized with the full raster image as the root node, a VLM controller iteratively selects active frontier nodes to expand based on their localized image crops and lineal execution history. For each selected node, the controller chooses an action from a discrete action space encapsulating specialized multimodal toolchains. Before committing newly generated children to the tree, a lightweight graceful verifier evaluates their physical coverage and spatial boundaries to immediately accept valid branches, prune redundant hallucinations, or request retries. Valid children continue to expand recursively until all frontier nodes reach atomic editable leaves (SVG paths, editable text layers with typography parameters, or localized raster images with layout coordinates).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raster Image Input (Canvas Root)"] --> B["Structured Tree Expansion<br/>VLM controller selects frontier nodes using lineal history"]
B --> C["Modular Action Space & Heterogeneous Tools<br/>Text extraction / Layer decomposition / CCL / Object segmentation"]
C --> D["Graceful Verification & Local Repair<br/>Coverage and boundary validation: accept / prune / retry"]
D -->|Non-atomic leaves| B
D -->|Passed validation| E["Leaf Parameterization & Vectorization<br/>SVG path fitting / Font attribute prediction / Raster placement"]
E --> F["Editable Layer Hierarchy (Figma / JSON Representation)"]
Key Designs¶
1. Structured Tree Expansion: Top-Down Topological Growth with Lineal Memory Decoupling
To eliminate the exponential search spaces and fragile trajectories characteristic of linear tool-using agents, ReDesign enforces that the reconstruction mirrors the hierarchical structure of a design document. Decompositions progress systematically from coarse layout groupings to fine-grained visual primitives. The VLM controller makes decisions conditioned on the node's local rendered patch and its "Lineal Memory." Crucially, lineal memory records only the ancestry trajectory along the path from the root (historical tool choices, hyperparameters, and specific failure diagnostics), deliberately omitting the states of sibling nodes. This architectural choice prevents memory bloat as the tree expands and eliminates concurrency contention. Because siblings maintain no inter-dependent locks, all active frontier nodes can be expanded concurrently in parallel. This reduces the theoretical critical path from the total number of tool invocations down to the maximum depth of the hierarchy, delivering immense throughput gains.
2. Modular Action Space: Tailored Decomposition for Heterogeneous Design Modalities
Real-world graphics interweave typography, vector paths, overlapping illustration layers, and photographic elements, rendering single monolithic models inadequate. ReDesign defines a modular discrete action space where each action encapsulates an atomic toolchain specialized for distinct modalities: - Text Extraction: Uses OCR to locate and read textual content, applies a text segmentation model to extract precise glyph masks, and deploys an inpainting model to cleanly reconstruct the occluded background, yielding an independent text layer and a clean remainder canvas. - Multi-layer Decomposition: Employs layered image generation models (such as Qwen-Image-Layered) to separate complex multi-element compositions into distinct transparent RGBA layers, adaptively selecting layer counts based on visual complexity. - Connected Component Labeling (CCL): When a layer contains spatially disjoint, non-touching graphical elements, a lightweight connected component labeling algorithm splits them into separate child nodes without consuming expensive foundation model calls. - Detection and Segmentation: For visually entangled foreground objects, a VLM identifies the foremost entity, an open-vocabulary detector localizes its bounding box, SAM segments its fine contour, and inpainting fills the vacated backdrop to cleanly separate foreground from background. - Vectorization and Typography Fitting: Leaf nodes containing geometric vectors are converted into resolution-independent SVG paths via VTracer. Leaf nodes containing text are processed by font recognition networks to infer font family, weight, size, and alignment attributes.
3. Graceful Verification: Local Self-Healing via Physical Coverage and Boundary Constraints
Tool outputs inevitably exhibit visual flaws, including hallucinated artifacts from inpainting, imperfect segmentation boundaries, or duplicated entities. In linear agent architectures, early errors corrupt downstream states and trigger catastrophic global restarts. ReDesign restricts the responsibility of each action to locally explaining its parent patch. Immediately following every node expansion, a Graceful Verifier inspects the candidate children against two deterministic geometric criteria: 1. Coverage Completeness Constraint: The pixel union of all proposed children must fully account for the visual information within the parent patch, preventing holes or missing content; 2. Spatial Boundary Constraint: No individual child may extend beyond the legitimate spatial boundary of the parent, catching inpainting hallucinations and preventing duplicate overlapping geometries between siblings.
The verifier produces three targeted outcomes: Accept commits valid splits; Prune discards redundant or hallucinated siblings while preserving valid children; and Retry rejects an improper decomposition and injects detailed failure diagnostics into the node's lineal memory. The controller then re-executes the expansion using alternative tool choices or adjusted hyperparameters without disrupting unrelated subtrees.
A Worked Example¶
Consider an e-commerce promotional banner containing a heading, illustration icons, and a decorative background:
1. Root Expansion: The controller processes the full canvas, recognizes a dominant heading, and invokes Extract Text. OCR and segmentation extract "75+ Free illustrations", while inpainting patches the backdrop. The graceful verifier confirms complete coverage and clean boundaries, accepting text node \(N_1\) and remainder node \(N_2\).
2. Parallel Frontier Expansion:
- For text node \(N_1\), the controller determines it is an atomic text entity, predicts font parameters (Gilroy, Bold, position [10, 20, 80, 120]), and finalizes it as an editable text object.
- For remainder node \(N_2\), the controller observes multi-layered character illustrations and invokes Multi-layer Decomposition.
3. Graceful Verification and Pruning: The decomposition outputs a complete green zombie hand, a broken variant artifact, and a spellbook. The verifier detects that the broken arm variant heavily overlaps with the primary hand and marks it as redundant. The broken artifact is pruned immediately, while the clean zombie hand and spellbook are accepted.
4. Final Parameterization: Leaf vector layers are fitted to SVG paths with extracted hex colors (#060016, #F2F2F2), and the final nodes are structured into a valid Figma hierarchy with correct z-order stacking.
Key Experimental Results¶
Main Results¶
To rigorously benchmark true editability, the authors collected 909 raw Figma files to create the Figma Edit Replay Benchmark, comprising 14,796 controlled edit instructions (repositioning, rotation, deletion, re-ordering, text rewriting, and recoloring). The exact same edit instructions were executed on both the ground-truth Figma designs and the reconstructed assets. Performance was quantified by structural similarity (SSIM) of post-edit rendered images, text recognition recall, and standard reconstruction metrics across Figma and Crello datasets.
Table 1: Reconstruction accuracy measured on the Figma dataset (Object-level, Global-level, and Layout)
| Method | Object-level L1 ↓ | Global PSNR ↑ | Global LPIPS ↓ | Layout PQ ↑ | Layout F1 ↑ |
|---|---|---|---|---|---|
| VTracer | 0.0977 | 20.487 | 0.1917 | 24.64 | 0.309 |
| LayerD | 0.0704 | 16.141 | 0.3381 | 30.09 | 0.350 |
| Qwen-Image-Layered | 0.0493 | 26.192 | 0.1073 | 35.37 | 0.429 |
| Tool Agent (ReAct) | 0.0493 | 13.923 | 0.3869 | 45.33 | 0.527 |
| ReDesign (Ours) | 0.0431 | 26.286 | 0.0883 | 45.37 | 0.535 |
Table 2: Reconstruction accuracy measured on the Crello dataset
| Method | Object-level L1 ↓ | Global PSNR ↑ | Global LPIPS ↓ | Layout PQ ↑ | Layout F1 ↑ |
|---|---|---|---|---|---|
| VTracer | 0.0953 | 18.805 | 0.2919 | 19.55 | 0.243 |
| LayerD | 0.0938 | 14.452 | 0.4585 | 30.98 | 0.369 |
| Qwen-Image-Layered | 0.0506 | 26.419 | 0.0985 | 40.19 | 0.496 |
| Tool Agent (ReAct) | 0.0767 | 12.011 | 0.5674 | 44.16 | 0.527 |
| ReDesign (Ours) | 0.0465 | 23.525 | 0.1249 | 49.57 | 0.587 |
Ablation Study¶
Ablation experiments evaluated the computational cost and variance trade-offs of the verification protocol alongside parallel execution scaling.
Table 3: Ablation study on verification strategies and parallel execution speedup
| Configuration / Mechanism | Runtime & Speedup | PSNR / Execution Variance | Functional Role & Observations |
|---|---|---|---|
| Terminal Verification | Baseline runtime (High variance) | Lower PSNR / Unstable tool counts | Errors cascade deeply, forcing full restarts with severe latency penalties |
| Graceful Verification (Ours) | Lower runtime (Low variance) | Highest PSNR / Stable execution | Step-level checking blocks cascading flaws and guides local repair |
| Serial Execution (Tool Agent) | 1.0× (Baseline) | - | Nodes expanded sequentially; execution time scales linearly with elements |
| Parallel Frontier = 2 | 1.7× Speedup | - | Lineal memory prevents sibling lock contention across 2 threads |
| Parallel Frontier = 4 | 2.2× Speedup | - | Concurrent execution on orthogonal tree branches |
| Full Parallel Expansion (Ours) | 7.1× Speedup | - | Critical path contracts to hierarchy depth, maximizing system throughput |
Key Findings¶
- Superior Edit Replay Fidelity: ReDesign achieves the highest post-edit SSIM across all 6 edit operations (reposition, rotation, deletion, re-ordering, text edit, and recolor). While layered models (LayerD, Qwen-Image-Layered) handle coarse geometric deletions reasonably well, their lack of clean element isolation causes severe color bleeding during attribute edits. Furthermore, on text edits, ReDesign achieves an OCR text recall of ~0.72 compared to <0.30 for the Tool Agent baseline, whose unconstrained serial tool chains frequently damage fine text glyphs in early steps.
- Frequent Local Verification Beats Sparse Terminal Validation: While adding intermediate verification steps might seem counter-intuitive for latency, experiments demonstrate that small local checks prevent expensive cascading reruns, yielding faster wall-clock execution and lower variance in total tool calls.
- Failures of End-to-End Generative Editing: Compared against cutting-edge raster image editing systems (e.g., Nano Banana 2), which often distort image aspect ratios and introduce unintended global hallucinations during simple repositioning tasks, ReDesign's explicit layer hierarchy ensures deterministic, non-destructive editing.
Highlights & Insights¶
- Hierarchy as an Agentic Scaffold: Casting raster de-rendering into a layer tree expansion replaces unstructured prompt chains with a mathematically sound decomposition, naturally bounding error propagation.
- Concurrency through Lineal Memory: Restricting node memories strictly to their ancestral path enables thread-safe, lock-free parallelization across tree frontiers, yielding a 7.1× speedup over serial tool execution.
- Physical Verification over Generative Critics: Employing unambiguous geometric coverage and boundary constraints provides reliable self-healing signals without relying on noisy secondary LLM judges.
Limitations & Future Work¶
- Subjective Hierarchy Granularity: The ideal granularity of an editable document depends heavily on user intent. A complex illustration could reasonably be treated as a single merged asset or hundreds of fine vector bezier curves. While ReDesign supports user-in-the-loop prompts to refine granularity, automated heuristics occasionally diverge from human authoring conventions.
- Scaling on Dense Micro-Elements: Highly congested documents (e.g., technical schematics or dense typographic tables) substantially increase tree depth and leaf counts, creating potential bottlenecks for OCR and segmentation tools.
- Future Directions: Integrating tighter human-in-the-loop conversational controls to guide subdivision depth, and natively compiling output hierarchies directly into executable Figma plugins.
Related Work & Insights¶
- vs Layered Image Decomposition (Qwen-Image-Layered, LayerD): These methods output a fixed set of RGBA raster layers that remain uneditable at the vector and typographic level. ReDesign incorporates them as sub-tools within an agentic hierarchy that yields fully parametric vector paths, text parameters, and structured JSON.
- vs Serial Tool-Using Agents (ReAct): Generic tool agents execute linear action sequences with terminal verification, suffering from error accumulation and high rerun costs. ReDesign introduces tree-structured planning with local graceful verification and achieves 7.1× parallel speedup.
- vs Classical Vectorization (VTracer): Global vectorization tools indiscriminately convert all pixels into complex polygons, destroying typography and semantic grouping. ReDesign routes modalities selectively, vectorizing only shape-like leaf nodes.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates raster-to-editable recovery as structured tree expansion with lineal memory and graceful verification.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Introduces the 14,796-instruction Figma Edit Replay Benchmark, combining visual fidelity with real-world editability metrics.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, rigorous methodology, and comprehensive experimental analysis.
- Value: ⭐⭐⭐⭐⭐ Bridges a critical gap between generative computer vision and production UI/UX workflows in tools like Figma.