Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback¶

Conference: ECCV 2026
Paper: ECCV Paper
Area: Multimodal VLM / Image Generation
Keywords: vector graphics generation, visual self-feedback, path decomposition, rendering verification, step-wise drawing

TL;DR¶

Render-in-the-Loop lets Qwen3-VL-8B observe the cumulative canvas after each SVG fragment, combining specialized multi-turn supervised training with inference-time repetition filtering to outperform several autoregressive baselines trained on larger datasets on MMSVGBench using 0.85M training samples, at approximately 1.50 times the inference latency for VSF.

Background & Motivation¶

SVG represents graphics through paths, fills, and layers, enabling lossless scaling and easier editing than pixel-only output. Methods such as StarVector and OmniSVG can already generate SVG code directly, but valid code does not necessarily produce a correct drawing: coordinates may form the wrong outline, later layers may cover earlier details, and individually valid paths may not compose into the intended object. During open-loop generation, the model must infer what it has drawn from previous code, without allowing its vision encoder to inspect the actual canvas continuously.

One improvement is reinforcement learning with rewards for renderability or semantic similarity, as explored by SVGen, Reason-SVG, and RLRF. This paper addresses a different issue: even when a reward indicates output quality, a scalar does not directly retain spatial information such as a misplaced lens or a foreground element covering the background. Since a vision-language model (VLM) can already interpret images, could it inspect the canvas after each drawing step? The difficulty is that pretrained models do not inherently know how to map an unfinished image to the next precise geometric code fragment. Simply supplying images across more turns does not reliably improve results.

The paper therefore changes more than inference prompts. It converts static SVG data into dense drawing trajectories, trains the model to predict the next stroke from intermediate canvases, and prevents ineffective strokes from being accepted at inference time. Core Idea: make rendered output a direct visual condition for the next generation step, and teach the model to use it through dedicated trajectory supervision instead of only compressing the final image into a reward score.

Method¶

Overall Architecture¶

The input is either a text description or a reference image, and the output is always renderable SVG code. Training first applies Fine-grained Path Decomposition and then Visual Self-Feedback Training to alternating code and canvas sequences. During inference, Rendering Verification decides whether each proposed fragment should join the accepted code.

The canvas is the cumulative rendering of all accepted fragments, not an isolated image of the latest path. This distinction matters because the presence of target parts, their occlusion, and missing layout elements must be assessed from the composition rather than the current stroke alone. The model retains the complete history of code and images; the latest screenshot does not replace the entire history.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    SOURCE["Static SVG data"] --> DECOMP["Fine-grained Path Decomposition"]
    DECOMP --> VSF["Visual Self-Feedback Training"]
    VSF -.->|Learned weights| GENERATE["Predict next fragment<br/>or end token"]
    PROMPT["Text or reference image<br/>and accepted history"] --> GENERATE
    GENERATE -->|Candidate fragment| VERIFY["Rendering Verification"]
    VERIFY -->|Accept and return cumulative canvas| GENERATE
    VERIFY -->|Reject and adjust sampling| GENERATE
    GENERATE -->|End token| OUTPUT["Output SVG"]
    VERIFY -->|Retry limit reached| OUTPUT

The dashed edge represents the effect of training on model parameters, not an extra training run during inference. Rendering Verification is used only at inference time. Training trajectories come from decomposing and cumulatively rendering existing SVGs, not from online reinforcement learning trials.

Key Designs¶

1. Fine-grained Path Decomposition: increase feedback frequency while preserving fill and opacity semantics

Original SVGs often place visually separate components in a single <path> to reduce storage, averaging only about 4 path elements per file. Generating one original path per step may draw the body, lens, and beams together before receiving feedback. This requires predicting many coordinates at once and omits intermediate states between components. The authors parse drawing commands in the d attribute and extract independent subpaths at discontinuous commands such as M/MoveTo, turning a large drawing action into finer steps.

However, finer splitting is not always better. Containment between inner and outer contours may define holes through fill rules; rendering intersecting translucent subpaths separately can also change color blending. The method treats subpaths as two-dimensional polygons, constructs a dependency graph using containment or spatial intersection under non-unity opacity, and merges each connected component back into one <path>. Visually coupled parts remain atomic, while genuinely independent parts become separate steps. Average path count rises from 4 to 6, providing denser feedback. A step can still contain several interdependent subpaths, so this should not be interpreted as generating every curve individually.

2. Visual Self-Feedback Training: condition the next code fragment on the current drawing state

Each processed sample becomes a complete multi-turn sequence of user condition, code fragment, cumulative canvas, next code fragment, next cumulative canvas, and so on, followed by an end token. Text-to-SVG uses a textual target, while Image-to-SVG uses a reference image. In both tasks, intermediate states are rendered from the code drawn so far. When predicting the next fragment, the model can observe the target, previous code, and the actual output of that code, reducing its reliance on mentally reconstructing geometry from coordinate strings. Images pass through the multimodal model's vision encoder and enter the context as visual tokens.

The task being learned is continuation from a partially completed state, not merely describing a finished image. Training consecutive steps together allows the model to use the current canvas while retaining the drawing sequence. The end token is also supervised, teaching the model when further additions are unnecessary. Intermediate canvases are fixed at 224 × 224 and encoded into only 49 visual tokens each to keep multi-turn images from exhausting the context. This limits fine-detail perception, while history continues to grow: the loop provides observable state, not unlimited resolution or memory.

3. Rendering Verification: check whether a candidate makes useful progress before accepting it into history

After the model proposes a fragment, Render-and-Verify (RaV) combines it with the accepted code and renders a hypothetical next canvas. The first check compares the old and new canvases and rejects changes below a pixel-difference threshold, targeting actions with no visible contribution, such as redundant overpainting or drawing outside the canvas. The second check compares the candidate with the previous accepted fragment and rejects excessive string similarity to prevent near-identical code loops. It inspects both output and adjacent code structure, but it is not a general evaluator of whether the target semantics are correct.

If either check fails, the candidate is discarded without entering the accepted drawing history. Sampling parameters are then adjusted, for example by slightly increasing temperature, before another attempt. Normal completion remains the model's prediction of the end token; RaV does not terminate the drawing immediately upon detecting a small change. Only after repeated failures reach the maximum retry count \(K_{\max}\) does the system force termination to avoid stalling. The main text specifies a pixel-difference threshold of \(\epsilon=0.001\) and a string-similarity threshold of \(\tau_{\mathrm{sim}}=0.98\), but the supplied text does not specify the difference normalization formula, string-similarity implementation, or numerical retry limit. These should not be filled in as though the exact algorithm were available.

A Worked Example¶

Consider the projector in the paper's Figure 1: the target specifies a white body, a dark lens, small purple legs, and gray dots on both sides of the lens. The following explains execution using that target; it is not presented as the paper's recorded token-by-token trajectory.

During preprocessing, if the body, lens, and beams belong to one complex path, subpaths are extracted first, then parts coupled by holes or translucent blending are merged. Cumulative rendering exposes states in which the body is present but the lens or decorations remain incomplete, instead of jumping from a blank canvas to the entire projector in one action.

At inference time, the model proposes the next code fragment from the description and accepted canvas. If it draws another path that almost exactly covers the existing lens, insufficient visual change or excessive code similarity causes RaV to reject it. A resampled fragment enters history only after passing verification, producing the cumulative image for the next turn. This prevents ineffective repetition from consuming generation opportunities needed by remaining components, but it does not guarantee that the accepted fragment depicts the requested legs or dots.

If the legs were placed incorrectly earlier, the current action space cannot delete or replace them. The model can only append paths to compensate visually. Self-feedback therefore means adjusting subsequent generation after observing the result, not general undo or editing capability. This distinction motivates the authors' proposed extension to DOM editing actions.

Loss & Training¶

Training uses multi-turn visual instruction tuning with a standard autoregressive language modeling objective. Only model-generated SVG code tokens and the end token contribute to the loss; user prompts and inserted canvas visual tokens are masked out. The objective is normalized by the number of supervised tokens. Equation (3) is corrupted in the cached text extraction, so this note retains the supervision scope explicitly described in the prose rather than reconstructing the damaged expression as the authors' exact formula.

The data comes from the released portion of OmniSVG: 0.9M icons and 0.25M illustrations, reduced by exact code-string deduplication to 0.65M icons and 0.2M illustrations, totaling 0.85M. This removes identical code, not necessarily all visually near-duplicate examples. Path decomposition increases supervised steps per drawing rather than adding independent images.

The model is initialized from Qwen3-VL-8B-Instruct and trained for 3 epochs on 8 NVIDIA H100 GPUs using AdamW with a learning rate of \(10^{-5}\), cosine decay, and a maximum sequence length of 10240. The paper does not jointly train VSF with reinforcement learning. Their compatibility is a proposed future direction, not a demonstrated gain.

Key Experimental Results¶

Main Results¶

Evaluation uses the official Icon and Illustration subsets of MMSVGBench for Text-to-SVG and Image-to-SVG. The following extracts results from the paper's Table 1. FID measures distribution-level image differences, with lower being better; CLIP measures text-semantic consistency, with higher being better; SSIM measures reconstruction structure similarity, with higher being better; LPIPS measures perceptual reconstruction distance, with lower being better. Metrics from the two tasks should not be treated as a single aggregate score.

Subset	Method / Training images	Text-to-SVG FID ↓	Text-to-SVG CLIP ↑	Image-to-SVG SSIM ↑	Image-to-SVG LPIPS ↓
Icon	OmniSVG-8B / 2M	130.56	0.276	0.893	0.235
Icon	InternSVG-8B / 16M	128.80	0.291	0.901	0.182
Icon	Ours VSF+RaV-8B / 0.85M	127.64	0.293	0.914	0.172
Illustration	OmniSVG-8B / 2M	138.42	0.231	0.907	0.231
Illustration	InternSVG-8B / 16M	138.10	0.229	0.915	0.205
Illustration	Ours VSF+RaV-8B / 0.85M	137.79	0.237	0.928	0.178

These results support advantages over the listed autoregressive baselines, not superiority over every reconstruction method. In the paper's Table 1, DiffVG achieves SSIM of 0.955 and LPIPS of 0.065 on Illustration, outperforming this method in reconstruction. The authors emphasize a different trade-off: optimization-based approaches produce many small paths with weaker editability, although this claim is supported primarily through qualitative comparisons.

Ablation Study¶

The following also comes from the paper's Table 1 and is restricted to the Illustration subset and Qwen3-VL-8B configurations. SFT only uses standard open-loop supervised fine-tuning, VSF introduces dedicated visual trajectory training, and VSF+RaV additionally enables inference-time verification.

Config	Text-to-SVG FID ↓	Text-to-SVG CLIP ↑	Image-to-SVG SSIM ↑	Image-to-SVG LPIPS ↓
SFT only	155.84	0.221	0.861	0.318
VSF	137.86	0.235	0.921	0.193
VSF+RaV	137.79	0.237	0.928	0.178

VSF improves FID by 17.98, whereas adding RaV provides a further improvement of only 0.07. RaV improves LPIPS by 0.015. Most of the quality gain comes from learning to exploit trajectories; verification is an incremental safeguard against generation degeneration, so their contributions should not be described as equal.

The paper's Table 3 reports the following MMSVGBench inference costs using the same Qwen3-VL backbone, vLLM serving stack, and NVIDIA V100 hardware. It compares VSF against open-loop SFT and does not separately report the retry cost of the complete VSF+RaV configuration.

Config	Time / seconds	Generated tokens	Latency multiplier	Rendering time / seconds
Open-loop SFT	28.77	550.94	1.00	0.000
VSF	43.19	804.31	1.50	0.038

Key Findings¶

Training image count is not the only factor: 0.85M is approximately 5.31% of InternSVG's data, but this is not a causal experiment controlling total training tokens, compute, and data distribution.
Observing the canvas requires corresponding training. Section 3.3 explicitly reports GPT-5's Illustration LPIPS increasing from 0.345 for one-shot generation to 0.388 for naive multi-turn prompting; feedback inputs alone do not guarantee improvement.
Additional cost primarily comes from step-wise autoregressive decoding, with generated tokens increasing by approximately 1.46 times, rather than slow rasterization. This constrains the practical efficiency of the closed-loop approach.

Highlights & Insights¶

A renderer can supply directly readable state for the next prediction, not just a training reward. This preserves spatial relations in images rather than requiring the model to infer occlusion entirely from code strings.
Path decomposition respects SVG semantics instead of simply splitting by length. Preserving holes and translucent blending through a dependency graph is essential to obtaining denser feedback without changing the original rendering.
Rendering candidates before accepting them offers a lightweight way to filter degeneration. However, visible change is only a necessary local progress signal, not proof of correct target semantics.

Limitations & Future Work¶

Author-acknowledged: inference is slower, and intermediate canvases are fixed at 224 × 224. The trade-off between higher-resolution detail perception and context or compute costs has not been systematically studied.
Author-acknowledged: actions only append primitives and cannot undo, delete, or replace accepted elements. DOM-level DELETE, REPLACE, and MODIFY could allow feedback to drive genuine correction.
The authors show failures involving repetition that omits a main object, drawing an ordinary clock instead of a melting clock, and oversimplifying complex scenes. Better geometric feedback does not yet solve abstract-state understanding or complete planning.
This note's evaluation caveat: Table 2 claims coverage of Qwen3-VL and GPT-5 but does not label the backbone for each row, so its full set of numbers cannot be independently assigned to both models. The supplied text also lacks a separate quantitative ablation isolating path decomposition, leaving room to strengthen the data-efficiency and editability evidence.

vs OmniSVG / InternSVG: these are strong direct SVG-code generation baselines. The key change here is to keep adding the actual canvas to the prediction conditions, rather than merely increasing static code data.
vs SVGen / Reason-SVG / RLRF: reinforcement learning mainly changes the policy optimization signal, while VSF changes the observable state and trajectory supervision. They can be complementary; the current experiments do not establish that visual feedback universally outperforms reinforcement learning.
vs DiffVG / LIVE: optimization-based vectorization can achieve strong pixel reconstruction scores, whereas this method emphasizes semantic generation and relatively compact path structure. Design-workflow evaluation should jointly measure visual quality, editability, and time rather than rank methods by LPIPS alone.
Transfer directions: the authors propose HTML/CSS, LaTeX/TikZ, 3D, and CAD program generation. These share the prerequisite of executing intermediate code and returning its output to the model, but effectiveness on these tasks is not experimentally established here.

Rating¶

Novelty: 4/5, integrating semantics-preserving trajectory construction, visual conditioning, and inference-time verification into a complete method.
Experimental Thoroughness: 3/5, covering two tasks, two subsets, and cost analysis, but lacking finer controlled ablations and full verification costs.
Writing Quality: 4/5, with a clear methodological narrative, though some backbone labels and verification implementation details need clarification.
Value: 4/5, providing a reusable closed-loop approach to structured visual code generation while requiring deployment trade-offs in latency and editing capability.