Teaching an Agent to Sketch One Part at a Time¶
Conference: ECCV2026
Paper: ECCV Paper
Project: https://xiaodan.io/teaching-an-agent-to-sketch/
Area: Reinforcement Learning
Keywords: vector sketches, part annotation, multi-turn generation, process reward, local editing
TL;DR¶
The paper automatically annotates vector sketches with semantic parts, then trains a vision-language agent with SFT and multi-turn process-reward GRPO to observe the canvas and add curves part by part; users prefer its final outputs over SketchAgent in 77.5% of comparisons.
Background & Motivation¶
A vector sketch is not an indivisible bitmap but a collection of curve paths that can be modified, scaled, and rearranged independently. However, when a generator emits every stroke at once, the editable file format does not ensure that stroke groups correspond to recognizable parts. For example, replacing a chair back while preserving its seat and armrests may still require identifying individual curves instead of expressing a local editing intention directly. CLIPDraw and DiffSketcher primarily optimize entire sketches jointly, while diffusion-based generation also typically lacks progressive part-level control. SketchAgent explores language-driven sequential drawing, but depends on a closed-source VLM's zero-shot capabilities, favors simple icon-like outputs, and can misplace components.
Obtaining more detailed progressive drawing requires more than splitting one generation into several calls. The model must know which curves belong to each semantic part and learn to add the next part under spatial constraints imposed by existing components. Ordinary image-sketch pairs do not provide this supervision directly, while manually labeling parts and path assignments in professional sketches is expensive. Even with part annotations, a model trained on ground-truth partial canvases encounters its own misplaced, missing, or disproportionate strokes at inference time. Later parts build on these imperfect states, so good single-turn performance does not guarantee a good completed sketch.
The paper therefore co-designs data structure and training: it first gives strokes semantic membership, then trains the agent on canvases generated by its own policy. Ground-truth paths can be recombined by part to provide a visual reference for every intermediate state, rather than scoring only the final result. Core Idea: use part-level data to construct supervised intermediate canvases, then compare trajectories within the same step to train a drawing policy that adapts to its own generated states.
Method¶
Overall Architecture¶
The input includes an overall caption, the next part description, the current canvas, previous part descriptions and paths, and the number of parts remaining after this turn. At each turn, the agent outputs vector paths for only the current part; a renderer adds them to the canvas and returns the updated image for the next turn. The resulting groups correspond naturally to the interaction history, allowing users to explore alternative designs at part boundaries without regenerating everything. Training proceeds through part annotation, single-turn supervised initialization, and multi-turn reinforcement learning; the annotation VLM is an offline data tool, not an additional teacher drawing alongside the agent at inference time.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw vector sketches"] --> B["Semantic part annotation"]
B --> C["Permutation-augmented<br/>single-turn SFT"]
C --> D["Multi-turn GRPO<br/>with stepwise rewards"]
B -.->|Ground-truth intermediate canvases for training| D
D -->|Trained policy| E["Inference: descriptions<br/>and current canvas"]
E --> F["Generate part paths<br/>and render"]
F -->|Next part and updated canvas| E
F --> G["Grouped vector sketch"]
Key Designs¶
1. Semantic part annotation: turn curve collections into actionable object structure
The pipeline first renders an SVG and asks a VLM to propose semantic part descriptions that do not overlap and collectively cover the object. The same VLM then acts as a critic, checking the descriptions against the requirements and returning violation types, severity, reasoning, and suggested fixes. The structured response also specifies whether revision is needed; if so, another pass revises the decomposition using the image and critique. Instead of trusting the first output, this separates proposing a decomposition from checking it, reducing missed requirements in a complex instruction set.
Given the part descriptions, the model examines both SVG text and the rendering to assign each path to a part. Schema constraints require every path to belong to exactly one part and every part to contain at least one path. A diagnostic image then shows uniquely colored part descriptions on the left and curves recolored by their current assignments on the right. The critic uses this visualization to identify misplaced paths, after which a refinement pass updates their assignments. Finally, the overall caption is generated solely from the refined part descriptions, keeping global and local semantics consistent.
The authors apply this pipeline to ControlSketch, whose underlying collection contains 35,000 image-sketch pairs across 15 object categories. The high-quality ControlSketch-Part annotations use Gemini 3.0 Pro and constrain each sketch to 2 through 5 parts. Category labels are not used in training: part descriptions and natural language provide direct supervision, although the object distribution still limits data coverage. The main text specifies a separate SFT dataset annotated through the same pipeline with Gemini 2.5 Flash, which is 6.7 times cheaper; the high-quality version is reserved for RL. Consequently, the two-stage procedure should not be understood as indiscriminately reusing exactly the same annotated examples at both stages.
2. Permutation-augmented single-turn SFT: learn to add one part to an existing canvas
Each output path is a cubic Bézier curve defined by a starting point, two control points, and an endpoint, totaling eight coordinate values.
Because stroke width, opacity, and other attributes are fixed, the model emits only the M and C commands with coordinates rather than a complete SVG wrapper.
Paths are separated by newlines and rendered with shared attributes, focusing learning on geometry instead of lengthy markup.
The current canvas supplies spatial context, previous paths record existing actions precisely, and the overall caption maintains object-level semantics.
The remaining-part count indicates progress so that the model does not treat the current component as a standalone complete drawing.
For each sketch, the authors sample up to 20 part permutations and construct incremental completion examples in each order. For an order C, B, D, E, A, the model first predicts C from an empty canvas, then B from a rendering of ground-truth C, and then D from ground-truth C+B. All permutations share the overall caption, but the canvas and requested part vary with the order, exposing the model to different partial contexts. The target is next-token prediction of the current part's ground-truth paths, without regenerating previously drawn components. Permutation augmentation also supports interactive changes by avoiding dependence on one fixed drawing order for an object.
The important boundary is that these intermediate canvases still consist of ground-truth paths. The model learns to continue from correct prefixes without adequately experiencing its own proportional errors or stroke defects. At inference time, its outputs become inputs to subsequent turns, and this state-distribution gap can accumulate errors throughout the drawing. SFT therefore initializes formatting and the local policy rather than replacing full multi-turn training.
3. Multi-turn GRPO with stepwise rewards: compare quality at the same drawing stage
Starting from the SFT model, RL samples a group of complete drawing trajectories for each input, with every trajectory observing its own generated canvas at each turn. For a given sketch, trajectories share the part count, so normally completed trajectories can be aligned at the same drawing step. Ground-truth paths can also be assembled for the parts drawn so far, yielding the appropriate reference partial canvas. The reward can therefore assess the result at the current stage instead of unfairly comparing an incomplete sketch with the finished reference object.
At every step, CairoSVG renders the current canvas, and a pretrained DreamSim ensemble compares it with the corresponding ground-truth intermediate rendering. The visual reward is the cosine similarity between the two DreamSim image embeddings; the meaning of Equation (7) can be stated unambiguously as:
Both images must contain the same set of parts drawn so far, so the reward assesses the new component together with its placement among existing content. DreamSim measures perceptual similarity between images rather than directly scoring text-sketch semantic alignment. The authors additionally reward an appropriate final path count to counter unnecessary stroke growth associated with GRPO's preference for longer outputs. This term regularizes only the whole-sketch path count because per-part count supervision was empirically found to be too noisy.
Advantage computation is another key choice: rewards are normalized across trajectories within the same step, not pooled across different steps. Following Equation (6) and its surrounding explanation, let \(r_g^t\) denote the combined reward for trajectory \(g\) at step \(t\):
Each step's advantage is applied to tokens in that response directly, without summing normalized rewards from subsequent steps. Intuitively, first parts are compared with other first parts and final states with other final states, avoiding reward-scale mixing across different drawing stages. This is still not a purely incremental score for the new part: the canvas contains earlier strokes, so previous mistakes can affect later visual rewards. A verifier checks formatting before reward computation; malformed responses receive a penalty and terminate their trajectories early, so these exceptions are not normally completed equal-length samples. The main text also states that path counts in these cases are cumulative through the last successful step.
A Worked Example¶
Consider the chair-editing example in Figure 9: the object includes a chair back, left and right armrests, a flat seat, and four legs. After the user specifies a circular chair back, the agent generates its curves and then adds subsequent parts while observing the rendering. When drawing an armrest, the input contains not just its description but also the existing chair-back placement and previous paths. Changing the initial canvas while preserving later descriptions can produce outputs adapted to that canvas rather than mechanically reusing fixed absolute coordinates. During training, each drawing stage can be compared with a canvas assembled from ground-truth parts; inference needs neither a reference image nor DreamSim rewards. Another Figure 9 example changes an early hair description while keeping subsequent descriptions fixed, illustrating differences largely localized to the affected part. These are qualitative demonstrations, not guarantees that arbitrary edits preserve every other curve, nor an interface for arbitrary fine-grained single-stroke editing.
Loss & Training¶
The main backbone is Qwen3-VL-30B-A3B, with rank 64 LoRA in both stages and training conducted through Tinker. SFT uses cross-entropy, a learning rate of \(2\times10^{-4}\), batch size 128, and 5400 steps. RL adds 1000 steps with a learning rate of \(3\times10^{-6}\), batch size 8, and a sampling group size of 8 per input. Both stages use Adam with \(\beta_1=0.9\), \(\beta_2=0.95\), and \(\epsilon=10^{-8}\); the KL-divergence loss is disabled during RL. Coordinates are rounded to the nearest ten during SFT, while RL retains the original integer coordinates; the former reduces initial numerical granularity and the latter permits finer geometric output. The main text reports a combined-reward weight of \(\lambda=1.0\), but symbols are missing from cached Equations (8) through (11), preventing reliable recovery of the path-count reward and full optimization objective. This note therefore retains the confirmed reward semantics and training settings without presenting guessed expressions as exact author formulas; handling of zero-variance reward groups is also unspecified in the readable main text.
Key Experimental Results¶
Main Results¶
The main experiments compare the full model, its SFT-only variant, SketchAgent, Gemini 3.1 Pro, and a cascade that generates an image with SDXL before converting it to a sketch with SwiftSketch. SketchAgent uses Claude Sonnet 4.5 because its original Claude Sonnet 3.5 backend is unavailable; whole-sketch baselines receive all part descriptions concatenated together. The automatic metric is Long-CLIP image-text embedding cosine similarity, supporting up to 248 text tokens, rather than the DreamSim reward used for training. Figure 5 reports the full model as the highest-scoring compared method; the main results table below instead uses the clearly recoverable user-study values.
The following table summarizes Figure 6 on page 12. All values are the proportions of double-blind forced-choice comparisons favoring the full model, not classification accuracies.
| Comparison baseline | Final quality preference | Step quality preference | Evaluation condition |
|---|---|---|---|
| SFT only | 84.1% | 83.1% | Training-stage comparison within the method |
| SketchAgent | 77.5% | 70% | Both support part-by-part generation |
| Gemini 3.1 Pro | 66.1% | Not evaluated | Whole-sketch generation baseline |
| SDXL + SwiftSketch | 91.1% | Not evaluated | Text-to-image followed by sketch conversion |
For step quality, each comparison collects 426 responses from 142 participants; for final quality, each comparison collects 560 responses from 146 participants. Together, the studies comprise 3,092 pairwise comparisons, which must not be interpreted as that many independent participants. Evaluating both the final sketch and the drawing process avoids assuming that an attractive final image necessarily follows the intermediate part descriptions.
Ablation Study¶
The following results come from Table 1 on page 14. Every configuration uses Qwen2.5-VL-3B rather than the main experiment's Qwen3-VL-30B-A3B; higher average Long-CLIP similarity is better.
| Config | Long-CLIP | Reward and trajectory setting |
|---|---|---|
| Single-turn RL | 0.281 | Entire drawing as one completion, final reward only |
| Multi-turn outcome reward | 0.286 | Multiple turns, final rendering reward determines all step advantages |
| Multi-turn process reward | 0.298 | Intermediate canvases contribute stepwise rewards |
Key Findings¶
- Multi-turn process reward improves by 0.012 over multi-turn outcome reward and by 0.017 over single-turn RL, supporting intermediate visual feedback beyond merely splitting the output into turns.
- The full model receives 84.1% final-quality preference and 83.1% step-quality preference against SFT only, indicating that RL benefits are not confined to the final frame.
- These comparisons do not independently isolate the path-count reward, visual input, or annotation critique stages, so they cannot assign separate gains to every component.
Highlights & Insights¶
- Part labels support interactive editing while also providing reference images for arbitrary intermediate states. Data organization directly determines whether dense supervision is available.
- Random permutations make the current state plus next part the learning unit instead of a fixed drawing order. This provides training support for continuing from different partial canvases.
- Same-step normalization restricts comparisons to matched drawing progress. A transferable insight is that aligned task stages can structure reward baselines instead of pooling every state together.
Limitations & Future Work¶
- The authors acknowledge weak generalization to unseen objects and out-of-domain parts; semantically related inputs can produce nearest-neighbor training categories. A natural-language interface does not imply open-world generation.
- Precise numerical coordinate prediction can still cause severe misplacement, and editing is currently part-level rather than fine-grained single-stroke editing. Qualitative examples should not be expanded into universal editing guarantees.
- From an evaluation perspective, the supplied main-text cache does not clearly specify test-set size, split details, or preference confidence intervals; fuller reproduction requires supplementary material.
- High-quality automatic annotation can still inherit VLM errors, and the main text does not provide human-audit statistics sufficient to independently assess path-assignment accuracy. Human spot checks and broader object coverage are reasonable reader-proposed priorities.
- The available source contains the complete main paper and references but not its cited supplementary material; partially corrupted extracted equations are identified in the Method section and have not been invented.
Related Work & Insights¶
- Compared with SketchAgent: both emphasize sequential drawing, while this paper adds trainable part-level data and RL on self-generated canvases; the trade-off is dedicated data and training requirements.
- Compared with Reason-SVG and Rendering-Aware RL: these methods also train vector generators using rendering quality, whereas this paper uses ground-truth intermediate states for multi-turn credit assignment rather than observing only completed outputs.
- Compared with SwiftSketch and DiffSketcher: the former is an image-to-sketch diffusion model and the latter uses test-time optimization; this paper directly conditions progressive path generation on free-form part descriptions.
- Transferable research direction: graphics editing or diagram construction with stage-specific reference states could use similar stepwise supervision; this is a reader inference, not a cross-task result demonstrated by the paper.
Rating¶
- Novelty: 4/5. Part annotation, interactive state representation, and stepwise GRPO form a coherent design that goes beyond replacing the drawing model.
- Experimental Thoroughness: 4/5. Automatic metrics, user studies, and reward comparisons are included, but controlled ablations use a smaller model and component and generalization analyses remain limited.
- Writing Quality: 4/5. The connection from data to training is clear; corrupted equations in the text cache hinder reproduction-oriented reading and should not be attributed directly to the original typesetting.
- Value: 4/5. Useful for structured visual generation and process-reward research, with practical utility still limited by object coverage and coordinate stability.