Skip to content

Vector Scaffolding: Inter-Scale Orchestration for Differentiable Image Vectorization

Conference: ECCV2026
arXiv: 2605.11913
Code: TBD
Area: Computer Graphics / Differentiable Rendering
Keywords: Differentiable vectorization, Hierarchical optimization, Bézier curves, Topological collapse, Image vectorization

TL;DR

This work proposes Vector Scaffolding, a hierarchical optimization framework that addresses the area-boundary gradient imbalance via Interior Gradient Aggregation, densifies curves from coarse to fine according to natural image power laws using Progressive Stratification, and applies a 50x learning rate scaling via Rapid Inflation Scheduling to achieve 2.5x speedup and a ~1.4dB PSNR improvement, reframing differentiable vectorization from "flat pixel matching" to "structured topological construction."

Background & Motivation

Differentiable vectorization is an important advancement in recent computer graphics—DiffVG first enabled gradients to flow through rasterization, LIVE introduced layer-wise path initialization, and Bézier Splatting brought the 3DGS Gaussian splatting pipeline to Bézier curves, achieving highly efficient parallel rendering. The core concept of these methods is consistent: placing hundreds or thousands of randomly initialized curves on a flat canvas, letting them compete with each other via pixel-level MSE loss, and ultimately fitting the target raster image. However, this "flat optimization" assumes all curves share equal status and update strategies, ignoring the intrinsic scale hierarchy of natural images.

This assumption leads to a failure mode the authors refer to as "topology collapse." Since curves start from pure randomness, large-scale curves are forced to distort their shapes to fit irrelevant details in order to reduce pixel errors in internal high-frequency texture regions. Meanwhile, small-scale curves overlap and entangle with each other, forming a structurally chaotic and uneditable "polygon soup." This undermines the core value of vector graphics—editability. The deeper mathematical cause is gradient imbalance: the analytical area gradient of a closed region scales as \(O(r^2)\), whereas the boundary gradient scales as \(O(r)\). When large curves attempt to cover massive regions, the interior gradients (originating from texture errors inside the region) dominate the optimization direction, degrading curves into superpixel fragments rather than semantically meaningful structural blocks.

Diagnosing the optimization dynamics, this paper proposes Vector Scaffolding, a purely optimization-driven hierarchical framework. Core Idea: Stabilize large curves by compensating for missing area gradients with Interior Gradient Aggregation, densify curves from coarse to fine according to natural image power laws via Progressive Stratification, and scale the learning rate by 50x using Rapid Inflation Scheduling, thereby reframing differentiable vectorization from a flat pixel-matching problem to a hierarchical topology construction problem.

Method

Overall Architecture

Vector Scaffolding is a hierarchical optimization pipeline coordinated by three components. Optimization starts with an extremely sparse initial set of curves (\(N_0 = 16\)). A "prune-and-densify" loop is executed every 100 iterations: first, useless curves with opacity lower than a threshold (\(\sigma(\alpha) < 0.01\)) are pruned; then, new curves are generated in regions with the highest residual error using an exponentially decaying radius (\(r_{\text{rad}} = 0.5\)) and positioned on the top layer of alpha-blending. Interior Gradient Aggregation consistently provides stable gradient signals throughout the process, ensuring large curves are not disrupted by high-frequency noise. Benefiting from the stable loss landscape established by progressive stratification, the learning rate can be safely scaled up to 50x compared to the baseline (color/opacity LR = 0.5, control points LR = 0.01).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Initial Sparse Curves<br/>N₀ = 16"] --> B["Bézier Splatting<br/>Differentiable Rasterization"]
    B --> C["Interior Gradient<br/>Aggregation<br/>Compensate Area Gradients"]
    C --> D{"Every 100 epochs?"}
    D -->|Yes| E["Pruning: Remove<br/>σ(α) < 0.01"]
    E --> F["Residual Heatmap Sampling<br/>Highest ‖Ī−I‖²"]
    F --> G["Exponentially Decaying Radius<br/>R_k = 0.5 × R_{k-1}"]
    G --> H["Z-ordering<br/>New Curves on Top"]
    H --> D
    D -->|No| I["50x Learning Rate<br/>Adan Optimizer"]
    I --> B
    B --> J["Final Hierarchical<br/>Vector Representation"]

Key Designs

1. Interior Gradient Aggregation: Stabilizing Large-scale Curves via Area Gradient Compensation

The backpropagation gradients of differentiable rasterization can be decomposed into two terms: the area gradient from the interior of the curve and the boundary gradient from its contour. According to the Reynolds transport theorem, the time derivative across a closed curve \(\partial\mathcal{A}\) can be split into an interior volume integral and a boundary surface integral. The scale of the area gradient is \(O(r^2)\), while the boundary gradient is \(O(r)\). Previous methods (e.g., Bézier Splatting) only sampled Gaussians along the boundaries to accelerate rendering, completely ignoring interior position gradients. Consequently, large curves are driven only by weak gradients (\(O(r)\)) at their boundaries, while high-frequency texture errors in interior regions indirectly affect optimization via the boundaries. Large curves are forced to distort themselves to fit high-frequency details that do not belong to their scale, falling into local optima. Interior Gradient Aggregation addresses this by uniformly sampling Gaussians within the enclosed region of each curve and explicitly accumulating these area gradients to the control point updates. Thus, large curves directly receive position gradients of matching magnitude (\(O(r^2)\)), securing them to the macroscopic structures they should cover. The area gradient itself does not determine which frequency band a curve should fit; it merely restores a forgotten gradient pathway, providing a stable foundation for subsequential frequency separation strategies.

2. Progressive Stratification: Coarse-to-Fine Hierarchical Driving via Natural Image Power Laws

Once Interior Gradient Aggregation stabilizes the landscape, Progressive Stratification constructs the hierarchical curve structure guided by three core rules. First, geomtrically increase curve counts—starting with \(N_0 = 16\), densification doubles the count at each step with \(r_{\text{num}} = 2.0\) (16 \(\rightarrow\) 32 \(\rightarrow\) 64 \(\rightarrow\) ... up to the target budget). Second, exponentially decay initialization radii—the initialization radius of newborn curves halves each generation with \(r_{\text{rad}} = 0.5\), naturally constraining the frequency range each layer can cover (large radius = low frequency, small radius = high frequency). Third, fixed temporal Z-ordering—newborn curves are always assigned lower depth values than older ones (meaning they are rendered on top of older curves) and are never reordered dynamically. This eliminates the optimization jiggling caused by dynamic depth sorting based on bounding box sizes in Bézier Splatting. Together, these three rules guarantee frequency separation: once large curves safely capture the low-frequency background, newly generated small-scale curves only fit residual hotspots on top, preventing backward interference with already anchored low-frequency structures. This mechanism naturally aligns with the power-law distribution of natural images, where most energy concentrates in low frequencies (a few curves cover the background), while smaller curves added layer-by-layer focus on finer textures.

3. Rapid Inflation Scheduling: Accelerating Convergence with 50x Learning Rate over Stable Landscapes

Since Progressive Stratification yields a stable and frequency-separated optimization landscape, the authors find that the learning rate can be safely scaled to 50 times that of Bézier Splatting (color/opacity LR from 0.01 to 0.5, control points LR from \(2 \times 10^{-4}\) to 0.01). Such an aggressive learning rate is only viable under a hierarchical framework; in flat optimization, scaling by 50x immediately triggers gradient oscillation and loss divergence, as validated by ablation studies. Rapid Inflation Scheduling further capitalizes on this by restructuring the schedule—curves complete all densifications within the first 600 epochs (doubling every 100 epochs, taking 6 steps from 16 to 1024), leaving the remaining iterations to fine-tune with the high learning rate. This "fast inflation, slow tuning" pacing reduces overall optimization time (wall-clock time) to 1/2.5 of the baseline while achieving higher PSNR under the same curve budget. Importantly, the computational cost per step does not increase, as Interior Gradient Aggregation does not introduce independent computational phases, and having fewer active curves in early stages actually reduces the rasterization overhead per step.

Loss & Training

The optimization target is a pixel-level MSE loss combined with three regularization terms:

\[ \mathcal{L}_{\text{total}} = \mathcal{L}_2(g(\mathcal{B}), \mathcal{I}) + 0.01 \mathcal{L}_{\alpha} + 0.02 \mathcal{L}_{\text{Xing}}(\mathcal{B}) + \mathcal{L}_{\text{BS}}(\mathcal{B}) \]

where \(\mathcal{L}_\alpha = \frac{1}{N}\sum_i |1 - \sigma(\alpha_i)|\) encourages opacities to approach 1 (solid vectors instead of blurry Gaussian splats); \(\mathcal{L}_{\text{Xing}}\) is the self-intersection avoidance loss introduced by LIVE; and \(\mathcal{L}_{\text{BS}}\) is the curve regularization term from Bézier Splatting. The optimizer is Adan (\(\beta_{1,2,3}=0.98/0.92/0.99\)), running for a total of 5000 iterations.

Key Experimental Results

Main Results

Evaluations on Kodak and DIV2K datasets compared against DiffVG, LIVE, LIVSS, and Bézier Splatting (all restricted to closed curves with budgets of 256/512/1024 curves, sharing the same rasterization backend):

Dataset Curves Metric Ours Bézier Splatting Gain
Kodak 256 PSNR 25.18 24.19 +0.99 dB
Kodak 512 PSNR 26.68 25.61 +1.07 dB
Kodak 1024 PSNR 28.30 26.91 +1.39 dB
DIV2K 256 PSNR 21.37 20.74 +0.63 dB
DIV2K 512 PSNR 22.64 22.11 +0.53 dB
DIV2K 1024 PSNR 23.99 23.45 +0.54 dB

LPIPS on Kodak also significantly outperforms all baselines (0.275 under 1024 curves vs 0.448 for Bézier Splatting). Since the proposed method shares exactly the same differentiable rasterization backend as Bézier Splatting, the performance gap stems purely from the optimization strategy restructuring—proving the decisive impact of the optimization structure on vectorization quality.

Ablation Study

Configuration Kodak PSNR (1024) Description
Full Model 28.30 All three components enabled
w/o Interior Gradients ~23.5 PSNR drops by ~5dB; large curves lose interior anchors, causing topology collapse
w/o Progressive Stratification (Flat + 50x LR) ~19.2 Gradients violently oscillate, and loss completely diverges
LR = 1x (rest as full model) 25.83 50x LR contributes the most to acceleration
r_rad = 1.0 (No radius decay) 25.30 Optimization quality drops significantly without scale constraints

Key Findings

  • Interior Gradient Aggregation is the largest individual contributor—removing it leads to a ~5dB drop in PSNR, proving that area gradients are critical for stabilizing large-scale curves.
  • Applying 50x LR directly under flat optimization leads to complete divergence; a high LR is only effective with the frequency separation brought by Progressive Stratification.
  • Hyperparameter sensitivity analysis shows that \(r_{\text{rad}}\) and \(\Delta T\) exhibit stable performance across wide ranges (PSNR variation < 0.2dB over 5k iterations), indicating that the framework is insensitive to hyperparameters.
  • Naturally supports LoD (Level-of-Detail) rendering—since curves are layered via temporal Z-ordering, users can select and edit curves of specific scales by layer, resulting in much better editability than a "polygon soup."

Highlights & Insights

  • Diagnosing gradient imbalance with the Reynolds transport theorem: This paper mathematically reveals the root cause of flat vectorization failure (the scale discrepancy between area and boundary gradients, i.e., \(O(r^2)\) vs \(O(r)\)), and delivers an elegant fix (Interior Gradient Aggregation). This "first diagnose, then repair" workflow is highly inspiring compared to purely stacking architectures.
  • The structure of optimization is more important than the rendering backend: By sharing the exact same rasterization backend with Bézier Splatting and only altering the optimization strategy, this method achieves a 2.5x speedup and +1.4dB improvement, proving that the "optimization structure" is a systematically designable dimension—opening a new research axis for future vectorization works.
  • Minimalist power-law heuristics: \(r_{\text{num}}=2.0, r_{\text{rad}}=0.5\) are extremely simple rules, yet they naturally align with the energy distribution of natural images. This philosophy of "guiding algorithm design with natural laws" has great potential for transferability.
  • Temporal Z-ordering: Binding depth to creation time instead of dynamic sorting resolves the most troublesome depth oscillation issues in previous methods while making LoD editing a natural byproduct.

Limitations & Future Work

  • The current method only supports closed Bézier curves as primitive shapes, without supporting open curves (stroking style) or richer SVG primitives (rectangles, circles, path morphing). Scaling to mixed primitives is a natural progression.
  • While the hierarchical strategy enables structural editability, semantic grouping of curves (e.g., "these 10 curves belong to the sky") still relies on manual identification, lacking automatic hierarchical semantic understanding.
  • The SSIM metric on DIV2K is slightly lower than some baselines—fast convergence driven by a high LR sometimes sacrifices local fine texture fitting. The authors attribute this to a design trade-off that "prioritizes editability," but this trade-off requires further optimization in scenarios demanding extreme fidelity.
  • All current experiments are evaluated under a fixed budget (up to 1024 curves). Whether higher-resolution images demand more curves or adaptive budget strategies remains to be explored.
  • vs Bézier Splatting [Liu 2025]: Both share the same rasterization backend (Bézier curves + 2D Gaussian splatting). Bézier Splatting adopts flat optimization—all curves are randomly initialized simultaneously, trained with the same learning rate, and dynamically sort depth by bounding box size. This work introduces area gradients through Interior Gradient Aggregation, achieves frequency separation with Progressive Stratification, and inflates learning rates by 50x via Rapid Inflation. Simply altering the optimization strategy yielded improvements more significant than replacing the backend.
  • vs LIVSS [Wang 2025]: LIVSS also uses hierarchical strategies but relies on external diffusion models to construct image hierarchies, incurring high computation costs. Ours is purely optimization-driven without external priors, running an order of magnitude faster than LIVSS.
  • vs Optimize & Reduce [Hirschhorn 2024]: O&R also attempts hierarchical path construction, but its "optimize then reduce" concept requires alternating pruning and reparameterization. Our monotonic densification strategy is more concise and does not require external semantic information.
  • vs DiffVG [Li 2020]: DiffVG pioneered differentiable vectorization but its curve-by-curve serial reconstruction is computationally expensive. Ours drastically improves efficiency through parallelized hierarchical optimization while maintaining editability.

Rating

  • Novelty: ⭐⭐⭐⭐ Applying the "gradient dynamics diagnosis + hierarchical optimization" system to vectorization is novel and mathematically self-consistent, though some individual components are not entirely first-of-their-kind.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive comparisons across 3 curve budgets × 5 baselines on two datasets, along with thorough hyperparameter sensitivity and LoD visualization, proving complete ablation of all components.
  • Writing Quality: ⭐⭐⭐⭐⭐ Motivation is clear (flat optimization \(\rightarrow\) topology collapse \(\rightarrow\) gradient imbalance), the three-component method is logically organized, and the experimental data and visualization are highly convincing.
  • Value: ⭐⭐⭐⭐⭐ Beyond achieving substantial empirical improvements (2.5x speedup + 1.4dB), it offers a methodological insight that "optimization structure itself is a critical design dimension."