SPECSIA: Stylization Dataset for Novel-View Enhancement in Drawing-based 3D Animation

Conference: ECCV2026
arXiv: 2607.00525
Code: None (To be released)
Area: 3D Vision
Keywords: Drawing-driven animation, novel-view enhancement, projection artifact correction, paired stylization dataset, data-level pretraining

TL;DR

This paper proposes the SPECSIA-15K paired stylization dataset (14,980 pairs of artifact-laden projections and clean targets) and DraViE, a lightweight plug-and-play post-correction module. By replacing sample-level frame-by-frame optimization with a data-level pretraining prior, it effectively eliminates projection artifacts in novel views for drawing-driven 3D animation while maintaining the original drawing style.

Background & Motivation

Generating 3D animations from a single 2D drawing holds significant value in fields such as character animation, content creation, and avatars. Current mainstream pipelines adopt a workflow of "single image \(\rightarrow\) 3D reconstruction \(\rightarrow\) rigging \(\rightarrow\) motion driven \(\rightarrow\) rendering back to 2D", where sample-level 2D refinement modules (e.g., DrawingSpinUp DSU, Occlusion-robust Stylization Framework OSF) are widely used to align the rendered results with the input drawing. These methods maintain the character's appearance well from the input viewpoint. However, they essentially perform pixel- or feature-level alignment under the constraints of a single character and a single input viewpoint. When the character rotates to an unseen novel view, 3D reconstruction defects and projection distortions amplify into obvious artifacts such as missing outlines, texture spots, and structural discontinuities, which sample-based alignment targets cannot reliably correct.

The core contradiction of this failure is that novel-view artifact correction is a "seen-to-be-corrected" problem—only when the model repeatedly observes "what kind of projection leads to what kind of error, and what the correct appearance should be" across a large number of diverse characters and viewpoints, can it learn a generalizable correction pattern. However, the hand-drawn character animation field lacks view-aligned "artifact-projection to clean-target" paired data, forcing existing methods to rely on sample-level optimization for each individual character, which inherently tends to overfit to limited observation views. The key insight of this paper is: since the bottleneck lies in the lack of scalable supervision signals, the author proactively constructs a large-scale paired dataset to provide this supervision. Core Idea: Construct the SPECSIA-15K dataset, render artifact-free GT for 10 views for each of the 1,498 3DBiCar characters, and then generate corresponding artifact-laden projections via single-view 3D reconstruction and reprojection, forming 14,980 pairs of "degraded \(\rightarrow\) clean" training samples. Based on this dataset, a lightweight post-correction module, DraViE, is trained to learn a generalizable projection error correction prior from the data distribution level. Finally, an optional lightweight self-adaptive fine-tuning step is applied to match the specific drawing style of the input character.

Method

Overall Architecture

The design philosophy of DraViE is modular post-correction: it does not alter upstream 3D reconstruction, rigging, and motion-driven pipelines, but instead intervenes as a plug-and-play refinement module after rendering the 2D projection. Given a single input drawing and a target motion sequence, the pipeline sequentially passes through 3D reconstruction (e.g., Wonder3D) and automatic rigging, renders a 2D projection \(Z_i\) at each timestep after applying motion. DraViE takes \(Z_i\), a foreground mask \(Z_{\text{mask}_i}\), and a position prompt map \(Z_{\text{pos}_i}\) as inputs to output the corrected frame \(Y_i\). The core of the entire process is that DraViE does not learn corrections on a per-character basis using single-sample optimization, but instead performs data-level pretraining on the SPECSIA-15K dataset to learn general projection error patterns across a large number of characters and viewpoints, followed by a lightweight self-adaptation step during inference if needed to align with the input drawing style.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Single 2D Drawing"] --> B["3D Reconstruction<br/>(Wonder3D)"]
    B --> C["Rigging + Motion Drive"]
    C --> D["2D Projection Rendering"]
    D --> E["DraViE<br/>Post-Correction"]
    E --> F["Refined Animation Frames"]
    D --> G["Projection Z_i"]
    G --> E
    H["Position Prompt<br/>Z_pos_i"] --> E
    I["Foreground Mask<br/>Z_mask_i"] --> E

    subgraph J["SPECSIA-15K Pretraining"]
        direction TB
        K["3DBiCar Character<br/>Original Assets"] --> L["10-View Rendering<br/>Artifact-Free GT"]
        M["Single-View Reconstruction<br/>→Reprojection"] --> N["Artifact-Laden Projection"]
        L --> O["Patch-wise<br/>Training of DraViE"]
        N --> O
    end

    O -.->|"Load Weights"| E

Key Designs

1. SPECSIA-15K Dataset: Constructing View-Aligned Paired Projection Correction Supervision

Existing novel-view artifacts (outline distortion, texture loss, spot noise) are systematic errors caused by imperfect 3D reconstruction, and different characters share many identical distortion patterns. However, there was previously no large-scale "artifact projection - clean reference" paired data aligned by viewpoint, making it impossible to learn a generalizable correction prior. The construction idea of SPECSIA-15K is as follows: select 1,498 characters from the 3DBiCar character asset library. First, the original assets are used to render artifact-free RGBA images as ground truth \(Y^*_v\) by uniformly sampling 10 viewpoints along the yaw direction (rendered with the Blender Cycles engine, with Freestyle used to overlay outline lines, where line thickness and grayscale are randomly sampled to increase diversity). Next, to simulate the real pipeline, the frontal view \(Y^*_0\) is selected as the single-view input and processed through Wonder3D reconstruction \(\rightarrow\) Mixamo automatic rigging \(\rightarrow\) reprojection, obtaining the corresponding artifact-laden projection \(Z_v\) under the same 10 viewpoints. Each sample forms a triplet input \((Z_v, Z_{\text{mask}_v}, Z_{\text{pos}_v})\) paired with \(Y^*_v\), totaling 14,980 pairs. The foreground mask \(Z_\text{mask}\) is obtained by rendering the pure silhouette and aligned using mask IoU to reduce structural misalignment. The position prompt \(Z_\text{pos}\) encodes the normalized mesh vertex coordinates \((\tilde{x}, \tilde{y}, \tilde{z})\) into RGB colors and renders them, providing the model with spatial clues about "which region of the character's surface this patch comes from", which is a critical design for the following patch-wise training to work effectively.

2. Data-Level Pretrained Prior: Replacing Overfitting with Diversity

Traditional sample-level refinement optimizes each character independently, causing the model to overfit to the specific appearance under the input view, which often fails when encountering large-angle changes. DraViE performs data-level pretraining on SPECSIA-15K to learn a general correction prior covering different characters and views. Since hand-drawn styles are mainly determined by local cues (line thickness, texture granularity) and full-image training easily leads the model to overfit to specific poses (such as T-pose), a patch-wise strategy is adopted during pretraining: \(32\times32\) patches are cropped from \((Z, Z_\text{mask}, Z_\text{pos})\) and \(Y^*\) under mask guidance, forcing the model to learn "how the same local texture should behave in different views" rather than memorizing global poses. After predicting the patches, weighted combinations of L1 loss, VGG19 perceptual loss (layers {0,3,5}), and LSGAN adversarial loss are used for supervision within the masked area. Each training step samples a patch from a random character and a random viewpoint, with a batch size of 1000, for 3 pretraining epochs. During inference, since the network is fully convolutional (based on a ResNet encoder-decoder structure with 7 residual blocks), it can directly process full-resolution inputs without being limited by patch size. Experiments show that this data-level training significantly reduces the novel-view FID from around 270 to 203 compared to the baseline, showing that the model learns a truly generalizable artifact elimination ability.

3. Lightweight Self-Adaptation: One-step Fine-tuning to Align Input Drawing Style

Although data-level pretraining brings cross-character generalization performance, test-time input drawings may have unique line thicknesses or color tones (such as thick outlines, hand-drawn color tendencies). To align with a specific drawing style while maintaining the pretrained correction prior, DraViE provides an optional lightweight self-adaptation step: using the pretrained weights as initialization, patch-wise fine-tuning is performed on the input character's reference drawing for only 1 epoch. The generator's learning rate is reduced to \(1/10\) of the pretraining rate (\(4\times10^{-5}\)), the discriminator's learning rate is kept at \(4\times10^{-4}\), and other hyperparameters remain consistent with pretraining. Fine-tuning does not alter the network architecture or loss functions, nor does it add extra parameters. Its key design boundary is: sampling a large number of patches from only one reference image (each character only has the original drawing of the frontal view) to perform a 1-epoch update—letting the model learn the color and texture preferences of this character without forgetting the general correction prior learned on SPECSIA-15K due to over-tuning. Ablation experiments show that without self-adaptation, the novel-view FID deteriorates from 206 to 221, and the CLIP score drops from 0.859 to 0.838, with color deviation and texture over-smoothing appearing visually, indicating that although self-adaptation is lightweight, it plays a key role.

Loss & Training

The same loss function combination is used for both pretraining and self-adaptation: \(\mathcal{L} = \lambda_{\text{recon}}\mathcal{L}_{\text{recon}} + \lambda_{\text{perc}}\mathcal{L}_{\text{perc}} + \lambda_{\text{adv}}\mathcal{L}_{\text{adv}}\), where \(\lambda_{\text{recon}}=4.0\), \(\lambda_{\text{perc}}=6.0\), and \(\lambda_{\text{adv}}=0.5\). \(\mathcal{L}_{\text{recon}}\) is the L1 loss, \(\mathcal{L}_{\text{perc}}\) represents the mean squared distance on VGG19 feature layers {0,3,5}, and \(\mathcal{L}_{\text{adv}}\) is the LSGAN (MSELoss). All losses are calculated only within the masked area. During the self-adaptation phase, the generator's learning rate is \(4\times10^{-5}\), while the discriminator is kept at \(4\times10^{-4}\). The Adam optimizer is used with \((\beta_1,\beta_2)=(0.9,0.999)\) and a weight decay of \(10^{-5}\).

Key Experimental Results

Main Results

Quantitative comparison on the 3DBiCar test set (100 characters × 20 motion sequences = 2,400 animation samples). The following table shows the key metrics of each method in the novel and frontal views:

View	Method	CLIP↑	SSIM↑	LPIPS↓	FID↓
Frontal View	DSU	0.902	0.840	0.212	205.19
	OSF	0.905	0.841	0.211	202.24
	DraViE	0.905	0.846	0.208	201.31
Novel View	DSU	0.847	0.833	0.250	280.83
	OSF	0.850	0.835	0.249	270.21
	DraViE	0.859	0.840	0.245	206.13

DraViE leads across all metrics in novel views: FID drops dramatically from 270+ to 206 (a reduction of about 24%), and CLIP semantic consistency improves by about 1 percentage point. On the frontal view, it performs comparably or slightly better than baselines, indicating that the correction module does not introduce degradation to the observed views. Consistent improvement trends are also observed on the Amateur Drawings hand-drawn test set, demonstrating that the results are not limited to the training domain assets.

Ablation Study

Configuration	Novel View CLIP↑	Novel View FID↓	Description
Full model	0.859	206.13	Full model (Pretraining + Adaptation + Position Prompt)
w/o Lightweight Adaptation	0.838	221.48	Style alignment degrades, color/texture deviations appear
w/o Position Prompt	0.844	318.39	Novel view FID surges by 54%, losing spatial awareness
w/o Position Prompt + w/o Adaptation	0.834	358.87	Most severe degradation

Key Findings

• Position prompt is a core component: Removing it causes the novel-view FID to spike from 206 to 318, an increase of 54%. Without spatial coordinates, the model can only conservatively smooth the artifact areas, failing to recover geometric structures (such as mouth and leg outlines).
• Lightweight self-adaptation is highly cost-effective: Fine-tuning for only 1 epoch with a 10× reduced learning rate brings a gain of +2.1 points in CLIP and -15 points in FID for novel views, making line thickness and color tones visually much closer to the input drawings.
• Strong cross-pipeline generalization ability: The same SPECSIA pretrained weights, without any backend-specific retraining, can consistently improve projections from three different 3D reconstruction backends: Wonder3D, InstantMesh, and CRM. This indicates that the model learns general projection-level artifact patterns rather than backend-specific noise.
• Overwhelming preference in blind tests: User studies on 30 participants show that preferences in artifact removal, style preservation, and overall quality are 73.3%, 60.0%, and 86.7% in favor of DraViE.
• View sector analysis: Even in large-angle backward views of \(90^\circ - 180^\circ\), DraViE's CLIP/LPIPS scores remain the best or on par with the state of the art, showing that the correction performance does not degrade as the view angle increases.

Highlights & Insights

• Solving test-time optimization using a data-driven approach: This paper reformulates "novel-view artifact correction" from a sample-level optimization per character to data-level learning—after one-time large-scale pretraining, the learned prior can be reused on any character and motion, serving as a classic paradigm of "exchanging data for generalization."
• Clever coupling of position prompts and patch-wise training: Patch-wise training naturally loses global spatial context. Position prompts fill in this missing piece of "which region of the character's surface this patch comes from," and their combination allows the model to focus on local style details without losing spatial rendering awareness.
• Boundary-aware design of lightweight self-adaptation: Fine-tuning for only 1 epoch with a 10× reduced learning rate strikes a practical balance between "preserving general priors" and "aligning individual styles," preventing catastrophic forgetting associated with full fine-tuning.
• Step-by-step construction of auxiliary signals is highly transferable: The pipeline design of hierarchically deriving mask \(\rightarrow\) position encoding \(\rightarrow\) final correction data from 3D assets provides constructive reference value for other 3D-aware 2D enhancement tasks.
• Engineering integrity of dataset construction: Incorporating randomized outline lines (1-4 pixels thickness, 0-255 grayscale), mask IoU alignment, and 10-view uniform sampling ensures the quality and diversity of training data.

Limitations & Future Work

• Constrained by upstream 3D quality: As a 2D post-correction module, DraViE cannot fix fatal upstream errors—when the reconstruction mesh fuses both legs into one block, it cannot reliably separate them and may even "hallucinate" non-existent crotch gaps. This calls for establishing a feedback loop between 2D correction and 3D reconstruction/rigging in the future.
• Frame-by-frame independent processing lacks temporal modeling: Although sharing the 3D proxy and position prompts provides implicit spatial consistency, flickering may still occur during rapid rotations, large-scale occlusions, or high-frequency details. Explicit temporal attention or optical flow constraints should be future directions for improvement.
• Dataset bias is not fully eliminated: SPECSIA-15K is built based on 3DBiCar assets and the Wonder3D pipeline, which may encode systematic biases from specific asset sources and reconstruction backends. Cross-pipeline experiments support generalization to some extent, but biases are not completely eradicated. Broader asset sources and backend types will help further improve generalization.
• Limited improvement margin on reasonable projections: When the upstream rendering projection is already of high quality, the margin of improvement by DraViE is relatively moderate, which is an intrinsic constraint of post-correction methods.

Related Work & Insights

• vs DrawingSpinUp (DSU): DSU performs sample-specific feature-level alignment on the input view of a single character. Its core limitation is overfitting to the observed views. DraViE replaces this with data-level pretraining, learning general correction priors from a large number of characters/viewpoints, which significantly improves novel-view performance.
• vs Occlusion-robust Stylization Framework (OSF): OSF implements robustness improvements (e.g., mask-aware refinement) targeted at occluded scenarios but still adopts the single-character, sample-by-sample optimization paradigm. DraViE provides stronger view generalization through diverse supervision on the SPECSIA dataset.
• vs General Image Stylization: Traditional stylization methods (AdaIN, CycleGAN) focus on texture/color transfer rather than artifact correction. DraViE redefines stylization as "correcting projection noise while maintaining style," addressing a different problem with a solution closer to image restoration than traditional style transfer.

Rating

• Novely: ⭐⭐⭐⭐ [The dataset construction idea effectively fills the training data vacancy, shifting the problem from sample-level optimization to data-level learning which represents a fundamental paradigm shift; however, the model architecture is inherited from DSU, meaning the key innovation lies in the data and training paradigm]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative and qualitative comparisons were conducted on 3DBiCar and hand-drawn datasets, covering cross-pipeline generalization, view-sector analysis, resolution scalability, ablation studies, and user studies. The appendix provides complete extended analyses]
• Writing Quality: ⭐⭐⭐⭐ [The motivation is clearly stated, and the experiments are solidly organized. The methodology section could be further streamlined by minimizing implementation details shared with baselines to emphasize the differentiating contribution of the dataset construction and training paradigm]
• Value: ⭐⭐⭐⭐⭐ [Provides a much-needed, high-quality paired dataset and a plug-and-play post-correction module to the drawing-driven 3D animation community, offering direct practical value to both academic research and industrial applications]

Related Papers

• [ECCV 2026] Capacity-Controlled Multi-View Stylization of 3D Gaussian Splatting
• [ICCV 2025] MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization
• [CVPR 2026] Charge: A Comprehensive Novel View Synthesis Benchmark and Dataset to Bind Them All
• [ICLR 2026] BigMaQ: A Big Macaque Motion and Animation Dataset Bridging Image and 3D Pose Representations
• [ICLR 2026] Stylos: Multi-View 3D Stylization with Single-Forward Gaussian Splatting