SPECSIA: Stylization Dataset for Novel-View Enhancement in Drawing-based 3D Animation¶

Conference: ECCV2026
arXiv: 2607.00525
Code: None (To be released)
Area: 3D Vision
Keywords: Drawing-driven animation, Novel-view enhancement, Projection artifact correction, Paired stylization dataset, Data-level pre-training

TL;DR¶

This paper proposes the SPECSIA-15K paired stylization dataset (14,980 pairs of artifact-laden projections and clean targets) and a lightweight plug-and-play post-correction module DraViE. By replacing instance-level frame-by-frame optimization with a data-level pre-training prior, it effectively eliminates projection artifacts under novel views in drawing-driven 3D animation while preserving the original drawing style.

Background & Motivation¶

Generating 3D animations from a single 2D drawing holds significant value in fields such as character animation, content creation, and virtual avatars. Current mainstream pipelines adopt the flow of "single image \(\rightarrow\) 3D reconstruction \(\rightarrow\) skeletal binding \(\rightarrow\) motion application \(\rightarrow\) rendering back to 2D", wherein instance-level 2D refinement modules (e.g., DrawingSpinUp DSU, Occlusion-robust Stylization Framework OSF) are widely used to align execution rendering results with the input drawing. While these methods adequately preserve the character's appearance from the input viewpoint, they essentially perform pixel- or feature-level alignment under the constraints of a single character and individual input viewpoint. When the character rotates to unseen novel views, 3D reconstruction defects and projection distortions are amplified into noticeable artifacts, such as missing contours, texture speckles, and structural discontinuities. Single-sample-based alignment objectives cannot reliably correct these projection-induced errors.

The key challenge of this failure lies in the fact that correcting novel-view artifacts is a "must-see-to-correct" problem—only by repeatedly observing "what kind of projections lead to what kind of errors, and what the correct appearance should be" across a large number of different characters and views can the model learn generalizable correction patterns. However, the hand-drawn character animation field lacks such viewpoint-aligned "artifact-projection vs. clean-target" paired data, and existing methods must rely on individual instance-level optimization for each character, which inherently tends to overfit to limited observation views. The key insight of this paper is that since the bottleneck lies in the lack of scalable supervisory signals, one should proactively construct a large-scale paired dataset to provide this supervision. Core Idea: Build the SPECSIA-15K dataset, rendering artifact-free GT for 10 viewpoints of each 3DBiCar character, and then generate corresponding artifact-projection targets via single-view 3D reconstruction and reprojection, forming 14,980 'poor-to-excellent' training pairs. Based on this dataset, a lightweight post-correction module, DraViE, is trained to learn generalizable projection error correction priors from the data distribution level. Finally, an optional lightweight adaptive fine-tuning is employed to match the specific drawing style of the input character.

Method¶

Overall Architecture¶

The design philosophy of DraViE is modular post-correction: it does not alter the upstream 3D reconstruction, skeletal binding, and motion-driven pipelines, but instead intervenes as a plug-and-play refinement module after rendering the 2D projections. Given a single input drawing and a target motion sequence, the pipeline sequentially undergoes 3D reconstruction (e.g., Wonder3D) and automatic skeletal binding. After applying motion, it renders 2D projections Z_i at each timestep. DraViE takes Z_i, the foreground mask Z_mask_i, and the position prompt map Z_pos_i as inputs to output corrected frames Y_i. The mechanism of the entire process is that DraViE does not learn corrections frame-by-frame using single-sample optimization, but rather performs data-level pre-training on the SPECSIA-15K dataset to learn generalized projection error patterns across a massive number of characters and views, followed by a lightweight adaptive session during inference to align with the style of the input drawing if necessary.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Single 2D Drawing"] --> B["3D Reconstruction<br/>(Wonder3D)"]
    B --> C["Skeletal Binding + Motion Driven"]
    C --> D["2D Projection Rendering"]
    D --> E["DraViE<br/>Post-Correction"]
    E --> F["Refined Animation Frames"]
    D --> G["Projection Z_i"]
    G --> E
    H["Position Prompt<br/>Z_pos_i"] --> E
    I["Foreground Mask<br/>Z_mask_i"] --> E

    subgraph J["SPECSIA-15K Pre-training"]
        direction TB
        K["3DBiCar Character<br/>Original Assets"] --> L["10-View Rendering<br/>Artifact-Free GT"]
        M["Single-View Reconstruction<br/>→ Reprojection"] --> N["Artifact-Laden Projection"]
        L --> O["Patch-wise<br/>DraViE Training"]
        N --> O
    end

    O -.->|"Load Weights"| E

Key Designs¶

1. SPECSIA-15K Dataset: Constructing novel-view paired projection correction supervision

The artifacts generated by existing pipelines in novel views (contour distortion, texture loss, speckle noise) are essentially systematic errors caused by imperfect 3D reconstruction, and many identical distortion patterns are shared across different characters. However, there was previously a lack of large-scale "artifact-projection vs. clean-reference" paired data aligned by viewpoint, making it impossible to learn a generalizable correction prior. The construction concept of SPECSIA-15K is to select 1,498 characters from the 3DBiCar character asset library, and use the original assets to uniformly sample 10 viewpoints in the yaw direction to render无伪影的 RGBA images as GT Y_v (using the Blender Cycles rendering engine, with an additional Freestyle contour overlay where line thickness and grayscale are randomly sampled to increase diversity). It then simulates the real pipeline by selecting the frontal view Y_0 as the single-view input, running Wonder3D reconstruction \(\rightarrow\) Mixamo automatic skeletal binding \(\rightarrow\) reprojection to obtain the artifact-laden projection Z_v under the same 10 viewpoints. Each sample forms a triplet input (Z_v, Z_mask_v, Z_pos_v) paired with Y*_v, totaling 14,980 pairs. The foreground mask Z_mask is obtained by rendering pure silhouettes and is aligned using mask IoU to minimize structural misalignment. The position prompt Z_pos encodes normalized mesh vertex coordinates (\(\tilde{x}\), \(\tilde{y}\), \(\tilde{z}\)) into RGB colors and renders them, providing the model with spatial cues about "which region of the character's surface this patch originates from," which is a crucial supporting design for the subsequent patch-wise training to work effectively.

2. Data-Level Pre-Training Prior: Replacing Overfitting with Diversity

Traditional instance-level refinement optimizes each character independently, causing the model to overfit to the specific appearance under the input view, which often fails when encountering large-angle viewpoint changes. DraViE performs data-level pre-training on SPECSIA-15K to learn a generalized correction prior covering different characters and viewpoints. Since hand-drawn styles are mainly determined by local cues (line thickness, texture grain size) and full-image training makes the model prone to overfitting to specific poses (such as the T-pose), pre-training adopts a patch-wise strategy: under mask guidance, 32×32 patches are cropped from (Z, Z_mask, Z_pos) and Y*, forcing the model to learn "how local textures should behave under different views" rather than memorizing global poses. After the model predicts the patch, it is supervised within the masked region using a weighted combination of L1 loss, VGG19 perceptual loss (layers {0, 3, 5}), and LSGAN adversarial loss. Each training iteration samples a patch from a random character and viewpoint with a batch size of 1000 for 3 pre-training epochs. During inference, since the network is fully convolutional (based on a ResNet encoder-decoder structure containing 7 residual blocks), it can directly process full-resolution inputs without being restricted by patch size. Experiments show that this data-level training significantly reduces the novel-view FID from approximately 270 to 203 compared to the baseline, indicating that the model has acquired a truly generalizable artifact elimination capability.

3. Lightweight Adaptation: One-Step Fine-Tuning for Style Alignment

Although data-level pre-training brings generalization performance across characters, the testing input drawings may have unique line thicknesses or color tones (such as thickened contours or hand-drawn color preferences). To align with a specific drawing style while retaining the pre-trained correction prior, DraViE provides an optional lightweight adaptation step: using the pre-trained weights as initialization, it performs patch-wise fine-tuning for only 1 epoch on the reference drawing of the input character. The generator learning rate is reduced to 1/10 of the pre-training rate (\(4 \times 10^{-5}\)), the discriminator learning rate remains at \(4 \times 10^{-4}\), and other hyperparameters are consistent with pre-training. Fine-tuning does not alter the network architecture or loss functions, nor does it add extra parameters. Its key design boundary is: sampling a large number of patches from only one reference image (each character only has the original drawing of the frontal viewpoint) for a 1-epoch update—allowing the model to learn the character's color and texture preferences without forgetting the generalized correction prior learned on SPECSIA-15K due to over-tuning. Ablation studies show that without adaptation, the novel-view FID degrades from 206 to 221, and CLIP decreases from 0.859 to 0.838, alongside visual color shifts and over-smoothed textures, proving that although the adaptation is lightweight, its role is critical.

Loss & Training¶

Pre-training and adaptation utilize the same loss function combination: \(\mathcal{L} = \lambda_{recon}\mathcal{L}_{recon} + \lambda_{perc}\mathcal{L}_{perc} + \lambda_{adv}\mathcal{L}_{adv}\), where \(\lambda_{recon}=4.0\), \(\lambda_{perc}=6.0\), and \(\lambda_{adv}=0.5\). \(\mathcal{L}_{recon}\) represents the L1 loss, \(\mathcal{L}_{perc}\) represents the mean squared distance on VGG19 feature layers {0, 3, 5}, and \(\mathcal{L}_{adv}\) represents the LSGAN (MSELoss). All losses are calculated solely within the masked region. Performance details for the adaptive phase include a generator learning rate of \(4\times10^{-5}\) and a discriminator learning rate maintained at \(4\times10^{-4}\), using the Adam optimizer with \((\beta_1,\beta_2)=(0.9,0.999)\) and a weight decay of \(10^{-5}\).

Key Experimental Results¶

Main Results¶

Quantitative comparison on the 3DBiCar test set (100 characters \(\times\) 20 motion sequences = 2,400 animation samples). The following are the key metrics of various methods under novel views and frontal views:

View	Method	CLIP↑	SSIM↑	LPIPS↓	FID↓
Frontal View	DSU	0.902	0.840	0.212	205.19
	OSF	0.905	0.841	0.211	202.24
	DraViE	0.905	0.846	0.208	201.31
Novel View	DSU	0.847	0.833	0.250	280.83
	OSF	0.850	0.835	0.249	270.21
	DraViE	0.859	0.840	0.245	206.13

DraViE achieves overall leadership in novel views: FID is significantly reduced from 270+ to 206 (a drop of about 24%), and CLIP semantic consistency increases by approximately 1 percentage point. Performance on frontal views is comparable to or slightly better than the baseline, indicating that the correction module does not introduce degradation in observed views. A consistent trend of improvement is also observed on the Amateur Drawings hand-drawn test set, demonstrating that the results are not confined to assets within the training domain.

Ablation Study¶

Configuration	Novel-View CLIP↑	Novel-View FID↓	Description
Full model	0.859	206.13	Full model (Pre-training + Adaptation + Position Prompt)
w/o Lightweight Adaptation	0.838	221.48	Decreased style alignment, with deviations in color/texture
w/o Position Prompt	0.844	318.39	Novel-view FID spikes by 54%, losing spatial awareness
w/o Position Prompt + w/o Adaptation	0.834	358.87	Most severe degradation

Key Findings¶

Position prompt is a core component: Removing it causes the novel-view FID to spike from 206 to 318, an increase of 54%. Without spatial cues, the model can only perform conservative smoothing on artifact regions and fails to reconstruct geometric structures (such as mouth and leg contours).
Lightweight adaptation is highly cost-effective: Fine-tuning for only 1 epoch with a 10\(\times\) reduced learning rate brings an increase of +2.1 points in CLIP and a decrease of -15 points in FID under novel views, with line thickness and color tones visually aligning much closer to the input drawing.
Strong cross-pipeline generalization capability: Without specific backend retraining, the same SPECSIA pre-training weights consistently improve projections across three different 3D reconstruction backends: Wonder3D, InstantMesh, and CRM. This indicates that the model learns generalized projection-level artifact patterns rather than backend-specific noise.
Overwhelming blind test preference: In a blind test of 30 participants, the preferences for artifact elimination, style preservation, and overall quality were 73.3%, 60.0%, and 86.7% respectively in favor of DraViE.
View sector analysis: Even at large oblique views ranging from 90° to 180°, DraViE's CLIP/LPIPS remains optimal or tied with the best, showing that the correction effect does not degrade as the viewpoint angle increases.

Highlights & Insights¶

Solving test-time optimization via data-driven methods: This paper shifts "novel-view artifact correction" from instance-level optimization for each character to data-level learning. After a single large-scale pre-training session, the prior can be reused for arbitrary characters and motions, representing a classic paradigm of "trading data for generalization".
Ingeneous coupling of position prompts and patch-wise training: Patch-wise training inherently loses global spatial contexts. The position prompt compensates for this missing information by indicating "which region of the character's surface this patch belongs to." This cooperation allows the model to focus on local style details without losing spatial orientation.
Boundary-conscious design of lightweight adaptation: Fine-tuning for only 1 epoch with a 10\(\times\) reduced learning rate strikes a practical balance between "retaining the general prior" and "aligning with personalized drawing styles," avoiding catastrophic forgetting caused by full fine-tuning.
Transferable step-by-step construction of auxiliary signals: The pipeline design of hierarchically exporting mask \(\rightarrow\) position encoding \(\rightarrow\) final correction data from 3D assets serves as a valuable reference for other 3D-aware 2D enhancement tasks.
Engineering thoroughness of dataset construction: Details such as contour line randomization (thickness 1-4 pixels, grayscale 0-255), mask IoU alignment, and uniform 10-viewpoint sampling ensure the quality and diversity of the training data.

Limitations & Future Work¶

Limited by upstream 3D quality: As a 2D post-correction module, DraViE cannot repair severe upstream errors. When the reconstructed mesh blends two legs into one block, they cannot be reliably separated, and the model may even "hallucinate" non-existent thigh gaps. This calls for establishing a feedback loop between 2D correction and 3D reconstruction/binding in the future.
Frame-by-frame independent processing lacks temporal modeling: Although sharing a 3D proxy and position prompts provides implicit spatial consistency, flickering can still occur during rapid rotations, large-scale occlusion, or high-frequency detail loss. Explicit temporal attention or optical flow constraints should be future directions for improvement.
Dataset bias is not completely eliminated: SPECSIA-15K is constructed based on 3DBiCar assets and the Wonder3D pipeline, which may encode systematic biases of specific source assets and reconstruction backends. Cross-pipeline experiments support generalizability to some extent, but the bias is not eradicated. Richer asset sources and backend types will help to further enhance generalization.
Limited room for improvement on reasonable projections: When the upstream projection quality itself is already high, the improvement margin brought by DraViE remains relatively modest, which is an inherent constraint of post-correction methods.

vs DrawingSpinUp (DSU): DSU performs instance-level feature alignment on the input viewpoint of a single character, with the core limitation being overfitting to the observed view. DraViE replaces this with data-level pre-training, learning a generalized correction prior from a massive number of characters/viewpoints, significantly improving novel-view performance.
vs Occlusion-robust Stylization Framework (OSF): OSF makes robustness improvements for occlusion scenarios (e.g., mask-aware refinement) but still employs the single-character instance-level optimization paradigm. DraViE provides stronger view generalization through diverse supervision using the SPECSIA dataset.
vs General Image Stylization: Traditional stylization (AdaIN, CycleGAN) focuses on texture/color transfer rather than artifact correction. DraViE redefines stylization as "eliminating projection noise while preserving style". The problem formulation is different, and the solution path is closer to image restoration than style transfer.

Rating¶

Novelty: ⭐⭐⭐⭐ [The dataset construction idea effectively fills the gap in training data, and shifting the problem from instance-level optimization to data-level learning is a fundamental paradigm shift; however, the model structure itself is inherited from DSU, so the innovation lies primarily in the data and training paradigms.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Detailed quantitative and qualitative comparisons were conducted on 3DBiCar and hand-drawn datasets, covering cross-pipeline generalization, view sector analysis, resolution scalability, and blind ablation testing. The appendix provides a complete set of extended analyses.]
Writing Quality: ⭐⭐⭐⭐ [The motivations are clearly articulated and the experiments are solidly organized. The methodology section could further streamline common implementation details shared with baselines to highlight the distinct contributions of the dataset construction and training paradigms.]
Value: ⭐⭐⭐⭐⭐ [Provides a desperately needed high-quality paired dataset and a plug-and-play post-correction module in the field of drawing-driven 3D animation, presenting direct practical value for both academic research and industrial applications.]