PeCA: Palette Context Assisted Inference for Test-Time Paint-Bucket Colourisation on Animation Videos¶
Conference: ECCV 2026
Paper: ECCV 2026
Project: https://rathgrith.github.io/PeCA/
Area: Others
Keywords: Video Colourisation, Paint-Bucket Colourisation, Region Correspondence, Test-Time Inference, Palette Context
TL;DR¶
Addressing ambiguous region matching under view/pose variations and fragmented regions in animation paint-bucket colourisation, PeCA introduces a training-free, plug-and-play test-time inference framework combining target-conditioned active reference expansion, Top-k soft voting probability aggregation, and cycle-consistent temporal fusion to consistently boost colourisation accuracy across both task-trained and frozen foundation models.
Background & Motivation¶
Hand-drawn animation colourisation is not an unconstrained image synthesis problem. In industrial production pipelines, region colours must strictly follow a discrete celluloid palette defined by character design sheets, and colours must be correctly assigned without bleeding across ink boundaries despite non-rigid character deformations, frequent occlusions, and severe layout shifts. While recent pixel-generative methods (including modern DiT-based models) produce visually appealing renders, they frequently violate production constraints by bleeding colours across boundaries and failing to strictly adhere to fixed discrete palettes. Consequently, the industry relies on segment-based paint-bucket colourisation, which treats colourisation as assigning discrete palette colour labels to enclosed line-sketch regions via regional visual correspondence with coloured reference frames or design sheets.
However, existing segment-based pipelines struggle with brittle correspondences caused by region ambiguity. When characters undergo large pose or viewpoint variations, or when regions are segmented into thin fragments under occlusion, individual target regions lack distinctive visual features in isolation, causing multiple reference candidates to display deceptively similar feature similarities. Relying on isolated Top-1 retrieval or unconstrained feature matching frequently leads to spurious correspondences and severe colour errors. Furthermore, empirical findings reveal that indiscriminately increasing the number of reference views quickly saturates performance gains, as expanding the candidate pool introduces high-similarity visual distractors that compound matching noise.
Human colourists resolve ambiguous patches by drawing on both spatial and temporal contexts—leveraging multiple reference angles to disambiguate local geometry and relying on adjacent video frames to confirm colour continuity. Rather than modifying the underlying representation or training an ever-larger correspondence backbone, this work approaches the problem from a test-time inference perspective. Core idea: reframe isolated region matching into palette-space belief reasoning by actively expanding reference views via submodular facility-location selection, aggregating noisy correspondences into continuous palette probabilities via Top-k soft voting, and refining cross-frame colour beliefs via cycle-consistent temporal gating.
Method¶
Overall Architecture¶
PeCA operates as a model-agnostic, plug-and-play test-time inference framework that enhances any feature-matching colourisation pipeline, seamlessly integrating with both colourisation-trained models (e.g., DACoN) and frozen foundation backbones (e.g., DINOv2, SAM 2.1). Given a target animation clip of \(T\) line-sketch frames \(\{I_t\}_{t=1}^T\), each frame is partitioned via flood fill into closed regions \(S_t = \{s_{t,i}\}_{i=1}^{N_t}\). A reference set \(R = \{(I^{(r)}, S^{(r)}, Y^{(r)})\}_{r=1}^R\) provides reference regions with ground-truth colour labels drawn from a finite palette \(C = \{c_1, \dots, c_{|C|}\}\).
The overall workflow proceeds through three sequential stages: first, active reference expansion generates a pool of geometric augmentations and greedily selects a budgeted subset that maximizes feature coverage over the target video; second, dense features extracted by the backbone are average-pooled across region masks to compute cosine similarities, which are transformed via Top-k temperature-scaled soft voting into discrete palette probability distributions; third, a bidirectional temporal sweep propagates colour beliefs across adjacent frames strictly through cycle-consistent links, reinforcing reliable colour transfers while suppressing error propagation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Target Sketch Video & Reference Set Input"] --> B["Active Reference Expansion<br/>Augmentation pool & greedy facility-location"]
B --> C["Correspondence Candidate Voting<br/>Top-k filtering & soft voting into palette simplex"]
C --> D["Cyclic-Gated Temporal Fusion<br/>Bidirectional cycle check & product probability update"]
D --> E["Final Discrete Palette Colour Output"]
Key Designs¶
1. Active Reference Expansion: Submodular facility-location balancing coverage against distractor exposure While providing diverse reference views increases the probability of finding geometrically aligned regions, naively expanding the reference set via test-time augmentation indiscriminately enlarges the candidate pool, drastically increasing exposure to high-similarity distractor regions. To maximize relevant spatial context under a fixed computational budget, PeCA introduces Active Reference Expansion (ARE). Beginning with base reference views \(V_0\), an augmented candidate pool \(V_{\text{aug}}\) of size \(|V_{\text{aug}}| = mB\) is generated via geometric transformations (rotations, flips, affine transforms). For each candidate view \(v \in V_{\text{aug}}\), its support to target region \((t, i)\) is quantified by the maximum cosine similarity among its regions: $\(\text{score}_v(t, i) = \max_{1 \le j \le N_v} \langle \bar{f}_{t,i}, \bar{f}_j^{(v)} \rangle\)$ To select a budgeted subset \(V \subseteq V_{\text{aug}}\) of size \(B\) that maximally covers target frames sampled uniformly over time (\(T_s \subset \{1, \dots, T\}\)), the selection is formulated as a monotone submodular facility-location objective: $\(F(V) = \sum_{t \in T_s} \sum_{i=1}^{N_t} \max_{v \in V} \text{score}_v(t, i), \quad \text{s.t.} \quad |V| = B\)$ Solving this objective with a greedy algorithm guarantees a standard \((1 - 1/e)\) approximation factor. Performed only once per video, ARE dynamically curates a support bank \(V_0 \cup V\) that significantly reduces appearance gaps while strictly controlling candidate pool expansion.
2. Correspondence Candidate Voting: Mapping discrete region matches to continuous palette consensus When multiple reference views are available, a target region naturally receives multiple candidate matches with high similarity scores. Standard top-1 hard assignment easily commits to isolated spurious matches, while global linear combination dilutes confidence by aggregating weak candidates. PeCA addresses this through Probability Aggregation (PA) over the discrete palette. For each target region \((t, i)\), candidate matches are restricted to the top-\(k\) nearest reference regions \(N_k(t, i)\). Using a temperature parameter \(\tau \in (0, 1]\), local normalized weights are computed: $\(p_{t,i}(r, j) = \frac{\exp(S_{(t,i),(r,j)} / \tau)}{\sum_{(r', j') \in N_k(t, i)} \exp(S_{(t,i),(r',j')} / \tau)}, \quad (r, j) \in N_k(t, i)\)$ These candidate weights are directly voted into the finite palette colour space \(C\): $\(P_{t,i}(c) = \sum_{(r, j) \in N_k(t, i)} p_{t,i}(r, j) \cdot \mathbb{I}[y_j^{(r)} = c], \quad c \in C\)$ This soft voting scheme projects unstructured region-to-region correspondence scores onto a standardized probability simplex \(\Delta^{|C|}\). This consensus mechanism dampens the impact of individual visual distractors and yields a shared, frame-independent probability space ideal for subsequent temporal fusion.
3. Cyclic-Gated Temporal Fusion: Bidirectional cycle-consistent propagation of palette beliefs Due to the continuous nature of animation videos, adjacent frames typically exhibit substantially smaller visual and layout disparities than distant design sheets. When direct reference matching is ambiguous, adjacent frames offer an effective transitive shortcut for colour propagation. However, unconstrained temporal propagation risks cascading single-frame matching errors throughout the entire sequence. PeCA resolves this dilemma with Cyclic-gated Temporal Fusion (CT). Given adjacent-frame region similarity \(A_t[i, j] = \langle \bar{f}_{t,i}, \bar{f}_{t-1,j} \rangle\), forward nearest-neighbor match \(\pi_t(i) = \arg\max_j A_t[i, j]\) and backward match \(\rho_t(j) = \arg\max_i A_t[i, j]\) are computed. A temporal correspondence link is deemed stable only if it satisfies bidirectional cycle consistency: $\(\rho_t(\pi_t(i)) = i\)$ For stable links, palette probabilities are updated via a product fusion rule followed by normalization: $\(\tilde{P}_{t,i}(c) \propto P_{t,i}(c) \cdot P_{t-1,\pi_t(i)}(c), \quad P_{t,i} \leftarrow \text{Normalize}(\tilde{P}_{t,i})\)$ The update is executed through a forward pass (\(t=2 \to T\)) followed by a backward pass (\(t=T-1 \to 1\)). The bidirectional passes condition on complementary temporal neighbors, effectively resolving ambiguous regions that possess reliable evidence in earlier or later frames.
Loss & Training¶
PeCA is an entirely training-free test-time inference method requiring zero gradient backpropagation, architectural modification, or parameter updates. Feature representations are computed directly using pre-extracted or frozen backbone representations (e.g., DACoN 1.1 task-trained weights or frozen DINOv2 ViT-L/14, SAM 2.1-Large, SigLIPv2) through region-wise average pooling. Hyperparameters are fixed globally across all benchmarks: top-\(k = 64\), temperature \(\tau = 0.05\), expanded reference view budget \(B = 31\), and candidate pool multiplier \(m = 4\).
Key Experimental Results¶
Main Results¶
The framework is comprehensively evaluated on the PaintBucket-Character benchmark (PBC-3D, 3,000 test frames), the professional hand-drawn testset PaintBucket-Real (PBC-Real, 200 frames across 20 clips), and a newly curated challenging long-shot benchmark Anita-Pirate (206 frames, ~140 segments per frame). Evaluation metrics follow standard conventions: segment Accuracy (Acc), Accuracy on segments larger than 10 pixels (Acc-Thresh), Pixel Accuracy (Pix-Acc), Foreground Pixel Accuracy (Pix-F-Acc), and Background Mean IoU (Pix-B-MIoU).
| Benchmark & Protocol | Backbone / Method | Training-Free | Acc (%) | Acc-Thresh (%) | Pix-Acc (%) | Pix-F-Acc (%) | Pix-B-MIoU (%) |
|---|---|---|---|---|---|---|---|
| PBC-3D 1-Shot Design-Sheet | BasicPBC-Ref [6] | Trained | 52.55 | 56.73 | 90.53 | 72.33 | 94.56 |
| PBC-3D 1-Shot Design-Sheet | DACoN 1.1 [23] | Trained | 68.01 | 72.87 | 96.97 | 91.03 | 99.11 |
| PBC-3D 1-Shot Design-Sheet | DACoN 1.1 + PeCA | Trained (Inference) | 72.04 (+4.03) | 77.08 (+4.21) | 97.90 (+0.93) | 94.04 (+3.01) | 99.42 (+0.31) |
| PBC-3D 1-Shot Design-Sheet | SAM2.1-Large (Base) | Yes | 34.54 | 38.95 | 86.76 | 54.12 | 88.37 |
| PBC-3D 1-Shot Design-Sheet | SAM2.1-Large + PeCA | Yes | 46.65 (+12.11) | 49.92 (+10.97) | 88.70 (+1.94) | 66.96 (+12.84) | 96.70 (+8.33) |
| PBC-3D 1-Shot Design-Sheet | DINOv2 ViT-L/14 (Base) | Yes | 57.49 | 61.86 | 95.35 | 87.24 | 97.45 |
| PBC-3D 1-Shot Design-Sheet | DINOv2 ViT-L/14 + PeCA | Yes | 61.38 (+3.89) | 65.58 (+3.72) | 96.25 (+0.90) | 89.31 (+2.07) | 98.62 (+1.17) |
| Anita-Pirate In-Between | BasicPBC [7] | Trained | 28.54 | 28.97 | 88.52 | 39.77 | 96.63 |
| Anita-Pirate In-Between | DACoN 1.1 [23] | Trained | 38.16 | 39.36 | 94.29 | 61.65 | 99.16 |
| Anita-Pirate In-Between | DACoN 1.1 + PeCA | Trained (Inference) | 41.24 (+3.08) | 42.16 (+2.80) | 94.29 (+0.00) | 62.78 (+1.13) | 99.43 (+0.27) |
| Anita-Pirate In-Between | DINOv2 ViT-L/14 (Base) | Yes | 28.55 | 29.30 | 93.06 | 53.88 | 99.40 |
| Anita-Pirate In-Between | DINOv2 ViT-L/14 + PeCA | Yes | 31.01 (+2.46) | 31.81 (+2.51) | 93.39 (+0.33) | 57.18 (+3.30) | 99.49 (+0.09) |
Ablation Study¶
On the PBC-3D one-shot design-sheet reference setting, the individual and cumulative contributions of ARE, PA, and CT are shown below:
| Backbone | ARE | PA | CT | Acc (%) | Acc-Thresh (%) | Pix-Acc (%) | Pix-F-Acc (%) | Pix-B-MIoU (%) |
|---|---|---|---|---|---|---|---|---|
| DACoN 1.1 | ✗ | ✗ | ✗ | 68.01 | 72.87 | 96.97 | 91.03 | 99.11 |
| DACoN 1.1 | ✓ | ✗ | ✗ | 69.04 | 74.02 | 97.22 | 91.67 | 99.24 |
| DACoN 1.1 | ✗ | ✓ | ✗ | 70.61 | 75.40 | 97.43 | 92.52 | 99.28 |
| DACoN 1.1 | ✗ | ✗ | ✓ | 70.30 | 75.27 | 97.21 | 92.47 | 99.38 |
| DACoN 1.1 | ✓ | ✓ | ✗ | 69.02 | 74.07 | 97.19 | 91.68 | 99.26 |
| DACoN 1.1 (Full) | ✓ | ✓ | ✓ | 72.04 | 77.08 | 97.90 | 94.04 | 99.42 |
| DINOv2 ViT-L/14 | ✗ | ✗ | ✗ | 57.49 | 61.86 | 95.35 | 87.24 | 97.45 |
| DINOv2 ViT-L/14 | ✓ | ✗ | ✗ | 58.74 | 63.43 | 95.96 | 88.14 | 98.57 |
| DINOv2 ViT-L/14 | ✗ | ✓ | ✗ | 60.64 | 64.66 | 95.94 | 88.32 | 98.60 |
| DINOv2 ViT-L/14 | ✗ | ✗ | ✓ | 58.68 | 62.35 | 94.56 | 84.52 | 97.99 |
| DINOv2 ViT-L/14 | ✓ | ✓ | ✗ | 58.51 | 63.29 | 95.89 | 88.08 | 98.52 |
| DINOv2 ViT-L/14 (Full) | ✓ | ✓ | ✓ | 61.38 | 65.58 | 96.25 | 89.31 | 98.62 |
Component-level design ablations further substantiate the specific algorithmic choices: 1. Facility-Location vs. Random Selection in ARE: Under SAM2.1-Large, greedy facility-location selection reaches 50.69% Acc (54.59% Acc-Thresh), significantly outperforming random selection with identical view budget (47.90% Acc), verifying that performance gains stem from targeted coverage rather than mere data expansion. 2. Cycle Gating in CT: On SAM2.1-Large, disabling cycle-consistency gating and unconditionally fusing nearest-neighbor predictions degrades Acc sharply to 45.92% (Acc-Thresh 49.11%), demonstrating that strict bidirectional verification is indispensable for blocking error propagation.
Key Findings¶
- Disproportionate Gains on Frozen Foundation Models: PeCA provides massive boosts to frozen backbones without task-specific tuning. For instance, SAM2.1-Large sees a +12.11% Acc surge (34.54% \(\to\) 46.65%) and a +12.84% gain in foreground pixel accuracy. This indicates that foundation representations inherently contain sufficient discriminative power, but were previously hamstrung by fragile point-to-point retrieval that PeCA successfully rectifies.
- Context Utility Correlates with Reference Modality: When design sheets serve as references, large spatial and pose discrepancies make Active Reference Expansion (ARE) the primary contributor. Conversely, in first-frame reference protocols where temporal proximity preserves layout, Cyclic-Gated Temporal Fusion (CT) yields the dominant share of improvement.
- Interlocking Synergy Across Stages: Applying ARE in isolation yields modest gains (~1% on DACoN) because additional views introduce matching noise. Only when coupled with Top-k soft voting (PA) to consolidate candidate evidence on the palette simplex does the full potential of reference expansion materialize.
Highlights & Insights¶
- Projection from Feature Metric Space into Palette Simplex: Rather than struggling with cross-image region alignment where region counts and scales vary arbitrarily, PeCA cleverly leverages the discrete colour palette as a unified canvas, transforming complex matching into soft voting and Bayesian-style updates on a shared probability simplex.
- Submodular Facility-Location for Pruned Test-Time Augmentation: Instead of unconstrained test-time augmentation that clutters the feature space with distractors, PeCA applies submodular facility-location to greedily pick only the views that best cover the target shot's missing poses under a strict budget.
- Practical Training-Free Modularity: By operating strictly at inference without architectural assumptions, parameter tuning, or backpropagation, PeCA offers immediate drop-in utility for real-world studio pipelines with zero retraining overhead.
Limitations & Future Work¶
- Irreversible Vulnerability to Line Leakage: Paint-bucket colourisation intrinsically relies on closed flood-fill segmentation. If hand-drawn sketches contain broken ink lines, flood fill merges disjoint anatomical parts, causing averaged region features to corrupt multiple semantic regions simultaneously.
- Closed-World Palette Assumption: The pipeline strictly assigns colours from the reference palette. If a target animation sequence introduces novel props, unseen characters, or dramatic lighting variations absent from reference sheets, the model cannot synthesize plausible out-of-palette colours.
- Future Directions: Developing open-world or interactive reference expansion, incorporating artist-in-the-loop stroke feedback and localized generative reference synthesis while strictly enforcing boundary constraints.
Related Work & Insights¶
- vs BasicPBC / BasicPBC-Ref [6, 7]: BasicPBC relies on topological inclusion matching and local temporal overlap, which breaks down under large non-rigid pose shifts typical of design sheets. PeCA maintains strict region assignment while using active reference expansion and probability aggregation to eliminate the requirement for rigid temporal overlap.
- vs DACoN [23]: DACoN adapts DINO features for anime paint-bucket matching via specialized training, but retains isolated Top-1 retrieval. PeCA addresses this decision layer bottleneck at test-time, pushing DACoN 1.1 SOTA even higher while enabling frozen foundation models to approach trained baselines.
- vs Pixel Generative Models (ColorFlow [52], AniDoc [22], LVCD [11]): Generative diffusion and autoregressive approaches synthesize smooth pixels that frequently bleed across ink lines and violate strict production colour palettes. PeCA adheres to discrete segment-level assignment, mathematically preventing boundary leakage.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Ingenious reframing of animation paint-bucket colourisation from a test-time inference perspective, elegantly unifying submodular selection and cycle-gated temporal fusion in palette space]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensively validated across 3D renders, professional hand-drawn shorts, and a new 206-frame long-shot benchmark, evaluating both task-trained and frozen backbones with meticulous ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, mathematically crisp formulations, and lucid analysis of failure modes and hyperparameter dynamics]
- Value: ⭐⭐⭐⭐⭐ [A plug-and-play, training-free toolkit that substantially improves production-level animation colourisation with immediate real-world applicability]