---

title: >- [Paper Note] Recolour What Matters: Region-Aware Colour Editing via Token-Level Diffusion description: >- [ECCV 2026][Image Generation][Colour Editing] ColourCrafter unifies token-level RGB feature fusion in latent space with a masked Lab-space perceptual loss to achieve structure-preserving continuous local recolouring. tags: - ECCV 2026 - Image Generation - Colour Editing - Diffusion Models - Lab Loss date: 2026-09-19 content_hash: cfbcbc709612644c

Recolour What Matters: Region-Aware Colour Editing via Token-Level Diffusion

Conference: ECCV 2026
Paper: ECCV 2026 Official
Project: https://yangyuqi317.github.io/ColourCrafter.github.io/
Area: Image Generation
Keywords: image editing, fine-grained colour control, diffusion model, token-level feature fusion, Lab perceptual loss

TL;DR

Addressing the limitations where discrete text prompts fail to capture continuous hues and global image conditioning incurs colour drift, ColourCrafter performs token-level fusion of RGB and image features in latent space under a masked Lab-space perceptual loss, enabling fine-grained, structure-preserving local recolouring supported by a newly curated dataset, ColourfulSet.

Background & Motivation

Colour is one of the most perceptually salient yet technically difficult attributes to control within generative diffusion systems. In high-precision downstream applications such as fashion prototyping, industrial industrial product design, and digital artwork rendering, users demand exact reproduction of designated hues while strictly preserving underlying material textures, specular highlights, and structural details. Nonetheless, conventional text-guided editing frameworks inevitably represent chromatic signals through natural language tokens. Because language is inherently discrete whereas human colour perception spans a continuous spectrum, even rich descriptors (e.g., "Tiffany blue", "Ancora red") fail to specify subtle tint shifts, and interpolating inside text embedding spaces routinely triggers discontinuous or erratic colour artifacts.

To transcend the expressiveness bottleneck of natural language, recent image-driven paradigms substitute linguistic prompts with explicit colour patches or reference exemplars. However, standard feature adapters—most notably exemplified by IP-Adapter frameworks—inject visual chromatic cues as global image-level style conditioning into cross-attention layers. In the absence of localized correspondence between the chromatic reference and the target spatial entity, generation degenerates into global tone transfer: the entire scene suffers hue drift, unintended background contamination, and texture degradation. While spatial masks can demarcate where editing occurs, masking alone fails to govern how accurately the target continuous chromaticity is reconstructed within the boundary. Compounding this challenge, the field lacks large-scale, aligned datasets featuring dense chromatic variations on identical visual instances, preventing models from learning robust disentanglement between object identity and colour transformation.

Resolving this tension requires an architecture that couples semantic localization with continuous chromatic fidelity at the representation level. Core idea: encode explicit RGB colour patches directly as visual tokens concatenated with image tokens within latent space, enabling selective region-aware colour propagation via multi-modal attention, while enforcing pixel-level chromatic accuracy through a luminance-chrominance decoupled, masked Lab-space perceptual loss alongside the curated ColourfulSet benchmark.

Method

Overall Architecture

ColourCrafter builds upon the Flux.1-Kontext backbone, which pairs a Diffusion Transformer (DiT) denoiser with Rectified Flow Matching (RFM). The framework accepts three distinct inputs: an original source image \(I_{\text{orig}}\), a reference RGB colour patch \(I_{\text{colour}}\), and a concise text localization instruction \(T\) (e.g., "Colour the chair"). Pre-trained, frozen encoders transform text into linguistic tokens and map images and colour patches into latent tokens. These representations are concatenated with temporal-spatial positional embeddings before entering the DiT's multi-modal attention layers, where token-level interactions selectively route chromatic information exclusively toward semantically target regions. Optimization is jointly driven by a rectified flow latent diffusion loss and a masked perceptual Lab-space loss, after which the VAE decoder synthesizes the fine-grained, structure-preserving recoloured output.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Source Image + Reference Colour + Text Prompt"] --> B["Latent Multi-Modal Token Unification<br/>VAE tokenization + 3D RoPE spatial-temporal alignment"]
    B --> C["Token-Level Multi-Modal Attention Fusion<br/>Full self-attention for selective colour propagation"]
    C --> D["Flow Matching Diffusion Backbone<br/>LoRA-adapted RFM velocity field estimation"]
    D --> E["Masked Lab-Space Perceptual Loss<br/>Decoupled luminance and chrominance on edited regions"]
    E --> F["Output: Structure-Preserving Recalibrated Image"]

Key Designs

1. Latent Multi-Modal Token Unification: Resolving Spatiotemporal Semantic Ambiguity
To handle multi-modal inputs without destructive structural interference, ColourCrafter utilizes a frozen text encoder \(E_T\) to map text instruction \(T\) into token sequence \(C_T\), while a frozen VAE encoder projects source image \(I_{\text{orig}}\) and reference colour patch \(I_{\text{colour}}\) into latent codes \(C_{I_o}\) and \(C_{I_c}\), which are concatenated into a joint context sequence \(C_I = \text{Concat}(C_{I_o}, C_{I_c})\). To prevent the diffusion denoiser from conflating noisy latent states with static conditional anchors, the model assigns 3D Rotary Position Embeddings (3D RoPE) across spatial coordinates and temporal offsets: noisy image tokens \(x\) are assigned a temporal index \(t = 0\) (\(u_x = (0, h, w)\)), whereas conditioning source tokens \(C_{I_o}\) and colour reference tokens \(C_{I_c}\) are both assigned \(t = 1\) (\(u_{C_I} = (1, h, w)\)). This formulation preserves 2D geometric topology while explicitly distinguishing conditional context from iterative generation targets.

2. Token-Level Multi-Modal Attention Fusion: Eliminating Global Drift via Selective Propagation
Unlike conventional adapters that inject reference features via separate cross-attention branches, ColourCrafter draws inspiration from unified DiT control (such as OminiControl) by concatenating noisy tokens \(x\), text tokens \(C_T\), and context tokens \(C_I\) into an all-to-all Multi-Modal Attention (MMA) block: $$ \text{MMA}([x; C_T; C_I]) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d}}\right) V $$ Under this unified sequence, colour tokens dynamically compute attention affinity scores guided by semantic text tokens against spatial image tokens. Consequently, chromatic information is propagated specifically to regions exhibiting strong semantic correspondence, while attention weights over background regions decay naturally, preventing tone pollution. To retain the base generative prior and ensure parameter efficiency, all backbone weights are frozen, and Low-Rank Adaptation (LoRA, rank=16) is injected exclusively into the attention projections.

3. Masked Lab-Space Perceptual Loss: Decoupling Luminance from Chrominance for Perceptual Fidelity
Because Euclidean distance in latent space does not correspond to uniform human perceptual chromatic sensitivity, diffusion loss alone leaves subtle hue drift; conversely, enforcing raw RGB mean-squared error collapses high-frequency contrast toward a desaturated grey mean. ColourCrafter resolves this by applying supervision within the perceptually uniform CIE Lab colour space. Decoded predicted images \(I_{\text{pred}}\) and ground-truth targets \(I_{\text{gt}}\) are converted from RGB to Lab space, isolating lightness \(L\) from chromatic opponent channels \(a\) (green-red) and \(b\) (blue-yellow). Guided by an automated foreground binary change mask \(M \in [0, 1]^{H \times W}\), channel-wise squared errors are restricted strictly to editing targets: $$ \mathcal{L}L^{\text{mask}} = |M \odot L|}} - M \odot L_{\text{gt}2^2, \quad \mathcal{L}|}^{\text{mask}} = |M \odot a_{\text{pred}} - M \odot a_{\text{gt}2^2 + |M \odot b|}} - M \odot b_{\text{gt}2^2 $$ $$ \mathcal{L}}}^{\text{mask}} = \lambda_L \mathcal{LL^{\text{mask}} + \mathcal{L} $$ This loss penalizes colour discrepancies solely within the mask, allowing natural luminance and shadow variations to persist across the object surface while forcing exact chromatic adherence on }^{\text{mask}\(a\) and \(b\) coordinates.

4. Automated ColourfulSet Benchmark Pipeline: Overcoming Paired Chromatic Data Scarcity
To supply the paired supervision required for fine-grained recolouring, the authors engineered ColourfulSet. Selecting 1,200 source images across 40 object categories from COCO 2017, they combined 144 W3C CSS-standard color names to generate synthesis instructions and rendered candidate samples using Flux.1-Kontext. An automated evaluation pipeline powered by Qwen2.5-VL scored samples across task alignment, background preservation, and chromatic realism, filtering the pool down to 80,000 paired instances. For every retained pair, Segment Anything (SAM) delineated object outlines, pixel difference maps \(D(x, y) = \|I_e(x, y) - I_o(x, y)\|_2\) with percentile thresholding established mask \(M\), and RGB histogram clustering identified the primary target vector \(c_{\text{target}}\), establishing a rigorous closed-loop training dataset.

Loss & Training

The overall training objective combines the latent rectified flow diffusion loss with the masked Lab-space perceptual loss: $$ \mathcal{L} = \mathcal{L}{\text{diffusion}} + \lambda}} \mathcal{L{\text{Lab}}^{\text{mask}} $$ where the velocity field diffusion objective is formulated as: $$ \mathcal{L}}} = \mathbb{E{t, x, C, C_T) - (\epsilon - x) |_2^2 \right] $$ Hyperparameter studies identified optimal weight balances at }, C_{I_c}, C_T} \left[ | v_\theta(z_t, t, C_{I_o}, C_{I_c\(\lambda_L = 0.5\) and \(\lambda_{\text{Lab}} = 0.8\). ColourCrafter was trained on a single NVIDIA A800 GPU (80GB) using the Adam optimizer with a learning rate of \(3 \times 10^{-5}\), batch size 1, classifier-free guidance scale 3.5, and LoRA rank 16 over 10,000 optimization steps (~40 hours). Denoising inference utilizes 28 sampling steps.

Key Experimental Results

Main Results

Quantitative evaluations were conducted on a curated benchmark derived from Unsplash, featuring 38 object categories across 90 high-resolution (\(1024 \times 1024\)) images selected for crisp contours and distinct colour properties. Evaluated metrics encompass CIE Lab total colour difference \(\Delta E^*_{ab}\), chroma-hue deviation \(\Delta E_{Ch}\), sRGB mean angular error \(\text{MAE}_{\text{RGB}}\), hue angular error \(\text{MAE}_{\text{Hue}}\), along with LPIPS, SSIM, and FID.

Method	\(\Delta E^*_{ab} \downarrow\)	\(\Delta E_{Ch} \downarrow\)	\(\text{MAE}_{\text{RGB}} \downarrow\)	\(\text{MAE}_{\text{Hue}} \downarrow\)	LPIPS \(\downarrow\)	SSIM \(\uparrow\)	FID \(\downarrow\)
ColorPeel [ECCV 2024]	60.96	53.40	77.79	61.95	0.38	0.72	79.30
ColorBind [WACV 2026]	53.42	47.26	62.78	44.12	0.36	0.70	69.68
Control-Color [IJCV 2025]	32.82	28.44	71.90	20.31	0.28	0.82	63.87
Flux.1-Kontext [2025]	41.36	36.31	48.34	23.19	0.29	0.74	63.01
FlowEdit [ICCV 2025]	66.26	56.32	83.85	61.59	0.29	0.79	81.32
UniEdit-Flow [2025]	73.26	63.87	91.60	79.47	0.31	0.80	46.57
ColourCrafter (Ours)	28.52	23.55	35.84	14.88	0.28	0.83	51.91

Ablation Study

Ablation experiments evaluated the sensitivity of the luminance loss coefficient \(\lambda_L\) and overall Lab loss weighting \(\lambda_{\text{Lab}}\).

Config (\(\lambda_L\) Ablation, fixed \(\lambda_{\text{Lab}}=0.8\))	\(\Delta E^*_{ab} \downarrow\)	\(\Delta E_{Ch} \downarrow\)	\(\text{MAE}_{\text{RGB}} \downarrow\)	\(\text{MAE}_{\text{Hue}} \downarrow\)	Note
\(\lambda_L = 0\)	45.15	39.18	56.06	34.47	Absence of luminance control degrades overall chromatic consistency
\(\lambda_L = 0.2\)	35.34	29.96	44.39	21.36	Gradual stabilization of colour accuracy
\(\lambda_L = 0.5\) (Default)	34.54	29.51	42.70	18.49	Optimal trade-off between luminance fidelity and chromatic vibrancy
\(\lambda_L = 0.8\)	40.79	33.93	51.37	23.37	Excessive brightness focus compromises saturation
\(\lambda_L = 1.0\)	35.19	27.87	50.09	23.10	Over-constrained lightness distorts hue tone

Config (\(\lambda_{\text{Lab}}\) Ablation, fixed \(\lambda_L=0.5\))	\(\Delta E^*_{ab} \downarrow\)	\(\Delta E_{Ch} \downarrow\)	\(\text{MAE}_{\text{RGB}} \downarrow\)	\(\text{MAE}_{\text{Hue}} \downarrow\)	Note
\(\lambda_{\text{Lab}} = 0\) (Diffusion loss only)	32.87	26.09	48.59	20.48	Unsupervised chromatic space leads to noticeable hue deviation
\(\lambda_{\text{Lab}} = 0.3\)	30.36	24.42	43.20	20.02	Marked improvement in chromatic precision
\(\lambda_{\text{Lab}} = 0.5\)	34.54	29.51	42.70	18.49	Balanced intermediate constraint
\(\lambda_{\text{Lab}} = 0.8\) (Full Model Default)	28.53	23.55	35.84	14.88	Best chromatic precision aligned with visual realism
\(\lambda_{\text{Lab}} = 1.0\)	37.50	31.67	47.34	20.72	Over-constraining diffusion suppresses texture realism

Key Findings

• Substantial Gains in Chromatic Precision: ColourCrafter surpasses existing baselines by a wide margin, reducing hue angular error \(\text{MAE}_{\text{Hue}}\) to 14.88 (compared to 20.31 for Control-Color and 23.19 for Flux.1-Kontext) and total colour deviation \(\Delta E^*_{ab}\) to 28.52, confirming the efficacy of token-level attention fusion over global feature injection.
• Critical Role of Masked Lab Space Constraint: Omitting the Lab constraint (\(\lambda_{\text{Lab}}=0\)) produces coarse colour matching with notable hue drift. Conversely, omitting the spatial mask (Ours w/o mask) forces the loss to penalize unedited pixels, diluting saturation and resulting in a washed-out appearance. Replacing Lab with an RGB MSE loss causes pixel regression toward the mean, generating desaturated, greyish imagery.
• Continuity Under Arithmetic Colour Progressions: As demonstrated in Fig. 1, when conditioning RGB references follow an arithmetic progression, ColourCrafter produces seamless, smooth chromatic transitions without discrete jumps or structural artifacts, proving strong manifold stability.

Highlights & Insights

• Token-Level Chromatic Conditioning: Rather than treating colour as an unaligned external style embedding or discrete text token, ColourCrafter treats colour patches as native visual tokens inside unified DiT self-attention, establishing natural localized colour-pixel correspondences.
• Decoupled Perceptual Supervision: Utilizing CIE Lab space decouples lightness \(L\) from opponent colour channels \(a\) and \(b\), enabling the model to faithfully enforce target hue and saturation while letting realistic surface illumination, reflections, and cast shadows remain intact.
• Automated Multi-Model Data Flywheel: The strategy of using Flux.1-Kontext for initial synthesis, Qwen2.5-VL for perceptual quality filtering, and SAM for mask extraction provides an exemplary blueprint for bootstrapping paired editing benchmarks without costly manual annotation.

Limitations & Future Work

• Synthetic Domain Gap: The training corpus relies entirely on pseudo-paired images generated by Flux.1-Kontext, which may not encompass the full gamut of real-world lens optics, lighting aberrations, or non-Lambertian material responses.
• Attenuation Under Small Colour Differences (Failure Mode): Empirical analysis reveals that when target colour difference is small (\(\Delta E^*_{ab} < 35.7\)), the gradient signal driving flow matching denoising becomes weak, occasionally causing under-editing or drifting toward adjacent colour clusters.
• Single-Entity Sequential Bottleneck: The current model handles only one target object and one reference colour per forward pass. Multi-object editing scenarios require sequential iterative passes, accumulating inference overhead.

Related Work & Insights

• vs ColorPeel [ECCV 2024]: ColorPeel disentangles colour from shape via text prompt embeddings, but text conditioning lacks localized spatial grounding, frequently inducing global saturation bleed; ColourCrafter uses latent token-level visual fusion and local Lab losses, yielding far superior colour accuracy (\(\Delta E^*_{ab}\): 28.52 vs 60.96).
• vs Control-Color [IJCV 2025]: Control-Color enforces spatial locality via external interactive hints, yet its chromatic conditioning remains coarsely injected; ColourCrafter's attention-driven token routing produces significantly tighter hue alignment (\(\text{MAE}_{\text{Hue}}\): 14.88 vs 20.31).
• vs IP-Adapter Baselines: IP-Adapter projects reference images as global style vectors via decoupled cross-attention, producing severe hue mismatch in object-level histograms; ColourCrafter grounds reference RGB tokens directly against image tokens within 3D RoPE-indexed self-attention, ensuring exact colour mapping.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Cohesive formulation of token-level RGB visual conditioning and masked Lab perceptual loss within DiT flow matching)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exhaustive quantitative and qualitative benchmarking, attention map inspection, user studies, and complex edge-case evaluations)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear logical progression, well-defined mathematical formulations, and rigorous ablation analyses)
• Value: ⭐⭐⭐⭐☆ (Sets a high standard for fine-grained image recolouring and offers a reusable methodology for paired generative dataset curation)