Skip to content

Multi-Block-Attention-based Color Constancy

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/OguzhanUlucan/HybridCC
Area: Interpretability
Keywords: Color Constancy, Illuminant Estimation, Hybrid Model, Multi-Block Attention, Soft-Maximum

TL;DR

HybridCC introduces a compact 1.1 MB hybrid architecture that guides a differentiable soft-maximum Retinex prior using learned saliency maps and refines block features via multi-head self-attention to eliminate aggregation bias, achieving state-of-the-art cross-camera illuminant estimation without requiring camera metadata or calibration.

Background & Motivation

The human visual system exhibits a remarkable ability known as color constancy, perceiving true surface reflectance consistently across diverse and fluctuating environmental illumination conditions. In sharp contrast, digital imaging sensors directly capture scene radiance modulated by the spectral power distribution of the illuminant and the camera sensor's spectral sensitivities. Consequently, captured images frequently suffer from pronounced color casts, making downstream vision tasks unreliable. Computational color constancy aims to estimate this global illuminant vector and neutralize it to obtain canonical, white-balanced images. Over the past decades, research has bifurcated into two dominant paradigms: classical statistical heuristics and deep learning models. Traditional methods like White-Patch Retinex assume that the highest-intensity pixel per color channel reflects a white or specular surface revealing the illuminant color; they are computationally negligible and sensor-agnostic, yet brittle when scenes lack bright cues or contain non-achromatic specular reflections. Conversely, deep neural networks directly map scene textures to illuminant vectors, delivering high in-distribution accuracy. However, they implicitly overfit the spectral profiles of training sensors, leading to severe performance collapse on unseen hardware and imposing a cumbersome dependency on pre-calibrated color correction matrices or camera-specific retraining.

At the intersection of these two paradigms lies an essential observation: not all pixels in a scene contribute equally to illumination estimation. While dividing an image into local spatial blocks enables algorithms to isolate spatially varying illumination cues, block-based methods consistently suffer from two unresolved challenges. First, within each individual block, determining which pixels are genuinely informative remains difficult without relying on fragile, hand-tuned percentile thresholds. Second, scenes in natural settings are overwhelmingly populated by expansive uniform regions—such as open skies, grass fields, pavement, or large foreground shadows. Because these monochromatic regions produce an overwhelming majority of blocks in a spatial grid, naive block-averaging or conventional pooling causes severe "aggregation bias," allowing uninformative background areas to dominate and degrade the final global illuminant estimate.

To resolve this fundamental tension, this paper proposes HybridCC, an end-to-end learnable hybrid framework that marries physics-based statistical priors with expressive deep contextual representations. Core idea: guide a differentiable soft-maximum Retinex statistical prior using learned pixel saliency maps, and deploy multi-head block-level self-attention to facilitate global cross-block information exchange, thereby suppressing aggregation bias from dominant uniform regions and enabling robust, calibration-free cross-camera illuminant estimation.

Method

Overall Architecture

HybridCC follows a two-branch hybrid topology. Given an input linear image, the model routes information concurrently through a backbone branch and a statistics branch. The backbone branch employs a compact feature extractor to predict unconstrained, pixel-level reliability weights (saliency maps) and encode spatially smoothed block representations. Guided by this saliency map, the statistics branch computes block-level statistical illumination priors using a temperature-parameterized differentiable soft-maximum and local intensity scaling. Block representations from both branches are subsequently concatenated and fed into an illumination estimation module, where a multi-head self-attention mechanism enables long-range information exchange across the block grid, effectively down-weighting uninformative and uniform regions. Finally, lightweight dual prediction heads yield per-block illuminant vectors and sparse combination coefficients to produce the final unit-norm global illuminant vector.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    IN["Input Image I<br/>384×384×3"] --> BB["Backbone Branch<br/>MobileNetV3-S Feature Extractor"]
    IN --> ST["Statistics Branch<br/>Non-overlapping Grid P=144"]
    BB --> SAL["Adaptive Saliency Prediction<br/>Bottleneck Dilated Convolutions"]
    BB --> SPM["Block Pooling & Spatial Mixing<br/>3×3 Depthwise Residual Updates"]
    SAL -->|Pixel Saliency Weights sk| ST
    ST --> SM["Differentiable Soft-Maximum & Intensity Scaling<br/>Learned Temperature Parameter τ"]
    SM --> MLP["Prior Feature Projection<br/>2-Layer MLP to D Dimensions"]
    SPM --> CAT["Block Feature Concatenation<br/>[F_stat ; F_backbone]"]
    MLP --> CAT
    CAT --> SA["Multi-Head Block Self-Attention<br/>Cross-Block Exchange Suppresses Bias"]
    SA --> HEAD["Lightweight Dual Prediction Heads<br/>Softplus Illuminant & Softmax Weights"]
    HEAD --> OUT["Final Normalized Global Illuminant ℓ̂"]

Key Designs

1. Adaptive Saliency Prediction and Local Spatial Mixing: Unsupervised Reliable Pixel Discovery To bypass hand-crafted percentile filtering and discover informative pixels adaptively across diverse scenes, the backbone branch leverages MobileNetV3-Small truncated at depth 8 as an efficient feature extractor. The branch splits into two distinct processing components: the saliency module and the feature processing module. The saliency module adopts a bottleneck design consisting of a \(1 \times 1\) channel-reducing convolution, a \(3 \times 3\) dilated depthwise convolution (dilation rate 2) for an extended receptive field, a \(1 \times 1\) channel expansion convolution, and a terminal \(1 \times 1\) projection producing a single-channel reliability map \(S = \sigma(f(F)) \in [0, 1]^{h \times w}\). Without any direct ground-truth saliency supervision, the network learns to highlight informative chromatic surfaces and penalize saturated artifacts solely through the end-to-end illuminant loss. In parallel, the feature processing module pools the feature map into an \(N_h \times N_w\) grid (\(P = 144\) blocks for \(384 \times 384\) inputs with \(32 \times 32\) block size) and applies two sequential spatial mixing layers with a learnable scaling factor \(\gamma\) (initialized to 0.1): $\(F_b^{(t+1)} = F_b^{(t)} + \gamma \cdot \phi \left( \left\{ F_{b'}^{(t)} \right\}_{b' \in \mathcal{N}(b)} \right)\)$ where \(\mathcal{N}(b)\) denotes the local \(3 \times 3\) spatial block neighborhood. This spatial mixing explicitly exploits illumination spatial smoothness, producing contextual block representations \(F_{\text{backbone}} \in \mathbb{R}^{P \times D}\).

2. Differentiable Soft-Maximum and Brightness-Scaled Priors: Continuous Generalization of Retinex To resolve the vulnerability of the classical hard maximum (Max-RGB) to sensor noise and uncalibrated specular highlights, the statistics branch formulates a temperature-controlled soft-maximum operator parameterized by a learnable scalar \(\tau\). Within each block \(b\), the upsampled saliency weights \(s_k\) are normalized such that \(\sum_{k \in b} s_k = 1\), yielding the chromatic descriptor \(m_b \in \mathbb{R}^3\): $\(m_b = \tau \log \sum_{k \in b} s_k \exp\left( \frac{x_k}{\tau} \right)\)$ As \(\tau \to 0\), \(m_b\) smoothly approaches the hard channel-wise maximum; for larger \(\tau\), it distributes mass across multiple prominent reflective pixels. Because chromatic direction alone fails to signal whether a block contains rich illumination information or dim shadow, the model additionally computes the saliency-weighted mean intensity \(\mu_b = \frac{1}{3} \sum_{k \in b} s_k \sum_{c \in \{r,g,b\}} x_{k,c}\), defining the block-level illumination prior vector as: $\(c_b = \frac{\mu_b}{\|m_b\|_2^2} m_b\)$ Crucially, \(c_b\) is not treated as an unrefined prediction—which would incur substantial error in monochromatic patches—but is projected via a 2-layer perceptron into a dense feature representation \(F_{\text{stat}} \in \mathbb{R}^{P \times D}\). This converts raw statistical priors into high-level features for downstream contextual refinement.

3. Multi-Head Block Self-Attention and Entropy-Regularized Aggregation: Eradicating Aggregation Bias When large monochromatic expanses such as clear sky or expansive terrain occupy the field of view, the sheer number of uninformative blocks biases conventional global pooling schemes. To overcome this aggregation bias, the fused block features \(F_b = [F_b^{\text{stat}} ; F_b^{\text{backbone}}] \in \mathbb{R}^{2D}\) are projected to \(D\) dimensions and processed through a 2-head self-attention module. Self-attention permits any block, regardless of spatial distance, to attend to informative, well-illuminated blocks elsewhere in the image. Blocks in shadows or uniform skies effectively query chromatically diverse surfaces to refine their internal state. Following self-attention, two lightweight linear heads compute the block estimates: an illuminant head with Softplus activation predicts strictly positive local vectors \(\hat{\ell}_b \in \mathbb{R}^3\), while a weight head predicts scalar logits \(w_b\). The global illuminant vector is obtained through temperature-scaled convex combination: $\(\hat{\ell} = \frac{\ell'}{\|\ell'\|}, \quad \ell' = \sum_{b=1}^P \alpha_b \hat{\ell}_b, \quad \alpha_b = \frac{\exp(w_b / T)}{\sum_{j=1}^P \exp(w_j / T)}\)$ where \(T\) is a learnable temperature parameter (initialized to 1.2) that controls the concentration of the weighting distribution, allowing the model to decisively isolate and discard uninformative block predictions.

Loss & Training

HybridCC is trained end-to-end by minimizing the angular error between the predicted unit-norm illuminant \(\hat{\ell}\) and the ground truth \(\ell^*\), combined with an entropy sparsity regularizer over the block weights: $\(\mathcal{L}(\ell^*, \hat{\ell}) = \arccos\left( \frac{\hat{\ell} \cdot \ell^*}{\|\hat{\ell}\| \|\ell^*\|} \right) + \lambda \mathcal{L}_{\text{sp}}(\alpha)\)$ where \(\mathcal{L}_{\text{sp}}(\alpha) = -\sum_{b=1}^P \alpha_b \log(\alpha_b)\). Because uniform weight distributions maximize entropy, penalizing \(\mathcal{L}_{\text{sp}}\) forces the network to concentrate combination weights onto a small, highly informative subset of reliable blocks. The hyperparameter is set to \(\lambda = 5 \times 10^{-4}\). The network is optimized via AdamW for up to 300 epochs with a batch size of 32 on a single NVIDIA RTX 4090 GPU, utilizing a base learning rate of \(3 \times 10^{-4}\), cosine decay, and a \(0.1\times\) scaled learning rate for the backbone parameters under balanced multi-dataset sampling.

Key Experimental Results

Main Results

Under the rigorous Leave-One-Dataset-Out protocol (evaluating cross-camera transfer with no sensor overlap) and the NUS-8 Cross-Sensor (CS) fold protocol, HybridCC consistently achieves the lowest mean angular error, exhibiting an especially pronounced performance margin on the Worst 25% hard cases. On the unseen mobile camera benchmark TA-AWB, it delivers state-of-the-art competitive performance without any in-domain exposure.

Dataset / Protocol Metric Ours (HybridCC) Prev. SOTA (CCMNet / C5 / FFCC) Gain / Margin
Gehler-Shi (Leave-out) Mean / Median / Worst 25% 2.11° / 1.60° / 4.65° 2.23° / 1.53° / 5.46° (CCMNet) Mean drops by 0.12°, Worst 25% improves by 0.81°
NUS-8 (Leave-out) Mean / Median / Worst 25% 2.26° / 1.79° / 4.78° 2.32° / 1.71° / 5.18° (CCMNet) Mean drops by 0.06°, Worst 25% improves by 0.40°
Cube+ (Leave-out) Mean / Median / Worst 25% 1.56° / 1.11° / 3.49° 1.68° / 1.16° / 3.89° (CCMNet) Mean drops by 0.12°, Worst 25% improves by 0.40°
INTEL-TAU (Leave-out) Mean / Median / Worst 25% 2.21° / 1.70° / 4.75° 2.52° / 1.70° / 5.96° (C5) Mean drops by 0.31°, Worst 25% improves by 1.21°
NUS-8 (Cross-Sensor) Mean / Median / Worst 25% 1.58° / 1.36° / 2.97° 1.71° / 1.31° / 3.62° (CCMNet) Mean drops by 0.13°, Worst 25% improves by 0.65°
TA-AWB (Zero-shot) Mean / Median / Worst 25% 2.63° / 1.95° / 5.80° 2.62° / 1.46° / 6.89° (FFCC) Competitive with DSLR-trained baselines; Worst 25% leads by 1.09°

Ablation Study

Ablations evaluated across Gehler-Shi, NUS-8, and Cube+ benchmarks demonstrate the necessity of each architectural component, block partitioning granularity, and backbone depth choice.

Config Variant Gehler-Shi NUS-8 Cube+ Average Mean Error (Avg) Note
Full model (Ours, \(32 \times 32\), Depth 8) 2.11° 2.26° 1.56° 1.97° Optimal configuration at 1.1 MB
w/o Stats 2.41° 2.59° 2.00° 2.33° Error surges by +0.36°; proves physical prior essential
w/ \(m_b\) only 2.23° 2.21° 1.86° 2.10° Omitting brightness scaling \(\mu_b\) discards informativeness
w/ smooth \(\tau\) (\(\tau_{\text{init}} = 0.5\)) 2.11° 2.44° 1.77° 2.11° Over-smoothed soft-max dilutes peak reflective cues
w/o Backbone 2.51° 2.57° 1.95° 2.34° Largest performance drop (+0.37°); loses saliency & context
w/o Saliency 2.13° 2.32° 1.73° 2.06° Reduces prior extraction to unweighted block statistics
w/o Self-Attention 2.16° 2.36° 1.69° 2.07° Lacks cross-block exchange; uniform bias re-emerges
w/o Sparsity Loss 2.33° 2.32° 1.78° 2.14° Weight entropy increases, accumulating noise from bad blocks
w/ equal block weights 2.36° 2.35° 1.81° 2.17° Uniform weighting suffers directly from aggregation bias
Block size \(16 \times 16\) 2.13° 2.41° 1.65° 2.06° Context within each block is too narrow for reliable statistics
Block size \(96 \times 96\) 2.21° 2.56° 1.91° 2.23° Coarse spatial resolution fails to isolate local outliers
MobileNetV3-S Depth 12 (5.9 MB) 2.16° 2.27° 1.85° 2.09° Deeper layers discard low-level color appearance cues

Key Findings

  • The Statistics Branch is the Pillar of Sensor Independence: Under the NUS-8 Cross-Sensor protocol, removing the statistics branch causes generalization error to degrade by 22% (rising from 1.58° to 2.01°), proving that pure deep feature regression memorizes sensor spectral biases while statistical priors provide camera-agnostic anchors.
  • Dramatic Gains on Worst-Case Scenarios: By empowering blocks to dynamically suppress uninformative areas via self-attention, HybridCC achieves its most decisive margins in the Worst 25% metric (e.g., reducing error from 5.96° down to 4.75° on INTEL-TAU), confirming extreme robustness against outlier scenes.
  • Empirical Stability of Temperature \(\tau\): Throughout training across disparate camera datasets, the learned temperature \(\tau\) consistently converges to a tight range near its lower bound (0.010 for Cube+ and Gehler-Shi, 0.014 for INTEL-TAU and TA-AWB, and 0.013–0.019 across NUS folds), validating that peaked, near-maximum color statistics are universally effective across camera sensors.

Highlights & Insights

  • Differentiable Generalization of Classical Retinex: Replaces brittle, hand-tuned percentile thresholds with a learnable, temperature-controlled Log-Sum-Exp formulation guided by unconstrained saliency maps, preserving the physical grounding of White-Patch theory while enabling gradient-based optimization.
  • Direct Solution to Block Aggregation Bias via Self-Attention: Rather than applying self-attention over abstract feature bags, HybridCC introduces it across spatial block representations, allowing monochromatic patches (sky, grass, shadow) to borrow illumination context from chromatically rich surfaces.
  • Ultra-Compact Footprint for Edge ISP Deployment: Requiring only 1.07 MB storage and executing in ~2.2 ms per image on an RTX 4090 GPU (nearly \(100\times\) faster than diffusion-based counterparts such as GCC at 180 ms), the architecture is exceptionally well-suited for integration into mobile camera pipelines.

Limitations & Future Work

  • Performance Degradation in Severely Monochromatic Environments: When scenes lack color diversity and contain virtually zero informative or contrasting surfaces (e.g., bare white walls, thick fog, or completely monochromatic studio setups), the model finds insufficient salient cues to propagate through attention, leading to residual variance.
  • Uniform Illuminant Assumption Under Multi-Light Scenes: The formulation presumes a single global illuminant under Lambertian reflectance. In the presence of mixed-color lighting (such as simultaneous indoor tungsten and outdoor daylight) or strong active light emitters (e.g., neon signs), a global single-vector estimate cannot represent spatially non-uniform illumination.
  • Future Directions: Extending the block-level self-attention mechanism to predict spatially varying illuminant maps, combined with physical illumination masks to separate direct light sources from reflected surface illumination.
  • vs Classical Heuristics (White-Patch Retinex [47], Shades of Gray [25]): Classical methods rely on rigid unweighted global/local statistics that easily fail under noise or large saturated regions. HybridCC continuously parameterizes Retinex via soft-maximum and end-to-end saliency, substantially enhancing robustness.
  • vs Fully Convolutional Color Constancy (FC4 [38], C5 [1]): FC4 performs confidence-weighted pooling and C5 enforces cross-camera constraints; however, they rely heavily on purely learned representations that risk camera bias, whereas HybridCC anchors representations with physical statistics at only 1.1 MB.
  • vs Generative Diffusion Models (GCC [18]): While GCC formulates color constancy as an inpainting diffusion process achieving strong representation capability, it suffers from heavy compute and ~180 ms latency. HybridCC delivers superior accuracy with a ~2.2 ms single-pass forward execution.

Rating

  • Novelty: ⭐⭐⭐⭐ [Clever differentiable soft-maximum Retinex formulation combined with spatial block self-attention to eliminate aggregation bias]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive validation across five benchmarks, leave-one-dataset-out, cross-sensor folds, and zero-shot mobile transfer]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, elegant mathematical formulation, well-structured figures, and self-contained empirical analyses]
  • Value: ⭐⭐⭐⭐⭐ [Ultra-compact 1.1 MB footprint and 2.2 ms inference speed offer immense practical value for mobile and edge photography pipelines]