Difficulty-Conditioned Attribute-Specific Restoration for Low-Light Image Enhancement

Conference: ECCV 2026
Paper: ECCV Official Poster
Code: https://github.com/mingzhuzhang1/DCASR-LLIE
Area: Image Restoration
Keywords: Low-light image enhancement, difficulty-conditioned restoration, attribute-specific enhancement, diffusion-based prior learning, HVI color space

TL;DR

Addressing spatial degradation heterogeneity in low-light imagery, DCASR decouples restoration in HVI space using a Global Intensity Modulator for coherent brightness adjustment and a DCT-driven Structural Refinement Modulator for localized detail recovery, complemented by a two-stage privileged teacher-student diffusion strategy for residual-free inference.

Background & Motivation

Low-light image enhancement (LLIE) aims to recover natural visibility and structural details from images captured under severely insufficient illumination. While recent deep learning approaches have advanced the field through deterministic Retinex decompositions, structural cues, or generative GAN and diffusion models, virtually all existing solutions apply a globally uniform enhancement mapping across the entire image. However, real-world low-light images exhibit pronounced spatial degradation heterogeneity: mildly under-exposed areas are relatively straightforward to brighten, whereas heavily shadowed, high-noise, or chromatically distorted regions present severe challenges. Applying a spatially uniform mapping inevitably leads to suboptimal compromises, either over-smoothing fine textures in hard areas or amplifying noise and producing color casts in easier regions. Residual diagnostics reveal that high-error regions follow structured, content-dependent topological patterns rather than random noise distributions.

Allocating model capacity adaptively according to regional complexity requires an explicit restoration difficulty prior. Yet integrating such a prior is hindered by two fundamental tensions. First, degradation attributes are inherently heterogeneous: illumination degradation is predominantly low-frequency and globally coherent, whereas structural textures and chromatic fidelity are localized and high-frequency sensitive. Imposing a uniform scalar difficulty guidance in RGB space is easily overwhelmed by overall intensity variations, providing insufficient steering for fine-grained structural recovery. Second, while an oracle difficulty prior can be directly derived from supervision residuals during training, ground-truth residuals are absent at test time, making inferring an ill-posed, one-to-many difficulty distribution from a single low-light input a significant barrier.

DCASR addresses these challenges by decoupling attributes in the HVI color space and treating test-time prior acquisition as a conditional generative distribution learning task. A privileged teacher first distills oracle difficulty cues from training residuals, while a latent conditional diffusion student synthesizes aligned priors directly from single low-light inputs during inference. Core idea: decouple low-light restoration into global intensity rectification and high-frequency structural refinement in HVI space, modulate each branch with a learned difficulty prior via affine scaling and DCT-based attention gating, and synthesize test-time priors without residuals via latent conditional diffusion.

Method

Overall Architecture

DCASR consists of an HVI-based dual-branch enhancement backbone, two difficulty-driven modulators (Global Intensity Modulator and Structural Refinement Modulator), an Adaptive Prior Weighting (APW) supervision scheme, and a two-stage teacher-student prior learning pipeline. The degraded input image is first converted into the HVI color space and mapped to intensity features \(F_i\) and hue-saturation features \(F_{hv}\). A compact difficulty prior vector \(P\) (extracted by the privileged teacher during training or generated by the diffusion student at inference) guides both branches: the Global Intensity Modulator (GIM) predicts channel-wise affine parameters for global brightness calibration, while the Structural Refinement Modulator (SRM) uses orthonormal Discrete Cosine Transform (DCT) filtering to extract high-frequency responses that sharpen cross-attention in the HV branch. The modulated features are subsequently fused and projected back through an inverse HVI transformation to yield the enhanced image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input low-light image I_low<br/>HVI color space transform"] --> B["Dual-branch feature decoupling<br/>Intensity branch F_i / Structure branch F_hv"]
    P_in["Difficulty prior P / ˆP<br/>(Teacher residual extraction / Diffusion student synthesis)"] --> C["Global Intensity Modulator (GIM)<br/>Channel-wise affine parameters α, β"]
    P_in --> D["Structural Refinement Modulator (SRM)<br/>DCT frequency response P_hf + cross-attention"]
    B -->|F_i| C
    B -->|F_hv| D
    C --> E["Modulated feature fusion<br/>Inverse HVI transform & residual addition"]
    D --> E
    E --> F["Enhanced normal-light image I_output"]

Key Designs

1. Global Intensity Modulator (GIM): Spatially invariant illumination calibration Because illumination degradation is governed by low-frequency, globally coherent variations, intensity restoration benefits from holistic rectification rather than fragmented spatial adjustments. GIM takes the compact \(D\)-dimensional difficulty prior \(P \in \mathbb{R}^D\) and passes it through a lightweight linear mapping network \(\mathcal{M}_{gain}\) to generate channel-wise affine scaling \(\boldsymbol{\alpha}\) and shift \(\boldsymbol{\beta}\): $\([\boldsymbol{\alpha}, \boldsymbol{\beta}] = \mathcal{M}_{gain}(P), \quad \boldsymbol{\alpha}, \boldsymbol{\beta} \in \mathbb{R}^{C' \times 1 \times 1}\)$ These parameters modulate the latent intensity feature map \(F_i \in \mathbb{R}^{C' \times H' \times W'}\) via residual broadcasting: $\(\tilde{F}_i = F_i \odot (\mathbf{1} + \boldsymbol{\alpha}) + \boldsymbol{\beta}\)$ Here, \(\boldsymbol{\alpha}\) controls global dynamic contrast gain and \(\boldsymbol{\beta}\) regulates overall brightness offset. By enforcing spatially invariant channel modulation, GIM maintains illumination consistency across large areas and avoids localized patching artifacts.

2. Structural Refinement Modulator (SRM): DCT-driven high-frequency attention gating The HV branch focuses on suppressing noise, correcting chromatic distortion, and restoring fine-grained textures. Because a flat global embedding lacks high-frequency selectivity, SRM treats the 1D prior embedding \(P \in \mathbb{R}^D\) as a discrete signal and computes its orthonormal Discrete Cosine Transform (DCT) coefficients \(\mathcal{X}_k\): $\(\mathcal{X}_k = c_k \sum_{n=0}^{D-1} P_n \cos\left[\frac{\pi}{D}\left(n + \frac{1}{2}\right)k\right]\)$ A high-pass filter \(m_\tau[k] = \mathbb{I}(k \ge \tau)\) retains components above cutoff frequency \(\tau\), followed by inverse transformation to reconstruct a high-frequency structural response \(P_{hf} \in \mathbb{R}^D\). This response is broadcast to modulate the Query projection in the cross-attention mechanism: $\(\tilde{F}_{hv} = \mathrm{FFN}\left(\mathrm{Attn}\left(\phi_q(\bar{F} \odot (1 + \lambda P_{hf})), \phi_k(\bar{F}), \phi_v(\bar{F})\right)\right) + F_{coarse}\)$ where \(\bar{F} = \mathrm{Norm}(F_{coarse})\), \(F_{coarse} = F_{hv} \odot \boldsymbol{\gamma} + \boldsymbol{\delta} F_{hv}\), and \(\lambda\) dictates modulation strength. This frequency-aware modulation sharpens spatial attention, directing restoration capacity toward texture-rich and heavily degraded structural regions.

3. Adaptive Prior Weighting (APW): Hard-region focused supervision reallocation Standard uniform L1 loss over-indexes on easily restored flat areas while under-penalizing challenging dark and noisy patches. APW derives an initial spatial difficulty map \(M_{diff}\) from the estimated residual map \(I_{res}\), scales it to \(\tilde{M}_{diff} \in [0, 1]\) via min-max normalization, and generates a base importance weight map \(\tilde{W}\) via a scaled sigmoid function with a stop-gradient operator: $\(\tilde{W} = \mathrm{sg}\left[\frac{\sigma(\omega \tilde{M}_{diff})}{\mathbb{E}[\sigma(\omega \tilde{M}_{diff})]}\right]\)$ with sharpness factor \(\omega = 6.0\). A learnable scalar \(\xi \in [0, 1]\) balances uniform and hard-mining supervision to construct \(W = \xi \cdot \tilde{W} + (1 - \xi)\). The prior-guided objective is formulated as: $\(\mathcal{L}_{apw} = \|W \odot (\hat{I} - I_{gt})\|_1\)$ This allows the network to transition progressively from uniform initialization (\(\xi \approx 0\)) to hard-example mining (\(\xi \to 1\)) without destabilizing optimization.

4. Privileged Teacher-Student Prior Learning: Residual-free test-time inference Because true residuals are inaccessible during deployment, DCASR implements a two-stage distillation framework. In Stage I (Privileged Teacher), an oracle difficulty prior \(P \in \mathbb{R}^{256}\) is extracted using ground-truth residuals \(I_{res} = |I_{gt} - \Phi_{llie}(I_{low})|\) via dual encoders \(E_1(I_{low} \oplus I_{res})\) and \(E_2(I_{res})\), global pooling, and an MLP. In Stage II (Generative Student), the teacher is frozen and provides target distribution \(P\) for forward diffusion \(q(P_t | P)\). A conditional student encoder extracts features \(C\) from \(I_{low}\), enabling a lightweight denoising network \(\epsilon_\theta(P_t, C, t)\) to iteratively denoise pure Gaussian noise \(\epsilon_0 \sim \mathcal{N}(0, \mathbf{I})\) into pseudo-prior \(\hat{P}\), supervised by alignment loss \(\mathcal{L}_{cdp} = \|\hat{P} - P\|_1\). At test time, \(\hat{P}\) is synthesized from \(I_{low}\) alone.

Loss & Training

The framework is optimized over 1000 epochs with Adam (\((\beta_1, \beta_2) = (0.9, 0.99)\)). Stage I (epochs 1-500) trains the enhancement backbone and teacher extractor using: $\(\mathcal{L}_{stage1} = \mathcal{L}_1 + \mathcal{L}_{ssim} + \mathcal{L}_{edge} + \mathcal{L}_{perc}\)$ Stage II (epochs 501-1000) freezes the teacher, substitutes the L1 loss with \(\mathcal{L}_{apw}\), and incorporates the diffusion prior alignment constraint: $\(\mathcal{L}_{stage2} = \mathcal{L}_{apw} + \mathcal{L}_{ssim} + \mathcal{L}_{edge} + \mathcal{L}_{perc} + \eta \mathcal{L}_{cdp}\)$ with balancing hyper-parameter \(\eta = 0.01\). The learning rate starts at \(1 \times 10^{-4}\) and decays to \(1 \times 10^{-7}\) via cosine annealing.

Key Experimental Results

Main Results

Quantitative evaluations across LOL benchmarks and unpaired real-world datasets show that DCASR consistently outperforms state-of-the-art methods (Table 1 and Table 3 in the paper).

Dataset	Metric	Ours (DCASR)	Prev. SOTA (CIDNet / CWNet)	Gain
LOL-v1 (Paired)	PSNR (dB) ↑	24.042	23.830 (Retinexformer) / 23.809 (CIDNet)	+0.212 dB
LOL-v1 (Paired)	SSIM ↑	0.850	0.857 (CIDNet)	-0.007
LOL-v1 (Paired)	LPIPS ↓	0.082	0.086 (CIDNet)	-0.004
LOL-v2-real (Paired)	PSNR (dB) ↑	24.053	23.920 (URetinex++) / 23.904 (CIDNet)	+0.133 dB
LOL-v2-real (Paired)	SSIM ↑	0.874	0.866 (CIDNet)	+0.008
LOL-v2-real (Paired)	LPIPS ↓	0.109	0.122 (CIDNet)	-0.013
LOL-v2-synthetic (Paired)	PSNR (dB) ↑	25.610	25.501 (CWNet) / 25.129 (CIDNet)	+0.109 dB
LOL-v2-synthetic (Paired)	SSIM ↑	0.940	0.939 (CIDNet)	+0.001
LOL-v2-synthetic (Paired)	LPIPS ↓	0.045	0.044 (CWNet) / 0.045 (CIDNet)	Par
Unpaired (Mean over 4 sets)	NIQE ↓	3.76	4.06 (CIDNet)	-0.30

Ablation Study

Ablations on LOL-v2-real demonstrate the incremental contributions and placement necessity of each component (Table 4 and Table 5 in the paper).

Config	GIM	SRM	\(\mathcal{L}_{apw}\)	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	Note
(a) Baseline (U-Net)	✗	✗	✗	23.498	0.857	0.123	Base dual-branch architecture without prior modulation
(b) + GIM	✓	✗	✗	23.625	0.868	0.113	Intensity branch global affine modulation only (+0.127 dB)
(c) + SRM	✗	✓	✗	23.765	0.868	0.111	HV branch structural refinement only (+0.267 dB)
(d) + GIM + SRM	✓	✓	✗	23.894	0.870	0.112	Joint dual-branch modulation (+0.396 dB)
(e) Full model (DCASR)	✓	✓	✓	24.053	0.874	0.109	Integration of adaptive prior-weighted supervision (+0.555 dB)
* Inverted Placement (SRM-I, GIM-HV)	-	-	-	22.662	0.850	0.171	Swapping branch modulators severely degrades quality (-1.232 dB)
* Generic Substitution (CA-I, CA-HV)	-	-	-	23.012	0.864	0.131	Standard Channel Attention is sub-optimal (-0.882 dB)

Key Findings

• Attribute alignment is essential for difficulty modulation: Assigning GIM to the intensity branch and SRM to the HV branch matches the underlying physical characteristics of the degradation. Swapping their roles (SRM on intensity, GIM on HV) causes PSNR to drop sharply by 1.232 dB (from 23.894 dB to 22.662 dB), falling well below the unguided baseline.
• Generative diffusion handles prior uncertainty better than regression: When evaluating prior generator architectures (Table 6a), conditional latent diffusion achieves 24.053 dB PSNR, clearly surpassing a Transformer (23.611 dB), ResBlock (23.534 dB), and direct MLP regression (23.490 dB). This confirms that diffusion handles the multimodal ambiguity of predicting degradation difficulty from a single image more effectively.
• Residual error supervision outperforms hand-crafted priors: Comparing difficulty guidance against SNR maps, brightness maps, and attention maps (Table 6b), the residual-derived prior achieves the highest reconstruction fidelity and perceptual scores, proving that error-based guidance targets actual failure modes more accurately.

Highlights & Insights

• 1D DCT filtering on latent prior vectors: Rather than executing computationally heavy 2D frequency transforms over large spatial feature maps, SRM computes an orthonormal 1D DCT directly on the 256-dimensional latent prior vector, extracting high-frequency components to modulate spatial attention queries efficiently.
• Bypassing the inference-time residual paradox: Leveraging privileged learning to extract oracle difficulty priors during training and distilling them into a compact latent diffusion model neatly circumvents the impossibility of observing ground-truth residuals during testing.
• Stabilized difficulty reweighting: The combination of min-max normalization, a stop-gradient operator, and a learnable scheduling scalar \(\xi\) in APW prevents difficult outliers from dominating early gradients and causing optimization collapse.

Limitations & Future Work

• Diffusion sampling latency: Although the diffusion process is restricted to a compact 256-dimensional latent space, iterative reverse sampling adds computational overhead, resulting in an inference time of 0.148 seconds per 256×256 image. Accelerating this via one-step consistency distillation remains desirable for real-time mobile deployment.
• Extreme mixed lighting and specular highlights: The global affine modulation in GIM assumes spatially coherent low-frequency lighting; in scenes with harsh directional glare or saturated specular highlights alongside deep shadows, it may slightly amplify high-intensity artifacts.
• Future work: Integrating the learned difficulty prior into downstream detection, tracking, or multi-modal vision-language models to explore uncertainty-guided perception in adverse visual conditions.

Related Work & Insights

• vs CIDNet: CIDNet established the decoupled HVI color representation for low-light enhancement but treated all image regions uniformly. DCASR builds upon HVI decomposition while introducing explicit difficulty conditioning to adaptively reallocate restoration capacity.
• vs Diff-Retinex / LightenDiffusion: Standard diffusion-based LLIE models perform iterative denoising in image or high-dimensional feature spaces, incurring massive inference latencies. DCASR confines diffusion exclusively to a 256-dimensional latent prior, maintaining the fast forward execution of a feed-forward CNN while reaping the benefits of generative modeling.

Rating

• Novelty: ⭐⭐⭐⭐ [Elegant synergy between HVI attribute decoupling, 1D DCT prior filtering, and latent diffusion distillation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive evaluations across paired benchmarks, unpaired generalization, cross-domain shifts, and modulator placement ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-formulated methodology, and cohesive narrative]
• Value: ⭐⭐⭐⭐ [Offers a practical paradigm for incorporating privileged residual guidance into low-level vision tasks]