Color Pass-Through via Camera-Display Coupling

Conference: ECCV 2026
Paper: ECCV Official
Project Page: Project Page
Area: Others (Computational Photography / Color Reproduction)
Keywords: Computational Photography, Color Pass-Through, Camera-Display Coupling, Metameric-Black, Spectral Reconstruction

TL;DR

Addressing noticeable chromaticity and lightness gaps when capturing real-world scenes on smartphone cameras and viewing them on screens, this paper proposes an end-to-end Color Pass-Through framework that couples camera and display as a unified system, utilizing a lightweight neural projector and rank-1 camera-null space color correction with a single-step observer calibration to faithfully reproduce perceived scene colors.

Background & Motivation

A core aspiration of photography and display systems is to faithfully reproduce a real scene's colors and contrast when presenting them on a screen. In daily life, however, what we see in the real world often differs substantially from what a smartphone screen presents or what an observer perceives in mixed reality pass-through headsets (such as Apple Vision Pro's video pass-through or virtual try-on AR mirrors). As users view live scenes on displays, noticeable shifts in chromaticity and lightness frequently occur: object hues drift, and displayed scenes appear overly bright, washed out, or desaturated compared to naked-eye observation. Historically attributed to sensor dynamic range limits or panel gamut constraints, this perceptual gap stubbornly persists even with modern high-end sensors and displays operating under standard ISP calibration and post-processing.

Analyzing the conventional International Color Consortium (ICC) workflow explains why existing approaches fail to close this perceptual gap. Conventional pipelines factor the high-dimensional capture-to-display process into two decoupled and separately calibrated stages: camera calibration that maps continuous scene radiance to three-channel intermediate color spaces (e.g., CIE XYZ), and display calibration that maps those coordinates back to emitted display radiance. Because physical scene radiance is intrinsically high-dimensional (\(\mathbf{s}_i \in \mathbb{R}^L, L \gg 3\)) while sensor measurements are inherently three-dimensional, this two-stage decoupling suffers from an intrinsic rank-deficiency bottleneck (\(\mathrm{rank}(\mathbf{D}\mathbf{F}\mathbf{C}) \le 3\)), leading to unavoidable cascading error accumulation across stages. Even advanced multi-illuminant auto white balance (AWB) or per-scene ColorChecker calibrations cannot eliminate this discrepancy because they rely on constrained illumination assumptions and standard color space transformations that cannot guarantee end-to-end fidelity for a specific camera–display hardware pair.

For typical pass-through scenarios where images are captured and immediately viewed on the same device, treating the capture and display components independently is suboptimal. Core idea: treat the camera and display as a single coupled system, learning a non-linear inverse projector via a physical re-capture protocol and compensating for residual observer color casts through a rank-1 approximation of the camera's metameric-black (camera-null) subspace with a single-step 3D observer calibration vector.

Method

Overall Architecture

The framework learns a corrective transformation \(\mathbf{F}\) applied to captured camera images before display such that display emissions, when integrated by a target observer (whether a human eye or a reference camera \(\mathbf{M} \in \mathbb{R}^{3 \times L}\)), match direct observation of the physical scene, i.e., \(\mathbf{M}\mathbf{s}_i \approx \mathbf{M}\mathbf{s}_i^*\). Because directly estimating an observer-specific high-dimensional operator is severely ill-posed, the system decomposes the problem into two distinct, cooperative modules: 1. Camera-Display Projector (\(\widehat{\mathcal{F}}_{\mathbf{C}}\)): Solves the pass-through mapping under the assumption that the observer's spectral sensitivity matches the capture camera (\(\mathbf{M} := \mathbf{C}\)), modeling the non-linear pseudo-inverse \((\mathbf{C}\mathbf{D})^\dagger\); 2. Camera-Null Color Correction (\(\boldsymbol{\delta}_i\)): Eliminates the perceptual color mask caused by spectral sensitivity discrepancies between human/observer vision and the camera (\(\mathbf{M} \ne \mathbf{C}\)) by estimating camera-null subspace components and applying a 3D observer-specific calibration vector \(\boldsymbol{\varphi} \in \mathbb{R}^{3 \times 1}\).

The end-to-end inference pipeline flows as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Scene Radiance & Capture<br/>Acquire raw camera measurement Cs"] --> B["Camera-Null Coefficient Prediction<br/>Spectral CNN predicts scalar a_i"]
    B --> C["Camera-Null Color Correction<br/>Combine PCA basis e and calibration vector φ to cancel cast δ_i"]
    C --> D["Camera-Display Projection<br/>Lightweight MLP models non-linear inverse (CD)†"]
    D --> E["Display Emission<br/>Observer M perceives color-accurate scene reproduction"]

Key Designs

1. Re-capture Driven Camera-Display Projector: Overcoming Non-Linearity Bottlenecks

Under the baseline setting where the observer shares identical spectral sensitivities with the camera (\(\mathbf{M} := \mathbf{C}\)), the theoretical corrective mapping reduces to \(\mathbf{F}_{\mathbf{C}} = (\mathbf{C}\mathbf{D})^\dagger \in \mathbb{R}^{3 \times 3}\). However, consumer smartphone displays introduce significant non-linearities in their default operational modes—including proprietary gamma curves, tone mapping, and color gamut mapping—that cannot be disabled. Consequently, a linear \(3 \times 3\) affine transform or standard 3D-LUT fails to invert \((\mathbf{C}\mathbf{D})\) accurately.

To resolve this, the authors establish a physical re-capture training protocol: high-quality clean images (denoised DIV2K) serve as digital supervision \(\boldsymbol{\sigma}_i \in \mathbb{R}^3\), are rendered on target display \(\mathbf{D}\), and are re-captured by paired camera \(\mathbf{C}\) to obtain measurements \((\mathbf{C}\mathbf{D})\boldsymbol{\sigma}_i\). After sub-pixel optical flow alignment, a compact pixel-wise neural network is trained to map \((\mathbf{C}\mathbf{D})\boldsymbol{\sigma}_i \to \boldsymbol{\sigma}_i\). The architecture employs an efficient MLP (2 hidden layers, width 128, only 30.8K parameters) augmented with two critical designs: - Positional Encoding (PE): Applied to input RGB channels to capture sharp non-linear transitions and color-vanishing regions where diverse sensor inputs compress into similar display colors; - Average Pooling on Green Channel: Filters out spatial interpolation artifacts introduced by Bayer array demosaicing before color transformation.

The projector \(\widehat{\mathcal{F}}_{\mathbf{C}}\) is optimized via an \(L_1\) objective: $\(\theta^* = \arg\min_\theta \sum_i \left\| \boldsymbol{\sigma}_i - \widehat{\mathcal{F}}_{\mathbf{C}}\big((\mathbf{C}\mathbf{D})\boldsymbol{\sigma}_i; \theta\big) \right\|_1\)$ On an NVIDIA RTX 4090 GPU, it processes 2K-resolution frames in 41.3 ms while achieving superior fidelity.

2. Null-Space Geometric Decomposition and Rank-1 Approximation: Isolating Metameric-Black Mismatches

When evaluated by an external observer such as human eyes or a reference DSLR (\(\mathbf{M} \ne \mathbf{C}\)), even a perfectly fitted \(\widehat{\mathcal{F}}_{\mathbf{C}}\) leaves a noticeable, tinted "color mask" over the display. The authors theoretically analyze this discrepancy by decomposing scene radiance \(\mathbf{s}_i\) along projection operator \(\mathbf{P}_{\mathbf{C}} = \mathbf{D}(\mathbf{C}\mathbf{D})^\dagger\mathbf{C}\): $\(\mathbf{s}_i = \mathbf{r}_i + \mathbf{n}_i, \quad \mathbf{r}_i = \mathbf{P}_{\mathbf{C}}\mathbf{s}_i \in \mathrm{Range}(\mathbf{D}), \quad \mathbf{n}_i = \mathbf{s}_i - \mathbf{r}_i \in \mathrm{Null}(\mathbf{C})\)$ Mathematical derivation shows that the residual color error term \(\boldsymbol{\delta}_i = \mathbf{C}(\mathbf{I} - \mathbf{P}_{\mathbf{M}})\mathbf{n}_i\) originates entirely from the camera's metameric-black space (\(\mathrm{Null}(\mathbf{C})\)), where \(\mathbf{C}\mathbf{n}_i = \mathbf{0}\). These spectral components are invisible to the camera sensor, yet once displayed through RGB emitters, they stimulate the observer's visual system \(\mathbf{M}\), causing perceived chromaticity shifts.

Because full spectral reconstruction of \(\mathbf{n}_i \in \mathbb{R}^L\) is ill-posed and the observer operator \(\mathbf{C}(\mathbf{I} - \mathbf{P}_{\mathbf{M}}) \in \mathbb{R}^{3 \times L}\) cannot be directly calibrated, the authors exploit the low intrinsic dimensionality of natural reflectance and radiance spectra. Performing PCA on camera-null spectra across diverse hyperspectral datasets reveals that the first principal component \(\mathbf{e} \in \mathbb{R}^{1 \times L}\) accounts for approximately 93% of the total variance! This motivates an elegant rank-1 approximation \(\mathbf{n}_i \approx a_i \mathbf{e}\), simplifying the correction term to: $\(\boldsymbol{\delta}_i = \mathbf{C}(\mathbf{I} - \mathbf{P}_{\mathbf{M}})\mathbf{n}_i \approx \boldsymbol{\varphi}(a_i \mathbf{e})\)$ Here, \(\mathbf{e}\) is an offline-computed global basis vector, \(a_i \in \mathbb{R}^1\) is a per-pixel scalar coefficient, and the high-dimensional observer interaction collapses into a compact 3-dimensional observer-specific calibration vector \(\boldsymbol{\varphi} \in \mathbb{R}^{3 \times 1}\).

3. Spectral Prior Network Prediction and One-Step Observer Calibration: End-to-End Synthesis

To estimate per-pixel camera-null coefficients \(a_i\), the authors train an efficient spectral reconstruction transformer network (MST++) using simulated camera raw measurements \(\mathbf{C}\mathbf{s}_i\) derived from hyperspectral datasets (ARAD-1K, CAVE, ICVL). The training loss combines spectral reconstruction, scalar coefficient fidelity, and total-variation spatial smoothness: $\(\mathcal{L} = \sum_i \Big( \lambda_1 \|\hat{\mathbf{s}}_i - \mathbf{s}_i\|_1 + \lambda_2 |\hat{a}_i - a_i| + \lambda_3 \|\nabla \hat{a}_i - \nabla a_i\|_1 \Big)\)$ Retaining the full-spectrum reconstruction term \(\|\hat{\mathbf{s}}_i - \mathbf{s}_i\|_1\) regularizes the feature representation and directly improves coefficient prediction accuracy.

At deployment time, calibrating for a new observer \(\mathbf{M}\) requires only a one-time grid search over candidate \(\boldsymbol{\varphi} \in \mathbb{R}^{3 \times 1}\) vectors on a single reference scene. Once calibrated, \(\boldsymbol{\varphi}\) remains constant across unseen scenes and varying illuminants. The complete pass-through pipeline evaluates as: $\(\mathbf{M}\hat{\mathbf{s}}_i^* = \mathbf{M}\mathbf{D} \cdot \widehat{\mathcal{F}}_{\mathbf{C}}\Big( \mathbf{C}\mathbf{s}_i - \boldsymbol{\varphi}(\hat{a}_i \mathbf{e}) \Big)\)$ enabling real-time, observer-aligned color reproduction.

Key Experimental Results

Main Results

Experiments were conducted using paired smartphones (HUAWEI Pura 70 Pro and Xiaomi 17 Pro Max), where one served as capture camera \(\mathbf{C}\) and the other as display \(\mathbf{D}\). A Sony ILCE-7M4 DSLR operated as a controlled digital observer \(\mathbf{M}\). Quantitative evaluations evaluated 24 ColorChecker patches under 15 challenging lighting conditions (10 correlated color temperatures from 2500K to 9000K, plus 5 narrow-band RGB-LED colors). Metrics include PSNR, CIE 2000 color difference (\(\Delta E_{mean}\)), and normalized stress (STRESS, where lower indicates better chromatic consistency).

The table below summarizes performance under diverse unseen lighting (gray numbers denote results with display brightness normalized to match the proposed method):

Methods	PSNR ↑ (Huawei)	PSNR ↑ (Xiaomi)	\(\Delta E_{mean}\) ↓ (Huawei)	\(\Delta E_{mean}\) ↓ (Xiaomi)	STRESS ↓ (Huawei)	STRESS ↓ (Xiaomi)
Default smartphone camera	13.78 (15.80)	14.61 (15.93)	14.62 (12.31)	13.49 (11.54)	26.23 (27.58)	25.07 (25.27)
ColorChecker calibration	15.02 (15.23)	16.36 (16.85)	18.49 (18.40)	15.32 (15.09)	38.85 (38.56)	36.42 (36.04)
Multi-illuminant Auto-WB	12.84 (12.95)	13.92 (14.41)	17.84 (17.37)	17.08 (16.26)	31.12 (30.48)	30.05 (29.66)
Ours w/o camera-null correction	27.32	27.84	6.49	5.97	17.27	16.39
Ours (Full Model)	28.65	29.10	5.18	4.79	17.48	16.12

In subjective user studies involving 10 human participants rating brightness and color accuracy on a 5-point Likert scale across 10 unseen scenes, the default camera scored 1.90 / 2.07, ColorChecker calibration scored 1.66 / 1.90, multi-illuminant Auto-WB scored 2.06 / 1.84, while the proposed method achieved 4.32 / 4.03, delivering an average gain of over +2.0 points.

Ablation Study

The authors ablated architectural options for the camera-display projector \(\widehat{\mathcal{F}}_{\mathbf{C}}\) and evaluated the contribution of individual loss components for the camera-null coefficient predictor.

Table 1a: Performance comparison of camera-display projector \(\widehat{\mathcal{F}}_{\mathbf{C}}\) architectures on DIV2K (2K resolution)

Methods	Params (K)	Runtime (ms)	PSNR ↑	\(\Delta E_{mean}\) ↓	\(\Delta E_{p95}\) ↓	STRESS ↓
IA-3DLUT	593.0	339.92	26.69	5.66	9.22	9.22
NiLUT	33.9	31.16	31.03	3.59	9.39	5.02
CSRNet	36.5	253.78	31.12	3.59	9.34	5.00
Ours (MLP + PE + G-AvgPool)	30.8	41.30	32.13	3.34	8.81	4.69

Table 1b: Loss function ablation for learned camera-null coefficients \(\hat{a}_i\)

Loss Configuration (\(\lambda_1\) spectral, \(\lambda_2\) scalar, \(\lambda_3\) TV)	\(\mathrm{PSNR}(\hat{a}_i, a_i)\) ↑	\(\mathrm{PSNR}(\hat{\mathbf{s}}_i, \mathbf{s}_i)\) ↑
\(\lambda_1\) only (spectral reconstruction)	28.27	14.94
\(\lambda_1 + \lambda_2\) (adding scalar direct supervision)	29.42	32.69
\(\lambda_1 + \lambda_3\) (adding TV smoothness)	29.00	31.49
Full Loss \(\lambda_1 + \lambda_2 + \lambda_3\) (1, 0.01, 1)	29.68	31.95

Key Findings

• High-Frequency Inversion Capability: While traditional 3D-LUTs struggle with non-linear tone curves (\(\Delta E_{mean} = 5.66\)), the lightweight MLP with positional encoding resolves the color-collapsing "vanishing" boundary, lowering color error to 3.34 with minimal parameters (30.8K).
• Critical Role of Camera-Null Correction: On the full pass-through task, relying solely on \(\widehat{\mathcal{F}}_{\mathbf{C}}\) yields \(\Delta E_{mean} = 6.49\) on Huawei; incorporating camera-null compensation drops the error to 5.18 (a 20.2% improvement), directly validating that metameric-black spectral mismatches are the primary source of perceptual color casts.
• Cross-Illuminant Stability of Calibrated \(\boldsymbol{\varphi}\): Sensitivity tests perturbing \(\boldsymbol{\varphi}\) by \(\pm 0.015\) demonstrated that observers consistently preserved their original calibration across diverse indoor and outdoor scenes, confirming that \(\boldsymbol{\varphi}\) captures observer-display ocular interactions independent of scene illumination.

Highlights & Insights

• Unified Capture-to-Display Coupling: Replacing decoupled ICC-style calibrations with an end-to-end framework overcomes the fundamental rank-deficiency bottleneck and avoids cascading color distortions.
• Metameric-Black PCA Simplification: Demonstrating that the camera-null subspace is dominantly explained (93% variance) by a single principal component provides a practical, rank-1 path to bypass otherwise intractable full-spectrum calibration.
• Practical One-Step Deployment: The framework requires only a single-step 3-parameter adjustment per observer, enabling seamless integration into mobile camera viewfinders and AR/VR pass-through engines.

Limitations & Future Work

• Device-Pair Re-capture Dependency: Each new camera-display hardware pairing requires a dedicated re-capture dataset and pre-training procedure, lacking zero-shot cross-device generalization.
• Manual Observer Calibration: Although \(\boldsymbol{\varphi}\) is stable once set, initial calibration requires user interaction or a reference camera; integrating gaze-tracking or ambient sensors could enable automated user profiling.
• Hardware Dynamic Range Bounds: Under extreme contrast where real-world luminance exceeds display peak nits, perceptual tone mapping inevitably encounters physical clipping limitations.

Related Work & Insights

• vs Computational Color Constancy / AWB: Prior methods (e.g., FC4, Cross-Camera AWB) focus exclusively on sensor-side illuminant estimation while neglecting display re-emission distortions. This work treats capture and display as an indivisible loop.
• vs Display Color Management (ICC / GoG Models): Standard display characterization maps through 3D reference spaces, suffering from metameric-black information loss. This approach directly addresses null-space discrepancies to ensure human-perceived color consistency.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering formulation of camera-display coupled color pass-through with rigorous null-space decomposition and rank-1 reduction]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative benchmarks via DSLR proxy, 15 unseen illuminants, and a 10-subject perceptual user study]
• Writing Quality: ⭐⭐⭐⭐⭐ [Elegant mathematical modeling, strong geometric intuition, and cohesive narrative linking theory to practical engineering]
• Value: ⭐⭐⭐⭐⭐ [High practical impact for VR/MR video pass-through, mobile camera viewfinders, and color-critical computational photography]