RawGen: Learning Camera Raw Image Generation

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://dy112.github.io/rawgen-page/
Area: Image Generation
Keywords: Camera Raw Generation, Diffusion Models, Inverse ISP, CIE XYZ, Many-to-One Reconstruction

TL;DR

RawGen presents the first diffusion-based generative framework that synthesizes physically consistent, sensor-level linear camera raw images from either text prompts or sRGB images, decoupling unknown and heterogeneous photo-finishing effects through a many-to-one inverse-ISP formulation anchored in canonical CIE XYZ space.

Background & Motivation

Modern digital camera sensors natively record scene-referred linear raw data that faithfully captures physical radiometric quantities and sensor irradiance. These linear measurements are foundational for computational photography, low-level vision algorithms, and learned image signal processor (ISP) pipelines. However, curating large-scale, high-quality raw image datasets is prohibitively expensive and labor-intensive. Because raw observations are inherently tied to specific sensor architectures, color filter arrays, and optical hardware, any change in camera hardware or onboard pipeline requires re-capturing data from scratch. This pervasive data scarcity severely hinders the development and generalization of sensor-centric low-level vision systems.

Generative diffusion models have demonstrated unmatched capability in synthesizing high-fidelity, semantically rich imagery, but their generation manifold is primarily confined to the display-referred 8-bit sRGB domain. Web-scale training corpora consist of images heavily altered by proprietary, diverse, and unknown in-camera ISP pipelines—involving nonlinear tone curves, white balance gains, gamma compression, and aesthetic color grading. Attempting to reverse-engineer linear raw measurements from diffusion-synthesized sRGB images using conventional inverse-ISP approaches inevitably fails. Traditional inverse-ISP models are trained on paired datasets generated under fixed and known software ISPs (such as RawPy or DCRAW), assuming a strict one-to-one mapping that breaks down completely when confronted with the uncontrolled and heterogeneous photo-finishing artifacts embedded in generative outputs.

Bridging the divide between high-capacity generative priors and physically grounded sensor data requires factoring out arbitrary post-processing variations and isolating an invariant scene-referred linear reference. Core idea: establish the device-independent linear CIE XYZ color space as a canonical radiometric anchor, introduce a many-to-one inverse rendering supervision that maps multiple stylized sRGB variations of a scene to this common target, fine-tune a rectified-flow DiT denoiser and a dedicated VAE decoder to reconstruct linear XYZ latents, and deterministically project the canonical output to arbitrary target camera raw spaces using standard calibration metadata.

Method

Overall Architecture

The core philosophy of RawGen is to decouple content synthesis from hardware-specific sensing: suppress post-processing variations within the latent space, establish radiometric linearity in a canonical device-independent domain, and handle sensor specificity via downstream calibration metadata. The system is built on a pretrained rectified-flow Diffusion Transformer (FLUX.1-Kontext) and a pretrained VAE, structured into two offline fine-tuning stages followed by a unified inference flow.

During training, real multi-camera raw images are first transformed using their native calibration metadata into canonical, scene-referred linear CIE XYZ anchor images \(I_{\mathrm{XYZ}}\). A parametric software ISP then generates multiple diverse sRGB renditions per scene by randomizing white balance gains, tone mapping curves, and contrast pivots. The frozen VAE encoder maps these stylized variations and the XYZ anchor into the latent space. Stage A fine-tunes the DiT via LoRA to denoise random sRGB-conditioned latents toward the invariant XYZ anchor latent, while Stage B fine-tunes the VAE decoder with an \(\ell_1\) reconstruction loss to accurately decode linear XYZ images from XYZ-domain latents. During inference, either an input sRGB image (Image-to-Raw, I2R) or an intermediate text-conditioned latent (Text-to-Raw, T2R) conditions the DiT to synthesize an XYZ latent, which is decoded into a linear CIE XYZ image and deterministically rendered into any target camera's raw space using its DNG metadata.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    subgraph S1["Data Preparation & Anchoring"]
        direction TB
        A["Multi-Device Raw Captures<br/>and DNG Metadata"] --> B["Many-to-One Dataset Construction<br/>Decouple ISP variations via XYZ anchor"]
    end
    subgraph S2["Model Fine-Tuning Stages"]
        direction TB
        B --> C["Conditional Flow-Matching DiT Tuning<br/>Predict velocity target via LoRA"]
        B --> D["Linear-Domain VAE Decoder Tuning<br/>Retarget decoder to physical linear XYZ"]
    end
    subgraph S3["Unified Inference & Target Mapping"]
        direction TB
        E["Input sRGB Image (I2R)<br/>or Text Prompt (T2R)"] --> F["Flow-Matching DiT generates XYZ Latent"]
        C -.-> F
        F --> G["Fine-Tuned Decoder reconstructs linear XYZ"]
        D -.-> G
        G --> H["Decoupled Camera Raw Rendering<br/>Interpolate color matrix & inject noise"]
    end
    H --> I["Arbitrary Target Camera Raw Output"]

Key Designs

1. Many-to-One Dataset Construction: Establishing Invariant Radiometric Anchors

Standard inverse-ISP approaches fail on web-scale or generative imagery because they rely on a fragile one-to-one assumption between an sRGB rendition and its underlying sensor measurements. In reality, a single scene can yield drastically disparate sRGB outputs depending on in-camera tuning and user grading. RawGen circumvents this ambiguity by posing a many-to-one inverse reconstruction objective. Using datasets such as MIT-Adobe FiveK and RAISE, the authors extract the AsShotNeutral and ForwardMatrix tags from DNG raw files to calculate a canonical, scene-referred linear CIE XYZ image \(I_{\mathrm{XYZ}}\) that precedes all nonlinear photo-finishing. A differentiable software ISP then applies randomized perturbations across three key parameter groups to produce \(N\) distinct sRGB variants \(\{I_{\mathrm{sRGB}}^{(n)}\}_{n=1}^N\): white balance gains perturbed around the DNG neutral initialization, diverse per-channel tone-curve shapes, and contrast scaling pivoted around mid-gray. The frozen VAE encoder projects each pair into latent representations: $\(z_{\mathrm{sRGB}}^{(n)} = E_{\mathrm{VAE}}(I_{\mathrm{sRGB}}^{(n)}), \quad z_{\mathrm{XYZ}} = E_{\mathrm{VAE}}(I_{\mathrm{XYZ}})\)$ By pairing multiple heterogeneous sRGB observations with a single shared XYZ target, the network is explicitly trained to collapse stylistic variations and recover an invariant physical anchor.

2. Conditional Flow-Matching DiT Tuning: Latent-Level Physical Alignment

To preserve broad semantic generative priors while learning the inverse-ISP mapping, RawGen adopts FLUX.1-Kontext, a rectified-flow DiT capable of native image context conditioning. During training, one sRGB variant latent \(z_{\mathrm{sRGB}}^{(n)}\) is randomly sampled as the condition for the anchor latent \(z_{\mathrm{XYZ}}\). Noise corruption follows a standard linear interpolation: \(z_t = (1-t)z_{\mathrm{XYZ}} + t\epsilon\), where \(\epsilon \sim \mathcal{N}(0, I)\) and \(t \in [0, 1]\). The DiT parameterized by \(\theta\) is trained to predict the rectified-flow velocity target \(v_{\mathrm{gt}} = \epsilon - z_{\mathrm{XYZ}}\): $\(\mathcal{L}_{\mathrm{denoise}} = \mathbb{E}_{n, t, \epsilon} \left\| v_{\mathrm{gt}} - v_\theta\left(z_t, t; z_{\mathrm{sRGB}}^{(n)}\right) \right\|_2^2\)$ The conditioning latent and corrupted target latent are patchified and concatenated along the sequence dimension, enabling bidirectional joint attention across tokens in the DiT transformer blocks. To prevent catastrophic forgetting and maintain generative stability, the backbone transformer weights remain frozen; trainable LoRA adapters of rank \(r=64\) and scaling \(\alpha=64\) are inserted into the attention projection layers to learn the transformation from nonlinearly styled latents to linear XYZ latents.

3. Linear-Domain VAE Decoder Tuning: Retargeting to Continuous Radiometry

Off-the-shelf VAE decoders in pretrained diffusion frameworks are optimized exclusively for 8-bit perceptual sRGB images. Consequently, feeding decoded XYZ latents directly into an unmodified decoder produces clipping, severe tonal banding, and color cast distortions because the decoder's normalization layers and output activations assume gamma-compressed distributions. To overcome this domain mismatch, RawGen introduces a targeted second fine-tuning stage for the VAE decoder \(D_{\mathrm{VAE}}\). Ground-truth anchor latents \(z_{\mathrm{XYZ}}\) serve as inputs, and the decoder is optimized against the ground-truth linear CIE XYZ images \(I_{\mathrm{XYZ}}\) via an \(\ell_1\) reconstruction loss: $\(\mathcal{L}_{\mathrm{recon}} = \left\| D_{\mathrm{VAE}}(z_{\mathrm{XYZ}}) - I_{\mathrm{XYZ}} \right\|_1\)$ This step retargets the decoder to continuous linear radiometric output while preserving the spatial resolution and structural fidelity captured by the encoder during pretraining.

4. Decoupled Camera Raw Rendering: Metadata-Driven Target Device Projection

Rather than forcing the neural network to memorize proprietary sensor characteristics for every camera brand, RawGen decouples generative synthesis from device physics. Once a canonical linear CIE XYZ image \(\widehat{I}_{\mathrm{XYZ}}\) is synthesized, it is deterministically mapped to the linear raw RGB space of any specified device using standard DNG metadata parsed from a single sample file. Specifically, given a chosen correlated color temperature (CCT), the system interpolates the forward calibration matrices corresponding to standard CIE illuminants (such as Standard Illuminant A and D65), projects \(\widehat{I}_{\mathrm{XYZ}}\) into white-balanced camera RGB coordinates, and scales the channels by the sensor illuminant gains. When required by downstream applications, synthetic heteroscedastic Gaussian-Poisson noise is added to emulate sensor shot and read noise, and the linear RGB image is re-mosaiced according to the target camera's Bayer CFA pattern (e.g., RGGB), achieving fully camera-agnostic synthesis without retraining.

Loss & Training

The framework optimizes two separate objectives: the DiT LoRA parameters are trained using the flow velocity loss in Eq. (3), while the VAE decoder weights are updated using the \(\ell_1\) reconstruction loss in Eq. (4). Both stages employ the AdamW optimizer. Training is conducted on the MIT-Adobe FiveK and RAISE datasets, covering a comprehensive spectrum of urban scenes, natural landscapes, portraits, and wildlife.

During inference, Image-to-Raw (I2R) extracts the conditioning latent from an input sRGB photograph via the frozen \(E_{\mathrm{VAE}}\) and solves the reverse ODE over 50 Euler steps. Text-to-Raw (T2R) taps directly into FLUX.1's native text-to-latent pathway to generate an intermediate sRGB-domain latent from a prompt, passing it through the identical conditional DiT to produce a linear XYZ latent, which is decoded and projected into the target camera raw format.

Key Experimental Results

Main Results

To evaluate robustness under real-world photo-finishing variations, RawGen was evaluated on the unseen test split of the MIT-Adobe FiveK dataset. Each scene in this benchmark was retouched independently by five professional photo editors (Experts A–E) with divergent stylistic and tonal preferences. RawGen was compared against established inverse-ISP methods—CIE XYZ Net, InvISP, and Raw-Diffusion—as well as an ablated variant trained with conventional one-to-one pairs (RawGen 1-to-1). Performance is measured against the ground-truth linear CIE XYZ reference using PSNR (dB) and SSIM.

Method	Expert A (PSNR / SSIM)	Expert B (PSNR / SSIM)	Expert C (PSNR / SSIM)	Expert D (PSNR / SSIM)	Expert E (PSNR / SSIM)
CIE XYZ Net (PAMI 2021)	19.60 / 0.7861	21.04 / 0.8431	19.49 / 0.7795	19.44 / 0.8058	18.64 / 0.7918
InvISP (CVPR 2021)	16.04 / 0.6854	16.04 / 0.6854	14.92 / 0.6323	14.24 / 0.6802	13.30 / 0.6473
Raw-Diffusion (WACV 2025)	19.30 / 0.7832	20.66 / 0.8397	18.79 / 0.7705	18.69 / 0.7966	17.49 / 0.7751
RawGen (1-to-1 baseline)	19.57 / 0.7782	21.21 / 0.8349	19.48 / 0.7741	18.74 / 0.7929	18.07 / 0.7839
RawGen (tuned denoiser only)	23.36 / 0.8407	24.35 / 0.8540	23.40 / 0.8390	23.50 / 0.8476	23.77 / 0.8448
RawGen (Full Model)	23.20 / 0.8432	24.35 / 0.8581	23.37 / 0.8387	23.51 / 0.8531	23.89 / 0.8500

Ablation Study

The ablations investigate whether traditional deterministic inverse-ISP architectures can match RawGen when provided the identical many-to-one (N-to-1) training data, and assess the degree of latent-space convergence across color grading variations. Table 2A reports average reconstruction over Experts A–E on MIT-Adobe FiveK along with perceptual color error \(\Delta E_{00}\). Table 2B evaluates the compactness of the VAE latent representations across 100 color-graded variants per prompt, measured as mean \(L_2\) distance to the centroid under PCA, t-SNE, and UMAP projections.

Table 2A: Training Protocol and Baseline Retraining Ablation (Average over Experts A–E)

Configuration / Method	PSNR (dB) ↑	SSIM ↑	\(\Delta E_{00}\) ↓	Note
CIE XYZ Net (N-to-1 retrained)	21.69	0.8070	9.66	Significant +2.05 dB boost over 1-to-1 version
InvISP (N-to-1 retrained)	8.92	0.2713	44.10	Invertible bijective architecture collapses under non-injective data
RawGen (1-to-1 training)	19.41	0.7928	13.85	Lacks invariance when tested against diverse expert stylizations
RawGen (Full Model N-to-1)	23.66	0.8486	8.39	Outperforms retrained baselines across all metrics

Table 2B: Latent Space Compactness Across 100 Color Variations (Mean \(L_2\) Distance to Centroid ↓)

Representation / Method	PCA Distance ↓	t-SNE Distance ↓	UMAP Distance ↓	Observation
Input sRGB variants	286.8	27.65	2.099	Substantial dispersion due to diverse color shifts
InvISP	265.4	24.33	1.913	Minimal style suppression
CIE XYZ Net	256.1	25.52	2.014	Weak convergence toward shared canonical anchor
RAW-Diffusion	318.2	19.58	2.557	Diffusion inversion preserves significant styling variance
RawGen (Ours)	160.0	10.69	1.067	Over 50% distance reduction, forming tight single-mode clusters

Table 3: Downstream Low-Level Vision Tasks Trained on Synthetic Raw Data

Training Data Source	Illuminant Estimation (NUS-8, 9 cams) Mean Angular Err ↓ / Worst 25% ↓	Neural ISP (PyNet) PSNR (dB) ↑ / \(\Delta E\) ↓	Sensor Denoising (SIDD) ISO 1600 PSNR ↑ / ISO 3200 PSNR ↑
EnlightenGAN	7.01° / 11.07°	35.58 / 3.137	48.82 / 47.25
UPI (CVPR 2019)	6.26° / 10.33°	36.43 / 2.907	49.05 / 47.51
Graphics2RAW (ICCV 2023)	4.21° / 8.57°	38.10 / 2.301	49.37 / 48.16
RawGen (3K Synthetic Samples)	3.14° / 7.37°	38.42 / 2.183	50.63 / 48.57
Real Data Reference (Real)	3.02° / 6.77°	38.32 / 2.133	49.80 / 48.25

Key Findings

• Many-to-one supervision is essential for undoing ISP styling: Training RawGen with single one-to-one pairs yields only 19.41 dB PSNR under expert retouches, whereas the many-to-one strategy elevates performance to 23.66 dB. When retrained on many-to-one pairs, CIE XYZ Net gains over 2 dB (19.64 to 21.69 dB). However, invertible networks like InvISP experience severe representational collapse (dropping to 8.92 dB) because many-to-one mapping violates bijective invertibility assumptions, confirming that a generative diffusion backbone is uniquely suited for this underdetermined mapping.
• Dense convergence in latent projections: In Table 2B, when 100 distinct color-graded variants of a single scene are mapped to XYZ, RawGen shrinks the latent distance to the centroid to 10.69 (t-SNE) and 1.067 (UMAP), reducing variance by over 50% compared to input sRGB and baseline models. This confirms that the model actively filters out stylistic noise rather than preserving input color biases.
• Synthetic raw data rivals and surpasses real dataset baselines: On raw sensor denoising at ISO 1600 (Table 3), models trained with RawGen synthetic data achieve 50.63 dB, surpassing models trained on real paired captures (49.80 dB). This advantage arises from the unbounded scene diversity, lighting variability, and semantic breadth provided by text-prompted diffusion generation, overcoming the physical coverage limits of small sensor calibration datasets.

Highlights & Insights

• Decoupling generative synthesis from sensor hardware: Rather than training separate diffusion models for each camera brand or CFA pattern, RawGen isolates content and lighting generation within the device-independent CIE XYZ space, delegating sensor specifics to linear forward matrices and parametric noise models. This enables zero-shot raw generation for any new camera given a single DNG file.
• Unprocessing via many-to-one reverse rendering: RawGen addresses the fundamental domain gap between web-scale generative models and scientific vision by treating post-processing as an unwanted style variable, training a flow-matching denoiser to project diverse stylized renditions into an invariant linear manifold.
• Physically grounded, non-destructive image editing: Unlike existing camera-controllable models that require repeating 50 diffusion steps for every exposure or white balance adjustment, RawGen generates the linear raw representation once. Downstream users can then perform white balance adjustment, exposure compensation, and custom tone curves in standard software ISPs (e.g., Lightroom or RawPy) in real time with zero quality degradation.

Limitations & Future Work

• Lack of optical aberration and sensor fingerprint modeling: The current mapping from CIE XYZ to raw employs a linear color matrix and a global heteroscedastic noise model, without accounting for spatial optical phenomena such as lens vignetting, chromatic aberrations, point spread functions (PSF), or non-uniform sensor microlens responses.
• Saturation and extreme highlight recovery: Because the underlying generative priors originate from 8-bit display-referred latent models, regions with complete highlight blowout rely on generative hallucination rather than true physical HDR irradiance reconstruction.
• Future directions: Integrating learnable sensor-specific priors, physically modeled wave-optics point spread functions, and continuous spectral sensitivity function estimation into the DiT framework to achieve end-to-end multi-spectral raw synthesis.

Related Work & Insights

• vs CIE XYZ Net / InvISP: Traditional inverse-ISP models rely on strict one-to-one paired datasets and struggle when applied to unknown, nonlinearly styled images. RawGen introduces the many-to-one training paradigm within a generative diffusion backbone, achieving superior stability and accuracy across unseen professional edits.
• vs Raw-Diffusion: Raw-Diffusion relies on fixed one-to-one software ISP mappings (e.g., RawPy) and synthesizes raw data tied to a specific camera. RawGen maps diverse inputs to a device-independent CIE XYZ space and deterministically projects to arbitrary camera targets.
• vs Generative Photography: Generative Photography simulates camera parameter adjustments by guiding iterative diffusion within the 8-bit sRGB space, requiring full diffusion passes for each setting modification. RawGen reconstructs true linear raw data, enabling immediate, non-destructive editing through conventional software ISPs without re-invoking the generative backbone.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering framework for direct text-to-raw diffusion synthesis and many-to-one inverse-ISP alignment in CIE XYZ space.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation including expert retouch benchmarks, latent clustering analyses, multiple downstream low-level vision tasks, and physical phone sensor renderings.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with rigorous color-science formulations, clear pipeline breakdowns, and thorough ablation studies.
• Value: ⭐⭐⭐⭐⭐ Substantially alleviates the acute scarcity of multi-camera raw data for computational photography, offering a scalable synthetic pipeline for real-world sensor applications.