Latent Space Imaging

Conference: CVPR 2025
arXiv: 2407.07052
Code: https://github.com/vccimaging/latent-imaging
Area: Image Generation / Computational Imaging
Keywords: Latent Space Imaging, single-pixel camera, generative models, extreme compression, optical-software co-design

TL;DR

Latent Space Imaging (LSI) proposes a new imaging paradigm that integrates optical encoding with generative model decoding. By directly encoding image information into the semantic latent space of StyleGAN, it achieves extreme compression ratios from 1:100 to 1:16384, while still enabling downstream tasks such as face reconstruction, attribute classification, segmentation, and landmark detection.

Background & Motivation

1. Background: Traditional digital imaging systems rely on brute-force pixel sampling on a regular grid. Although compressed sensing (CS) methods leverage sparsity and structural priors to reduce the number of measurements, reconstruction quality severely degrades at extremely low sampling rates. In recent years, co-design of optics and algorithms has made progress in fields such as color restoration, depth estimation, and super-resolution.
2. Limitations of Prior Work: Traditional CS methods suffer from severe over-smoothing in image reconstruction under extreme compression (e.g., 1:1000+), losing crucial identity features such as facial hair and eye shapes. Existing single-pixel camera methods have limited compression rates, making it difficult to break through the trade-off between reconstruction quality and compression rate.
3. Key Challenge: Pixel-space reconstruction is inherently limited by Shannon's sampling theorem. However, the human visual system compresses information from 120 million photoreceptors to 0.7–1.7 million optic nerve fibers while maintaining highly efficient perception. Biological vision achieves extreme compression by encoding visual information into a "latent representation" suitable for processing by the brain.
4. Goal: How to design a new imaging system that encodes images directly into a semantically rich, low-dimensional latent space at the hardware level, rather than a pixel grid?
5. Key Insight: The latent space of generative models (e.g., StyleGAN) is semantically rich, well-disentangled, and compact, making it suitable as a target representation for imaging systems. Linear boundaries within the latent space naturally support downstream tasks like classification and segmentation.
6. Core Idea: Use an optical encoder (physical mask) + a digital encoder (small DNN) to map the scene directly into the StyleGAN latent space, bypassing pixel reconstruction to perform downstream tasks directly within the latent space.

Method

Overall Architecture

The LSI pipeline comprises three core components: (1) Optical Encoder \(O\)—a physical mask matrix that linearly projects high-dimensional images \(I \in \mathbb{R}^{mn}\) into a low-dimensional measurement vector \(J \in \mathbb{R}^d\); (2) Digital Encoder \(\mathcal{D}_\theta\)—a non-linear DNN that maps the measurement vector into the StyleGAN latent space \(L \in \mathbb{R}^{512 \times 18}\); (3) Generative Model \(\mathcal{G}\)—a pre-trained StyleGAN-XL that decodes images from latent codes. The core formulation is \(L = \mathcal{D}_\theta(O \cdot I)\), where \(O\) and \(\mathcal{D}_\theta\) are jointly optimized for latent space reconstruction.

Key Designs

1. Optimizable Optical Encoder:

   • Function: Implement extreme data compression at the physical level by projecting high-dimensional images into a very small number of scalar measurements.
   • Mechanism: Each row of the optical encoder \(O \in \mathbb{R}^{d \times mn}\) corresponds to a physical mask pattern. In the single-pixel camera implementation, masks are realized using a Digital Micromirror Device (DMD) for time-multiplexed measurement. During training, a Straight-Through Estimator (STE) is used to handle binary quantization (constraining mask values to 0/1), ensuring binary masks during the forward pass and smooth gradient flow during the backward pass. An energy-efficiency loss is also introduced to ensure a \(1\%\) intensity difference for each mask, enabling the system to effectively distinguish between different patterns.
   • Design Motivation: Physical realizability is the key constraint—mask values must be non-negative (amplitude modulation) and quantized to binary values to match the high frame rate capabilities of DMDs. The optimized mask patterns automatically focus on crucial facial regions (eyes, nose, mouth contours), showing that the system learns domain-specific information compression strategies.

2. Hierarchical Digital Encoder:

   • Function: Non-linearly expand the linearly compressed low-dimensional measurement vector into StyleGAN's full latent space representation.
   • Mechanism: Inspired by the multi-resolution hierarchical structure of StyleGAN, the digital encoder consists of a multi-stage network. The measurement vector \(J\) passes sequentially through linear layers and attention mechanisms, with each level corresponding to a resolution layer of StyleGAN (18 layers from coarse to fine). The stacking depth increases from low-resolution to high-resolution layers. Finally, a Mixer Block is used to learn weighted cross-hierarchy blending, outputting \(L \in \mathbb{R}^{512 \times 18}\).
   • Design Motivation: Different layers of StyleGAN control attributes of different granularities (e.g., coarse layers control pose, while fine layers control texture). The encoder must match this structure, allocating more computational resources to process fine-detail layers.

3. Linear Projections for Downstream Latent Tasks:

   • Function: Directly complete multiple high-level vision tasks using simple linear transformations on the latent space without performing image reconstruction.
   • Mechanism: Capitalize on the linearly separable boundaries found in the GAN latent space: (1) Attribute Classification: A fully connected layer \(P_A\) projects \(L\) to \(\mathbb{R}^{40}\) (40 facial attributes like age, gender, facial hair, and smile), maintaining over 80% accuracy even under 1:16384 compression; (2) Face Segmentation: Multi-scale feature maps from the generative model are extracted and linearly projected via \(P_S\), combined with bilinear interpolation and convolutional layers to yield pixel-level segmentation; (3) Landmark Detection: A linear projection \(P_L\) from a coarse feature map \(\mathbb{R}^{1024 \times 36 \times 36}\) to 2D coordinates of 68 landmarks. All three tasks share the exact same \(O\) and \(\mathcal{D}_\theta\).
   • Design Motivation: The semantic linear separability of the GAN latent space eliminates the need for complex, task-specific models; simple linear transformations can handle downstream tasks. More importantly, \(O\) only needs to be optimized once to serve multiple tasks.

Loss & Training

The core loss is the latent space reconstruction loss \(\mathcal{L}_{lat} = \|\mathcal{D}_\theta(O \cdot I) - \mathcal{E}(I)\|_1\), where \(\mathcal{E}\) is a pre-trained StyleGAN encoder providing pseudo-ground truth latent representations. Auxiliary losses include identity loss \(\mathcal{L}_{id}\) (ArcFace feature cosine distance), pixel loss \(\mathcal{L}_{l2}\), perceptual loss \(\mathcal{L}_p\) (DINO/LPIPS features), and energy efficiency loss. The training set comprises FFHQ + CelebAHQ, and evaluation is conducted on a holdout set of 2,000 images from CelebAHQ.

Key Experimental Results

Main Results

Compression Ratio	Measurements	VGGFace Rec. Rate↑	Dlib Rec. Rate↑	FID↓	Attribute Acc.↑	Segmentation F1↑	Landmark NME↓
1:128	512	91.97%	92.74%	27.38	89.07%	70.00%	1.48
1:256	256	90.98%	92.68%	26.62	89.15%	70.94%	1.43
1:512	128	89.61%	91.67%	28.66	89.20%	70.25%	1.48
1:1024	64	81.12%	87.44%	28.79	88.74%	69.18%	1.52
1:2048	32	54.72%	77.77%	35.89	88.06%	65.81%	1.67
1:4096	16	27.22%	59.21%	46.18	86.44%	60.63%	2.01
1:16384	4	N/A	N/A	N/A	81.75%	46.36%	2.47

Only 64 measurements (1:1024) are needed to achieve an 81%+ face recognition rate, and just 4 measurements (1:16384) still yield an 81.75% attribute classification accuracy.

Ablation Study

Method	VGGFace↑	Dlib↑	FID↓
FSI-DL (Fourier Single-Pixel)	3.30%	13.15%	123.5
SAUNet (1-bit Quantized Deep Unfolding)	~60%	~70%	~50
SAUNet (8-bit Quantized)	~75%	~85%	~35
LSI (1-bit, 1:512)	89.61%	91.67%	28.66

Even under the more restrictive 1-bit quantization constraint, LSI significantly outperforms conventional methods.

Key Findings

• Downstream tasks are more robust to compression: While facial reconstruction essentially fails above 1:4096, attribute classification maintains an 81.75% accuracy at 1:16384, demonstrating that semantic tasks require far less information than pixel-level reconstruction.
• Optimized masks show semantic focus: Visualizations show that mask patterns automatically target facial contours and key feature regions, indicating that the system learns domain-specific sampling strategies.
• One optimization, multi-task multiplexing: The same set of \(O\) and \(\mathcal{D}_\theta\) can serve four completely different tasks—reconstruction, classification, segmentation, and landmark detection—eliminating the need to re-optimize the optical encoding for each individual task.
• Hardware prototypes validated the feasibility of the approach in physical experiments—successfully reconstructing faces and completing downstream tasks via actual capture using a DMD and a single-pixel detector.

Highlights & Insights

• Paradigm Shift: Shifting from "capture pixels \(\rightarrow\) compress \(\rightarrow\) process" to "directly capture latent space representations \(\rightarrow\) process within the latent space." This is a fundamental shift in imaging design, analogous to the information compression mechanism of the human visual system.
• New Possibilities for Optical-Generative Co-design: Joint training of mask optimization and latent space reconstruction adapts physical hardware to the generative model, rather than the traditional way of adapting generative models to raw hardware. A single mask optimization run can serve multiple models and tasks.
• Face attribute classification with only 4 scalars: Achieving over 80% accuracy under an extreme compression ratio of 1:16384 demonstrates that semantic information is highly compact within the GAN latent space. This has significant implications for privacy-preserving imaging (transmitting attributes rather than raw images).

Limitations & Future Work

• Domain limited by generative models: Current training is based on StyleGAN for the face domain. Extending to other domains requires corresponding pre-trained generative models and re-optimizing the masks.
• Speed limitations of single-pixel cameras: Under time-multiplexed operation, the imaging speed is constrained by the DMD switching rate and the number of measurements, which remains insufficient for real-time applications.
• Unsuitable for complex scenes: Currently, the method is only verified on the human face domain. For natural scenes with multiple objects and complex backgrounds, StyleGAN's domain restrictions might cause failures.
• Biases in the generative model itself (e.g., better reconstruction for certain ethnicities) will propagate to the imaging system.
• Future work can explore extending LSI to the latent spaces of diffusion models (e.g., Stable Diffusion) to achieve more generalized domain coverage.
• Combining programmable optical elements with the LSI framework can lead to the development of privacy-preserving cameras that capture only task-relevant information rather than complete images.

Related Work & Insights

• vs. Traditional Compressed Sensing: CS methods perform reconstruction in pixel space and are bound by RIP constraints and sparsity assumptions. LSI operates in the semantic latent space, leveraging generative model priors to achieve compression rates far beyond the limits of CS.
• vs. Deep Unfolding Networks (SAUNet): Even when SAUNet utilizes 8-bit quantized masks, its recognition rate under 1:512 compression remains far below that of LSI using 1-bit masks, as LSI targets the latent space rather than the pixel space.
• vs. Minimal Camera Designs (Freeform Pixels): These approaches utilize a small number of freeform pixels for task-specific cameras, whereas LSI enables finer-grained downstream task support (e.g., face recognition vs. simple occupancy monitoring) through the latent space.
• The LSI paradigm has inspiring implications for privacy-preserving imaging, high-speed imaging (requiring very few measurements), and task-specific sensor design.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Paradigm-level innovation. Introducing the generative model latent space into imaging system design is a highly visionary concept.
• Experimental Thoroughness: ⭐⭐⭐⭐ Validated with both simulation and real hardware, multi-task evaluations, though limited to the facial domain.
• Writing Quality: ⭐⭐⭐⭐ Concepts are clearly explained, and the analogy to biological vision is highly engaging.
• Value: ⭐⭐⭐⭐ Standard-setting work opening a new direction, but utility is currently constrained by domain limits and hardware speed.

Related Papers

• [CVPR 2025] Probability Density Geodesics in Image Diffusion Latent Space
• [NeurIPS 2025] NPN: Non-Linear Projections of the Null-Space for Imaging Inverse Problems
• [CVPR 2025] UltraFusion: Ultra High Dynamic Imaging using Exposure Fusion
• [ICCV 2025] What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalization
• [CVPR 2025] Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward