λSplit: Self-Supervised Content-Aware Spectral Unmixing for Fluorescence Microscopy

Conference: ECCV 2026
Paper: ECCV Official Page
Code: https://github.com/juglab/lambdaSplit
Area: Image Restoration
Keywords: spectral unmixing, self-supervised learning, fluorescence microscopy, hierarchical variational inference, physical consistency

TL;DR

λSplit learns cellular structural priors with a hierarchical VAE and remixes predicted fluorophore concentration maps through known spectra for training without unmixed ground truth; on simulated CellAtlas data with 5 ms exposure and 32 bands, it achieves 34.03 dB PSNR versus 28.91 dB for RLU, the strongest comparator by PSNR in that setting.

Background & Motivation

Fluorescence microscopy must distinguish biological structures associated with different labels. Sequential channel acquisition conveniently supports only a limited number of structures and increases acquisition time, photobleaching, and phototoxicity. Spectral acquisition instead records emissions from multiple fluorophores across wavelength bands and delegates separation to computation. Each band generally contains contributions from several fluorophores, so similar emission spectra and low photon counts make pixel-wise least-squares estimates susceptible to noise amplification and residual crosstalk.

Classical methods such as LU and NNLU mainly use the spectral vector at an individual pixel, without knowing that neighboring pixels should form continuous microtubules or discrete cellular structures. Learning-based methods can exploit these spatial regularities, but supervised models require paired mixtures and clean concentration maps, which are difficult to obtain experimentally. Remote-sensing unmixing networks are not a direct solution either: microscopy differs in noise, spatial resolution, and the meaning of concentration. In particular, fluorophore concentrations at a pixel should not automatically be assumed to sum to 1.

The paper retains known emission spectra as accessible physical information and replaces the missing target concentration maps with an observable constraint: predicted maps should reproduce the input when remixed. This constraint alone remains ambiguous, so hierarchical latent variables learn the distribution of biological structures. Core Idea: use content-aware structural priors to select plausible unmixed solutions and a fixed spectral forward model to verify that they explain the observations, enabling joint separation and implicit denoising without paired unmixed ground truth.

Method

Overall Architecture

The input is a two- or three-dimensional spectral image with \(L\) bands, and the output contains concentration maps for \(F\) fluorophores. λSplit first generates plausible concentration distributions through a hierarchical structural prior, then reconstructs the input through a physical spectral loop. Training compares spectral observations rather than using the original concentration maps as supervision targets.

At test time, the encoder and concentration decoder produce 50 posterior samples whose average forms the prediction. The spectral mixer supplies the physical constraint during training; it is not a module that directly inverts the mixture at test time. Dashed edges in the diagram denote training supervision, while solid edges denote image and prediction data flow.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Spectral observation"] --> Prior["Hierarchical<br/>structural prior"]
    Prior --> Maps["Concentration samples"]
    Maps --> Mixer["Physical spectral loop"]
    Spectra["Known emission spectra"] --> Mixer
    Mixer --> Recon["Reconstructed spectra"]
    Input -. "Observation target" .-> Loss["Spectral MSE + KL"]
    Recon -. "Reconstruction error" .-> Loss
    Prior -. "Hierarchical KL" .-> Loss
    Loss -. "Training update" .-> Prior
    Maps --> Average["Average 50 posterior<br/>samples at test time"]
    Average --> Output["Unmixed concentration maps"]

Key Designs

1. Hierarchical structural prior: resolve ambiguous pixel assignments through spatial context

The backbone is a Ladder Variational Autoencoder (LVAE). A bottom-up convolutional encoder extracts observation features at multiple resolutions. The highest latent level uses a standard multivariate Gaussian prior; learnable priors at lower levels are conditioned on higher-level states and combined with corresponding encoder features to form posteriors. Decoding samples and propagates features down the hierarchy, and a final unmixing convolutional block converts them into \(F\) concentration channels. The network therefore predicts source images directly under structural constraints rather than applying LU and then denoising its output.

This allows the interpretation of a local pixel to draw on surrounding morphology. When spectra do not distinguish two labels well, the network can also exploit the structural distribution learned from training data instead of relying solely on that pixel's band intensities. The four stochastic levels progressively halve spatial resolution while retaining 128 feature channels; the two- and three-dimensional versions use convolutions of the corresponding dimensionality. The KL term connects observation posteriors to hierarchical priors, but this regularization does not guarantee a unique correct solution in an underdetermined setting.

Inference averages 50 posterior samples to approximate a minimum mean square error (MMSE) estimate. These are alternative unmixed predictions, not repeated microscope acquisitions. Averaging attenuates stochastic details while retaining structures consistently present across samples. The paper demonstrates implicit denoising, but the available main text does not report uncertainty calibration; the ability to sample should not be equated with trustworthy confidence estimates.

2. Physical spectral loop: replace unavailable concentration supervision with observable spectral reconstruction

Each column of the known mixing matrix \(M\in\mathbb{R}^{L\times F}\) represents a fluorophore's discretized emission spectrum across acquisition bands. The authors average continuous spectra within each band and apply column-wise \(\ell_1\) normalization so that each column sums to 1. Spectra can come from FPBase or measurements of single-fluorophore control samples. The matrix remains fixed during training while gradients propagate through mixing to the predicted concentration maps.

At any spatial location \(p\), the mixer and its intensity conservation property can be expressed compactly as:

\[ \widehat S_{\ell,p}=\sum_{j=1}^{F}M_{\ell j}\widehat U_{j,p},\qquad \sum_{\ell=1}^{L}\widehat S_{\ell,p}=\sum_{j=1}^{F}\widehat U_{j,p}. \]

Here, \(\widehat U\) denotes predicted concentrations and \(\widehat S\) denotes reconstructed spectra. Normalization applies to columns of the mixing matrix, not to the sum of concentrations at each pixel. The latter would incorrectly restrict intensity variation in microscopy images. Because mixing is differentiable, spectral reconstruction error can train concentration prediction end-to-end without clean concentration targets or an arbitrary learned spectral decoder.

The physical loop tests whether predictions explain observations; it does not mathematically eliminate ambiguity. In particular, when \(L<F\), different concentration combinations can yield the same spectral observation, leaving the structural prior essential for selecting a solution. This is therefore unmixing with known spectra, not blind unmixing that jointly estimates spectra and concentrations. Incorrect reference spectra would directly undermine the validity of the loop.

A Worked Example

Consider the low-band CellAtlas experiment, which reconstructs 4 concentration channels corresponding to nuclei, microtubules, endoplasmic reticulum, and mitochondria. With only 3 observed bands, each pixel supplies fewer spectral constraints than unknown concentrations, so a simple pixel-wise inversion cannot uniquely determine the solution.

λSplit encodes a three-band image patch, forms posteriors from multiscale features, and decodes four-channel concentration samples. A fixed \(3\times4\) mixing matrix projects these samples back into the three-band observation domain. Reconstruction error penalizes predictions that fail to explain the input, while the hierarchical prior favors cellular structures supported by the training distribution.

At test time, 50 samples are averaged. In Table 2's fixed per-band SNR setting, this three-band task yields 27.25 dB PSNR, compared with 26.39 dB for RLU. The result demonstrates improved underdetermined reconstruction, not lossless replacement of richer spectral acquisition or unique recovery of every output structure from the observations.

Loss & Training

Starting from a hierarchical variational objective, the authors replace the unobservable concentration reconstruction term with spectral-domain MSE and add hierarchical KL regularization. The main text explicitly states that no probabilistic noise model is specified, so this is more accurately described as an MSE approximation to the reconstruction term than as an exact derivation of a Poisson photon-counting likelihood.

The cached equations contain extraction-related character loss. The following summarizes the training mechanism described in the prose and is not a verbatim transcription of the paper's Equation (6):

\[ \mathcal L=\mathbb E_{q_\phi(\mathbf z\mid S)}\left[\frac{1}{LP}\sum_{\ell=1}^{L}\sum_{p=1}^{P}\left(S_{\ell,p}-\widehat S_{\ell,p}(\mathbf z)\right)^2\right]+\beta\mathcal L_{\mathrm{KL}},\qquad \beta=1. \]

Here, \(P\) is the total number of pixels or voxels. In the implementation, KL is estimated with Monte Carlo sampling, averaged over latent tensor entries at each level, and then averaged across levels. Its weight should not be compared directly with KL coefficients from other VAEs without accounting for this normalization.

Training uses Adamax with a batch size of 32, an initial learning rate of \(10^{-3}\), and a schedule of 150 epochs. Mixed precision, early stopping with patience 30, and learning-rate reduction on plateau with patience 15 are used. Training patches are \(64\times64\) for two-dimensional inputs and \(8\times64\times64\) for three-dimensional inputs; inference uses inner tiling with one-quarter overlap between neighboring tiles.

Global mean-standard-deviation normalization preserves relative band intensities rather than independently rescaling each band. Models were trained on 18 GB and 35 GB MIG partitions of an NVIDIA DGX H200 system for two- and three-dimensional data, respectively. Reported training time ranges from approximately 2 hours to one day, depending on dataset size and dimensionality.

Key Experimental Results

Main Results

Experiments use structural images from three real microscopy datasets, BioSR, CellAtlas, and HHMI25, and synthesize spectral observations with microsim. They construct 58 controlled benchmarks and compare against 10 baselines across the experiments. Simulation includes point spread functions, photon statistics, readout noise, and band configuration. Real structural source data should not be mistaken for quantitative validation on native real spectral acquisitions.

The following selection comes from Table 1's CellAtlas 32-band results. PSNR and MS-SSIM use range-invariant evaluation to compensate for global intensity scaling; PSNR is measured in dB, and lower LPIPS is better. The comparator at each exposure is the non-λSplit method with the highest PSNR, and the gain is the difference between their PSNR values.

Exposure	Method	PSNR ↑	MS-SSIM ↑	LPIPS ↓	λSplit PSNR Gain
2 ms	TAEU	24.52	0.761	0.520
2 ms	λSplit	27.14	0.904	0.373	+2.62 dB
5 ms	RLU	28.91	0.927	0.316
5 ms	λSplit	34.03	0.980	0.155	+5.12 dB
10 ms	RLU	33.14	0.971	0.221
10 ms	λSplit	36.66	0.989	0.137	+3.52 dB
20 ms	RLU	36.68	0.988	0.125
20 ms	λSplit	36.46	0.988	0.155	-0.22 dB

Advantages are clear at lower exposures, but RLU has better PSNR and LPIPS at 20 ms. At 2 ms in Table 1, λSplit's MicroMS-SSIM is 0.625, below TAEU's 0.652. PSNR gains therefore do not establish superiority on every quality metric.

Ablation Study

The supplied cache contains only the main paper, not the cited supplementary Section S.5, so numerical ablations of stochastic depth and KL weight cannot be verified. The table below instead presents the controlled band-count analysis from Table 2, not a component ablation. RLU is the fixed comparator, and the target always contains 4 concentration channels.

Acquisition Control	Bands	λSplit PSNR ↑	RLU PSNR ↑	λSplit MS-SSIM ↑	RLU MS-SSIM ↑	λSplit LPIPS ↓	RLU LPIPS ↓
Fixed per-band SNR	3	27.25	26.39	0.878	0.866	0.283	0.238
Fixed per-band SNR	4	29.95	27.56	0.960	0.921	0.267	0.258
Fixed per-band SNR	5	31.12	28.91	0.971	0.943	0.313	0.220
Fixed per-band SNR	32	36.46	36.68	0.988	0.988	0.155	0.125
Fixed total photon budget	3	27.45	27.13	0.868	0.882	0.273	0.187
Fixed total photon budget	4	31.54	28.63	0.967	0.938	0.238	0.229
Fixed total photon budget	5	31.12	28.91	0.971	0.943	0.313	0.220
Fixed total photon budget	32	31.32	26.15	0.971	0.872	0.218	0.387

The two controls answer different questions: the first isolates spectral dimensionality, whereas the second lets fewer bands receive a higher per-band SNR. They share the five-band reference setting, but their 32-band photon conditions differ. Corresponding columns across the two groups should not be treated as identical acquisitions.

Key Findings

• Structural priors help in difficult settings, but fewer bands are not inherently better. At fixed per-band SNR, λSplit falls from 36.46 dB with 32 bands to 27.25 dB with 3 bands.
• At fixed total photon budget, λSplit achieves 31.54 dB with 4 bands and 31.32 dB with 32 bands. Better spectral separability can therefore be offset by lower per-band SNR.
• Metrics expose genuine trade-offs. With 3 bands and a fixed total photon budget, λSplit has higher PSNR than RLU but lower MS-SSIM, 0.868 versus 0.882, and worse LPIPS, 0.273 versus 0.187.
• The overlap study uses spectral shifts of 2, 5, 10, 20, and 50 nm in 32- and 5-band configurations, totaling 40 experiments. Figure 4 shows larger advantages under stronger overlap, but the cache does not contain the supplementary tables needed to quote exact results.

Highlights & Insights

• Use the physical model as a bridge to supervision. Mixtures, rather than pure channels, are observable, making projection back to the observation domain more practical than demanding paired labels. A fixed mixer also prevents an arbitrary decoder from bypassing the concentration interpretation.
• Exploit morphology as well as spectral differences. Multiscale priors make underdetermined unmixing less dependent on isolated pixel equations. Their value is distribution-dependent solution selection, not a new uniqueness guarantee.
• Control spectral information and photon budget separately. The two band-count experiments distinguish reduced dimensionality from increased photons per band. This makes the evaluation more informative for acquisition design than varying channel count alone.

Limitations & Future Work

• The method assumes linear mixing and known reference spectra. The authors identify spectrum estimation as future work; practical validation should also test spectral drift, reference-spectrum errors, and changes in fluorophore environment.
• Quantitative evidence comes from synthetic spectral observations driven by real structures. Compatibility with standard confocal hardware does not establish reliable transfer across instruments, laboratories, or real acquisition domains.
• Learned priors may favor familiar training morphologies. Rare structures, strong co-localization, and distribution shifts require independent tests for incorrect separation, especially because low spectral reconstruction error does not establish biological fidelity.
• Posterior averaging increases inference work, and the main text does not provide a full latency comparison. Approximately 3M parameters do not automatically imply real-time execution once 50 samples and tiling are included.
• Supplementary material is absent from the supplied cache, preventing verification of architectural ablations, detailed metric implementations, and additional supervised-baseline tables. The paper also discloses a related patent application; code availability does not imply unrestricted permission for every use case.

Related Work & Insights

• vs LU / NNLU / RLU: These methods mainly determine outputs from spectra and pixel observations, whereas λSplit additionally learns spatial structural distributions. RLU remains competitive at high SNR, so the learning-based model's principal advantage lies in noisier or more ambiguous regimes.
• vs MicroSplit: MicroSplit already uses hierarchical variational architectures to separate structures from a single fluorescence image. λSplit's key addition is explicit use of multiband measurements and known emission spectra within a self-supervised spectral forward loop.
• vs AutoUnmix: The supervised comparator uses noise-free concentration targets from the simulator, giving it additional supervision. The main text reports approximately 450M and 10B parameters for its 5-band and 32-band models, respectively, versus approximately 3M for λSplit in both cases. These figures do not establish higher λSplit accuracy on every supervised task.
• vs FCLU / TAEU: Their weaker adaptation in these microscopy experiments highlights differences between remote-sensing abundance constraints and microscopy concentration statistics. It does not invalidate their performance on their original remote-sensing tasks.

Rating

• Novelty: 4/5. Combining a hierarchical VAE with a known-spectrum loop directly addresses microscopy unmixing, although the underlying components are established.
• Experimental Thoroughness: 4/5. The 58 controlled settings cover major difficulties, but real spectral-domain validation and the unavailable supplementary ablations remain evidence boundaries.
• Writing Quality: 4/5. The problem, physical module, and controlled experiments connect clearly, but broad claims of superiority require attention to metric-specific exceptions.
• Value: 4/5. Training without paired concentration targets fits standard spectral microscopy workflows, while deployment still depends on accurate reference spectra and cross-domain validation.