title: >- [Paper Note] Progression as Latent Drift: Generative Forecasting of Slow-Evolving Pathologies description: >- [ECCV 2026][Medical Imaging][Slow-Evolving Pathologies] Generative forecasting of longitudinal 3D brain MRI via latent residual drift and FSQ dead-zone filtering to eliminate identity collapse and nuisance noise. tags: - ECCV 2026 - Medical Imaging - Slow-Evolving Pathologies - Latent Drift - Finite Scalar Quantization - 3D Brain MRI date: 2026-09-19 content_hash: 9e48ff117b95a7e6

Progression as Latent Drift: Generative Forecasting of Slow-Evolving Pathologies¶

Conference: ECCV 2026
Paper: ECCV Official Link
Area: Medical Imaging
Keywords: Slow-Evolving Pathologies / Latent Drift / Finite Scalar Quantization / Topological Dead-Zone / 3D Brain MRI Simulation

TL;DR¶

To overcome the dominance of stationary anatomy and dense imaging noise in longitudinal brain MRI forecasting, Latent Drift reformulates generation to target temporal latent residuals combined with Finite Scalar Quantization (FSQ) acting as a topological dead-zone filter, enabling high-fidelity and clinically aligned individualized neurodegeneration forecasting.

Background & Motivation¶

Dynamic anatomical forecasting for slow-evolving neurodegenerative disorders (such as Alzheimer's and Parkinson's diseases) is crucial for clinical intervention and disease-modifying therapy design. Because neuronal loss and irreversible structural degeneration (e.g., hippocampal atrophy and ventricular expansion) develop gradually over years, substantial permanent damage has typically accumulated by the time cognitive symptoms become clinically observable. Anticipating personalized 3D brain trajectories can open earlier therapeutic windows and enrich clinical trials by pinpointing participants most likely to exhibit measurable disease progression.

However, directly adopting state-of-the-art generative sequence models from video or audio synthesis fails when applied to longitudinal 3D brain MRIs. Natural video generation relies on salient and large-scale temporal motions; in contrast, over a typical one-year clinical follow-up interval, morphological changes caused by neurodegeneration account for less than 1% of the total volumetric variance in 3D MRI scans, rendering current and future anatomical states statistically almost identical. In this low-signal regime, standard continuous sequence models suffer from two optimization pathologies: identity collapse, where optimization gradients are overwhelmed by stationary background anatomy (e.g., invariant skull and unaffected tissue), causing the network to degenerate into an approximate identity mapping; and the continuous interpolation trap, where Lipschitz-continuous deep neural networks cannot disentangle sparse biological atrophy from pervasive, sample-specific scanner noise, diffusing spurious variance across the entire brain volume.

To resolve these joint bottlenecks of source dominance and noise diffusion, the predictive paradigm must shift away from naive full-volume synthesis toward targeted dynamic decomposition. Core idea: shift the generative objective from synthesizing absolute future anatomy to forecasting temporal residuals within a compressed semantic space (Latent Drift), using Finite Scalar Quantization (FSQ) as a non-Lipschitz topological dead-zone filter to suppress dense measurement noise while preserving sparse pathological progression.

Method¶

Overall Architecture¶

Latent Drift uses a two-stage decoupled generative pipeline: Stage 1 trains a Latent Drift Tokenizer that compresses longitudinal 3D MRI pairs into continuous latent spaces, computes the temporal residual, and applies Finite Scalar Quantization (FSQ) as a topological dead-zone filter to quantize continuous drift into discrete tokens; Stage 2 trains an autoregressive Decoder-Transformer conditioned on baseline latent anatomy and multimodal clinical metadata (demographics, neuropsychological metrics, time horizon) to predict the discrete drift token sequence. Finally, the sampled drift is summed with the baseline latent representation and reconstructed into a 3D MRI by a spatial de-tokenizer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Longitudinal 3D MRI Pair<br/>(Current Scan x_cur, Future Scan x_fut)"] --> B["Spatial Compression<br/>Extract continuous latents z_cur, z_fut"]
    B --> C["Temporal Residual Decoupling<br/>Compute raw drift delta_z_raw = z_fut - z_cur"]
    C --> D["FSQ Dead-Zone Filtering<br/>Non-linearly annihilate noise to delta_z_q"]
    D --> E["Autoregressive Drift Generation<br/>Forecast drift tokens conditioned on z_cur and metadata"]
    E --> F["Latent Residual Fusion<br/>z_pred = z_cur + delta_z_q"]
    F --> G["Spatial De-tokenizer<br/>Reconstruct personalized future 3D MRI (x_fut_hat)"]

Key Designs¶

1. Temporal Residual Decoupling: Bypassing the Gradient Monopoly of Stationary Anatomy

Under standard conditional generative objectives, models directly optimize $p_\theta(x_{\mathrm{fut}} \mid x_{\mathrm{cur}}, c, \Delta t)$ or predict future latents $\hat{z}_{\mathrm{fut}} = \mathcal{G}_\theta(z_{\mathrm{cur}}, h_{\mathrm{cond}})$. Because the energy magnitude of stationary background structures vastly exceeds the subtle pathological signal ($\|z_{\mathrm{cur}}\| \gg \|\delta_{\mathrm{pathology}}\|$), back-propagation gradients are dominated by static neuroanatomy (such as skull contours and unaffected white matter), forcing $\mathcal{G}_\theta$ toward a trivial identity function $f(z) \approx z$. To address this, the future representation is formally decomposed into three distinct components: $$z_{\mathrm{fut}} = z_{\mathrm{cur}} + \delta_{\mathrm{pathology}} + \eta_{\mathrm{nuisance}}$$ By defining the continuous temporal residual $\Delta z_{\mathrm{raw}} = z_{\mathrm{fut}} - z_{\mathrm{cur}}$ as the optimization target, the static baseline $z_{\mathrm{cur}}$ is explicitly subtracted from the objective. Spatial gradient analysis confirms that this subtraction shifts training gradients away from static regions and concentrates them squarely on active neurodegenerative zones like the hippocampus, completely bypassing identity collapse.

2. FSQ Dead-Zone Filtering: Breaking the Continuous Interpolation Trap

Targeting the raw temporal residual $\Delta z_{\mathrm{raw}}$ exposes the network to sample-specific imaging noise $\eta_{\mathrm{nuisance}}$. According to statistical learning theory (Theorem 1), Lipschitz-continuous neural networks optimized via Empirical Risk Minimization (ERM) are mathematically forced to interpolate dense nuisance noise, resulting in an expected error lower bounded by $\mathbb{E}\|f_\theta(x) - \delta(x)\|_2^2 \gtrsim \mathbb{E}\|\eta(x)\|_2^2$ and dispersing spurious variations across the volume. To resolve this, Finite Scalar Quantization (FSQ) is deployed as a non-Lipschitz topological dead-zone filter: $$\Delta z_q = \Delta z_{\mathrm{raw}} + \text{sg}\left(s \cdot \text{clip}\left(\left\lfloor\frac{\Delta z_{\mathrm{raw}}}{s}\right\rceil, -L_{\max}, L_{\max}\right) - \Delta z_{\mathrm{raw}}\right)$$ Calibrated such that the quantization step satisfies $h > 2\varepsilon$, any coordinate lying outside the true pathology support contains only nuisance noise with $|\eta| \le \varepsilon < h/2$ and is mapped strictly to zero ($\Delta z_{q,j} = 0$). Conversely, true biological drift exceeds the threshold and preserves its exact support geometry (Theorem 2). This step discontinuity breaks the smooth interpolation trap and collapses continuous nuisance fluctuations into a sparse discrete set prior to sequence learning.

3. Autoregressive Drift Generation: Multimodal Conditioning Over Discrete Trajectories

Once continuous temporal progressions are discretized by the FSQ dead-zone tokenizer, forecasting is formulated as discrete causal sequence modeling over an isolated vocabulary of disease dynamics. An autoregressive Decoder-Transformer $G_\theta$ processes the discrete tokens, conditioned on the patient's continuous baseline anatomy $z_{\mathrm{cur}}$ and an MLP projection of clinical metadata (age, sex, education, cognitive assessments, APOE genotype, and target time horizon $\Delta t$): $$P(\Delta \hat{z}_q \mid x_{\mathrm{cur}}, c, \Delta t) = \mathcal{G}_\theta(\Delta \hat{z}_q \mid z_{\mathrm{cur}}, \text{MLP}(c, \Delta t))$$ Trained with cross-entropy loss, the model captures the conditional probability distribution of biological decline. At inference time, predicted discrete drift tokens $\Delta \hat{z}_q$ are mapped to the latent domain, added directly to $z_{\mathrm{cur}}$, and passed through the pre-trained spatial decoder to reconstruct the future 3D brain volume.

Loss & Training¶

Latent Drift is trained through a two-stage schedule: - Stage 1 (Tokenizer & Dead-Zone Filter): The spatial 3D encoder, FSQ bottleneck, and spatial decoder are jointly trained on paired scans $(x_{\mathrm{cur}}, x_{\mathrm{fut}})$ using a composite objective comprising Mean Squared Error (MSE), 3D perceptual loss (LPIPS), and adversarial patch-GAN loss, with the Straight-Through Estimator (STE) propagating gradients through the non-differentiable rounding operator. - Stage 2 (Autoregressive Sequence Model): With the first stage frozen, discrete drift tokens $\Delta z_q$ are extracted across all training pairs. The Decoder-Transformer is trained via standard categorical cross-entropy loss to predict the discrete sequence conditioned on baseline latents and clinical embeddings.

Key Experimental Results¶

Main Results¶

Experiments were conducted on the ADNI and AIBL longitudinal cohorts, comprising 3,981 longitudinal 3D MRI pairs (with 1,140 patient-disjoint pairs reserved for the test set) resampled to $93 \times 112 \times 96$ voxels. Downstream diagnostic utility was assessed using a frozen, patient-disjoint ViViT-style Alzheimer's disease classifier (exhibiting $>91\%$ accuracy on authentic scans).

Method	Generative Fidelity FID (i3d) $\downarrow$	Generative Fidelity FID (cls.) $\downarrow$	Structural Similarity Diff-SSIM $\uparrow$	Structural Similarity Pix-SSIM $\uparrow$	Structural Similarity NCC $\uparrow$	Clinical Utility F1 (%) $\uparrow$	Clinical Utility Precision (%) $\uparrow$	Clinical Utility Recall (%) $\uparrow$	Clinical Utility Accuracy (%) $\uparrow$
CycleGAN	12.13	7.03	0.7713	0.9612	0.9611	75.44	85.95	67.22	79.39
Palette (Diffusion)	46.74	9.27	0.8030	0.8088	0.9810	80.36	87.50	74.30	82.89
I2I-DiT (Diffusion)	25.65	7.63	0.7799	0.7979	0.9807	62.37	87.00	48.60	72.36
BrLP (Diffusion)	19.59	10.14	0.8063	0.8574	0.9819	80.36	86.98	74.67	82.81
VQGAN (AR)	30.11	18.29	0.7625	0.8612	0.9576	70.55	78.23	64.24	74.74
RQ-Transformer (AR)	27.71	10.89	0.7071	0.7507	0.8666	60.78	62.82	58.84	64.21
NeuroAR (AR)	22.28	15.34	0.8128	0.8554	0.9782	81.86	87.50	76.91	83.95
Latent Drift (Ours)	13.92	12.77	0.8204	0.9483	0.9880	87.51	88.26	86.78	88.33

Ablation Study¶

1. Prediction Target and Quantization Mechanism Ablations

Ablation Category	Configuration	rFID $\downarrow$	Diff-SSIM $\uparrow$	Diagnostic Acc. (%) $\uparrow$	Diagnostic F1 (%) $\uparrow$	Note
Target Ablation	Pixel-level Synthesis	44.38	0.7625	77.19	72.75	Suffers from identity collapse; misses faint atrophy
	Latent Drift (Ours)	13.92	0.8205	88.33	87.52	Decouples static background; focuses on pathology
Quantization Mechanism	Vector Quantization (VQ)	14.04	0.8249	86.67	85.52	Discrete codebook without geometric dead-zone control
	Residual VQ (RVQ)	29.06	0.7798	84.56	83.67	Multi-scale residual steps re-introduce nuisance noise
	Lookup-Free Quantization (LFQ)	19.09	0.8253	85.09	84.02	Moderate structural score but inferior fidelity
	Binary Spherical Quantization (BSQ)	20.70	0.7795	84.56	82.75	Coarse binary projection discards subtle variations
	FSQ Dead-Zone (Ours)	13.77	0.8224	88.24	87.41	Optimal trade-off between fidelity and clinical diagnostic utility

2. FSQ Grid Capacity and Step-Size Trade-off (Stage 2 Generation)

FSQ Grid Specification	Codebook Capacity	rFID $\downarrow$	Diff-SSIM $\uparrow$	Diagnostic Acc. (%) $\uparrow$	Diagnostic F1 (%) $\uparrow$	Note
`[8, 6, 5]`	Coarse low-capacity grid	7.07	0.8128	86.84	85.27	Overly wide dead-zone discards early subtle atrophy
`[8, 5, 5, 5]`	4-channel compact grid	15.58	0.8240	82.46	79.51	Significant drops in fidelity and clinical diagnostic score
`[7, 5, 5, 5, 5]`	5-channel extended grid	7.00	0.8253	85.35	83.64	High fidelity but suboptimal clinical diagnostic transfer
`[8, 8, 8, 6, 5]`	High-capacity fine grid	27.07	0.8273	82.81	80.00	Small step size captures imaging noise; falls into interpolation trap
`[8, 8, 8, 5, 5, 5]`	Calibrated Grid (Ours)	12.78	0.8224	88.24	87.41	Correct noise cutoff barrier; achieves peak clinical performance

Key Findings¶

Failure of Direct Pixel Synthesis: Predicting absolute voxel intensities results in severe identity collapse (FID deteriorates to 44.38 and Diff-SSIM drops to 0.7625), wasting model capacity on skull and invariant white matter; shifting to latent drift improves downstream diagnostic accuracy from 77.19% to 88.33%.
Validation of the Continuous Interpolation Trap: Expanding FSQ capacity to [8, 8, 8, 6, 5] narrows the quantization step $h \le 2\varepsilon$, invalidating the dead-zone condition. The model interpolates high-frequency noise, causing rFID to spike to 27.07. The calibrated [8, 8, 8, 5, 5, 5] configuration suppresses noise and maximizes clinical fidelity.
Robustness Across Heterogeneous Brain Regions: Across the hippocampus, cerebellum, and ventricles (which exhibit a 23× dynamic range in atrophy rates), the ratio of recovered-to-ideal morphological change remains tightly within $[0.93, 1.18]$ with a mean absolute deviation of only 0.12, outperforming baselines by 2.6× to 5.8×.

Highlights & Insights¶

Formalizing Pathological Evolution as Latent Drift: The paper insightfully diagnoses the fundamental limitation of generative sequence modeling for slow diseases as an energy imbalance ($\|z_{\mathrm{cur}}\| \gg \|\delta\|$), cleanly bypassing identity collapse by shifting to temporal residual forecasting.
FSQ as a Non-Lipschitz Topological Dead-Zone Filter: Re-evaluates scalar quantization through non-smooth operator theory, demonstrating that an FSQ step size exceeding twice the noise magnitude ($h > 2\varepsilon$) rigorously collapses dense perturbations to zero while preserving sparse pathology.
A Generalizable Paradigm for Low-Signal Forecasting: The combined residual decoupling and dead-zone filtering design can be directly transferred to other slow-evolving medical domains, such as longitudinal retinal OCT for glaucoma and chest CT for pulmonary fibrosis.

Limitations & Future Work¶

Author-Acknowledged Limitations: The globally shared scalar quantization step assumes a relatively uniform background rate, potentially leading to slight under-representation in rapidly expanding structures (e.g., late-stage lateral ventricles); evaluations were limited to well-registered, downsampled datasets rather than raw multi-site scans.
Independent Observations: Downsampling 3D volumes to $93 \times 112 \times 96$ restricts sub-millimeter cortical thickness assessment; downstream evaluation relies on categorical diagnostic classification rather than tracking continuous multi-timepoint cognitive scores.
Future Directions: Developing anatomical region-aware adaptive quantization grids (Region-Aware FSQ) tailored to regional atrophy rates, and integrating neural implicit representations to scale toward full native-resolution voxel prediction.

vs CycleGAN / Static 3D Generation: While CycleGAN achieves competitive static distribution metrics (FID 12.13), it lacks temporal causality and patient-specific trajectory conditioning, yielding inferior Diff-SSIM (0.7713); Latent Drift explicitly integrates clinical covariates and time horizons.
vs Palette / I2I-DiT / BrLP (Diffusion Models): Continuous diffusion baselines are constrained by Lipschitz smoothness, causing them to either hallucinate noise or settle into near-identity mappings across time; Latent Drift eliminates spurious interpolation via its discrete FSQ dead-zone.
vs NeuroAR (Continuous Autoregressive Forecasting): NeuroAR operates autoregressively on continuous latent tokens, suffering from compounding noise drift over extended horizons (e.g., at 48 months); Latent Drift maintains structural stability across long-term trajectories.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates identity collapse and the continuous interpolation trap with rigorous proofs, pioneering FSQ dead-zone filtering for pathology.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation covering structural similarity, dual-feature FIDs, longitudinal stability, and frozen ViViT clinical diagnosis.
Writing Quality: ⭐⭐⭐⭐⭐ Logically structured, clear problem framing, and strong theoretical grounding.
Value: ⭐⭐⭐⭐⭐ Highly impactful for early disease screening, organ trajectory simulation, and clinical trial cohort enrichment.