Generative Manifold Distillation: Aligning Restoration Trajectories with the Natural Image Prior

Conference: ECCV 2026
Paper: ECCV Official
PDF: ECCV PDF
Area: Image Restoration / Image Generation
Keywords: Image Restoration, Unsupervised Domain Adaptation, Generative Prior Distillation, Flow Matching, Manifold Alignment

TL;DR

Generative Manifold Distillation (GMD) operates under a strictly unsupervised setting with only unannotated low-quality target observations, performing offline orthogonal projection onto the natural image manifold via a frozen 12B foundation model and distilling it into a lightweight restorer with quality-gated filtering and source trajectory anchoring, delivering state-of-the-art out-of-distribution performance with zero added inference latency and zero architectural changes.

Background & Motivation

Diffusion models and flow matching algorithms have achieved remarkable perceptual fidelity across inverse imaging problems such as super-resolution, deblurring, and inpainting. However, because these restoration networks are predominantly trained on synthetic degradation pairs, they experience severe performance degradation (manifold drift) when deployed to real-world environments characterized by complex, unknown out-of-distribution (OOD) degradations. Since collecting paired high-quality ground truth for these real-world scenarios is practically impossible, adapting an efficient baseline model to an unlabeled target domain without clean target data or invasive network alterations is a critical challenge.

Existing unsupervised domain adaptation (UDA) strategies suffer from two key paradigm bottlenecks: (1) Traditional unpaired UDA methods (e.g., CycleGAN-style dual frameworks or degradation translation pipelines) introduce intrusive structural changes like adversarial discriminators or auxiliary extractors, which destabilize training and disrupt the seamless deployment of standard pre-trained architectures while still requiring a curated pool of clean target images. (2) Zero-shot foundation model priors (e.g., deploying the 12B FLUX via SDEdit or rectified flow inversion at test time) achieve impressive perceptual realism, but they incur prohibitive computational cost and inference latency (multiplying inference time several-fold). Crucially, without task-specific constraints, these unconstrained generative priors frequently overpower the degraded input, causing severe semantic hallucinations and geometric drift.

These dilemmas establish a clear objective: we require the expressive generative prior of a massive foundation model to overcome severe OOD degradations, while strictly preserving the high structural fidelity and rapid inference speed of an efficient, unmodified restoration network. The paper addresses this by decoupling the heavy generative prior from test-time inference. Core idea: reframe unsupervised domain adaptation as an offline two-stage geometric manifold alignment process—utilizing a frozen 12B text-to-image foundation model to construct high-quality pseudo-targets via attention-injected orthogonal projection and quality-gated filtering, followed by trajectory-regularized distillation anchored to source-domain data to eliminate both generative hallucinations and test-time latency.

Method

Overall Architecture

GMD operates in the strictest unsupervised setting: the learner has access to paired source-domain data \(\mathcal{D}_{id} = \{(y_{id}^{(i)}, x_{id}^{(i)})\}_{i=1}^{N_{id}}\) and only unannotated low-quality target-domain observations \(\mathcal{D}_{ood} = \{y_{ood}^{(j)}\}_{j=1}^{N_{ood}}\), with no clean target images available. Geometrically, natural high-fidelity images lie on a low-dimensional manifold \(\mathcal{M}\). When exposed to real-world OOD inputs \(y_{ood}\), an in-distribution restorer \(f_{\theta_{id}}\) produces an initial estimate \(\tilde{x}_{ood}\) that preserves global layout but is pushed into an off-manifold artifact space (\(D_{KL}(f_{\theta_{id}}(\mathcal{P}_T(y)) \parallel \mathcal{M}) > \epsilon\)).

GMD establishes an offline adaptation pipeline consisting of three sequential phases: First, an initial off-manifold estimate \(\tilde{x}_{ood}\) is generated by the baseline restorer. Second, a frozen 12B generative model acts as an orthogonal manifold projector \(\Pi_\mathcal{G}\), lifting \(\tilde{x}_{ood}\) via forward ODE dynamics to marginalize artifacts into isotropic noise and projecting it back to \(\mathcal{M}\) through reverse ODE dynamics with attention injection to enforce structural identity. Third, candidate pseudo-targets are screened by a quality-gated manifold filter based on NIMA scoring to discard off-manifold outliers and hallucinations. Finally, the lightweight student model is fine-tuned via trajectory-regularized distillation on a mixed batch (90% source data, 10% filtered target pseudo-pairs), achieving natural texture alignment while strictly preventing error accumulation and structural collapse.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Unlabeled OOD LQ Input y_ood"] --> B["Initial Off-Manifold Inference<br/>Baseline Restorer Generates x_init"]
    B --> C["Orthogonal Manifold Projection<br/>Forward ODE Lifting + Reverse ODE Attention Injection"]
    C --> D{"Quality-Gated Manifold Filter<br/>NIMA Score s(x) ≥ α"}
    D -->|Pass| E["Filtered Pseudo-Target Set D_sel^ood"]
    D -->|Reject| F["Discard Off-Manifold Samples"]
    E --> G["Trajectory-Regularized Distillation<br/>90% Source Anchoring + 10% Manifold Alignment"]
    H["Paired Source Data D_id"] --> G
    G --> I["Adapted Lightweight Model f_θ<br/>Zero Added Latency / Unchanged Architecture"]

Key Designs

1. Orthogonal Manifold Projection: Forward Artifact Marginalization and Reverse Attention Injection

Directly projecting an off-manifold estimate \(\tilde{x}_{ood} = x_{clean} + \epsilon_{art}\) onto \(\mathcal{M}\) using standard generative sampling can land on an arbitrary high-density mode, altering semantic identities (e.g., hallucinating a completely different vehicle or face). GMD deploys a frozen 12B FLUX.1.dev Rectified Flow model as an orthogonal projection operator \(\Pi_\mathcal{G}\) via two coupled steps:

The first step is manifold lifting via a forward ODE under null-conditioning \(c_\emptyset\). Integrating \(\frac{d x}{d\tau} = v(x, \tau, c_\emptyset)\) from \(\tau=0\) to \(\tau=1\) pulls the latent representation into a high-noise state \(z := x(1)\), asymptotically absorbing the unstructured degradation artifact \(\epsilon_{art}\) into the Gaussian noise prior. Concurrently, the global spatial topology and low-frequency structures are encoded into the trajectory's self-attention maps.

The second step is guided reverse projection using a standard text prompt (Prompt: "A sharp, focused, high quality photograph") under a classifier-free guidance velocity field \(v_{cfg}\):

\[\hat{x}_{ood} = \Pi_\mathcal{G}(\tilde{x}_{ood}) = x(1) + \int_{1}^{0} v_{cfg}(x(\tau), \tau, c_{prompt}) d\tau\]

Because the velocity field \(v_{cfg}\) points directly toward high-density regions of \(p_{data}\), integrating this trajectory ensures the reconstruction lands squarely on \(\mathcal{M}\). To make this projection orthogonal—finding the point on \(\mathcal{M}\) closest in geometric structure to the degraded input—GMD replaces the self-attention keys and values during each reverse step with those cached from the forward pass. This attention injection rigidly freezes the spatial layout, compelling the generative flow to eliminate off-manifold artifacts while preventing semantic hallucinations and geometric distortions.

2. Quality-Gated Manifold Filtering: Objective Quality Thresholding to Stop Error Cascades

Because large generative priors can occasionally fail on severe, unstructured real-world corruptions, unfiltered pseudo-targets may contain residual artifacts or hallucinations. Training a student model on these flawed targets induces confirmation bias and model drift.

GMD integrates a quality-gated manifold filter prior to distillation. An objective no-reference image quality assessment model (NIMA) serves as a proxy metric \(s(\cdot)\) for manifold proximity. Candidate pseudo-pairs are admitted into the distillation set \(\mathcal{D}_{sel}^{ood}\) only if their score meets or exceeds a strict reliability threshold \(\alpha = 4.0\):

\[\mathcal{D}_{sel}^{ood} = \left\{ (y_{ood}^{(i)}, \hat{x}_{ood}^{(i)}) \;\middle|\; s(\hat{x}_{ood}^{(i)}) \ge \alpha \right\}\]

This quality gate acts as a robust firewall: sub-threshold samples are rejected, guaranteeing that the student restorer is supervised exclusively by high-confidence, photorealistic targets, thus insulating the network from teacher hallucinations.

3. Trajectory-Regularized Manifold Distillation: Source Anchoring Against Manifold Collapse

Fine-tuning the student network \(f_\theta\) exclusively on target pseudo-pairs \(\mathcal{D}_{sel}^{ood}\) leads to manifold collapse—the network learns to hallucinate plausible surface textures while forgetting the precise physical mappings needed for faithful reconstruction.

To prevent this catastrophic forgetting, GMD formulates fine-tuning as a constrained dual-objective optimization problem:

\[\mathcal{L}_{total}(\theta) = \underbrace{\frac{1}{B_{id}} \sum_{i=1}^{B_{id}} \left\| f_\theta(y_{id}^{(i)}) - x_{id}^{(i)} \right\|^2}_{\text{Trajectory Regularization } (\mathcal{L}_{id})} + \lambda \underbrace{\frac{1}{B_{ood}} \sum_{j=1}^{B_{ood}} \left\| f_\theta(y_{ood}^{(j)}) - \hat{x}_{ood}^{(j)} \right\|^2}_{\text{Manifold Alignment } (\mathcal{L}_{ood})}\]

Here, \(\mathcal{L}_{ood}\) pulls student predictions toward the natural image manifold \(\mathcal{M}\), while \(\mathcal{L}_{id}\) serves as a rigid structural anchor. By enforcing a high mixing ratio of in-distribution source data (a 9:1 ratio, meaning 90% source and 10% target), the optimization is confined to physically consistent inverse mappings. This enables the student to smoothly interpolate between exact physical reconstruction learned on the source domain and rich natural textures distilled from the target domain.

Loss & Training

The student model is a 1.3B parameter Latent Diffusion Model (LDM) built on an MMDiT backbone, pre-trained on the source domain for 500K iterations. The offline adaptation proceeds in two distinct stages: 1. Offline pseudo-target generation: uses a 12B FLUX.1.dev model with an Euler solver over \(N=50\) steps, classifier-free guidance scale \(w=3.5\), and attention injection. On 4 TPUv5p chips, processing an entire target dataset takes approximately 3 hours. 2. Knowledge distillation / fine-tuning: trains on 32 TPUv5p chips for approximately 4 hours with Adam, using a 9:1 data ratio (90% source, 10% filtered target). Once adapted, the 1.3B student model executes standard single-pass inference with zero test-time teacher invocation, zero added parameters, and zero additional latency.

Key Experimental Results

Main Results

GMD was evaluated across four challenging domain adaptation scenarios: 1. Deblurring adaptation: GoPro (source) \(\to\) REDS (video motion blur) and RealBlur-J (real-world camera shake and low-light noise). 2. Synthetic SR adaptation: Weak degradation (DIV2K low blur/noise) \(\to\) Strong degradation (Flickr2K severe blur/noise). 3. Real-world SR adaptation: Synthetic degradation \(\to\) DPED-iPhone (real sensor noise and ISP compression).

The following table summarizes quantitative performance on the real-world deblurring adaptation benchmark (GoPro \(\to\) REDS and RealBlur-J), reporting perceptual quality (LPIPS, NIMA, MUSIQ, FID, CLIPIQA) and distortion metrics (PSNR, SSIM):

Dataset	Method	LPIPS↓	NIMA↑	MUSIQ↑	FID↓	CLIPIQA↑	PSNR↑	SSIM↑
REDS	DeblurGANv2	0.190	4.350	53.17	35.77	0.313	27.07	0.805
REDS	Restormer	0.220	4.401	53.07	36.42	0.270	26.58	0.801
REDS	Uformer	0.197	4.315	54.24	35.11	0.283	27.20	0.825
REDS	Hi-Diff	0.205	4.305	54.04	39.26	0.277	27.21	0.831
REDS	DA-CLIP	0.223	4.294	48.43	42.90	0.258	25.82	0.764
REDS	LDM-Deblur (Baseline)	0.183	4.325	57.60	37.67	0.306	24.08	0.678
REDS	LDM-Deblur w/ GMD (Ours)	0.179	4.460	63.67	31.64	0.404	24.35	0.682
REDS	Fully Supervised (Upper Bound)	0.169	4.578	65.04	32.49	0.462	24.51	0.686
RealBlur-J	Restormer	0.150	4.060	47.71	23.97	0.245	27.07	0.824
RealBlur-J	Hi-Diff	0.145	4.142	50.07	22.28	0.254	27.12	0.831
RealBlur-J	DA-CLIP	0.239	3.859	38.96	42.26	0.236	20.53	0.680
RealBlur-J	LDM-Deblur (Baseline)	0.145	4.081	50.71	25.74	0.214	26.74	0.780
RealBlur-J	LDM-Deblur w/ GMD (Ours)	0.132	4.480	61.33	20.33	0.354	26.88	0.796
RealBlur-J	Fully Supervised (Upper Bound)	0.110	4.487	52.57	17.63	0.293	27.96	0.829

Across super-resolution benchmarks: - Synthetic 4× SR (Weak \(\to\) Strong DIV2K): GMD increased MUSIQ from 54.41 to 58.29 (+3.88), reduced FID from 30.11 to 28.30 (-1.81), lowered LPIPS from 0.386 to 0.368, and improved PSNR/SSIM from 22.03/0.564 to 22.32/0.578. - Real-World 2× SR (DIV2K \(\to\) DPED-iPhone): GMD reduced NIQE from 5.467 to 4.300, BRISQUE from 19.02 to 16.35, and boosted MANIQA from 0.543 to 0.627, outperforming specialized diffusion baselines such as StableSR (NIQE 4.475, MANIQA 0.607) and SeeSR (NIQE 5.110, MANIQA 0.619). - Human Evaluation: A 2AFC user study with 50 participants (deblurring) and 60 participants (super-resolution) showed that GMD was overwhelmingly preferred over all competitive baselines (>80% win rate).

Ablation Study

1. Impact of Quality-Gated Manifold Filtering (REDS Deblurring)

Variant	PSNR↑ / SSIM↑	LPIPS↓	MUSIQ↑	FID↓	CLIPIQA↑	Note
w/o filter	23.51 / 0.652	0.293	50.17	41.26	0.272	Severe degradation; drops well below unadapted baseline
w/ filter (Ours)	24.35 / 0.682	0.179	63.67	31.64	0.404	Discards off-manifold outliers; peak perceptual quality

2. Impact of Source-Anchoring Data Mixing Ratio (ID:OOD)

Source Ratio	LPIPS↓	MUSIQ↑	FID↓	CLIPIQA↑	Behavior & Empirical Observation
1.0 (Source only)	0.183	57.60	37.67	0.306	Unadapted baseline; leaves noticeable residual blur on OOD targets
0.95	0.188	58.10	36.75	0.320	Mild adaptation; conservative texture injection
0.90 (Ours)	0.179	63.67	31.64	0.404	Optimal trade-off: sharp natural details with rigid fidelity
0.60	0.187	62.15	39.09	0.379	Weakened physical regularization; FID drops to 39.09
0.30	0.198	62.03	39.73	0.375	Structural fidelity drifts; hallucinated artifacts appear
0.00 (Target only)	0.217	52.61	53.56	0.287	Catastrophic forgetting and manifold collapse; performance crashes

3. Test-Time Foundation Model Invocations vs. Offline Distillation (GMD)

Execution Mode	Model Params	Inference Latency	LPIPS↓	MUSIQ↑	FID↓	CLIPIQA↑	PSNR / SSIM
Baseline (GoPro)	1.3B	1.17s	0.183	57.60	37.67	0.306	24.08 / 0.678
+ SDEdit (online)	1.3B+12B	1.17s + 2.40s	0.201	63.50	40.09	0.404	20.18 / 0.619
+ FLUX.1 Kontext (online)	1.3B+12B	1.17s + 6.50s	0.180	63.60	32.10	0.400	21.50 / 0.630
+ RF-Solver (online)	1.3B+12B	1.17s + 4.79s	0.186	63.47	33.57	0.404	22.15 / 0.643
w/ GMD (Ours, offline)	1.3B	1.17s (+0s)	0.179	63.67	31.64	0.404	24.35 / 0.682

Key Findings

• Quality-gated filtering prevents toxic pseudo-supervision: Disabling the filter causes FID to spike from 31.64 to 41.26, which is considerably worse than the unadapted baseline (37.67). Unchecked generative hallucinations directly contaminate student training, underscoring quality gating as an indispensable safeguard.
• Source-anchoring regularization guards against manifold collapse: Setting the source ratio to 0 (training exclusively on pseudo-targets) causes catastrophic degradation (FID 53.56). Maintaining a 90% source data proportion anchors the student to exact physical inverse mappings while successfully absorbing target domain natural textures.
• Attention injection is essential for orthogonal projection: In the zero-shot alignment stage, omitting attention injection causes severe content drift and geometric distortions (e.g., reshaping vehicle bodylines). Reusing forward self-attention keys and values restricts generative flow to high-frequency detail restoration while freezing spatial layout.
• Architectural flexibility across teachers and students: Supplementary experiments show that GMD generalizes effectively when adapting lightweight non-diffusion backbones (SwinIR) or using smaller generative teachers (Stable Diffusion).

Highlights & Insights

• Translates prohibitive generative inference costs into a one-time offline training investment: By decoupling prior extraction from online deployment, GMD endows a 1.3B student model with the perceptual realism of a 12B generative model at an unchanged inference latency of 1.17s.
• Achieves true orthogonal manifold projection via attention injection: Using forward ODE lifting to marginalize unstructured noise and reverse ODE attention caching to lock spatial semantics, GMD resolves the fundamental realism-vs-fidelity trade-off in generative restoration.
• Highly reproducible 9:1 source-anchoring paradigm: The asymmetrical mixed-supervision strategy offers a robust, generalizable template for applying generative pseudo-labels to ill-posed inverse problems without suffering from catastrophic forgetting.

Limitations & Future Work

• Reliance on general text-to-image foundation priors: The offline teacher's guidance is bounded by its pre-training data; for specialized domains far removed from natural photography (e.g., medical pathology, satellite remote sensing), general-purpose foundation priors may provide limited utility.
• Static offline adaptation cost: Although test-time latency is zero, adaptation requires several hours of TPU computation, making it less suited for continuous, streaming domain shifts in real time.
• Global perceptual metric limitations in quality gating: Global quality scores (NIMA) may miss localized semantic inconsistencies or subtle structural deviations, highlighting the need for patch-level topological consistency metrics in future iterations.

Related Work & Insights

• vs Traditional Unpaired UDA (CycleGAN / DualGAN / AdaIR / DA-CLIP): Conventional methods require unstable adversarial training, specialized auxiliary modules, and clean target-domain references. GMD maintains an unmodified base architecture and operates strictly on low-quality target observations.
• vs Test-Time Foundation Model Inversion (SDEdit / RF-Solver / DiffBIR / SeeSR): Online zero-shot techniques multiply test-time latency and introduce semantic hallucinations. GMD moves the generative prior entirely offline, eliminating latency penalties and suppressing content drift through distillation.
• vs Pseudo-Labeling Pipelines: Naive pseudo-labeling suffers from cumulative error cascades. GMD resolves this via a three-tier defense: attention-guided orthogonal projection, objective quality filtering, and 90% source-domain trajectory regularization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Reframes domain adaptation as geometric manifold distillation, elegantly combining rectified flow dynamics, attention injection, and source anchoring.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across four distinct benchmarks with perceptual/distortion metrics, detailed ablations, compute cost profiles, and human evaluations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear geometric motivation, disciplined mathematical formulation, and well-structured empirical validation.
• Value: ⭐⭐⭐⭐⭐ Directly addresses industry demands for zero-latency, architecture-preserving domain adaptation in real-world visual enhancement.

Related Papers

• [ECCV 2026] When Distillation Breaks Motion Control: Restoring Generative Trajectories for Fast Video Generators
• [ECCV 2026] Learning to Corrupt for Better Restoration
• [ECCV 2026] Not All Prediction Targets Keep Training-Free Diffusion Guidance on the Manifold
• [ICLR 2026] Motion Prior Distillation in Time Reversal Sampling for Generative Inbetweening
• [ECCV 2026] Flow Straight to Reality: Perceptually Consistent Flow Matching for Efficient Image Restoration