Skip to content

Generalization and Memorization in Rectified Flow

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/mx-ethan-rao/rf_gen_mem.git
Area: Image Generation
Keywords: Rectified Flow, Membership Inference Attack, Model Memorization, Generalization Theory, Timestep Sampling Distribution

TL;DR

Addressing the underexplored memorization mechanisms in Rectified Flow generative models, this paper proposes the first complexity-calibrated membership inference attack test statistic, theoretically proves that memorization strictly peaks at the ODE trajectory midpoint (\(t=0.5\)) due to state-velocity statistical orthogonality, and introduces symmetric exponential U-shaped timestep sampling to suppress privacy vulnerabilities without degrading generative fidelity.

Background & Motivation

Generative modeling based on the Flow Matching (FM) objective, notably Rectified Flow (RF), connects standard Gaussian noise and target empirical distributions via straight-line ODE trajectories. By linearizing the generative dynamics, RF achieves few-step generation and training stability, underpinning modern state-of-the-art vision foundation models including Stable Diffusion 3, FLUX.1, and Ideogram. Despite intense interest in visual fidelity scaling and architectural expansions, the underlying dynamics of how RF models memorize individual training samples, when they overfit, and how sample-level memorization distributes across continuous integration trajectories remain poorly understood.

Conventional indicators of overfitting, such as the standard generalization gap between training and validation losses, fail to capture continuous-time memorization. Even when validation loss differences are negligible, target networks can still leak identity-level information about training instances through predicted velocity fields, leaving them vulnerable to Membership Inference Attacks (MIA). Existing diffusion model MIAs rely on discrete-step reconstruction perturbations or reverse-SDE stochastic fluctuations, which do not translate cleanly to deterministic straight-line flow ODEs. Furthermore, generative likelihood and reconstruction errors are heavily confounded by spatial complexity bias—low-complexity samples (e.g., flat backgrounds) exhibit artificially small errors, obscuring genuine memorization signals.

To address these fundamental issues, this paper constructs an MIA formulation directly from the first principles of Conditional Flow Matching (CFM), coupled with a spatial complexity calibration mechanism. By comparing the trained velocity network against an analytical Linear Minimum Mean Square Error (LMMSE) baseline, the authors rigorously dissect time-dependent memorization behavior along the continuous trajectory. Core idea: quantify sample memorization via a complexity-decoupled Monte Carlo flow matching test statistic, establish that memorization strictly peaks at \(t=0.5\) because input state and target velocity become statistically orthogonal (forcing the network into non-linear memorization), and introduce a symmetric exponential U-shaped timestep distribution to suppress privacy risks with zero visual quality degradation.

Method

Overall Architecture

Rectified Flow defines an ODE \(\mathrm{d}X_t = v(X_t, t)\mathrm{d}t\) connecting base noise \(X_0 \sim \mathcal{N}(0, \mathbf{I})\) and data \(X_1 \sim \pi_1\) via linear interpolation \(X_t = t X_1 + (1-t)X_0\), where the target velocity is constant along trajectories: \(v = X_1 - X_0\). During inference, a network \(v_\theta(X_t, t)\) predicts the marginal velocity field. The methodology comprises an analytical diagnostic track and a defensive regularization track: it derives a multi-step MIA test statistic using Monte Carlo integration to eliminate prior noise variance and JPEG compression bitrate normalization to remove low-frequency spatial biases; it then formalizes the LMMSE baseline to explain the midpoint non-linear surge; finally, it replaces uniform temporal sampling with a U-shaped distribution to shield vulnerable intermediate timesteps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input sample x and prior Gaussian noise ε"] --> B["Monte Carlo Integral Test Statistic<br/>Multi-sample noise averaging eliminates prior expectation bias"]
    B --> C["Spatial Complexity Decoupling Calibration<br/>JPEG compression bitrate normalization removes spatial texture artifacts"]
    C --> D["Orthogonality & Linearity Gap Analysis<br/>Comparison against LMMSE baseline proves statistical orthogonality at t=0.5"]
    D --> E["Symmetric Exponential U-shaped Timestep Sampling<br/>Concentrating density at boundaries suppresses vulnerable intermediate exposure"]
    E --> F["Output: Memorization-resistant Rectified Flow with preserved FID"]

Key Designs

1. Monte Carlo Integral Test Statistic: Eliminating Prior Gaussian Noise Bias via Conditional Flow Matching A naive attack derived directly from the marginal FM objective, \(T_{\text{naive}}(\mathbf{x}, t) = \|\mathbf{x} - (v_\theta(\mathbf{x}_t, t) + \varepsilon)\|_2^2\), depends heavily on an individual sampled noise vector \(\varepsilon \sim \mathcal{N}(0, \mathbf{I})\), resulting in high evaluation variance. The authors inspect the optimal minimizer of the Conditional Flow Matching (CFM) objective: when overfit such that \(\mathcal{L}_{\text{CFM}} = 0\), the condition \(\mathbb{E}_{X_0 \sim \pi_0}[\mathbf{x}_1 - X_0 - v_\theta(X_t, t)] = 0\) holds. Since the standard Gaussian prior has zero mean \(\mathbb{E}[X_0] = \mathbf{0}\), the empirical data point satisfies \(\mathbf{x}_1 = \mathbb{E}[v_\theta(X_t, t)]\). Leveraging this property, the authors formulate a Monte Carlo amortized test statistic over \(N\) independent noise draws: $\(T_{\text{mc}}(\mathbf{x}, t) = \left\| \mathbf{x} - \frac{1}{N}\sum_{n=1}^N v_\theta(t\mathbf{x} + (1-t)\varepsilon_n, t) \right\|_2^2\)$ For training points \(\mathbf{x} \in \mathcal{D}_{\text{train}}\), the overfitted model's average predicted velocity converges directly to \(\mathbf{x}\), yielding \(T_{\text{mc}}(\mathbf{x} \in \mathcal{D}_{\text{train}}) \ll T_{\text{mc}}(\mathbf{x} \in \mathcal{D}_{\text{val}})\). As \(N\) increases, the empirical mean of the sampled Gaussian vectors contracts at rate \(\mathcal{O}(1/\sqrt{N})\), significantly enhancing the separability between training and held-out distributions.

2. Spatial Complexity Decoupling Calibration: Normalizing Image Bitrate to Suppress Pseudo-Memorization Artifacts Likelihood scores and reconstruction errors in deep generative models suffer from input complexity bias: visually simple, smooth inputs naturally produce smaller velocity estimation errors, whereas intricate, high-frequency training images exhibit larger residuals. Consequently, raw \(T_{\text{mc}}\) scores correlate strongly with spatial complexity (Pearson correlation exceeding 0.8). To decouple inherent image redundancy from genuine model memorization, the authors introduce a complexity-calibrated statistic: $\(T_{\text{mc\_cal}}(\mathbf{x}, t) = \frac{T_{\text{mc}}(\mathbf{x}, t)}{C(\mathbf{x})}\)$ where \(C(\mathbf{x})\) denotes the spatial complexity of the image. While Kolmogorov complexity is uncomputable, compressed byte sizes from standard algorithms provide an effective upper bound. Under a log-linear model \(\log(T_{\text{mc}}) \sim \beta \log(C(\mathbf{x})) + c\), empirical results demonstrate that standard JPEG compression consistently yields \(\beta \approx 1\) across diverse datasets, eliminating the need for hyperparameter tuning. This normalization ensures the metric tracks parameter-induced sample memorization rather than low-level image smoothness.

3. Orthogonality & Linearity Gap Analysis: Revealing Peak Memorization at t=0.5 via LMMSE Degeneracy Scanning attack effectiveness across \(t \in [0, 1]\) reveals that MIA vulnerability strictly peaks at the integration midpoint \(t=0.5\). To explain this phenomenon, the authors compare \(v_\theta\) against an analytical Linear Minimum Mean Square Error (LMMSE) estimator \(v_{\text{linear}}(\mathbf{x}_t)\), representing a predictor with zero non-linear extraction capacity. Assuming standardized data with mean \(\mu_{x_1} = \mathbf{0}\) and covariance \(\Sigma_{x_1} = \mathbf{I}\), the covariance between state \(\mathbf{x}_t = t \mathbf{x}_1 + (1-t)\mathbf{x}_0\) and target velocity \(v = \mathbf{x}_1 - \mathbf{x}_0\) evaluates to: $\(\Sigma_{v, \mathbf{x}_t} = \mathbb{E}[v \mathbf{x}_t^\top] = t \mathbb{E}[\mathbf{x}_1\mathbf{x}_1^\top] - (1-t)\mathbb{E}[\mathbf{x}_0\mathbf{x}_0^\top] = (2t - 1)\mathbf{I}\)$ At exactly \(t = 0.5\), \(\Sigma_{v, \mathbf{x}_{0.5}} = \mathbf{0}\), proving that the noisy input state is statistically orthogonal to the target velocity. The optimal linear predictor degenerates entirely to the uninformative mean \(v_{\text{linear}}(\mathbf{x}_{0.5}) = \mathbf{0}\), providing zero linear guidance. To minimize flow matching loss at this juncture, the neural network is forced to exploit its non-linear capacity to memorize individual training samples, drastically widening the linearity gap \(\Delta v = \|v_\theta - v_{\text{linear}}\|_2\) for training data while collapsing on validation inputs. This proves that under uniform temporal sampling, continuous-time memorization risk is strictly upper-bounded by the midpoint vulnerability.

4. Symmetric Exponential U-shaped Timestep Sampling: Reshaping Training Density to Mitigate Intermediate Overfitting Standard differential privacy mechanisms (such as DP-SGD) inject destructive noise and clip gradients, severely impairing sample quality (elevating FID). Having proven that memorization is tightly concentrated near \(t=0.5\) while the boundaries (\(t \to 0\) and \(t \to 1\)) carry rich linear information with minimal overfitting risk, the authors replace uniform sampling \(t \sim \mathcal{U}(0, 1)\) with a Symmetric Exponential U-shaped distribution: $\(p(t; \alpha) = \frac{\alpha}{2(1 - e^{-\alpha})}\left(e^{-\alpha t} + e^{-\alpha(1-t)}\right), \quad t \in [0, 1]\)$ The parameter \(\alpha > 0\) controls the degree of probability mass concentrated near the extremes. Suppressing exposure to vulnerable intermediate timesteps (\(t \in [0.4, 0.6]\)) impedes non-linear memorization accumulation under equivalent SGD iteration budgets, preserving or even improving generative fidelity (FID).

Loss & Training

Models are trained under the Conditional Flow Matching objective paired with the reparameterized U-shaped temporal distribution: $\(\mathcal{L}_{\text{CFM}}(\theta) = \mathbb{E}_{t \sim p(t; \alpha), \mathbf{x}_1 \sim \pi_1, \mathbf{x}_0 \sim \mathcal{N}(0, \mathbf{I})}\left[ \left\| (\mathbf{x}_1 - \mathbf{x}_0) - v_\theta(t\mathbf{x}_1 + (1-t)\mathbf{x}_0, t) \right\|_2^2 \right]\)$ To prevent endpoint numerical instabilities, sampling intervals are clamped to \([t_{\text{min}}, t_{\text{max}}] = [10^{-5}, 1 - 10^{-5}]\). Attacks evaluate using \(N_{\text{mc}} = 5\) Monte Carlo samples; for high-resolution latent RF on MSCOCO, a fixed VAE encoder-decoder space is utilized.

Key Experimental Results

Main Results

Evaluations were conducted on CIFAR-10 (\(32\times 32\)), SVHN (\(32\times 32\)), and TinyImageNet (\(64\times 64\)), using balanced 50/50 splits for training (\(\mathcal{D}_{\text{train}}\)) and validation (\(\mathcal{D}_{\text{val}}\)). MIA performance across AUC (%) and the privacy-critical TPR@1%FPR (%) metric is reported below:

Method CIFAR-10 AUC↑ (%) CIFAR-10 TPR@1%FPR↑ (%) SVHN AUC↑ (%) SVHN TPR@1%FPR↑ (%) TinyImageNet AUC↑ (%) TinyImageNet TPR@1%FPR↑ (%)
\(T_{\text{naive}}\) 70.98 2.70 68.04 1.89 72.33 4.10
\(T_{\text{mc}}\) (\(N_{\text{mc}}=5\)) 75.12 3.05 70.92 1.54 76.44 5.33
\(\Delta\) (vs. \(T_{\text{naive}}\)) +4.14 +0.35 +2.88 -0.35 +4.11 +1.23
\(T_{\text{mc\_cal}}\) (\(N_{\text{mc}}=5\)) 84.89 27.88 79.43 16.46 92.96 50.03
\(\Delta\) (vs. \(T_{\text{mc}}\)) +9.77 +24.83 +8.51 +14.92 +16.52 +44.70

On latent RF models trained on MSCOCO (\(512\times 512\)) using a pretrained VAE, scanning attack performance across timesteps \(t\) (AUC %) confirms that peak attack success aligns with the theoretical midpoint:

Timestep \(t\) 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80
\(T_{\text{naive}}\) 57.94 70.63 75.50 77.04 76.14 73.16 68.94 48.46
\(T_{\text{mc}}\) (\(N_{\text{mc}}=5\)) 59.11 73.15 80.25 85.14 85.33 83.45 75.63 64.37
\(\Delta\) (vs. \(T_{\text{naive}}\)) +1.17 +2.52 +4.75 +8.10 +9.19 +10.29 +6.69 +15.91

Ablation Study

The strong confounding effect of spatial complexity on uncalibrated statistics is evidenced by correlation coefficients between compression bitrate and raw \(T_{\text{mc}}\):

Split CIFAR-10 Pearson \(\rho\) CIFAR-10 Spearman \(r_s\) SVHN Pearson \(\rho\) SVHN Spearman \(r_s\) TinyImageNet Pearson \(\rho\) TinyImageNet Spearman \(r_s\)
Train \(\mathcal{D}_{\text{train}}\) 0.6030 0.5917 0.8520 0.8699 0.5275 0.5535
Val \(\mathcal{D}_{\text{val}}\) 0.8318 0.8561 0.8842 0.9060 0.7837 0.8111

Ablations on Monte Carlo steps \(N_{\text{mc}}\) show that while \(N_{\text{mc}}=1\) exhibits high variance, \(N_{\text{mc}} \ge 2\) steadily surpasses \(T_{\text{naive}}\), stabilizing around \(N_{\text{mc}}=5\). Furthermore, scaling the U-shaped concentration parameter \(\alpha\) from 0 to 2 and 4 steadily depresses peak attack AUC without degrading generation quality (FID remains flat or slightly improves).

Key Findings

  • Complexity calibration unlocks high-precision attack capability: Uncalibrated statistics yield dismal TPR@1%FPR values (1.5%~5.3%), as false positives arise from low-complexity validation samples. Calibrating with image bitrate (\(T_{\text{mc\_cal}}\)) boosts TPR@1%FPR dramatically to 50.03% on TinyImageNet (+44.70% absolute gain), revealing severe underlying memorization.
  • Vulnerability peaks precisely at statistical orthogonality: Across pixel-level and latent-space models, peak attack sensitivity centers at \(t=0.5\). In latent space, \(T_{\text{mc}}\) peaks precisely at \(t=0.50\) (AUC 85.33%), confirming that Gaussian variance cancellation pins the vulnerability to the theoretical orthogonality point.
  • Linearity gap directly mirrors memorization strength: The trajectory of \(\|v_\theta - v_{\text{linear}}\|_2\) closely tracks the attack AUC curve, confirming that memorization emerges because linear regression collapses at the midpoint, forcing the network into overfitted non-linear representations.

Highlights & Insights

  • First-principles analysis of generative memorization: Rather than treating Flow Matching models as black-box denoisers, this work connects the CFM objective to LMMSE filtering, deriving the closed-form covariance \(\Sigma_{v, \mathbf{x}_t} = (2t-1)\mathbf{I}\) and mathematically pinning down the midpoint memorization spike.
  • Circumventing the privacy-utility trade-off: Traditional DP-SGD impairs generative fidelity by injecting global noise. By uncovering the temporal locality of memorization, U-shaped timestep sampling provides a non-invasive regularization mechanism that slows privacy leakage while maintaining image quality.

Limitations & Future Work

  • Admitted limitations: Bitrate calibration currently relies on spatial compression (JPEG), which is well-suited for pixel data but cannot directly operate on non-grid latent feature spaces produced by deep VAEs, necessitating external encoder-decoder calibration heuristics.
  • Open questions: The midpoint peak assumes a standardized isotropic covariance \(\Sigma_{x_1} = \mathbf{I}\). For datasets with highly skewed eigenvalue spectra, the minimum-variance timestep may shift away from \(t=0.5\). Exploring adaptive, spectrum-aware temporal distributions remains an intriguing future direction.
  • vs Carlini et al. (2022) [7] / Duan et al. (2023) [10] (Diffusion MIAs): Prior methods formulate test statistics for discrete noise-addition steps or reverse-SDE perturbations; this work derives tailored statistics directly from CFM deterministic flow identities and explicitly removes input complexity bias.
  • vs Lee et al. (2024) [26] (Symmetric Exponential Sampling): Lee et al. proposed U-shaped sampling heuristically to address large boundary loss in 2-Rectified Flow. This paper supplies a fundamental generalization and memorization theoretical justification, demonstrating its efficacy as a temporal regularizer against privacy leakage.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering theoretical analysis of memorization in Rectified Flow, proving midpoint orthogonality and deriving calibrated statistics]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across pixel and latent models, 4 datasets, metric ablations, and FID tracking]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical derivations seamlessly connected to empirical observations]
  • Value: ⭐⭐⭐⭐⭐ [High impact for generative AI copyright protection, privacy auditing, and principled training scheduling]