SD3.5-Flash: Distribution-Guided Distillation of Generative Flows

Conference: ECCV2026
Paper: ECCV Paper
Area: Image Generation
Keywords: rectified flow, distribution matching distillation, timestep sharing, split-timestep fine-tuning, mobile generation

TL;DR

SD3.5-Flash places distribution matching on the student's own generation trajectory and combines split-timestep fine-tuning with pipeline optimization to distill SD3.5M into a four-step text-to-image model, improving selected preference and alignment metrics while supporting phones, without improving every quality metric.

Background & Motivation

The cost of high-quality text-to-image models comes not only from their parameter counts but also from repeatedly invoking the denoising network for each image. Rectified flow models such as SD3.5 directly predict a velocity field that transports Gaussian noise toward samples in image latent space. Reducing dozens of iterations to four can substantially reduce computation, but it is not simply a matter of increasing the sampler's step size: the student must learn transitions that originally required multiple teacher updates. Trajectory imitation supplies an initialization, while distribution matching distillation further asks the student's overall output distribution to approach the teacher's without reproducing every teacher output pixel by pixel.

The difficulty concerns where distribution matching supervision is evaluated. Conventional DMD starts from a clean student-generated sample, adds noise at a random time, and uses the difference between target and student distribution scores to produce an update signal. The paper argues that, for a rectified flow student whose trajectory guidance establishes strongly coupled noise-data pairs, this re-noising can leave the learned ODE trajectory and make velocity estimates unreliable. With only four steps, subsequent iterations have little opportunity to correct errors; meanwhile, one parameter set must support both visual quality and complex prompt semantics across noise levels.

The authors therefore address supervision locations, training capacity, and deployment costs separately: supervision uses intermediate states reached by the student, and the final training stage temporarily separates models for different time ranges before merging them. Even with only four denoising steps, text encoders such as T5-XXL consume substantial memory, motivating optimization of the complete generation pipeline. Core Idea: supervise the few-step student on its own trajectory, temporarily increase time-specialized capacity during training, and deploy a single model with configurable text encoders.

Method

Overall Architecture

The inputs are a text prompt and Gaussian noise, and the output is a VAE-decoded image; intermediate states are noisy image latents processed by the flow model. Training proceeds through trajectory warm-up, on-trajectory distribution matching, and split-timestep fine-tuning; full-pipeline optimization determines the encoders and numerical precision used for deployment. The teacher supplies distribution supervision, while a continuously updated proxy tracks the student distribution and provides features for a multi-head discriminator. The proxy, discriminator, and split-timestep branches are training mechanisms, not additional generators that must run sequentially at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Initialize teacher and student"] --> WARM["Trajectory warm-up"]
    WARM --> MATCH["On-trajectory<br/>distribution matching"]
    AUX["Teacher, proxy, and<br/>multi-head discriminator"] -.->|Training supervision only| MATCH
    MATCH --> SPLIT["Split-timestep fine-tuning"]
    SPLIT --> OPT["Full-pipeline optimization"]
    OPT --> GEN["Single-model four-step generation<br/>and VAE decoding"]
    PROMPT["Inference: text and noise"] --> GEN
    GEN --> IMAGE["Image"]

Key Designs

1. Trajectory warm-up: first learn to span multiple teacher updates

The teacher, student, and proxy all start from pretrained teacher weights, avoiding the need for a randomly initialized few-step generator to learn the entire image distribution. The first-stage trajectory guidance loss, denoted \(L_{\mathrm{TG}}\), selects teacher trajectory states corresponding to student sampling times and supervises the student's larger updates against the teacher's accumulated motion over those intervals. It does not require the student to reproduce the teacher at every fine-grained time; it first makes transport between a small number of update nodes reasonable. The paper describes this stage as establishing reflow-like, strongly coupled noise-data pairs and a more stable initialization for subsequent training. The second stage therefore does not bear the full burden of establishing a usable generation trajectory from scratch and can concentrate on texture, sharpness, and distribution refinement.

The model generating synthetic training data must be distinguished from the distillation teacher. Section 5.1 uses the 8B SD3.5 Large model with 32 steps and CFG 4.0 to generate synthetic samples, whereas few-step distillation uses the 2.5B SD3.5M teacher. They provide the training samples and distillation guidance, respectively; the result should not be described as directly compressing the 8B model into four steps. The cached trajectory guidance equation is damaged, so this note explains the interval-imitation mechanism supported by the prose without reconstructing its exact summation.

2. On-trajectory distribution matching: supervise states the student actually reaches

DMD seeks to reduce the KL divergence of the student distribution relative to the teacher distribution, but their probability densities cannot be computed directly. Training therefore obtains gradients from distribution scores rather than explicitly assigning a probability to each image. The teacher supplies information about the target distribution, while the proxy velocity model \(v_{\mathrm{fake}}\) learns from student-generated samples to estimate the student distribution. The proxy is neither a single forward pass through the few-step student nor a permanently frozen teacher copy; it must keep adapting as the student's distribution changes. Section 4.2 trains it with the standard flow matching objective:

\[ L_{\mathrm{FM}}=\left\|v_{\mathrm{target}}-v_{\mathrm{fake}}(x_t,t)\right\|_2^2. \]

Here, \(v_{\mathrm{target}}\) is the target velocity associated with constructing the noisy sample, and \(x_t\) is the latent at that noise level. The background rectified flow interpolation is \(x_t=(1-t)x_0+t\epsilon\), where \(t=0\) is the data endpoint and \(t=1\) is the noise endpoint. Fitting this proxy and selecting the DMD supervision location are separate operations; the method does not prohibit adding noise in every training objective.

Timestep sharing changes the intermediate state at which the distribution matching gradient is evaluated. The student first follows its own denoising trajectory to the update being trained, enables gradients only for that update, and evaluates the distribution discrepancy at the resulting next noisy state. It no longer needs to extrapolate a clean sample and then add fresh noise to reach a randomly selected time. The trade-off is that DMD sees fewer time points; the authors exchange temporal coverage for more reliable local supervision of the trajectory the student actually follows. When the final update reaches \(t=0\), its DMD term is set to zero, but adversarial and other auxiliary objectives still train that step.

Distribution matching also relies on an adversarial objective for stable training. The proxy extracts latent-space features at multiple predefined noise levels and network layers, and attached MLP discriminator heads distinguish synthetic real samples from student samples. High-noise and low-noise features provide relatively coarse and fine visual signals, avoiding judgments based on only one scale. Training uses non-saturating GAN objectives and periodically reinitializes discriminator heads to reduce overfitting to limited training patterns. This discriminator feature path still noises samples; the removal of random re-noising should be attributed specifically to DMD estimation under timestep sharing.

3. Split-timestep fine-tuning: temporarily allocate training capacity across noise ranges

A few-step model must use one parameter set for stages ranging from global layout to local detail, and the authors attribute part of the alignment loss to this capacity constraint. The final stage copies the partially trained model into two branches: \(M_1\) handles \((0,500]\), and \(M_2\) handles \((500,1000]\). These are implementation-scale timesteps corresponding to the normalized interval above; the high-noise range executes before the low-noise range. Each branch is fine-tuned on its assigned range with an exponential moving average decay of \(\beta=0.99\) to stabilize training and keep weights close to the original checkpoint. After convergence, weight interpolation merges the branches at \(M_1:M_2=3:7\), with the ratio selected using GenEval alignment performance.

Inference loads only the merged checkpoint, rather than storing two complete models for different time ranges or interpolating two generated images. The additional effective capacity exists during training; it does not double the deployed network size. The approach relies on the branches remaining in a weight neighborhood that supports useful interpolation and does not imply that arbitrary independently trained models can be averaged successfully. The paper reports improvements in composition, background detail, and prompt alignment after this stage, but main-paper Table 5 does not provide a separate GenEval value for this ablation.

4. Full-pipeline optimization: include encoders and hardware precision in the deployment budget

Besides MM-DiT, the SD3.5 pipeline contains CLIP-L, CLIP-G, T5-XXL, and a VAE. T5-XXL contributes substantially to peak memory and inference cost, and the complete 16-bit distilled pipeline requires approximately 18 GiB of GPU memory. The authors exploit encoder dropout during SD3.5 pretraining to make T5-XXL optional: null embeddings replace its output while CLIP-L/G text conditioning remains active. This does not mean replacing the entire prompt with empty text or performing unconditional image generation.

Quantizing MM-DiT to 8-bit and removing T5 reduces memory requirements to approximately 8 GiB. For phones and tablets, the authors further quantize the 8-bit model to 6-bit through CoreML and rewrite operations such as RMSNorm to reduce precision loss on the Apple Neural Engine. The mobile result therefore depends on operator implementation and backend adaptation, not merely conversion of the weight file to a lower bit width. Quantization does not guarantee a speedup on every device; some 8-bit configurations are slower than their 16-bit counterparts on an RTX 4090. Removing T5 can also weaken conditioning for complex compositional prompts, so deployment choices must account jointly for memory, latency, and semantic requirements.

A Worked Example

Consider the training example in Section 4.2, where the current update is at implementation-scale time \(t=500\). Starting from pure noise, the student follows \(x_{1000}\rightarrow x_{750}\rightarrow x_{500}\) with gradients disabled. It then enables gradients, predicts a velocity, and updates to \(x_{250}\), where the teacher and proxy supply DMD supervision at the actual next state. The conventional route would first estimate a clean endpoint from the current velocity and re-noise it to a random time; the proposed method removes precisely this off-trajectory operation. Other auxiliary objectives can still train the update, while the subsequent \(x_{250}\rightarrow x_0\) update receives no DMD term. Deployment only follows the four-step student trajectory and decodes the result, without retaining the training teacher, proxy, or discriminator.

Loss & Training

The first stage performs 2K trajectory warm-up iterations, the second uses distribution matching and multi-head adversarial training for 800 iterations, and the third performs split-timestep fine-tuning for 400 iterations. Training uses a single H100 node; the main paper does not provide complete GPU hours or the node's GPU count for a reliable compute conversion. The proxy tracks the student distribution through \(L_{\mathrm{FM}}\), discriminator heads optimize their adversarial objective, and the student receives DMD and generator adversarial supervision. These objectives optimize different components, so proxy fitting error should not be interpreted directly as an inference-time image quality loss. Because the cached KL-gradient and GAN equations are partially corrupted, this note does not invent exact total-loss weights or unspecified optimizer hyperparameters.

Key Experimental Results

Main Results

The following results are selected from Table 3 on page 12. CLIP, AeS, IR, and FID use 30K samples generated from COCO captions; GenEval separately tests object, attribute, and relation alignment rather than measuring the same COCO statistic. CLIP measures image-text similarity in CLIP ViT-B/32 space; AeS and IR denote Aesthetic Score and ImageReward, with higher values preferred; FID compares Inception-V3 feature distributions, with lower values preferred.

Model	Steps	CLIP ↑	AeS ↑	IR ↑	GenEval ↑	FID ↓
SD3.5M	50	32.00	5.99	0.91	0.64	20.06
SDXL-DMD2	4	31.64	6.28	0.88	0.56	16.64
SWD-M	4	32.00	6.37	1.12	0.72	25.90
SANA-Sprint 0.6B	2	31.39	6.54	0.98	0.77	24.99
SD3.5-Flash 16-bit (+T5)	4	31.65	6.38	1.10	0.70	29.80
SD3.5-Flash 16-bit (No T5)	4	31.63	6.39	1.08	0.68	28.65

Relative to the 50-step teacher, the complete four-step model raises GenEval from 0.64 to 0.70 but changes FID from 20.06 to 29.80; this is not uniformly lossless compression. SWD-M has higher GenEval and IR, and SANA-Sprint 0.6B has higher GenEval than the complete proposed configuration, so no method wins every automated metric.

Ablation Study

The following results come from Table 5 on page 13, using the metric definitions in Section 5.4 and the 16-bit four-step SD3.5M distillation pipeline. The first four rows primarily test the second stage and omit final split-timestep fine-tuning; removing the adversarial objective also removes discriminator refresh, so differences from the final model are not all strict single-variable effects.

Config	AeS ↑	CLIP ↑	FID ↓	IR ↑
Without adversarial objective	4.72	28.50	97.40	-0.57
Without student warm-up	6.22	31.14	30.13	0.89
Without timestep sharing	5.97	30.97	35.61	0.67
Without discriminator refresh	6.07	31.22	41.84	0.52
Baseline: without split-timestep fine-tuning	6.32	31.46	30.76	0.93
Full model: 16-bit + T5	6.38	31.65	29.80	1.10

Against the baseline without split-timestep fine-tuning, removing timestep sharing gives IR of 0.67 rather than 0.93 and FID of 35.61 rather than 30.76, supporting the importance of supervision location. Split-timestep fine-tuning provides a more direct final-stage comparison: IR rises from 0.93 to 1.10, while FID falls from 30.76 to 29.80. Removing the adversarial objective produces FID of 97.40, showing that timestep sharing alone does not replace the stability supplied by adversarial training.

Key Findings

The deployment results below come from Table 1 on page 8, all using the four-step 6-bit configuration without T5; resolution labels such as "768 px" follow the source table. Both the M4 iPad and A17 iPhone are listed with 8 GB of unified memory, and the table does not report their latency at 1024 px.

Resolution	M4 iPad latency / seconds	A17 iPhone latency / seconds
768 px	6.44	8.32
512 px	2.62	3.25

Table 4 on page 12 reports 0.58 seconds for 16-bit + T5 on an RTX 4090 versus 0.66 seconds for 8-bit + 8-bit T5, illustrating that lower memory usage does not necessarily mean lower latency. The user study involves 178 annotators, 4 seeds per prompt, and 3 votes per image pair, with over 54K image-quality comparisons against 12 competitors. The authors report leading image-quality preference, but prompt-adherence preference differences across methods are below ±1.6%; quality preference should not be treated as an equally large semantic-alignment gain. Table 2 groups Elo ratings by deployment environment, with only two models in the mobile group; scores from different groups should not be combined into a universal ranking.

Highlights & Insights

• The central change is not a more complicated distance function but a different supervision location. For strongly path-coupled generators, evaluating gradients on the actual trajectory may matter more than covering additional random times.
• Split-timestep fine-tuning separates training capacity from inference capacity. It allows specialized optimization across noise ranges and avoids deploying two models through parameter merging.
• The proxy serves both distribution estimation and discriminator feature extraction, reducing the need for another feature network, but its objectives and noise constructions must be distinguished.
• Deployment experiments account for text encoders, backbone quantization, and operator adaptation within one budget. Reporting four denoising steps alone does not establish whether the pipeline can run on a phone.

Limitations & Future Work

• The authors acknowledge that distillation sacrifices some quality and diversity, while removing T5 particularly affects complex compositions. Small changes in average alignment metrics do not exclude severe individual failures on long prompts or compositional relations.
• FID degradation is an actual observation and should not be omitted. The authors attribute it to differences between low-level statistics and perceptual preferences, but this does not establish that all distribution-coverage issues are absent.
• Section 5.4 claims SOTA AeS, but Table 3 gives 6.38 for the proposed +T5 model versus 6.67 for SDXL-HyperSD; the specific table comparison should take precedence over that unconditional claim.
• Generalization to FLUX.1-dev is demonstrated only through qualitative samples after training through stage two, with minor artifacts remaining; it does not establish a complete cross-architecture quantitative result.
• Supplementary material is unavailable in the cache, and some equations and figure text are corrupted. Unverified Figure 8 percentages, loss weights, and complete training costs are not used as quantitative evidence.
• Future work could test timestep sharing across step counts, trajectory initializations, and distribution coverage; this is a mechanism-based research suggestion, not a completed experiment.

Related Work & Insights

• Versus DMD / DMD2 (paper references 56 / 55): the method retains score-based distribution supervision, a proxy, and adversarial training; its main change is supervision location for few-step rectified flows rather than a reinvention of distribution matching.
• Versus NitroFusion (paper reference 3): it adopts multiple discriminators and periodic refresh but also relies on trajectory warm-up and DMD, so the result should not be attributed to purely adversarial distillation.
• Versus SWD (paper reference 49): SWD organizes distillation across scales, whereas this work emphasizes temporal trajectories and split capacity; the main results show advantages on different metrics.
• Versus SANA-Sprint (paper reference 5): the latter uses continuous-time consistency distillation, and its two-step configurations offer speed and selected alignment advantages; this work emphasizes four-step SD3.5M quality and cross-device deployment.
• Research direction: separately ablate the student states used for supervision and the estimation of the student distribution to distinguish stability gains from changed state distributions versus improved proxy estimates.

Rating

• Novelty: 4/5. Timestep sharing and post-training branch fusion are targeted contributions within an established distribution matching and adversarial distillation framework.
• Experimental Thoroughness: 4/5. Automated metrics, human preference, component ablations, and device latency are included, while cross-architecture and diversity analyses remain limited.
• Writing Quality: 3/5. The training example is clear, but the AeS claim conflicts with the table, and some reproducibility details depend on supplementary material.
• Value: 4/5. The work provides useful training and deployment lessons for few-step rectified flows and mobile text-to-image generation.