DiffPro: Joint Timestep and Layer-Wise Precision Optimization for Efficient Diffusion Inference

Conference: ECCV2026
Paper: ECCV Paper
PDF: Official Linked Full Text
Area: Model Compression
Keywords: mixed-precision quantization, dynamic activation quantization, timestep pruning, hardware-aware optimization, diffusion models

Identity check: manifest ID 3584 matches the cached title and official entry. The authors are Farhana Amin, Sabiha Afroz, Kanchon Gharami, Mona Moghadampanah, and Dimitrios S. Nikolopoulos. PDF filename number 2066 is not the conference entry ID. Quantization and deployment optimization are the central contributions, so this note belongs in Model Compression.

TL;DR

DiffPro uses calibration data to identify quantization-sensitive layers and denoising steps, then jointly selects layer precision, dynamic activation quantization, and retained timesteps, reducing DiT-XL/2 from 2575 MB to 327 MB and A30 latency from 6.88 to 2.55 seconds/image while FID increases from 2.27 to 5.89.

Background & Motivation

Diffusion inference has two sources of cost: a large model executes at every denoising step, and generating an image repeats that execution many times. Post-training quantization reduces the cost of each pass, while timestep pruning reduces the number of passes. Optimizing them independently misses an important interaction: a quantized model no longer has the original full-precision error landscape, so previously dispensable steps may become important.

This interaction is especially relevant to DiTs. Attention, MLP, and embedding layers tolerate low precision differently, while the activation range within one layer also changes throughout denoising. Fixed scales may clip outliers at particular timesteps; uniform bit widths may waste precision on robust layers while damaging a few fragile ones. BitOps alone also misses tensor-shape, kernel-dispatch, and memory effects that determine real GPU latency.

DiffPro therefore searches for a deployment plan for an existing model instead of introducing a new generative backbone. Core Idea: jointly decide which layers deserve more precision and which steps remain necessary after quantization, using teacher-student output differences to estimate quality risk and measured hardware costs to allocate the budget.

Method

Overall Architecture

The inputs are a pretrained denoiser, calibration examples, and latency and memory budgets. The output specifies fixed weight bit widths, group sizes, and a retained timestep set. Candidate weight precisions are 4, 8, and 16 bits; candidate group sizes are 32, 64, 128, 192, and 288. These are deployment search variables, not architectural parameters of a retrained generator.

The pipeline first profiles sensitivity against a full-precision teacher and constructs precision candidates, then enables dynamic activation quantization and evaluates which steps can be removed. Its final budget decision considers bit reduction and timestep removal together instead of simply chaining two independently optimized compressors. The diagram groups the paper's five stages into four designs, combining the first two stages because both construct the layer-wise precision plan.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Pretrained model, calibration data<br/>Latency and memory budgets"] --> Precision["Sensitivity-Guided<br/>Precision Allocation"]
    Precision --> DAQ["Dynamic Activation<br/>Quantization"]
    DAQ --> Steps["Drift-Guided<br/>Timestep Selection"]
    Steps --> Joint["Joint Search under<br/>Hardware Budgets"]
    Joint --> Output["Fixed deployment plan<br/>Quantized denoising and VAE decoding"]

Key Designs

1. Sensitivity-Guided Precision Allocation: identify layers that cannot be aggressively compressed

Calibration records full-precision teacher noise predictions and layer activation statistics. The PCA statistics include the number of components needed to explain 95% of the variance, the feature dimension, and residual variance, characterizing the complexity of activation structure. The paper combines this information with normalized squared activation energy to form a curvature-and-structure fragility signal. Curvature here is an approximation, not the exact Hessian of the complete generation-quality objective.

Structural information alone cannot indicate whether a layer remains stable throughout denoising. The score therefore also incorporates drift across timestep phases, a sensitivity knee obtained by sweeping bit widths and group sizes, and estimated rounding noise amplified by Jacobian energy. The four signals represent structural fragility, temporal drift, knee sensitivity, and quantization noise, respectively. Their fusion is original Eq. (3):

\[ S^*_\ell = 0.4 S^x_\ell + 0.2 S^d_\ell + 0.25 S^k_\ell + 0.15 S^n_\ell. \]

The weights are not simply guessed: the authors quantize one layer at a time to W4 while keeping the others at FP16, collecting quality-degradation measurements for 113 layers and fitting ordinary least squares. Five-fold cross-validation gives \(R^2=0.76\pm0.04\). However, the readable main text does not fully specify every computational detail of the drift and knee signals, so the fusion equation alone is not a complete reproduction recipe.

This ranking drives risk tiers and knapsack-style precision allocation: robust layers can use W4 and coarse groups, while sensitive layers receive W8 or remain FP16. GPTQ then quantizes weights group by group, and evolutionary search modifies bit widths and group sizes. GPTQ uses an activation-derived second-order approximation and Cholesky decomposition to handle weight errors. Candidates are screened on a few batches before stronger candidates receive fuller evaluation, avoiding complete FID computation for every proposal. The final configuration reported in Figure 4 contains 77 W4 layers, 14 W8 layers, and 22 FP16 layers, so Mixed must not be interpreted as an entirely W4 model.

2. Dynamic Activation Quantization: adapt scales to the current sample and denoising step

Choosing weight precision does not make activations safe. The same layer can have substantially different numerical ranges at different timesteps, causing static scales to alternate between inadequate resolution and excessive outlier clipping. DAQ updates scales for each sample, timestep, and channel group. It first clips symmetrically at the 99.9th-percentile threshold, then determines the quantization range from the maximum absolute clipped value within the group. The 99.9 threshold was selected by sweeping 99.0 through 99.99 on calibration data; it is not a theoretical constant for all models.

The paper also uses early, middle, and late timestep bins to handle temporal variation, with calibration informed by the quantized student rather than only the teacher's activation distribution. Forward pre-hooks update scales before grouped linear layers, covering Q, K, V, the attention output projection, and both MLP layers. LayerNorm and residual paths remain at higher precision. The described compute path multiplies INT8 by INT8 and accumulates into INT32. Consequently, W4 weight storage does not establish that native INT4 arithmetic kernels are used. Table 3 reports 0.24 ms of DAQ overhead per step: 0.12 ms for percentiles, 0.04 ms for scale calculation, and 0.08 ms for requantization, totaling 3.6% in the reported setting.

3. Drift-Guided Timestep Selection: remove low-risk steps while protecting final refinement

Drift here does not mean the image change between adjacent timesteps. It measures the difference between quantized-student and full-precision-teacher outputs on the same noisy input, timestep, and condition. The screening loss below follows the noise-prediction notation of original Eq. (4). Conditioning the expectation on a fixed timestep gives the per-step drift concept in original Eq. (7).

\[ \mathcal{L}_{\mathrm{drift}} = \mathbb{E}_{(x_t,t,y)}\left\|\epsilon_{\mathrm{student}}(x_t,t,y)-\epsilon_{\mathrm{teacher}}(x_t,t,y)\right\|_2^2. \]

The selector first protects the last 20% of steps in denoising execution order, then fills the remaining quota with the highest-drift steps outside that tail. High drift indicates that the student already differs from the teacher, making further removal of computation risky; low-drift steps are better pruning candidates. The tail remains protected even when its drift is low because errors during final refinement can translate directly into visible image defects.

For DiT, retaining 500 of 1000 steps means protecting the final 200 and selecting another 300 outside the tail. Protected steps therefore constitute 40% of the retained schedule. For SDXL, retaining 50 of 100 steps similarly protects 20 and adds another 30. This interpretation follows the prose description of the final execution phase. Cached Eq. (8) has missing symbols, so its damaged inequality should not be used to infer the direction of the original diffusion timestep indices.

4. Joint Search under Hardware Budgets: compare the next bit reduction with the next removed step

Algorithm 2 starts with all steps retained and maximum layer precision. It maintains a drift-ranked queue of removable steps and a heap of bit-reduction moves ranked by cost savings relative to estimated quality loss. At each iteration, it compares removing the current step with applying the best bit-reduction move, selects the more efficient action, and updates latency and memory estimates until the budgets are satisfied. This is a different search level from the evolutionary refinement of layer-wise plans; their initialization procedures should not be conflated.

Latency comes from measured kernel timings with synchronized CUDA events, not a direct conversion of bits or FLOPs into seconds. Candidates still use teacher-student MSE as a quality proxy, and the prose describes penalties for latency and BitOps violations. Hardware awareness therefore does not mean that every proxy metric disappears from the search. The cached multiplication and addition symbols in Eq. (10) are incomplete, so this note does not reconstruct the scoring function. The prose gives both penalty weights as 0.5, but the exact implementation still requires checking the original PDF or code. Finalization applies lightweight bias and LayerNorm correction plus adaptive rounding on fragile layers. During inference, weight allocation and the timestep schedule remain fixed, while activation scales are computed dynamically for the current input.

Loss & Training

This is post-training optimization without retraining the diffusion backbone, but it is not cost-free: teacher calibration, quantization-parameter processing, sensitivity measurements, and plan search are required. Calibration uses 512 examples. One-time planning takes approximately 53 minutes, or 108 minutes on FLUX; the paper estimates amortization after approximately 1200 generated images. Planning is excluded from deployed per-image latency, but it must be included in total cost for small workloads or frequently changing models.

Key Experimental Results

Main Results

DiT-XL/2 performs class-conditioned generation on ImageNet at 256×256, with FID measured on 50K images and CFG=1.5. Its DDIM schedule is reduced from 1000 to 500 steps. SDXL U-Net uses 30K COCO 2014 prompts and 30K generated images at 1024×1024 with CFG=3.0, reducing 100 steps to 50. The main implementation uses NVIDIA A30 24 GB GPUs, CUDA 12.1.1, and PyTorch 2.1.2; Table 4 additionally reports A100 latency. Latency uses batch size 1, synchronized CUDA events, and the mean of 100 runs. Energy uses NVML with idle power subtracted, also averaged over 100 runs.

The following selection is from original Table 4. Size is in MB, latency is A30 seconds/image, and lower FID is better. FID values across the two architectures are not directly comparable because the datasets differ.

Architecture	Method	W/A	FID	Size (MB)	A30 (seconds/image)
DiT-XL/2	Full Precision	16/16	2.27	2575	6.88
DiT-XL/2	Q-DiT	4/8	6.40	348	3.21
DiT-XL/2	TQ-DiT	8/8	4.91	392	3.60
DiT-XL/2	Q-DiT+Uni.500	4/8	13.82	348	1.85
DiT-XL/2	DiffPro W8A8	8/8	5.12	392	2.68
DiT-XL/2	DiffPro	Mixed	5.89	327	2.55
SDXL	Full Precision	16/16	24.50	12877	9.84
SDXL	GPTQ	4/8	29.47	3568	6.54
SDXL	TMPQ-DM	Mixed	27.91	2854	3.12
SDXL	DiffPro	Mixed	26.50	2317	2.35

DiT's approximately 7.9-fold compression describes model size, not peak runtime memory. The approximately 2.7-fold acceleration comes from 6.88/2.55. FID worsens by 3.62 relative to full precision, so this is not lossless compression. Q-DiT+Uni.500 is faster but substantially worse in quality, whereas TQ-DiT offers better quality at higher latency.

Ablation Study

The following selection is the DiT-XL/2 group of original Table 7, following the main experimental protocol. Uni.50% retains DiffPro's bits and DAQ but replaces drift-guided selection with uniform step thinning.

Config	FID	Latency (seconds/image)	Energy (J)	Size (MB)
Full Precision	2.27	6.88	660.94	2575.42
Uniform W8/G32	8.68	2.72	389.61	429.95
No DAQ	7.09	2.64	359.64	327.24
No Prune	6.22	2.65	361.51	327.24
Uni.50%	8.41	2.55	358.72	327.24
Sequential	6.74	2.55	361.28	327.24
DiffPro	5.89	2.55	360.05	327.24

Key Findings

• With matched bits, DAQ, and retained step count, Uni.50% gives an FID gap of 2.52. The 7.93 gap between 13.82 and 5.89 also includes different quantization plans and cannot be attributed entirely to timestep selection.
• Sequential and the full method both take 2.55 seconds/image, while FID improves from 6.74 to 5.89, a joint-search contribution of 0.85. No DAQ has FID 7.09 but slightly lower energy, 359.64 J, than the full method, so the complete plan does not win every metric.
• Original Table 2 compares only precision plans, without DAQ or pruning: the best individual signal gives correlation/FID of 0.61/10.41, versus 0.81/8.86 for fusion. The 8.86 value is not the full DiffPro result.
• SDXL energy in original Table 7 falls from 675.92 J to 369.24 J, approximately 45%, while FID rises from 24.50 to 26.50. Integer model sizes in Table 4 and decimal sizes in Table 7 have different reporting precision.

Highlights & Insights

• Quantization changes timestep value. Selecting steps using differences between the quantized student and teacher targets the model that will be deployed rather than only the original teacher.
• Tail protection addresses a blind spot of the error proxy. Low current drift does not guarantee that removing a step preserves final details, motivating a structural protection rule.
• Practical gains depend on interactions between layers, time, and hardware. Bit widths and step counts are both resource-allocation decisions and should be compared under one budget instead of optimized separately and then combined.

Limitations & Future Work

• Quality loss remains substantial: DiT FID increases from 2.27 to 5.89. The authors propose stronger block-wise quantization, video diffusion support, and lightweight fine-tuning; video-generation benefits are not demonstrated here.
• Calibration and search are nontrivial. The 53-minute planning cost and approximately 1200-image amortization estimate depend on the workload and are not guarantees across devices, batch sizes, or prompt distributions.
• A reproduction ambiguity observed during reading concerns initialization: Figure 4 describes an all-W4 seed, Stage 2 describes a tiered seed, and Algorithm 2 starts at maximum precision. The illustrated experiment and the two search levels should be distinguished and checked against implementation.
• Table 7 gives No Prune latency of 2.65 seconds/image versus 2.55 for the full method despite the stated removal of half the steps. This does not support a claim that halving steps necessarily halves latency; actual ablation schedules and timing boundaries need clarification.
• Cached Eqs. (6), (8), and (10) are incomplete. This note does not invent the exact quantization denominator, timestep-index inequality, or joint scoring function. Original Table 6 supplies FLUX/PixArt quality-transfer evidence, not evidence of identical speedup factors.

Related Work & Insights

• Versus GPTQ: DiffPro reuses layer-wise second-order quantization as a tool and additionally addresses diffusion-specific temporal variation, precision allocation, and scheduling. It is not a replacement invention of weight quantization.
• Versus Q-DiT / PTQ4DiT: the method adds timestep selection to quantization and uses student-aware DAQ to handle activation changes. Comparisons must account for step count, precision, and end-to-end latency together.
• Versus TMPQ-DM: both consider joint timestep and precision selection, while DiffPro emphasizes composite sensitivity and measured kernel costs. Its contribution should not be presented as the first proposal to optimize both dimensions jointly.
• Research implications: agreement between the search-quality proxy and final FID, benefits with short-schedule samplers, and cross-hardware replanning costs deserve separate evaluation rather than only pursuing lower nominal bit widths.

Rating

• Novelty: 4/5. Combines composite sensitivity, student-aware DAQ, and hardware-budgeted scheduling, while the foundational components have precedents.
• Experimental Thoroughness: 4/5. Includes two-backbone main results, matched-plan ablations, and newer-architecture extensions, with some timing and initialization details still needing verification.
• Writing Quality: 3/5. The overall pipeline is understandable, but initialization descriptions and incomplete cached equations reduce reproduction clarity.
• Value: 4/5. Relevant to high-volume compressed deployment; small workloads require careful accounting for calibration and search costs.