Vitality-Aware Compression for Efficient Image-to-Shape Diffusion Transformers¶

Conference: ECCV 2026
arXiv: 2607.00382
Code: None
Area: 3D Vision
Keywords: Image-to-3D, Diffusion Transformer, Structured Pruning, Adaptive Quantization, Layer Importance

TL;DR¶

For image-to-3D Diffusion Transformers (DiTs), this paper introduces a "layer vitality" metric based on Earth Mover's Distance to measure each layer's contribution to geometric synthesis. Based on this, a pipeline of structured pruning, layered mixed-precision quantization, and targeted distillation that fine-tunes only the lowest-vitality layers is developed. This compresses backbones by up to 66% on state-of-the-art models like Step1X-3D and Hunyuan3D while maintaining geometric fidelity.

Background & Motivation¶

Generating geometrically consistent 3D meshes from a single image has made rapid progress recently, with backbones evolving from early GAN priors and Large Reconstruction Models (LRMs) to 3D-native Diffusion Transformers and flow-matching frameworks, enabling the recovery of view-consistent meshes from a single image. However, the cost is increasingly massive model sizes—the DiT backbones in these pipelines often exceed 2.5 GB in parameters alone, making them almost impractical for real-time, edge-device, and memory-constrained scenarios. While image and video diffusion model compression is well-researched, those approaches exploit spatial/temporal redundancy. Directly applying them to 3D fails: the paper demonstrates that applying image-generation methods like TinyFusion and Diff-Pruning to shape-generating DiTs leads to severe geometric degradation, such as structural collapse, topological distortion, and loss of detail.

The root cause lies in the fundamental difference between 2D and 3D generation. Image compression only needs to preserve visual perception, whereas 3D models must maintain globally consistent geometry across all viewpoints, where minor perturbations during the denoising process can propagate along the sampling path and amplify into structural artifacts. Meanwhile, existing 3D efficiency efforts (e.g., Turbo3D, FlashVDM) focus almost entirely on inference acceleration rather than fundamentally compressing backbone parameters and bit-widths, offering limited relief for GPU memory bottlenecks. In other words, physical compression of 3D DiTs under the constraint of "preserving geometric fidelity" remains an unchartered territory.

The authors' entry point stems from an observation: studies in text-to-image and text-to-video domains have shown that only a small subset of layers in DiTs truly determine output quality. However, those works measure perceptual editing or image-domain quality rather than "geometric fidelity under permanent structural compression." Consequently, this work directly employs layer-wise ablation to measure each layer's contribution to 3D synthesis quality—calculating the EMD distance between the point cloud generated after removing a specific layer and the point cloud generated by the full model, where a larger distance indicates a more critical layer. The core idea of this paper is: using an EMD-based layer vitality metric to classify 3D DiT layers into 'critical' or 'redundant' based on their geometric contribution, performing structured pruning with separate thresholds for double-blocks and single-blocks, allocating 8/4-bit mixed-precision quantization according to vitality, and finally conducting targeted distillation fine-tuning on only the lowest-vitality retained layers within each block, thereby physically compressing the backbone by nearly half while preserving geometric quality.

Method¶

Overall Architecture¶

The input is a pre-trained image-to-3D DiT (e.g., Step1X-3D, Hunyuan3D 2.0/2mini), and the output is a lightweight model with significantly reduced parameters and bit-widths whose geometric synthesis quality remains on par with the original model. This type of DiT backbone categorizes layers into two kinds of blocks: double blocks, where the noise stream and conditioning tokens remain separate and interact only via shared attention; and single blocks, which process a unified latent representation after modal fusion. The two exhibit different sensitivity patterns to geometry and must be treated separately during compression.

The entire workflow is a three-stage sequential pipeline: first, layer-wise vitality analysis is conducted (Sec. 3.1), running an "ablation-comparison" process on each layer to obtain vitality scores and identify safe-to-prune redundant layers; next, these vitality scores simultaneously guide structured pruning and layered adaptive quantization (Sec. 3.2), removing low-vitality layers while preserving 8-bit precision for critical layers and compressing secondary layers to 4-bit; finally, targeted distillation fine-tuning is applied to the compressed model (Sec. 3.3), updating only the lowest-vitality retained layer in each module to encourage the student to replicate the teacher's (original model) behavior, bridging the performance gap caused by compression.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Pre-trained<br/>Image-to-3D DiT"] --> B["EMD Layer Vitality Analysis<br/>Layer-wise ablation and point cloud comparison"]
    B --> C["Vitality-Guided Structured Pruning<br/>Separate thresholds for double/single blocks"]
    C --> D["Adaptive Mixed-Precision Quantization<br/>Critical layers: 8-bit / Secondary layers: 4-bit"]
    D --> E["Min-vital Targeted Distillation Fine-tuning<br/>Tune only the lowest-vitality retained layer per block"]
    E --> F["Output: Compressed ~50-66%<br/>Lightweight DiT"]

Key Designs¶

1. EMD Layer Vitality Metric: Using optimal transport instead of nearest neighbors to determine critical geometric layers

Existing T2I methods measure layer importance using DINO perceptual distance, which is designed for the image domain and cannot directly indicate whether the 3D geometry collapses when a layer is removed. This work addresses this by using a conditional image \(y\) and generating point clouds using both the full model \(\theta_{\text{full}}\) and the model lacking the \(l\)-th layer \(\theta_{-l}\). The distance between these two point clouds serves as the vitality score for that layer. The crucial decision lies in selecting the distance metric—the authors opt for the Earth Mover's Distance rather than the common Chamfer Distance:

\[\text{vitality}(l)=\mathbb{E}_{y\sim\mathcal{D}}\left[\min_{\Gamma\in\mathcal{P}_{n}}\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{n}\Gamma_{ij}\bigl\|q_{\theta_{\text{full}}}^{(i)}(y)-q_{\theta_{-l}}^{(j)}(y)\bigr\|_{2}\right]\]

where \(\mathcal{P}_n\) is the set of all \(n{\times}n\) permutation matrices, requiring a one-to-one correspondence between the two point sets. The choice of EMD is highly practical: Chamfer Distance relies on nearest-neighbor association, highlighting only local surface accuracy while being extremely sensitive to sampling density. In contrast, EMD computes the optimal transport cost between two point sets to establish one-to-one correspondences, capturing the overall shape distribution. This enables it to detect global structural distortions such as "a layer responsible for geometric consistency is removed, leading to translation, asymmetry, or large-scale misalignment of the entire shape," while being less biased by sparse or unevenly sampled regions. Robustness experiments in the appendix corroborate this: when reducing sample points from 10k to 5k, the deviation of EMD remains under 5%, whereas Chamfer's deviation fluctuates as high as 40%–50% in deeper layers.

2. Block-Type Separated Thresholding for Structured Pruning: Double and single blocks cannot be measured with the same yardstick

Once vitality scores are obtained, pruning can be performed: layers with scores exceeding a threshold \(\tau\) are retained as critical, while the rest are pruned. However, the authors find that using a single threshold for both double-blocks and single-blocks degrades performance—the appendix shows that strictly enforcing \(\tau_d=\tau_s=0.18\) on Hunyuan3D 2.0 severely distorts geometry. This happens because the two types of blocks play distinct roles: double-blocks govern global coherence, whereas single-blocks manage local details, leading to inherently different sensitivity profiles. Thus, distinct thresholds \(\tau_d\) and \(\tau_s\) are specified. How are they determined? Starting from the lowest-vitality layers and pruning upwards, while monitoring the distance to the original model, the threshold is anchored right before the elbow point where quality drops sharply. Fig. A in the appendix demonstrates that this EMD criterion indeed defines a smooth "safe compression interval"—increasing the number of pruned layers from 2 to 6 reduces model size near-linearly with negligible quality loss, whereas pruning beyond 6 layers triggers a performance cliff, showing that this metric cleanly demarcates "safe compression" from "over-pruning."

3. Vitality-Driven Adaptive Mixed-Precision Quantization: Budgeting bits on critical geometric layers

Following pruning, quantization is employed to compress size further. Here, vitality scores are reused: layers are split into two groups, high-vitality layers quantized to 8-bit and low-vitality layers pushed to a more aggressive 4-bit, applying separate thresholds for double/single blocks to avoid degradation. The necessity of an "adaptive" scheme over a one-size-fits-all approach is evident in the ablation studies—while full 8-bit quantization maintains original model quality, full 4-bit quantization causes a clear performance drop (and severe collapse in the Hunyuan3D series); adaptive mixed-precision minimizes this gap while reducing the footprint beyond full 8-bit. Since this work focuses on layer-wise analysis, quantization is weight-only, leaving activations untouched. An interesting by-product: calculating the parameter breakdown reveals that in these FLUX-style DiTs, FFN blocks occupy only about 40%–45% of total parameters, rather than the ~66% typical of standard Transformers, due to their heavier attention components—which explains why FFN-only pruning methods (e.g., Diff-Pruning) struggle here.

4. Min-Vital Targeted Distillation Fine-Tuning: Optimization is more stable when tuning only the least vital retained layers

Despite pruning and quantization, a behavioral gap remains between the compressed model and the original model, which requires fine-tuning to bridge. The authors design a distillation loss that encourages the student to mimic the teacher's ODE trajectory: along the sampling path of the full model, at each timestep \(t\), the compressed student \(v^c\) is trained to match the teacher's prediction \(v^f\) under the identical latent and conditioning, accounting for both conditional and unconditional paths:

\[\mathcal{L}_{\text{Distill}}(\theta_c)=\tfrac{1}{2}\bigl\|v^c(z^f_t,t,y)-v^f(z^f_t,t,y)\bigr\|_2^2+\tfrac{1}{2}\bigl\|v^c(z^f_t,t,\varnothing)-v^f(z^f_t,t,\varnothing)\bigr\|_2^2\]

where \(z^f_t\) represents the latent at timestep \(t\) sampled from the full model, and \(\varnothing\) denotes the null condition. After a single optimization step at each timestep, flow sampling continues to \(t-1\) using the teacher's prediction. The truly counterintuitive aspect lies in "which layers are tuned": fine-tuning all retained critical layers is not only computationally wasteful but occasionally drifts the student further from the teacher, dropping performance. Hence, the authors propose selective fine-tuning—updating only the retained layer with the lowest vitality score in each DiT module (denoted as Min-vital), avoiding excessive perturbation of the critical layers that actually sustain layout and geometry. Appendix ablations validate this choice: full fine-tuning destabilizes training, tuning only the Max-vital layers fails to recover details, while tuning solely the Min-vital layers maintains stability and restores quality close to the original model.

Loss & Training¶

The training objective is the distillation loss \(\mathcal{L}_{\text{Distill}}\), which trains the student to reproduce the teacher's ODE trajectory rather than optimizing standard flow matching. The training dataset consists of a 10K subset of Objaverse rendered images. Step1X-3D is trained for 30K steps using a learning rate of \(10^{-8}\) (on 2×A100, taking ~22 hours), while Hunyuan3D 2.0/2mini is trained for 20K steps with a learning rate of \(10^{-4}\). Thresholds are manually tuned per architecture; for example, Step1X-3D uses \(\tau_d=0.17\) and \(\tau_s=0.165\) for pruning, with quantization split boundaries at double 0.25 / single 0.185, while Hunyuan3D 2mini skips double-block pruning entirely as all its double-block layers are deemed critical, keeping them almost entirely at 8-bit.

Key Experimental Results¶

Main Results¶

Compression is evaluated on three SOTA image-to-3D models. Performance is measured via image-to-3D joint embedding similarities (Uni3D-I and OpenShape-I) along with model size, TFLOPs, latency, and peak GPU memory (evaluated on 200 image-shape pairs sampled from Objaverse; metrics cover only the backbone parameters).

Model	Uni3D-I ↑	OpenShape-I ↑	Size (GB) ↓	Latency (s) ↓	VRAM (GB) ↓
Step1X-3D	0.3586	0.1480	2.452	6.23	2.718
Step1X-3D + Ours	0.3580	0.1489	0.843 (−65.63%)	2.78	1.206
Hunyuan3D 2.0	0.3582	0.1487	2.704	5.85	2.463
Hy3D 2.0 + Ours	0.3601	0.1491	0.909 (−66.37%)	3.90	1.761
Hy3D 2mini	0.3614	0.1490	1.042	1.28	1.224
Hy3D 2mini + Ours	0.3608	0.1484	0.578 (−44.50%)	1.14	1.135

The parameter sizes of the three models are reduced by 65.63%, 66.37%, and 44.50% respectively, with Uni3D-I / OpenShape-I remaining virtually unchanged (and occasionally slightly improved), while TFLOPs, latency, and VRAM drop across the board. The compressed models also outperform recent DiT baselines like Craftsman3D and TRELLIS, as well as feed-forward/diffusion baselines like Splatter Image, TripoSR, and LGM. In user studies, the compressed versions are practically indistinguishable from the original models in terms of geometric fidelity and overall quality.

Comparison against existing diffusion compression methods (on the same 3D backbone for a fair comparison):

Method	Step1X-3D Rate	Hy3D 2.0 Rate	Hy3D 2mini Rate
TinyFusion	49.31%	49.82%	49.52%
Diff-Pruning (60%)	28.71%	28.92%	28.89%
Ours	65.63%	66.37%	44.50%

TinyFusion achieves around 50% compression but fails to recover quality; Diff-Pruning is restricted to pruning FFN layers, limiting its maximum compression rate (as FFNs constitute only 40%–45% of parameters). Ours secures a higher compression rate while fully retaining quality.

Ablation Study¶

Adding components step-by-step (taking Step1X-3D as an example):

Configuration	Uni3D-I	OpenShape-I	Size (GB)	Description
Original	0.3586	0.1480	2.452	Full-parameter original model
+ Pruning (random)	0.0829	0.0375	1.123	Random pruning; quality collapses
+ Vitality-Aware	0.3584	0.1472	1.123	Pruning only non-critical layers; negligible drop
+ Quantization (4b)	0.3489	0.1466	0.803	Full 4-bit; significant drop on Hunyuan
+ Adaptive Quant.	0.3579	0.1478	0.843	Mixed-precision; smaller size and lower quality drop
+ Fine-tuning (Ours)	0.3580	0.1489	0.843	Targeted fine-tuning recovery; close to original model

Ablation of fine-tuning strategies (Hunyuan3D 2.0): Full fine-tuning collapses the Uni3D-I score to 0.1766 and V-IoU to 28.69; tuning only the Max-vital layers yields 0.3541 / V-IoU 61.50; Ours (Min-vital) achieves 0.3601 / V-IoU 72.04, proving to be the most stable and effective strategy.

Key Findings¶

The most significant contribution comes from vitality-aware pruning: random pruning collapses the Uni3D-I score from 0.3586 to 0.0829, whereas vitality-based pruning maintains quality—demonstrating that a substantial portion of layers are indeed redundant, with only a select few being critical.
Separate thresholds are imperative for double and single blocks; utilizing a shared threshold leads to severe geometric distortion, highlighting a key difference between 3D and 2D compression.
"Fine-tuning only the lowest-vitality layers" is more stable and effective than tuning all critical layers: altering critical layers often causes the student to diverge from the teacher. For Step1X-3D, because the partition between critical and non-critical layers is exceptionally distinct, fine-tuning post-pruning has negligible effect (suggesting that the pruning itself is clean enough).
Quantization mainly alleviates VRAM overhead, yielding limited latency benefits (as low-bit inference speed heavily relies on hardware/kernel optimization).
The proposed method is orthogonal to inference acceleration: augmenting this with guidance/step distillation achieves up to 13.5× inference speedup on top of 2.5× memory savings, without any extra VRAM overhead.

Highlights & Insights¶

Transition from Image-Domain to 3D-Domain Layer Importance via Metric Shift (DINO → EMD): EMD's one-to-one mapping nature inherently adapts to the 3D-specific constraint of "global geometric consistency." It is far superior to Chamfer Distance at capturing large-scale structural misalignments. This concept of "aligning metrics with target failure modes" is highly transferable to generalized 3D quality assessments.
Counter-Intuitive "Min-Vital" Fine-Tuning: While basic intuition suggests tuning the most crucial layers, this work demonstrates that optimizing the least important retained layers maintains robustness. Tuning critical layers distorts the student's behavior, whereas the goal of distillation is exactly to preserve those core functions.
Plug-and-play post-processing pipeline: The entire method requires no architectural modifications or retraining from scratch. It is universally applicable across three structurally different pipelines (including Step1X/Hunyuan with separated structure/texture and TRELLIS with joint modeling), presenting high engineering deployment value.
Block-type separated thresholding insights: It illuminates a frequently neglected point: distinct blocks within DiT models play different semantic roles (global coherence vs. local details); thus, compression strategies must be customized by block type rather than applied uniformly.

Limitations & Future Work¶

Quantization is capped at 4-bit, leaving more extreme 1-2 bit configurations unexplored (as they require dedicated hardware-level implementation support).
The compressed model inherits the geometric/topological limitations of the base architectures. Since the distillation objective is to "preserve" the original model's behavior, inaccuracies present in the baseline (such as reconstruction failures on flat/stylized illustrations) persist in the compressed version.
The vitality thresholds must currently be manually tuned for each architecture, limiting generalization. The authors plan to automate this using cross-architecture relative vitality statistics to preserve the plug-and-play profile.
For TRELLIS, only the geometry-governing Sparse Structure Flow is compressed. The texture-governing SLAT Flow remains untouched because geometric metrics (EMD/CD) are relatively flat while it is highly sensitive to LPIPS, leaving this to future work. This suggests that pure geometric metrics are insufficient to guide the compression of texture-related layers and require joint geometry-appearance criteria.
The latency gains from quantization are currently weak, requiring highly optimized GPU-aware kernels to translate size advantages into practical speedups.

vs. Image Diffusion Compression (TinyFusion / Diff-Pruning): These methods exploit spatial redundancy with the goal of preserving visual appeal. This paper notes that "preserving visual appeal \(\neq\) preserving geometry," showing that direct transfer to 3D triggers structural collapse. By relying on EMD geometric metrics instead, this work achieves higher compression rates while strictly preserving geometry.
vs. 3D Efficiency Methods (Turbo3D / FlashVDM): These models focus on inference acceleration rather than compressing the backbone, offering limited relief for GPU RAM limits. This paper is the first to systematically compress both parameter scale and bit-widths of 3D DiTs, and its method is orthogonal and stackable with inference-only speedup methods.
vs. T2I/T2V Layer Vitality Analysis (Stable Flow / TV-LiVE): These utilize perceptual distances to evaluate layer contributions to image-domain quality for training-free editing. This work shifts to EMD to measure contributions to 3D geometry and deploys it in a novel pipeline of permanent structural compression (pruning, quantization, and distillation).

Rating¶

Novelty: ⭐⭐⭐⭐ First systematic attempt to physically compress image-to-3D DiTs. Both the EMD vitality metric and the Min-vital fine-tuning are clever additions, though the individual components (pruning/quantization/distillation) are compositions of established technologies.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Highly solid, featuring three SOTA models, user studies, comparisons with image compression baselines, and numerous appendix ablations (metric robustness, threshold analyses, fine-tuning policies, and TRELLIS generalization).
Writing Quality: ⭐⭐⭐⭐ Clear motivation, rich diagrams/tables, and a comprehensive ablation logic; minor typos in individual formulas and formulations do not hinder understanding.
Value: ⭐⭐⭐⭐ Plug-and-play, training-free, and universally applicable across models. It directly alleviates GPU memory bottlenecks for 3D generation, carrying high industrial value.