Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/csslc/Self-Transcendence
Area: Image Generation
Keywords: Diffusion Transformer, VAE Structure Guidance, Feature-Level CFG, Representation Alignment, Training Acceleration

TL;DR

Self-Transcendence first trains an internal teacher with clean VAE latents, strengthens its semantics using feature-level classifier-free guidance, and uses the frozen teacher to supervise a diffusion model trained from scratch; on ImageNet 256 without sampling CFG, SiT-XL/2 improves from FID 17.63 to 7.51 after 80 student epochs, with teacher warm-up charged separately.

Background & Motivation

The final denoising loss in a diffusion transformer must traverse a long gradient path before reaching shallow layers, which consequently learn slowly to separate useful structure from noisy inputs. REPA directly supervises intermediate representations with DINO features, providing earlier semantic and spatial organization, but introduces a dedicated external vision encoder and its pretraining dependencies.

SRA and LayerSync instead use the model's deeper representations to teach shallower ones. However, deep features are also immature early in training, and changing supervision targets provide an unreliable starting point. This paper separates two requirements for useful guidance: clean structure helps distinguish signal from noise, while semantic separability helps shallow layers identify condition-relevant content. VAE latents satisfy the former but cannot alone replace DINO's semantic capability.

The authors therefore train a structurally competent internal teacher before strengthening its conditional semantics, instead of having an immature model learn and teach simultaneously. Core Idea: warm up structural representations with the existing VAE's clean latents, then move CFG from prediction outputs to intermediate features so that a fixed internal teacher can replace external representation supervision.

Method

Overall Architecture

Inputs are the noisy latents and class or text conditions used in standard latent diffusion training; outputs remain the original backbone's diffusion predictions. The first stage in Figure 3 trains a guiding model with the same architecture as the target backbone, supervising an intermediate layer with clean VAE latents alongside the diffusion loss.

After warm-up, the guiding model is frozen. Conditional and unconditional features from the same input and layer are combined through feature-level CFG to form supervision targets. The second stage initializes a fresh student rather than continuing to fine-tune the teacher; a student intermediate layer is aligned with the target through an MLP while optimizing the diffusion loss. Alignment is active only early in training, after which the teacher branch is removed and standard diffusion training continues.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Noisy latents and conditions"] --> Structure["VAE Structure Guidance"]
    Clean["Clean VAE latents"] --> Structure
    Structure --> Frozen["Freeze teacher after warm-up"]
    Frozen --> Feature["Feature-Level CFG"]
    Feature --> Align["Early-Stopped Alignment"]
    Student["Freshly initialized student"] --> Align
    Align --> Output["Continue with diffusion loss only"]

Key Designs

1. VAE Structure Guidance: teach the teacher to recognize clean structure first

The first stage extracts an intermediate representation, projects it into the VAE latent space through a lightweight MLP, and aligns it with the current training image's clean latent using an L2 loss. Supervision comes from the VAE already required by latent diffusion, rather than an additional DINO encoder. This provides a shorter training signal path to shallow layers and helps recover spatial structure from noisy inputs.

The cached mathematical rendering of Equation (1) is incomplete. Based on the surrounding prose, the following restates the objective using conventional squared-L2 notation; the readable text does not establish the reduction convention, which requires checking the original PDF or implementation:

\[ \mathcal{L}_{\mathrm{VAE-guide}}=\|\mathrm{MLP}(\mathbf{f}_n)-\mathbf{z}\|_2^2. \]

Here, \(\mathbf{z}\) denotes the clean VAE latent, not the noisy training input, and \(\mathbf{f}_n\) is the representation at layer \(n\). The original diffusion loss remains active, so the auxiliary objective does not replace diffusion training with plain latent reconstruction.

Why not use VAE features to supervise the entire final model directly? They provide clear structure but relatively weak class semantics. Variant V1 in Table 4 confirms that this mechanism alone helps, yet falls substantially short of the full method. The first stage primarily prepares a suitable teacher rather than supplying the final semantic target.

The teacher is warmed up for 40 epochs by default and then held fixed. Its features still change with inputs and conditions, but no longer drift because of parameter updates, unlike an online EMA teacher or synchronized layer learning.

2. Feature-Level CFG: amplify conditional semantics inside the teacher

For the same noisy input, the teacher extracts conditional features \(\mathbf{f}_c\) and unconditional features \(\mathbf{f}_u\). The method moves the CFG combination normally applied to output predictions into an intermediate layer, amplifying their difference so that target features emphasize information associated with the class or text condition.

Following Equation (3) and its accompanying explanation, the combination is:

\[ \mathbf{f}_g=\mathbf{f}_u+\omega(\mathbf{f}_c-\mathbf{f}_u). \]

Here, \(\omega=1.0\) gives ordinary conditional features, rather than removing the teacher entirely. Larger weights strengthen the conditional direction but are not universally better: Table 5 reports that increasing the scale from 30.0 to 60.0 worsens FID from 22.91 to 23.57.

The default is \(\omega=30.0\) for SiT and \(\omega=10.0\) for LightningDiT. These are feature weights for training targets, not CFG scales for image sampling; the sampling visualizations in Figures 4 and 6, for example, use 4.0.

Targets come from a deeper teacher layer, but selecting the layer nearest the output is not optimal. For a backbone with \(N\) Transformer blocks, the default teacher layer is \(2N/3\) and the student layer is \(N/2\). This preserves a moderate abstraction gap without making the target too different from what shallow student layers need.

3. Early-Stopped Alignment: constrain the student only while guidance is useful

The selected layer of the fresh student passes through an MLP and is matched to the frozen teacher's \(\mathbf{f}_g\) using an L2 loss. The target combines structural priors learned in the first stage with conditional semantics strengthened by the second design, rather than simply copying unmodified teacher features.

The cached rendering of Equation (4) is also damaged. The following again restates the textual description using squared-L2 notation; it describes the mechanism without claiming to recover every implementation-level reduction detail:

\[ \mathcal{L}_{\mathrm{guide}}=\|\mathrm{MLP}(\mathbf{f}_m)-\mathbf{f}_g\|_2^2, \qquad \mathcal{L}=\mathcal{L}_{\mathrm{diff}}+\lambda_{\mathrm{guide}}\mathcal{L}_{\mathrm{guide}}. \]

Freezing the teacher prevents the target from being jointly optimized with the student. The MLP maps student intermediate representations into the teacher feature space; the main paper defers its architecture to the supplement, so layer counts and hidden dimensions cannot be specified from this cache.

The authors observe that enforcing shallow semantic alignment for too long destabilizes subsequent deep-layer training. Auxiliary supervision is therefore used only early on. Stopping alignment does not stop student training; training continues, and inference does not require the teacher or auxiliary supervision branch.

Loss & Training

The default is \(\lambda_{\mathrm{guide}}=0.5\), with diffusion-loss weight 1.0. Section 4.1 specifies alignment during the first 20 epochs for base models and the first 10 epochs for larger models, followed by diffusion-only optimization.

Experiments use SiT with patch size 2 and LightningDiT with patch size 1, following each backbone's original training and inference settings. Teacher and student share an architecture. The latent spaces include SD-VAE and VAVAE; avoiding an additional external representation teacher does not mean avoiding a pretrained VAE.

The default SiT-B/2 pairing in Table 5 guides student layer 6 from teacher layer 8, using a teacher trained for 200K iterations, corresponding to the stated 40-epoch warm-up. Longer warm-up is not consistently better: a 300K-iteration teacher gives FID 23.05, slightly worse than 22.91 at 200K.

The available cache contains the main paper and references only. The one-stage comparison, MLP details, 512-resolution settings, and fuller text-to-image protocol referenced by the paper require supplementary material; these parameters are not invented here.

Key Experimental Results

Main Results

The table selects key results from Tables 1, 2, and 3. Lower FID is better, and tasks and sampling conditions are stated separately. Tables 1 and 3 evaluate 50,000 generated samples; listed training lengths refer to student budgets and exclude the 40-epoch teacher warm-up.

Source / Task	Backbone and Setting	Baseline or REPA	Self-Transcendence	Comparison Boundary
Table 1 / ImageNet 256	SiT-B/2, no sampling CFG, 80 epochs	Baseline 36.14; REPA 24.40	FID 20.49	130M parameters
Table 1 / ImageNet 256	SiT-L/2, no sampling CFG, 80 epochs	Baseline 21.41; REPA 9.70	FID 8.74	458M parameters
Table 1 / ImageNet 256	SiT-XL/2, no sampling CFG, 80 epochs	Baseline 17.63; REPA 7.90	FID 7.51	675M parameters
Table 1 / ImageNet 256	LightningDiT-XL/1, no sampling CFG, 64 epochs	Baseline 5.30; REPA 4.09	FID 3.55	Uses VAVAE
Table 2 / Text-to-image	MMDiT, 150K iterations	REPA: FID 4.90, IS 32.55	FID 4.56, IS 33.08	Main text follows the REPA protocol
Table 3 / ImageNet 256	SiT-XL/2, with sampling CFG	REPA: 800 epochs, FID 1.42	400 epochs, FID 1.44	Similar, not lower, FID
Table 3 / ImageNet 256	LightningDiT-XL/1, with sampling CFG	Baseline: 800 epochs, FID 1.35	400 epochs, FID 1.25	Different student budgets

Ablation Study

Table 4 uses SiT-B/2 trained for 80 epochs and evaluates only 10,000 samples. Its FID 22.91 and the main table's 20.49 use different evaluation sample counts and should not be treated as contradictory measurements.

Config	VAE Structure Guidance	Self-Guided Representation	FID	IS
Baseline	No	No	38.60	41.95
V1	Yes	No	32.20	52.38
V2	No	Yes, teacher trained with diffusion loss only	25.21	63.83
Full method	Yes	Yes	22.91	70.37

Key Findings

• In Table 4, structural supervision alone reduces FID from 38.60 to 32.20; the full method improves over V2 by another 2.30, supporting complementarity between teacher warm-up quality and subsequent self-guidance.
• In Table 5, feature CFG scale 1.0 gives FID 29.30, versus 22.91 at the default 30.0. Ordinary conditional features do not reproduce the full gain.
• In the same table, teacher layer 8 guiding student layer 8 yields FID 24.29, worse than 22.91 when guiding layer 6. The supervision location is itself an important hyperparameter.
• Table 6 reports 58.87 hours for the baseline, 65.44 for REPA, and 60.45 for this method with 8 A800 GPUs, batch size 256, and 400 student epochs. Teacher training takes another 6.39 hours, giving 66.84 hours when charged from scratch; student-only timing does not establish lower total cost than REPA.
• Peak memory during the first 50K iterations in Table 6 is 12.34 GB per GPU, above REPA's 10.39; it returns to 9.03 after early stopping. This cost experiment reports the first 50K and remaining 1950K iterations separately and should not be rewritten as a universal stopping schedule for all settings.

Highlights & Insights

• Structure and semantics provide different forms of supervision. Clean VAE latents first establish where useful signals are, while feature CFG emphasizes their connection to conditions; V1 and V2 allow these roles to be tested separately.
• CFG is not restricted to final sampling. Amplifying the conditional-unconditional feature difference provides a testable way to construct internal teachers for other conditional generative models.
• Teacher classification performance is not the sole proxy for generation quality. Table 7 probes features at noise time 0.5: layer 8 reaches 41.18% for this method versus 51.87% for REPA, yet the former achieves lower FID in some generation settings.

Limitations & Future Work

• The authors acknowledge that internal supervision quality remains bounded by backbone capacity and introduces guidance weights, layer pairings, and warm-up lengths as additional hyperparameters. Defaults are not guaranteed to transfer across tasks.
• Evaluation covers image generation, including class-conditional and text-conditional tasks. Text-to-video and text-to-3D remain untested and cannot be inferred from image results.
• The two-stage procedure requires teacher warm-up, a more significant fixed cost under short training budgets or one-off teacher use. Complete cost comparisons should report both teacher and student training.
• The LightningDiT experiments use VAVAE, and the related-work discussion describes VA-VAE alignment with vision foundation models. Thus, self-contained is best understood as introducing no additional external feature teacher, not as removing all external representation dependencies from the existing backbone and tokenizer pretraining.
• The cache does not provide supplementary material or multi-seed error bars. Reproduction details and the statistical stability of small differences such as 7.90 versus 7.51 require further verification.

Related Work & Insights

• vs REPA: REPA aligns with external DINO representations; this method aligns with an internal teacher warmed up with VAE structure guidance and enhanced with feature CFG. Table 1 supports advantages in some matched-budget settings, but the SiT FID 1.44 in Table 3 is not better than REPA's 1.42.
• vs SRA / LayerSync: These methods exploit internal deep features during training. This paper first warms up and freezes a teacher, then trains a new student, exchanging an additional stage for stable, more condition-discriminative supervision.
• vs Dispersive Loss: That regularizer promotes dispersed internal representations without a pretrained teacher. This method introduces explicit target features and an alignment branch, improving generation results at the expense of a more involved training and tuning procedure.

Rating

• Novelty: 4/5, combines VAE structural warm-up and feature-level CFG into a concrete internal-teacher approach.
• Experimental Thoroughness: 4/5, covers multiple backbone scales, two generation tasks, and component ablations, while full costs and supplementary settings need careful checking.
• Writing Quality: 4/5, clearly motivates structure and semantics, but cost accounting and some implementation details need additional clarification.
• Value: 4/5, offers an empirically supported training approach that reduces additional external representation dependencies.