TiltDiff: Tilted Weight-Space Diffusion for Neural Network Generation

Conference: ECCV 2026
Paper: ECCV 2026
Area: Image Generation
Keywords: weight-space learning, neural network generation, latent diffusion model, performance-tilted diffusion, model interpretability

TL;DR

TiltDiff introduces a performance-tilted latent diffusion framework for neural network weight generation that aligns network parameters into a compact Transformer latent space and biases the diffusion denoising objective with validation accuracy, synthesizing high-performing, robust, diverse, and functionally interpretable model weights.

Background & Motivation

In modern deep learning, neural network parameters are increasingly treated as structured high-dimensional data rather than merely the static endpoints of numerical optimization. Under this "model weights as data" perspective, collections of models trained across different random initializations, hyperparameters, and optimization checkpoints form an empirical distribution spanning a specific architecture family. Generative modeling of these weight spaces offers exciting opportunities beyond conventional training, enabling model property prediction, smooth weight interpolation, and direct synthesis of functional neural networks without expensive retraining from scratch.

However, generating functional neural networks in weight space faces formidable technical barriers. Weight tensors are extremely high-dimensional, hierarchically organized by layers, and highly vulnerable to parameter perturbations. Furthermore, internal channel-permutation symmetries imply that functionally identical networks can be separated by large Euclidean distances in raw parameter space. The core tension is that realistic model zoos exhibit substantial quality heterogeneity: existing hyper-representation and autoencoding approaches (such as SANE) model the empirical distribution uniformly, dedicating equal generative capacity to all checkpoints. Consequently, generators waste modeling capacity on weak, degenerate, or partially converged weight regions rather than sampling from functional, high-performing regions.

To resolve this limitation, the key angle of attack is to retain the rich diversity of the model zoo while explicitly biasing the generative dynamics toward higher-performing regions. Core idea: map permutation-aligned network parameters into a compact Transformer latent space, tilt the diffusion denoising objective by validation accuracy, and extract causal decision pathways from diffusion cross-layer activations to verify internal functional alignment.

Method

Overall Architecture

TiltDiff operates via a cohesive three-stage pipeline comprising weight preprocessing and latent encoding, performance-tilted latent diffusion training, and reverse diffusion sampling with parameter reconstruction. Given a model zoo of trained checkpoints, the system first standardizes parameter layers and aligns their channel permutations against a canonical reference model. A Transformer autoencoder tokenizes and encodes these parameters into low-dimensional latent vectors. A latent U-Net diffusion model then learns to synthesize these latent codes, where the denoising loss is scaled by each checkpoint's normalized validation accuracy. Finally, during inference, reverse diffusion synthesizes clean latent codes from Gaussian noise, which are decoded and inverse-standardized into deployable neural network parameters.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Model Zoo Checkpoints<br/>(Parameters & Validation Acc)"] --> B["Weight Standardization & Alignment"]
    B --> C["Weight Tokenization & Latent Autoencoding"]
    C --> D["Accuracy-Weighted Latent Diffusion"]
    D --> E["Reverse Diffusion Sampling & Parameter Recovery"]
    E --> F["Decision Pathway Attribution & Causal Validation"]
    F --> G["High-Performing Functional Networks"]

Key Designs

1. Weight Standardization and Alignment: Eliminating Permutation Symmetries

Network weights vary considerably in scale across layers, and channel permutation symmetries mean that functionally equivalent networks often occupy distant coordinates in raw parameter space. To prevent these artificial geometric discrepancies from disrupting generative modeling, TiltDiff standardizes each checkpoint and aligns it to a fixed canonical reference checkpoint.

For standardized parameters, the optimal admissible channel permutation operator is determined by minimizing the squared Euclidean distance to the reference: $$ \pi_i^* = \arg\min_{\pi \in \Pi} |\operatorname{vec}(\check{\Theta}_{\mathrm{ref}}) - \operatorname{vec}(\pi \cdot \check{\Theta}_i)|_2^2 $$ Following Git Re-Basin principles, this permutation is propagated across adjacent layers to ensure that functional input-output mappings remain strictly invariant. For checkpoints belonging to the same optimization run, the permutation derived from the final checkpoint is shared across all earlier checkpoints, successfully anchoring the weight population to a unified coordinate system.

2. Weight Tokenization and Latent Autoencoding: Compressing Parameter Topology

Directly applying diffusion models to raw, high-dimensional parameter tensors is computationally prohibitive and prone to mode collapse. TiltDiff adopts the SANE tokenization protocol to slice aligned parameter tensors into uniform weight tokens, paired with layer-specific positional encodings.

Before autoencoding, severely undertrained or degenerate checkpoints are pruned to retain a higher-performing subset. A Transformer encoder then compresses the weight tokens into a compact latent code \(z_0^{(i)}\), while the decoder reconstructs the original token sequence. After training with a reconstruction objective, the autoencoder weights are frozen, establishing a smooth, low-dimensional manifold for subsequent diffusion learning.

3. Accuracy-Weighted Latent Diffusion: Biasing Generation Toward High Performance

Standard diffusion models assign equal weight to every training sample, naturally absorbing weak regions of the empirical model zoo. TiltDiff models a tilted probability distribution \(p_{\mathrm{tilt}}(\Theta) \propto p_{\mathrm{zoo}}(\Theta)\widetilde{w}(\Theta)\) by weighting the diffusion denoising loss according to validation performance.

Validation accuracy is first normalized into the unit interval: $$ v_i = \frac{a_i - a_{\min}}{a_{\max} - a_{\min}} $$ Unnormalized importance weights are computed via exponential weighting \(\widetilde{w}_i = \exp(\lambda v_i)\) or sigmoid weighting \(\widetilde{w}_i = \sigma(\beta(v_i - \tau))\), and rescaled to maintain unit mean. The performance-tilted diffusion loss is formulated as: $$ \mathcal{L}{\mathrm{tilt}} = \mathbb{E}, t) \right|_2^2 \right] $$ This loss tilts gradient updates toward high-performing models, guiding the denoising trajectory toward parameter regions associated with strong generalization.}} \left[ w_i \left| \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\psi(\mathbf{z}_t^{(i)

4. Decision Pathway Attribution and Causal Validation: Verifying Functional Structure

Aggregate test accuracy alone does not verify whether synthesized weights preserve coherent internal decision logic. TiltDiff inspects intermediate representations of the diffusion U-Net, leveraging cross-layer unit associations and Captum gradient attributions to uncover class-specific decision pathways.

At a fixed diffusion timestep (\(t=261\)), feature maps are extracted across U-Net blocks, and unit contributions to a target class score are calculated: $$ \operatorname{Attr}^{(l)}(x, c) = A^{(l)}(x) \odot \frac{\partial f_c(x)}{\partial A^{(l)}(x)} $$ Positively attributed units are traced backward from target outputs to construct class-specific pathways. Setting neurons along these pathways to zero demonstrates selective accuracy degradation on the target class, verifying that the generative process preserves causal, class-specific neural structures that transfer across independently sampled models.

Loss & Training

The framework is trained in two decoupled stages. In Stage 1, the Transformer autoencoder is trained using mean squared error reconstruction loss on weight tokens. In Stage 2, the autoencoder is frozen, and the latent U-Net is trained using the performance-tilted diffusion loss \(\mathcal{L}_{\mathrm{tilt}}\) with an \(\boldsymbol{\epsilon}\)-prediction objective. During inference, latent vectors are sampled from standard Gaussian noise via the reverse diffusion trajectory, decoded back into weight tokens, and inverse-standardized into complete functional neural networks.

Key Experimental Results

Main Results

TiltDiff is comprehensively evaluated on three weight-space benchmarks: the CIFAR-10 CNN model zoo (~12k parameters), MNIST CNNs (~5k parameters), and CIFAR-100 ResNet-18 classifier heads (~53k parameters). Table 1 presents test accuracy on CIFAR-10 across various retained data fractions, where TiltDiff consistently surpasses the SANE baseline.

Table 1: Test accuracy of generated models on CIFAR-10 across retained-data fractions

Retained Fraction	Metric	SANESub (Baseline)	TiltDiff (Unweighted)	TiltDiffSig (Sigmoid)	TiltDiffExp (Exponential)
30%	Top-1 / Top-3 (%)	59.77 / 57.37±2.24	58.74 / 58.14±0.71	58.83 / 58.13±0.78	60.39 / 59.63±0.66
25%	Top-1 / Top-3 (%)	58.01 / 57.23±0.69	57.61 / 56.58±0.95	55.84 / 55.69±0.78	58.45 / 58.01±0.57
20%	Top-1 / Top-3 (%)	58.93 / 57.59±1.17	56.95 / 56.79±0.15	61.02 / 59.08±1.69	59.93 / 58.73±1.06
15%	Top-1 / Top-3 (%)	55.53 / 55.13±0.36	59.93 / 59.76±0.27	60.13 / 59.60±0.13	60.16 / 59.03±0.98
10%	Top-1 / Top-3 (%)	57.80 / 56.79±0.89	60.61 / 58.70±2.43	61.54 / 60.76±0.62	58.88 / 58.54±0.40

Table 2 evaluates Top-3 accuracy on MNIST CNNs and CIFAR-100 classifier heads, demonstrating robust generalization to higher-dimensional parameter spaces.

Table 2: Top-3 test accuracy (%) of sampled models on MNIST CNNs and CIFAR-100 ResNet-18 classifier heads

Setting	Retained Fraction	SANESub	TiltDiff	TiltDiffSig	TiltDiffExp
MNIST CNNs	25%	85.87 ± 0.32	87.27 ± 0.17	87.68 ± 0.19	87.91 ± 0.19
MNIST CNNs	20%	85.84 ± 0.55	86.83 ± 0.60	88.02 ± 0.39	88.28 ± 0.04
MNIST CNNs	10%	86.57 ± 0.71	86.56 ± 0.06	86.68 ± 0.26	86.37 ± 0.01
CIFAR-100 Heads	30%	69.89 ± 0.07	74.59 ± 0.06	73.24 ± 2.11	71.45 ± 2.62
CIFAR-100 Heads	20%	69.39 ± 0.21	74.63 ± 0.04	74.51 ± 0.10	74.43 ± 0.00
CIFAR-100 Heads	15%	69.60 ± 0.19	74.36 ± 0.06	74.40 ± 0.04	74.40 ± 0.04

Ablation Study

The core design choices of TiltDiff—noise prediction parameterization, accuracy weighting, and validation-based data selection—are ablated on the CIFAR-10 model zoo.

Table 3: Ablation study of core components on the CIFAR-10 model zoo (Top-3 accuracy, %)

Prediction Target	Accuracy Weighting	Data Selection	Top-3 Accuracy (%)	Note
\(z_0\) (Direct Latent Prediction)	–	–	21.17 ± 3.24	Direct latent prediction fails to converge effectively
\(\boldsymbol{\epsilon}\) (Noise Prediction)	–	–	42.18 ± 1.11	Predicting noise stabilizes diffusion (+21.01%)
\(\boldsymbol{\epsilon}\)	✓	–	46.76 ± 0.70	Performance weighting alone yields +4.58% gain
\(\boldsymbol{\epsilon}\)	–	✓	58.70 ± 2.43	Filtering top-10% models brings large boost (+16.52%)
\(\boldsymbol{\epsilon}\) (Full TiltDiff)	✓	✓	60.76 ± 0.62	Joint selection and weighting achieves optimal accuracy

Key Findings

• Noise prediction is essential: Directly predicting clean latents \(z_0\) yields poor accuracy (21.17%), whereas \(\boldsymbol{\epsilon}\)-prediction achieves 42.18%, proving that noise prediction is crucial for modeling intricate weight manifolds.
• Selection and weighting are complementary: Data selection filtering out undertrained models provides the largest single gain (up to 58.70%), while continuous accuracy weighting pushes performance further to 60.76%, demonstrating that loss tilting provides fine-grained performance preference beyond discrete pruning.
• Robustness against parameter masking: When 1%–10% of convolutional weights are randomly masked to zero, TiltDiff models exhibit substantially smaller relative accuracy drops than SANE.
• High functional diversity via CKA: Evaluating pairwise centered kernel alignment (CKA) shows that TiltDiff samples exhibit lower maximum similarity (\(s_{\max}\)), demonstrating superior functional diversity (\(1 - s_{\max}\)) alongside higher accuracy.
• Causal specificity of decision pathways: Masking the Block 4 cat pathway results in a 98.21% relative drop in cat accuracy while degrading non-target classes by only 14.90% (specificity gap of 83.31 percentage points), confirming that generated weights encode genuine functional modularity.

Highlights & Insights

• Lightweight performance-tilted objective: Without modifying the U-Net architecture or adding complex reinforcement learning loops, scaling diffusion loss weights by normalized accuracy effectively steers generation toward high-performing parameter regions.
• Bridging generative diffusion with model interpretability: Extracting cross-layer pathways from the diffusion U-Net uncovers causal functional circuits in generated models, turning the "black-box generating black-box" dilemma into an interpretable process.
• Effective initialization for rapid adaptation: Generated weights serve as high-quality initializations, achieving 60.8% (CIFAR-10) and 88.0% (MNIST) zero-shot accuracy, and reaching 69.6% after fine-tuning.

Limitations & Future Work

• Evaluation on moderate network scales: The experimental validation is restricted to small CNNs (~12k parameters) and linear classifier heads (~53k parameters), leaving large-scale vision backbones or language models unexplored.
• Permutation alignment computational overhead: Solving channel alignment via Git Re-Basin poses nontrivial computational overhead on branching architectures such as DenseNets or Transformers.
• Future directions: Extending tilted diffusion to controllable weight generation, conditional editing of functional sub-modules, and automated synthesis of diverse model ensembles.

Related Work & Insights

• vs SANE (Schürholt et al., ICML 2024): While SANE pioneered parameter tokenization and autoencoding for empirical weight modeling, TiltDiff introduces performance-tilted diffusion objectives, achieving superior accuracy, perturbation robustness, and functional diversity.
• vs Hyper-Representations / D2NWG: Earlier weight generation methods focused on unconditional sampling or implicit neural fields. TiltDiff provides the first systematic causal analysis of emergent decision pathways across generated networks.

Rating

• Novelty: ⭐⭐⭐⭐ [Introduces performance-weighted latent diffusion to weight space with causal pathway analysis]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans multiple benchmarks, fine-tuning adaptation, weight masking robustness, CKA diversity, and causal pathway ablation]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, structured narrative, and rigorous empirical validation]
• Value: ⭐⭐⭐⭐ [Provides practical tools for model zoo synthesis, fast parameter initialization, and understanding weight space topology]

Related Papers

• [ECCV 2026] Spanning the Visual Analogy Space with a Weight Basis of LoRAs
• [ECCV 2026] NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment
• [CVPR 2025] Generation of Maximal Snake Polyominoes Using a Deep Neural Network
• [ICML 2026] DynaDiff: Generative Adaptation of Dynamics to Environmental Shifts via Weight-space Diffusion
• [ECCV 2026] Generative Refinement Network for Visual Synthesis