Topology-Weighted Effective Rank: A Zero-Cost Proxy for Training Dynamics Stability in Deep Vision Networks

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/Thiswycf/TER-Score
Area: Model Compression
Keywords: Neural Architecture Search (NAS), Zero-Cost Proxy (ZCP), Training Dynamics Stability, Effective Rank, Topology Weighting

TL;DR

To overcome the failure of conventional zero-cost proxies to capture optimization dynamics and architectural topological heterogeneity, this paper introduces ER-Score based on the effective rank of feature correlation matrices and its topology-weighted variant TER-Score, achieving state-of-the-art ranking consistency and ultra-low search overhead across diverse NAS benchmarks.

Background & Motivation

Neural architecture search (NAS) aims to eliminate the heavy trial-and-error process of manual deep network design, yet evaluating thousands of candidate architectures under early reinforcement learning or evolutionary frameworks incurred thousands of prohibitive GPU days. While weight-sharing one-shot supernets alleviated computational bottlenecks, they introduced persistent issues with sub-network gradient coupling and ranking disorder. Recently emerged zero-cost proxies (ZCPs) leverage single forward or backward passes to extract statistics such as parameter norms, gradient variances, or linear region activation patterns, estimating network performance without any training and substantially speeding up search pipelines.

However, existing zero-cost proxies suffer from two fundamental limitations. First, most methods rely exclusively on static architectural signals at initialization (such as parameter count, FLOPs, GradNorm, or synaptic flow SynFlow), or compute instantaneous local curvature at initialization (such as the NTK condition number), failing to reflect the actual evolution of training dynamics during backpropagation and gradient descent. In practice, the final generalization of deep networks is heavily governed by optimization stability, loss decay speed, and perturbation sensitivity, causing static metrics to exhibit severe ranking degradation across tasks and search spaces. Second, existing proxies universally adopt a "node-homogeneous" assumption, naively summing or averaging local statistics across layers or operators. This ignores the heterogeneous contributions of components in directed acyclic graph (DAG) structures like CNNs, where depth, connectivity patterns, and information aggregation hubs play profoundly different roles.

Starting from the training dynamics theory of over-parameterized networks, this work connects the Gram matrix eigenspectrum—which determines the loss decay rate in the infinite-width limit—to the channel correlation matrix of intermediate feature maps. It proposes an effective rank score (ER-Score) to measure spectral dispersion and incorporates graph-theoretic topological priors for edge-level weighting. Core idea: quantify training dynamics stability via the spectral entropy effective rank of feature correlation matrices, and heterogeneously weight computation units according to their topological information aggregation and diffusion roles in DAGs to construct a zero-overhead, data-independent, and topology-aware proxy termed TER-Score.

Method

Overall Architecture

The evaluation pipeline of TER-Score consists of three main stages: forward propagation on randomly perturbed inputs, operator-level feature correlation effective rank computation, and topology weight synthesis and aggregation based on the network DAG. Given a candidate architecture, the system feeds a single mini-batch of unlabeled random Gaussian noise, executes a single lightweight forward pass to extract intermediate feature maps from each operator edge, computes the channel covariance matrix and its exponential spectral entropy (ER-Score), and calculates the structural importance of each connection edge using graph metrics (PageRank or degree centrality) to produce the final architecture fitness score.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Random Tensor Input (Gaussian / Uniform)"] --> Fwd["Data-Free Random Mini-Batch Lightweight Proxy Protocol<br/>Single forward pass extracting edge feature maps A_e"]
    Fwd --> ER["Effective Rank Quantification of Training Dynamics Stability<br/>Feature covariance eigenspectrum entropy exponentiation ER(A_e)"]
    DAG["CNN Cell DAG Topology Analysis"] --> TW["Topology-Aware Weighting Mechanism for CNN DAGs<br/>Bidirectional PageRank / path-based weighting w_e"]
    ER --> TER["Topology-Weighted Effective Rank Score (TER-Score)<br/>Normalized weighted aggregation ∑ w_e ER(A_e)"]
    TW --> TER
    TER --> Downstream["Downstream NAS & Selection (Evolutionary Search / Predictor)"]

Key Designs

1. Effective Rank Quantification of Training Dynamics Stability: from Gram Matrix Spectrum to Local Feature Correlations

To resolve the inability of static parameter metrics to reflect gradient optimization dynamics, this design builds upon the dynamics theory of over-parameterized neural networks: in the infinite-width regime, network output evolution follows the differential equation \(\frac{d\mathbf{u}(t)}{dt} = \mathbf{H}(t)(\mathbf{y} - \mathbf{u}(t))\), where the instantaneous Gram matrix \(\mathbf{H}(t)\) rapidly converges to a stationary kernel \(\mathbf{H}^\infty\). Because loss residual decay is directly governed by the eigenspectrum of \(\mathbf{H}^\infty\), a broader and more balanced eigenvalue distribution yields more stable convergence across orthogonal gradient directions. Since computing the full-network \(\mathbf{H}^\infty\) is computationally prohibitive, the method exploits the equivalence between spatial feature dimensions in convolutional layers and multi-sample fully connected layers, flattening the \(l\)-th feature map \(\mathbf{A}_l \in \mathbb{R}^{h_l \times w_l \times c_l}\) into \(\mathbf{A}'_l \in \mathbb{R}^{(h_l w_l) \times c_l}\) to construct the channel correlation matrix: $\(\mathbf{C}_l = \frac{1}{h_l w_l} (\mathbf{A}'_l)^\top \mathbf{A}'_l \in \mathbb{R}^{c_l \times c_l}\)$ Eigendecomposing the positive semi-definite matrix \(\mathbf{C}_l\) yields normalized eigenvalues \(\tilde{\lambda}_{l, i} = \lambda_{l, i} / \sum_j \lambda_{l, j}\), from which the single-module effective rank is defined via the exponential of spectral entropy: $\(\mathrm{ER}(\mathbf{A}_l) = \exp\left(-\sum_{i=1}^{c_l} \tilde{\lambda}_{l, i} \log \tilde{\lambda}_{l, i}\right)\)$ This metric is strictly scale-invariant and reflects how evenly feature energy is distributed across orthogonal channel directions. Empirical tests confirm that architectures with higher initial ER-Scores exhibit significantly lower cosine similarity fluctuations in subsequent gradient steps, ensuring steady training dynamics.

2. Topology-Aware Weighting Mechanism for CNN DAGs: Breaking the Node-Homogeneity Assumption

Targeting the limitation that conventional proxies treat all computation nodes identically and fail to distinguish architectural bottlenecks, this design models the CNN cell structure as a directed acyclic graph \(G=(V, E)\). Decision tree impurity regression verifies that the relative importance of operator features to final accuracy varies substantially across topological locations. Four structural weighting schemes are established to map topological priors onto feature-propagating edges \(e=(u, v) \in E\): - Degree-based weighting (Degree): Assigns edge weights based on node in-degree \(k^{in}_v\) and out-degree \(k^{out}_u\) via \(w_e = k^{in}_v + k^{out}_u\), highlighting highly interactive connections. - Connectivity-based weighting (Connectivity): Computes the number of distinct paths from source \(s\) to \(u\) (\(c_{su}\)) and from \(v\) to sink \(t\) (\(c_{vt}\)), setting \(w_e = c_{su} \times c_{vt}\) to emphasize information transmission backbones. - Shortest-path-based weighting (Shortest-Path): Evaluates shortest path distances using Dijkstra, setting \(w_e = ((1 + d_{su})(1 + d_{vt}))^{-1}\) to discount isolated peripheral chains. - Bidirectional PageRank weighting (PageRank): Solves stationary distribution vectors \(\boldsymbol{\pi}^*\) and \(\boldsymbol{\pi}'^*\) on the original and reversed DAGs to simultaneously capture information aggregation and diffusion roles, defining edge weight \(w_e = \pi^*_e \times \pi'^*_e\).

Normalizing edge weights such that \(\sum_{e \in E} w_e = 1\) and weighting the effective rank of each corresponding feature map yields the topology-weighted score: $\(\mathrm{TER}\text{-}\mathrm{Score} = \sum_{e \in E} w_e \mathrm{ER}(\mathbf{A}_e)\)$ This strategy provides exceptional normalized discounted cumulative gain (nDCG) when isolating top-tier elite architectures.

3. Data-Free Random Mini-Batch Lightweight Proxy Protocol: Decoupling Semantic Content from Computational Overhead

To avoid the sensitivity of conventional proxies to specific training image distributions and the associated data-loading overhead, this design systematically explores input disturbance sources and spatial resolutions. Experiments demonstrate that feeding natural images degrades Spearman rank correlation, whereas injecting pure random noise from standard Gaussian \(\mathcal{N}(0, \mathbf{I})\) or uniform distributions \(\mathcal{U}[-1, 1]\) generates unbiased, high-dimensional activations that most cleanly expose intrinsic architectural stability. Furthermore, moderately downsampling spatial resolution preserves channel correlation spectra while eliminating spatial redundancy, enabling covariance decomposition and effective rank estimation within milliseconds per architecture and substantially decreasing NAS traversal time.

Loss & Training

As a training-free zero-cost proxy, TER-Score requires no parameter fine-tuning or gradient backpropagation. When evaluating any candidate architecture, network weights are randomly initialized with \(W \sim \mathcal{N}(0, \mathbf{I})\) and batch normalization running statistics are frozen. A single mini-batch of random noise tensors (defaulting to moderate-resolution Gaussian noise with batch size 32 or 64) is passed in a single forward inference step to collect edge feature maps. During downstream search, TER-Score seamlessly integrates into standard evolutionary algorithms, serving as the individual fitness evaluation function driving mutation and crossover selection.

Key Experimental Results

Main Results

Evaluations span NAS-Bench-201, NAS-Bench-301, and the DARTS search space, along with cross-architecture generalization tests on ViT-Bench-101. Architectures discovered using TER-Score guided evolutionary search in the DARTS space were trained from scratch and compared against classical NAS and leading zero-cost approaches.

Search Space / Task	Metric	Ours (TER-Score)	Best Baseline (AZ-NAS / SWAP)	Gain
DARTS / CIFAR-10 Full Training	Test Error (%)	2.41%	2.48% (SWAP-NAS)	-0.07% error
DARTS / ImageNet-1k Full Training	Top-1 Error (%)	23.55%	23.70% (AZ-NAS)	-0.15% error
DARTS / Search Cost	GPU Days	0.09 days	0.06 days (AZ-NAS) / 4.0 days (DARTS)	44x faster than DARTS
ViT-Bench (AutoFormer-C100)	Spearman Rank Correlation (%)	95.36%	63.87% (Auto-Prox)	+31.49%
ViT-Bench (PiT-Flowers)	Spearman Rank Correlation (%)	96.33%	92.94% (Auto-Prox)	+3.39%
ViT-Bench (PiT-Chaoyang)	Spearman Rank Correlation (%)	76.14%	55.09% (Auto-Prox)	+21.05%

Ablation Study

Ablations investigate the impact of mini-batch input distribution, input spatial resolution, and topological weighting strategies.

Configuration Dimension	Experimental Setting	Spearman Rank Correlation / Impact	Core Mechanism Note
Input Data Distribution	Gaussian Noise \(\mathcal{N}(0, \mathbf{I})\)	Optimal correlation (baseline)	Unbiased random perturbations activate intrinsic architectural dynamics
Input Data Distribution	Uniform Noise \(\mathcal{U}[-1, 1]\)	Parity with Gaussian input	Preserves flat spectral perturbation without semantic dataset bias
Input Data Distribution	Real Image Dataset Inputs	Noticeable drop vs. random noise	Semantic distributions bias and dilute underlying dynamical stability measurement
Input Data Distribution	Constant All-One Input	Severe performance degradation	Lacks channel variance, collapsing covariance matrix rank
Input Spatial Resolution	Medium Resolution	Best overall performance	Balances inter-channel structural info with local spatial redundancy removal
Input Spatial Resolution	Low / High Resolution	Slight drop at low / redundant compute at high	Verifies full-resolution images are unnecessary for spectral entropy estimation
Topology Weighting Strategy	PageRank (P.R.) Weighting	Best elite discrimination (nDCG)	Balances aggregation and diffusion, yielding stable evolutionary convergence
Topology Weighting Strategy	Unweighted (ER-Score)	Solid baseline, lower elite separation	Confirms the substantial benefit of incorporating DAG structural weights

Key Findings

• Asymmetric Value of Topological Flow: Among the four graph weighting strategies, PageRank and degree centrality significantly outperform shortest-path weighting on downstream tasks and elite discrimination. This indicates that importance is not simply a function of proximity to inputs/outputs, but rather depends on structural hubs that possess high capacity for information gathering and distribution.
• Eliminating Parameter Count Bias: Certain existing proxies (e.g., GradNorm, SynFlow) tend to favor architectures with disproportionately large parameter counts during evolutionary search, causing negative optimization on small datasets like CIFAR-10. By leveraging the intrinsic normalization of channel effective rank, TER-Score maintains monotonically improving accuracy and minimal variance across search trajectories.
• Cross-Architectural Generalization: Although topology weighting is designed for CNN DAGs, the foundational ER-Score is model-agnostic. When evaluated across Vision Transformer search spaces in ViT-Bench-101, its rank correlation with distillation accuracy exceeds \(95\%\) on both AutoFormer and PiT, significantly outperforming dedicated ViT search metrics such as TF-TAS and Auto-Prox.

Highlights & Insights

• Low-Cost Observability of Training Dynamics: Connects the abstract Gram matrix of over-parameterized gradient descent dynamics to the effective rank of single-layer feature covariance matrices, bridging theoretical training dynamics and training-free evaluation via lightweight linear algebra.
• Graph Topology Theory Empowering Architecture Evaluation: Introduces bidirectional PageRank steady-state probabilities on DAGs into architectural weight allocation, abandoning crude uniform averaging across layers and providing an interpretable formalization of structural heterogeneity.
• Data-Free Proxy Purity: Demonstrates that pure random noise perturbations outperform real images for probing optimization stability, eliminating data-loading overhead and privacy constraints while establishing a versatile data-agnostic evaluation paradigm.

Limitations & Future Work

• Heuristic Nature of Topology Weighting: The authors acknowledge that degree centrality and PageRank weighting remain empirical graph heuristics without formal theoretical proofs linking specific graph metrics to generalization bounds.
• Absence of Transformer Topology Modeling: TER-Score's topological weighting is tailored for CNN DAGs and does not directly extend to dense or sequential Vision Transformer graphs; future work could explore structural priors based on self-attention interaction graphs or token routing.
• Sensitivity in Extreme Quantization: For ultra-low-bit quantized or heavily pruned networks (e.g., 1-bit or 2-bit weights), severe activation clipping might cause pseudo-saturation in effective rank, warranting further stress-testing.

Related Work & Insights

• vs TE-NAS [6]: TE-NAS relies on the condition number and linear regions of the Neural Tangent Kernel (NTK) at initialization, but full NTK computation is expensive and ignores dynamic evolution; TER-Score approximates dynamic loss decay via local feature effective rank, offering faster evaluation and more stable ranking.
• vs ZiCo [23]: ZiCo uses the inverse coefficient of variation (ICV) of initial gradients to quantify landscape flatness, remaining a coarse scalar gradient statistic; TER-Score delves into channel principal component distributions in representation space, yielding more nuanced characterizations.
• vs ParZC [10]: ParZC identifies node-homogeneity issues but relies on simple sinusoidal positional encodings; TER-Score directly computes structural connectivity on the true computational DAG (e.g., PageRank and path topology), providing stronger interpretability and generalization.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Derives feature effective rank from Gram matrix spectral flatness and systematically introduces DAG graph-theoretic weighting]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-task and cross-modal validation across CNNs and ViTs, complete with full training and evolutionary search on DARTS]
• Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical foundation, well-structured motivation, self-consistent data, and clear presentation]
• Value: ⭐⭐⭐⭐☆ [Open-source code offering an ultra-low-latency, highly practical tool for lightweight and training-free NAS]

Related Papers

• [ECCV 2026] Task Alignment: A simple and effective proxy for model merging in computer vision
• [ICLR 2026] Distilling and Adapting: A Topology-Aware Framework for Zero-Shot Interaction Prediction in Multiplex Biological Networks
• [ICLR 2026] Training Dynamics Impact Post-Training Quantization Robustness
• [ECCV 2026] Puppet-CNN: Continuous Parameter Dynamics for Input-Adaptive Convolutional Networks
• [CVPR 2026] AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks