Robustness Emerges Early in Training Dynamics, but Is Not Preserved¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Pretraining
Keywords: training dynamics, natural robustness, shallow representations, robustness fading, trajectory intervention
TL;DR¶
This paper discovers that shallow layers spontaneously develop robust representations and flat loss landscapes in early training but lose them during standard convergence; to retain these priors without modifying architectures, it introduces two parameter-free trajectory interventions—Early-Phase Stabilization (EPS) and Asymmetric Weight Reversion (AWR).
Background & Motivation¶
Deep neural networks achieve remarkable performance on clean image classification benchmarks, yet their accuracy drops precipitously under real-world corruptions such as sensor noise, weather artifacts, and blur degradations. This fragility poses severe hazards for safety-critical deployments in autonomous driving and medical imaging. Prior robustness approaches largely focus on input data augmentations (such as AugMix or Mixup) and complex optimization recipes. However, layer-wise linear probing reveals that these state-of-the-art strategies deliver performance gains almost exclusively in deeper stages, while the front-end layers responsible for low-level texture and edge processing exhibit vulnerability indistinguishable from naive baselines. While biologically inspired designs (such as VOneNet's V1 module) demonstrated that hard-coding robust filters into front-end layers boosts overall robustness, they raise a fundamental question: if robust properties can be hard-wired into shallow layers, might they also emerge spontaneously during the training trajectory?
Tracking the directional cosine similarity of shallow-layer weight trajectories reveals that early optimization involves an active exploration phase characterized by substantial directional reorganization. Representation-level analysis using Centered Kernel Alignment (CKA) and InfoNCE-based Mutual Information demonstrates that shallow features attain peak structural similarity and information preservation under corruptions during this exploratory period. Causal freezing experiments further confirm that anchoring networks to early shallow configurations yields a significantly flatter loss landscape under input perturbations. Nevertheless, as standard Empirical Risk Minimization continues on clean data, the model gradually exploits non-robust, high-frequency shortcuts, causing shallow representation stability to deteriorate and the optimization landscape to sharpen—a phenomenon defined in this work as "Robustness Fading."
This paper approaches the problem from a training dynamics perspective: rather than altering architectures or introducing extra parameters, one can preserve or recover these transient early-emergent priors via lightweight trajectory interventions. Core idea: formalize corruption robustness as a trajectory intervention problem, introducing Early-Phase Stabilization (EPS) to freeze shallow weights early and Asymmetric Weight Reversion (AWR) to rewind shallow weights to early snapshots under dampened learning rates, thereby locking in flat minima and corruption-resilient priors with zero architectural overhead.
Method¶
Overall Architecture¶
The framework decomposes standard network parameters \(\theta\) into two functional components: the shallow subnetwork \(\theta_s\), which extracts low-level geometric and textural features, and the deep subnetwork \(\theta_d\), which handles high-level semantic abstraction. During standard training over \(T\) epochs, both components evolve concurrently to minimize clean empirical risk, inevitably eroding early-stage robust priors in \(\theta_s\). The proposed framework introduces a temporal intervention operator \(\mathcal{I}\) at a designated intervention time \(\tau\) to modulate the update trajectory of \(\theta_s\), while keeping \(\theta_d\) unconstrained to preserve semantic task adaptation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Data & Standard Loss L"] --> B["Decoupled Trajectory Intervention<br/>Parameter split into shallow θs and deep θd"]
B --> C{"Select Intervention Strategy"}
C -->|Truncation Mechanism| D["Early-Phase Stabilization<br/>Freeze θs at τeps to lock flat minima"]
C -->|Rewinding Mechanism| E["Asymmetric Weight Reversion<br/>Rewind θs to tearly snapshot with lr ηfine"]
D --> F["Deep Subnetwork θd continues optimization"]
E --> F
F --> G["Converged Robust Output Model"]Key Designs¶
1. Decoupled Trajectory Intervention: functional separation of shallow and deep update dynamics Conventional deep learning updates all layers synchronously across iterations, overlooking the distinct dynamical evolution between early and late layers. To prevent clean-data overfitting from washing out early robust shallow features, this design partitions parameters into \(\theta = \{\theta_s, \theta_d\}\) and modulates the shallow update trajectory via an intervention operator \(\mathcal{I}\): $$ \theta_s^{(t+1)} \leftarrow \mathcal{I}\left(\theta_s^{(t)}, \nabla_{\theta_s} \mathcal{L}, t, \tau\right) $$ Before time step \(t = \tau\), \(\theta_s\) and \(\theta_d\) undergo standard stochastic gradient descent. Once \(t \ge \tau\), the operator \(\mathcal{I}\) takes over the update policy of \(\theta_s\), isolating shallow geometric representations from late-stage task-specific overfitting while allowing \(\theta_d\) to realign classification boundaries with the stabilized front-end.
2. Early-Phase Stabilization: trajectory truncation to lock in flat robust minima To arrest the monotonic decline of representation CKA similarity and mutual information across training epochs, Early-Phase Stabilization (EPS) enforces a clean trajectory truncation. During the exploratory phase (up to epoch \(\tau_{eps}\)), shallow convolutional layers develop robust global representations and converge into smooth, flat loss regions. At threshold epoch \(\tau_{eps}\), EPS nullifies subsequent updates on the shallow subnetwork: $$ \forall t \ge \tau_{eps}, \quad \Delta \theta_s^{(t)} = 0 \implies \theta_s^{(t+1)} = \theta_s^{(\tau_{eps})} $$ The shallow parameters \(\theta_s\) remain permanently frozen in their peak robust state for the remaining iterations, while \(\theta_d\) continues minimizing empirical risk. This explicit freezing eliminates late-stage representation degradation and reduces backward computational and memory overhead since shallow gradients are no longer required.
3. Asymmetric Weight Reversion: historical snapshot rewinding with asymmetric learning rate damping For scenarios where early truncation was not deployed or extended joint training has already elapsed, Asymmetric Weight Reversion (AWR) offers an active trajectory recovery mechanism inspired by the Lottery Ticket Hypothesis. AWR records a snapshot of the shallow subnetwork during early exploration: \(S \leftarrow \theta_s^{(t_{early})}\). At a later intervention epoch \(t = \tau_{awr}\), once deep representations have developed sufficient semantic discriminability, AWR rewinds the shallow parameters back to this historical robust state: $$ \theta_s^{(\tau_{awr})} \leftarrow \theta_s^{(t_{early})}, \quad \text{where } t_{early} < \tau_{awr} $$ Following this state reversion, AWR suppresses the shallow learning rate to an attenuated scale \(\eta_s \leftarrow \eta_{fine}\) (typically \(10^{-6}\) to \(10^{-3}\)) to prevent shallow weights from drifting back into sharp minima, while the deep subnetwork continues optimizing at the standard learning rate \(\eta_d\). This asymmetric modulation enables the model to simultaneously inherit high-level semantic capacity and reclaim early-emergent low-level robustness.
Loss & Training¶
The framework trains entirely on clean data using the standard cross-entropy loss without auxiliary contrastive or regularization penalties: $$ \mathcal{L}{CE} = -\sum(F(x_i; \theta_s, \theta_d))\right) $$ The base training recipe trains ResNet-50 on ImageNet for 90 epochs with SGD (momentum 0.9, weight decay } y_i \log \left(\text{Softmax\(10^{-4}\)), starting at learning rate \(\eta=0.1\) and decaying by \(10\times\) at epochs 30 and 60. For EPS, the sole hyperparameter is the truncation epoch \(\tau_{eps}\). For AWR, the snapshot epoch \(t_{early}\) is captured early (e.g., epoch 5-10), the rewinding epoch \(\tau_{awr}\) is applied around epoch 20, and the fine-tuning learning rate \(\eta_{fine}\) is set to \(10^{-4}\).
Key Experimental Results¶
Main Results¶
On the ImageNet-C benchmark using a ResNet-50 backbone without complex data augmentations, EPS and AWR substantially outperform established optimization constraints and structural prior baselines:
| Model / Method | Category | Top-1 Acc (%) | Relative Gain (%) |
|---|---|---|---|
| Standard Baseline | Empirical Risk Minimization | 39.2 | - |
| DST | Dynamic Sparse Training | 38.7 | -0.5 |
| AdaSAP* | Adaptive Sparse Prior | 43.3 | +4.1 |
| EWS | Elastic Weight Saliency | 40.6 | +1.4 |
| SAM | Sharpness-Aware Minimization | 39.8 | +0.6 |
| DAMP | Directional Annealed Smoothing | 41.4 | +2.2 |
| DAT | Dynamic Adversarial Tuning | 41.1 | +1.9 |
| VOneNet | Bio-inspired V1 Front-end | 40.3 | +1.1 |
| Gabor Layers | Handcrafted Filter Bank | 37.5 | -1.7 |
| EPS (Ours) | Trajectory Truncation | 42.5 | +3.3 |
| AWR (Ours) | Asymmetric Weight Reversion | 43.1 | +3.9 |
| AWR* (Ours + Strong Aug) | Augmentation Synergy | 46.5 | +7.3 |
Across various architectures evaluated on ImageNet-100 and its corruption variants (C, \(\bar{\text{C}}\), 3DCC, V2-C), both interventions deliver consistent reductions in Average mean Corruption Error (Avg. mCE, lower is better):
| Backbone Architecture | Strategy | IN-100 (Clean) | IN-100-C | IN-100-¯C | IN-100-3DCC | IN-100V2-C | Avg. mCE (↓) | Change |
|---|---|---|---|---|---|---|---|---|
| MobileNetV2 | Baseline | 84.6 | 95.8 | 94.4 | 88.8 | 97.6 | 94.2 | baseline |
| MobileNetV2 | EPS | 84.2 | 92.8 | 93.9 | 87.8 | 96.0 | 92.6 | -1.6 |
| MobileNetV2 | AWR | 84.2 | 93.4 | 94.1 | 87.8 | 95.6 | 92.7 | -1.5 |
| WideResNet-50 | Baseline | 85.6 | 88.3 | 88.4 | 83.3 | 92.1 | 88.0 | baseline |
| WideResNet-50 | EPS | 84.9 | 72.6 | 83.1 | 72.5 | 80.5 | 77.2 | -10.8 |
| WideResNet-50 | AWR | 85.3 | 71.8 | 82.3 | 71.9 | 80.1 | 76.5 | -11.5 |
| MobileViT-S | Baseline | 85.4 | 88.8 | 92.5 | 84.8 | 92.9 | 89.8 | baseline |
| MobileViT-S | EPS | 85.8 | 87.8 | 90.5 | 84.8 | 93.6 | 89.2 | -0.6 |
| MobileViT-S | AWR | 85.2 | 87.0 | 91.7 | 84.5 | 90.7 | 88.5 | -1.3 |
| EfficientFormer-L1 | Baseline | 91.6 | 73.7 | 66.3 | 65.2 | 80.4 | 71.4 | baseline |
| EfficientFormer-L1 | EPS | 92.5 | 68.0 | 62.9 | 60.0 | 75.8 | 66.7 | -4.7 |
| EfficientFormer-L1 | AWR | 92.0 | 72.8 | 64.3 | 64.1 | 79.2 | 70.1 | -1.3 |
| MambaOut-femto | Baseline | 94.1 | 60.0 | 54.7 | 57.7 | 68.2 | 60.2 | baseline |
| MambaOut-femto | EPS | 93.4 | 57.2 | 50.8 | 52.9 | 66.5 | 56.9 | -3.3 |
| MambaOut-femto | AWR | 93.1 | 58.5 | 52.8 | 55.0 | 67.3 | 58.4 | -1.8 |
Ablation Study¶
When combined with representative data augmentation and regularization techniques, EPS and AWR exhibit additive improvements across ImageNet-1K benchmarks:
| Baseline Technique | Added Strategy | Clean IN Acc (%) | IN-C mCE | IN-¯C mCE | IN-3DCC mCE | INV2-C mCE | Avg. mCE (↓) | Relative Gain |
|---|---|---|---|---|---|---|---|---|
| AugMix | None | 76.1 | 71.7 | 74.0 | 69.6 | 78.4 | 73.4 | baseline |
| AugMix | EPS | 76.2 | 70.8 | 73.6 | 69.1 | 78.2 | 72.9 | -0.5 |
| AugMix | AWR | 74.1 | 68.9 | 75.1 | 69.1 | 76.4 | 72.4 | -1.0 |
| AutoAug | None | 76.4 | 73.2 | 76.9 | 69.6 | 79.8 | 74.9 | baseline |
| AutoAug | EPS | 74.3 | 70.6 | 78.6 | 69.5 | 77.7 | 74.1 | -0.8 |
| AutoAug | AWR | 74.3 | 69.7 | 76.6 | 68.7 | 76.7 | 73.0 | -1.9 |
| CutMix | None | 76.9 | 76.9 | 76.4 | 72.5 | 82.5 | 77.1 | baseline |
| CutMix | EPS | 74.3 | 73.4 | 76.9 | 72.4 | 79.9 | 75.7 | -1.4 |
| CutMix | AWR | 74.4 | 74.1 | 76.5 | 73.1 | 80.4 | 76.0 | -1.1 |
| Label Smoothing | None | 76.6 | 75.2 | 77.1 | 72.1 | 81.5 | 76.5 | baseline |
| Label Smoothing | EPS | 74.5 | 72.7 | 77.7 | 71.8 | 79.3 | 75.4 | -1.1 |
| Label Smoothing | AWR | 74.7 | 72.6 | 77.0 | 71.9 | 79.3 | 75.2 | -1.3 |
| Dropout | None | 76.3 | 76.4 | 78.6 | 72.9 | 82.4 | 77.6 | baseline |
| Dropout | EPS | 73.9 | 73.0 | 77.7 | 72.4 | 79.8 | 75.7 | -1.9 |
| Dropout | AWR | 74.0 | 74.1 | 78.8 | 73.0 | 80.6 | 76.6 | -1.0 |
Downstream dense predictions and adaptation tasks also confirm strong transferability: - Object Detection (COCO-C): Faster R-CNN mAP improves from 17.5 to 17.9 (+0.4); YOLOv5 mAP increases significantly from 19.9 to 24.3 (+4.4). - Semantic Segmentation (Cityscapes-C / ADE20K-C): DeepLabV3+ mIoU increases from 36.2 to 39.9 (+3.7) on Cityscapes-C; GCNet gains +3.1 mIoU on Cityscapes-C and +0.8 mIoU on ADE20K-C. - Continual Online Test-Time Adaptation (ImageNet-C, severity 5): Under severe shift, AdaCont combined with AWR gains +2.6% accuracy (34.9% \(\to\) 37.5%), and ROTTA combined with EPS gains +1.8% (32.6% \(\to\) 34.4%).
Key Findings¶
- Hyperparameter insensitivity: EPS exhibits stable performance across intervention timings \(\tau_{eps} \in [0, 40]\); AWR shows minimal sensitivity to the snapshot anchor \(t_{early} \in [0, 30]\), achieves optimal balance at trigger epoch \(\tau_{awr} = 20\), and remains robust across learning rates \(\eta_s \in [10^{-6}, 10^{-3}]\).
- Flatter loss geometry and invariant rank: EPS and AWR suppress local sharpness \(\Delta L_{max}\) and gradient norm \(\|\nabla L\|\) under increasing perturbation scales; they also drastically reduce discrepancies (\(\Delta\)) in Effective Rank and Intrinsic Dimension between clean and corrupted data.
- Global structure bias over texture: Patch shuffling evaluations indicate that intervened models rely more on global shape rather than local patch cues; frequency-domain analysis shows error rates in mid-to-high frequencies are substantially reduced.
Highlights & Insights¶
- Paradigm shift in robustness understanding: Highlights the "Robustness Fading" phenomenon—shallow networks naturally learn robust filters early on, challenging the assumption that natural robustness requires explicit architectural priors or heavy augmentations.
- Zero-cost plug-and-play intervention: Operates strictly via scheduling without extra forward branches, loss terms, or parameters, while reducing backpropagation costs when shallow layers are frozen.
- Broad versatility across models and tasks: Successfully scales across CNNs, ViTs, and Mamba models, transferring seamlessly to object detection, semantic segmentation, real-world extreme weather (ACDC), and online test-time adaptation.
Limitations & Future Work¶
- Heuristic boundary for shallow layers: Shallow parameters \(\theta_s\) are empirically chosen as Stem or Stage-1 layers, which may require manual tuning on non-hierarchical or multi-scale architectures.
- Scope limited to low-level corruption shifts: Trajectory interventions specifically preserve low-level frequency and textural robustness, offering limited protection against complex high-level semantic domain shifts (such as DomainBed-style contextual changes).
- Future automated scheduling: Potential follow-ups could explore gradient curvature or spectral norm triggers to automate truncation and reversion timings adaptively.
Related Work & Insights¶
- vs VOneNet / Bio-inspired Architectures: VOneNet hard-codes biological Gabor filters into the front-end, compromising end-to-end flexibility; this work demonstrates that standard optimization spontaneously forms robust shallow representations that can be preserved via simple trajectory scheduling.
- vs SAM / DAMP Flatness Optimizers: SAM seeks flat minima across the entire parameter space at double the computational cost; this work identifies that flat geometry primarily stems from early shallow layers, achieving comparable landscape flatness by intervening only on shallow dynamics without computation overhead.
- vs Lottery Ticket Hypothesis / Rewinding: LTH applies rewinding post-pruning to locate sparse subnetworks; this work repurposes asymmetric rewinding to rescue early-emergent robust shallow states during continuous training.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First systematic revelation of early-emergent shallow robustness and its subsequent fading, supported by elegant dynamic interventions.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage spanning classification, detection, segmentation, online TTA, real-world weather, loss landscapes, effective rank, and frequency analysis.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from motivational probing and causal freezing to formal methodology and extensive validation.
- Value: ⭐⭐⭐⭐⭐ A zero-overhead, architecture-agnostic training paradigm that can be immediately integrated into modern vision pretraining pipelines.