Geometry-Anchored Transport Framework for Exemplar-Free Class-Incremental Learning

Conference: ECCV 2026
arXiv: 2606.25347
Code: https://github.com/HXuSz11/GATF_ECCV2026
Area: Continual Learning / Class-Incremental Learning
Keywords: Exemplar-Free Class-Incremental Learning, Feature Transport, Topology Preservation, Mahalanobis Classification, Representation Drift

TL;DR

This paper proposes the Geometry-Anchored Transport Framework, which shifts feature transport from a decoupled post-processing paradigm to training-time geometric constraints. By using analytical geometric anchors (closed-form solutions of the Sylvester equation) to suppress macro-anisotropic drift, and combining with topology-aware evolution to constrain local manifold degradation, it stably transports old-class Gaussian statistics under exemplar-free constraints to achieve reliable Mahalanobis evaluation.

Background & Motivation

Exemplar-Free Class-Incremental Learning (EFCIL) requires continuously learning new categories without storing any old samples. This implies that the model must rely on previously learned statistical knowledge (such as class-conditional means and covariances) to distinguish old classes. Mainstream approaches use prototype-plus-Gaussian parameter inference: maintaining a multivariate Gaussian distribution for each class and performing Bayesian classification in the new task's feature space using the Mahalanobis distance. This method is elegant and efficient, but it implicitly assumes that the Gaussian statistics of old classes can be accurately "transported" to the current feature space.

However, as the backbone finishes learning each new task, its embedding space undergoes a continuous non-linear deformation—referred to as representation drift. Mapping the statistics of old classes into the deformed space is non-trival. Existing methods generally adopt a decoupled two-stage paradigm: first training the backbone on new data with cross-entropy and knowledge distillation, and then training a post-processing adapter to project old class prototypes into the new space. This "train-first, adapt-later" pipeline exhibits two structural deficiencies.

First, topological degradation. The optimization of the backbone only focuses on the discriminative and distillation losses of the new classes, ignoring its disruption to the neighborhood structure of old classes. When the embedding space undergoes rotation and non-uniform displacement due to parameter updates, the relative spatial relationships between old classes can be severely distorted—leaving the post-processing adapter to deal with an already "topologically collapsed" manifold, which naturally limits mapping accuracy. Second, anisotropic drift is amplified by Mahalanobis. The Mahalanobis distance relies on the inverse covariance matrix. Microscopic drift along low-variance directions (i.e., the "narrow axes" in the feature space) is dramatically amplified by the inverse covariance, leading to decision boundary shifts. Adapters under decoupled paradigms typically optimize isotropic Euclidean objectives, which apply almost no penalty to these sensitive errors along low-variance directions.

The core insight of this paper is that rather than repairing the feature space after it has degraded, it is better to directly embed transport constraints into the backbone training process—allowing features to evolve while maintaining consistency with the geometry of old classes. Core Idea: The authors propose the Geometry-Anchored Transport Framework, which transforms feature transport from a post-processing adaptation into an endogenous training constraint. It suppresses anisotropic macro-drift using Analytical Geometric Anchors (AGA) and limits local manifold degradation using Topology-Aware Evolution. This co-optimizes backbone training, linear anchoring, and non-linear residual calibration, ultimately outputting an off-the-shelf, geometry-aware adapter after training without requiring additional decoupled fine-tuning.

Method

Overall Architecture

The core problem setting of this paper is as follows: when the \(t\)-th task arrives, the old backbone \(f_{t-1}\) is frozen, and the new backbone \(f_t\) is trained using the current data \(D_t\). For the same input \(x\), \(f_{t-1}\) and \(f_t\) generate features \(z_{\text{old}}\) and \(z_{\text{new}}\), respectively. The evaluation phase follows the Mahalanobis classification rule, but the class-conditional Gaussian statistics \((\mu_c, \Sigma_c)\) of the old classes must be "pushed" into the feature space of \(f_t\).

The traditional decoupled approach is to first train \(f_t\) in the forward pass (focusing only on new classes) and then additionally train a post-processing network to project old-class statistics. The approach in this paper is completely different: while training \(f_t\), a geometry-aware adapter \(A_t\) is jointly optimized to ensure that the subsequent pushing of old-class statistics is inherently accurate.

The framework comprises two core components. At the macro level, Analytical Geometric Anchors (AGA) obtain a closed-form affine mapping \(P_t z + b_t\) by solving the Sylvester equation to capture the dominant global drift from the old space to the new space and apply anisotropic shrinkage along low-variance directions to suppress Mahalanobis-sensitive errors. At the micro level, Topology-Aware Evolution constrains the local deformation of the backbone through a pair of loss terms: forward residual calibration uses a lightweight MLP to learn non-linear displacements that the linear anchor fails to model, while backward manifold anchoring employs an auxiliary distiller to guarantee that the new features can still reconstruct the old features, forcing the backbone updates to remain topology-preserving.

The two work cooperatively during training: AGA provides a zero-gradient closed-form baseline under whose constraints the backbone and residual network evolve; AGA itself is recomputed at intervals using the current paired features and is smoothly updated via Exponential Moving Average (EMA). Ultimately, \(A_t(z) = \bar{P}_t z + \bar{b}_t + g(z; \theta_t)\) is directly used for pushing old-class Gaussian statistics after training, eliminating the need for any post-processing fine-tuning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Task t Dataset D_t"] --> B["Active backbone f_t<br/>Frozen f_{t-1}"]
    B --> C["Paired Features<br/>(z_old, z_new)"]

    subgraph D["Analytical Geometric Anchors (AGA)"]
        direction TB
        D1["Periodically Compute GLS Closed Solution<br/>Sylvester Equation"]
        D2["EMA Smoothing<br/>P̄_t, b̄_t"]
        D1 --> D2
    end

    C --> D
    C --> E["Topology-Aware Evolution"]

    subgraph E["Topology-Aware Evolution"]
        direction TB
        E1["Forward Residual Calibration<br/>MLP g learns z_new - AGA(z_old)"]
        E2["Backward Manifold Anchoring<br/>Distiller D(z_new) -> z_old"]
    end

    D2 --> E1
    E1 --> F["Geometry Adapter A_t<br/>AGA + Residual MLP"]
    E2 --> F
    F --> G["Push Old-class Gaussian Statistics<br/>z̃ ~ N(μ,Σ) -> A_t(z̃)"]
    G --> H["Mahalanobis Classification"]

Key Designs

1. Analytical Geometric Anchors (AGA): Capturing Macro-Drift with the Sylvester Equation

What is the most intuitive correspondence between old and new features? For the same input \(x\), the outputs of \(f_{t-1}\) and \(f_t\) should exhibit some predictable shift. This paper proposes using an affine mapping \(P_t z + b_t\) to approximate this global relationship. Crucially, however, this mapping must "understand" the Mahalanobis geometry: transport errors along low-variance directions are amplified by the inverse covariance, and thus stronger regularization penalties are required in these directions.

The detailed formulation is as follows. Denoting the paired new and old features as \((z_{\text{old}}, z_{\text{new}})\), the affine mapping is solved using Generalized Least Squares (GLS). However, the error metric employs the weighted Mahalanobis norm \(||\cdot||^{2}_{\Sigma_{\text{new}}^{-1}}\)—which penalizes deviations along low-variance directions more heavily. Incorporating the Frobenius norm regularization term \(\rho ||P||_F^2\), the optimal solution \(P^{*}\) satisfies a Sylvester equation: \(P^{*} \Sigma_{\text{old}} + \rho \Sigma_{\text{new}} P^{*} = C_{\text{on}}^T\), where \(C_{\text{on}}\) is the cross-covariance between \(z_{\text{old}}\) and \(z_{\text{new}}\). This equation has a closed-form spectral solution \(X_{ij} = (U^T C_{\text{on}}^T V)_ij / (\gamma_j + \rho \lambda_i)\). The denominator \(\gamma_j + \rho \lambda_i\) is crucial: when an eigenvalue \(\lambda_i\) of \(\Sigma_{\text{new}}\) along some direction is very small (a narrow axis), or \(\gamma_j\) of \(\Sigma_{\text{old}}\) is very small (a low-variance direction in the old space), the penalty term \(\rho \lambda_i\) and the corresponding term in the denominator naturally restrict the mapping amplitude along that direction, achieving anisotropic shrinkage. This essentially means: do not attempt to map old features across large distances in "dimensions where the data provides almost no information"—any estimation error in those dimensions will be heavily amplified by the subsequent Mahalanobis evaluation.

AGA is utilized as a zero-gradient prior during training: its parameters are determined by the closed solution and do not participate in gradient propagation, but the optimization of the backbone and the residual network is calibrated against it. Moreover, because \(f_t\) continuously changes during SGD, the joint distribution of \((z_{\text{old}}, z_{\text{new}})\) also evolves. Consequently, AGA updates its instantaneous closed solution every \(\Delta\) steps using the current paired features and is smoothed via EMA to yield a stable mapping baseline.

2. Topology-Aware Evolution: Forward Residual Calibration + Backward Manifold Anchoring

AGA only models global linear drift, whereas backbone evolution introduces local non-linear deformations that cannot be captured by a simple affine mapping. Topology-Aware Evolution constrains backbone evolution within a topology-preserving range through a pair of loss terms \(\mathcal{L}_{\text{top}}\).

The first component is forward residual calibration. The adapter \(A_t\) is decomposed into a linear AGA part and a lightweight MLP \(g(z; \theta_t)\): \(A_t(z) = \bar{P}_t z + \bar{b}_t + g(z; \theta_t)\). During training, the objective of \(g\) is to fit the "residual shift" of \(z_{\text{new}}\) minus the AGA prior: \(||g(z_{\text{old}}) - \text{stopgrad}(z_{\text{new}} - \bar{A}_t^{\text{lin}}(z_{\text{old}}))||^2_2\). The \(\text{stopgrad}\) operator prevents the residual calibration from backpropagating constraints onto the plasticity of the backbone: \(g\) catches up with changes in the current feature space without feeding this pursuit back to \(f_t\). In this way, the linear prior carries the global rigid drift while the MLP absorbs local flexible deformations, dividing the labor.

The second component is backward manifold anchoring. This is crucial for preventing topological degradation: since directly penalizing the Euclidean distance between old and new features would limit plasticity, a more ingenious constraint is employed—training an auxiliary distiller \(D\) to reconstruct \(z_{\text{old}}\) from \(z_{\text{new}}\): \(||D(z_{\text{new}}) - z_{\text{old}}||^2_2\). Note that while \(D\) itself is optimized, its gradients backpropagate to \(f_t\). This means that the backbone is free to learn discriminative features for new classes but cannot distort the manifold in a way that renders old features "unrecoverable". If updates to \(f_t\) lead to topological degradation—e.g., if two old classes that were originally close are pulled far apart in the new space—then the distiller will fail to accurately recover \(z_{\text{old}}\) from \(z_{\text{new}}\), and the loss will penalize this deformation. This essentially utilizes the reconstruction capability of a companion (oracle) as a proxy metric for manifold integrity.

The combined \(\mathcal{L}_{\text{top}}\) from both components, along with cross-entropy and anti-collapse loss (introduced by AdaGauss to prevent Gaussian distributions of different classes from collapsing together), constitutes the total training objective: \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \lambda_{\text{top}} \mathcal{L}_{\text{top}} + \mathcal{L}_{\text{ac}}\).

3. EMA Smoothing and Periodic Prior Refreshing

The SGD evolution of \(f_t\) implies that the joint distribution of \((z_{\text{old}}, z_{\text{new}})\) changes after every mini-batch. If the Sylvester equation is re-solved based on the instantaneous distribution in every step, the estimation variance of \(P_t\) will be large, which interferes with training. This paper's strategy is: every \(\Delta\) steps (with \(\Delta = 10\) epochs in experiments), paired features are sampled from the current batch to compute an instantaneous GLS estimate \((\hat{P}_t, \hat{b}_t)\), which then updates the EMA using momentum \(m\): \(\bar{P}_t \leftarrow m \bar{P}_t + (1-m) \hat{P}_t\). Between two refreshes, the adapter computes \(\mathcal{L}_{\text{top}}\) using the stable and smooth EMA prior. This design provides a steady geometric baseline for backbone optimization, avoiding training instability caused by high-frequency jitter of the prior.

Loss & Training

The total loss comprises three components:

• \(\mathcal{L}_{\text{CE}}\): Cross-entropy loss on the new classes of the current task, which drives the backbone to learn discriminative features for new categories.
• \(\mathcal{L}_{\text{top}}\): Topology-aware loss containing forward residual calibration (which makes MLP \(g\) fit the residue of \(z_{\text{new}}\) minus the AGA prior) and backward manifold anchoring (where the distiller reconstructs \(z_{\text{old}}\) from \(z_{\text{new}}\)), both of which are L2 losses.
• \(\mathcal{L}_{\text{ac}}\): Anti-collapse loss (the original from AdaGauss), which prevents various class Gaussian distributions from collapsing together in the feature space.

The hyperparameter \(\lambda_{\text{top}}\) controls the strength of the topological constraint. Training utilizes the SGD optimizer with a ResNet-18 backbone, with the 512-dimensional feature space projected to a 64-dimensional prototype space, and is conducted for 200 epochs on CIFAR-100/TinyImageNet/ImageNet-100. The AGA refresh interval is \(K=10\) epochs, sampling approximately 1 epoch's worth of steps at each refresh, incurring an additional computational overhead of around 6.7% (theoretical analysis shows that the overhead primarily stems from two extra forward passes, whereas solving the GLS matrix equation over the low-dimensional covariance has near-zero cost).

Key Experimental Results

Main Results

This paper evaluates the proposed method against 11 baseline methods including EWC, LwF, SDC, PASS, FeTrIL, FeCAM, EFC, ADC, LDC, AdaGauss, and DPCR across four benchmarks: CIFAR-100, TinyImageNet, ImageNet-100 (from scratch), and CUB-200 (pretrained backbone). Below are the main results on CIFAR-100 and TinyImageNet (from scratch):

Method	CIFAR-100 T=10 Alast	CIFAR-100 T=10 Ainc	TinyImageNet T=10 Alast	TinyImageNet T=10 Ainc
AdaGauss (NeurIPS24)	46.8	60.9	32.9	45.8
DPCR (ICML25)	50.2	62.8	34.3	46.9
Ours	52.0	64.4	37.2	50.6

Under a longer task sequence (\(T=20\)), the advantage is even more pronounced: on CIFAR-100 \(T=20\), our \(A_{\text{last}}\) of 43.0 significantly outperforms DPCR's 39.8 (+3.2 pp); on TinyImageNet \(T=20\), our \(A_{\text{last}}\) of 31.5 outperforms the runner-up EFC's 28.4 (+3.1 pp). On ImageNet-100, our method also achieves the highest \(A_{\text{last}}\) (53.2 for \(T=10\), 44.7 for \(T=20\)). On CUB-200, which has a pretrained backbone resulting in minor representation drift, our method performs on par with AdaGauss, while EFC is slightly superior in this setting.

Ablation Study

Configuration	CIFAR-100 T=10 Alast	Description
w/o Top-Aware + w/o AGA + FT	46.8	Equivalent to AdaGauss: Decoupled training + post-processing adaptation
w/ Top-Aware + w/o AGA + FT	50.3	Topology constraints only, showing pronounced improvement
w/ Top-Aware + w/ AGA + FT	51.7	Both components enabled
w/ Top-Aware + w/ AGA + w/o FT	52.0	Full framework, performing even better without post-processing fine-tuning

Key Findings: - Independent Contribution of AGA: Comparing "w/ Top-Aware + w/o AGA" (50.3) with "w/ Top-Aware + w/ AGA + w/o FT" (52.0), AGA brings a +1.7 pp gain. - Post-processing Fine-tuning is Unnecessary: The adapter coupled during training is directly usable post-training; fine-tuning might instead disrupt the consistency established during training (it decreases from 52.0 to 51.7 on CIFAR-100 when incorporating fine-tuning). - TinyImageNet \(T=20\) is an Exception: The drift accumulated over long task sequences makes isolated fine-tuning slightly beneficial (31.7 vs 31.5), though the margin is minimal. - Prototype Drift Analysis: Tracking the drift distance between the old-class mean features and the transported prototypes, the full framework significantly outperforms AdaGauss and the topology-only variant, exhibiting the most pronounced suppression effect on earlier tasks where drift accumulation is larger. - Mahalanobis Tail Errors: Along directions with decreasing eigenvalues (narrow axes), the suppression effect of AGA intensifies as eigenvalues decrease—consistent with the spectral analysis of anisotropic shrinkage in Theorem 3.1. - Boundary of Old Classes: After incorporating AGA, the margin distribution of the old classes systematically shifts to the right (more positive values), meaning old classes are less likely to be misclassified as new classes.

Highlights & Insights

• Crucial Paradigm Shift: From Post-processing Repair to Training-time Constraints. The most fundamental contribution of this paper is not designing a better adapter architecture, but migrating the "feature transport" task from decoupled post-processing steps into the main loop of backbone training. This is analogous to transforming "correcting mistakes post-hoc" to "maintaining constraints during evolution"—which is more intuitive in continual learning scenarios: the backbone should be aware of the evaluation it will face next.

• Sylvester Equation as a Geometric Prior. Solving the cross-space mapping as a closed-form solution of weighted GLS cleverly exploits the geometric meaning of the Mahalanobis norm—errors along low-variance directions are naturally penalized more heavily. The spectral form \(X_{ij} = \dots / (\gamma_j + \rho \lambda_i)\) intuitively demonstrates the mechanism of anisotropic shrinkage, presenting an elegant design.

• Surrogating Topology Preservation via Reversibility. Backward manifold anchoring measures topological integrity by whether \(z_{\text{old}}\) can be reconstructed from \(z_{\text{new}}\), rather than supervising feature distances directly. This is an indirect yet effective design: it allows the backbone to move feature points freely while prohibiting "irreversible" deformations (such as dimensional collapse and inter-class merging).

• EMA Refreshing of AGA as a Key Engineering Detail. If AGA is solved only once at the start of training and kept fixed, the prior will quickly become outdated as the backbone evolves. If it is re-solved at every step, high-variance fluctuations will interfere with training. The scheme of refreshing every 10 epochs paired with EMA smoothing nicely balances both ends, incurring an extra overhead of only 6.7%.

Limitations & Future Work

• Limitations of the Linear Prior. AGA is a global affine mapping. Although it can model local non-linearities when combined with the residual MLP, for highly non-stationary or heavily non-linear representation drift (e.g., where the feature space undergoes massive rotation or dimensional redistribution), the accuracy of the linear approximation might be insufficient.
• Theoretical Guarantees Tied to Gaussian Assumptions. The margin stability bound in Theorem 3.2 depends on the class-conditional Gaussian assumption and the non-singularity of the covariance. When covariance estimation is noisy or close to singular, actual performance may deviate from theoretical guarantees.
• Diminished Advantages with Pretrained Backbones. On CUB-200, representation drift is inherently mild. Our method shows no significant superiority over AdaGauss and is even outperformed by EFC, illustrating that complex geometric constraints yield diminishing returns when drift is minimal.

Related Work & Insights

• vs DPCR (ICML 2025): DPCR is the top-performing post-processing method, reconstructing the classifier with dual projection and ridge regression after training the backbone. The core difference between our method and DPCR lies not in the mapping form, but in the timing: our method protects the old-class manifold using geometric constraints during the training process, rendering the subsequent statistical push inherently more reliable, rather than repairing the space after it has already been distorted.
• vs AdaGauss (NeurIPS 2024): AdaGauss introduces class-conditional covariance and anti-collapse losses, but transport is still executed via decoupled post-processing. This paper directly inherits the Gaussian statistical bookkeeping and anti-collapse losses from AdaGauss, augmenting them with geometric anchoring and topology-aware training—meaning our ablation studies can cleanly isolate the contribution of the transport mechanism.
• vs EFC / LDC: These methods also employ auxiliary neural networks to map old prototypes but are likewise trained in a decoupled paradigm, optimizing Euclidean objectives rather than Mahalanobis geometry-sensitive objectives. Our AGA prior provides the geometric directionality that these methods lack.

Rating

• Novelty: ⭐⭐⭐⭐ Shifting feature transport from post-processing to training-time constraints is a simple yet powerful paradigm shift, and utilizing the Sylvester equation for geometric anchoring is elegant. Overall, it is an incremental improvement but with deep insights.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thoroughly evaluated across 4 benchmarks \(\times\) 2 task configurations \(\times\) 5 repetitions. The ablation and diagnostic analyses (prototype drift, Mahalanobis tail, decision margins, old-new confusion rate) are highly systematic. Quantitative analysis of computational overhead is also provided.
• Writing Quality: ⭐⭐⭐⭐ The theoretical derivations (Sylvester equation + margin stability theorem) are rigorously formulated, and the logical progression of ablation and diagnostic experiments is clear. The Introduction provides a spot-on dissection of the "structural flaws of decoupled paradigms". The topology visualization in Fig. 1 is intuitive and convincing.
• Value: ⭐⭐⭐⭐ Provides the EFCIL community with a generalizable design principle of training-time geometry anchoring, which does not rely on specific post-processing architectures and has the potential to migrate to more continual learning settings.

Related Papers

• [ECCV 2024] ABC Easy as 123: A Blind Counter for Exemplar-Free Multi-Class Class-Agnostic Counting
• [ICLR 2026] Consistency-Driven Calibration and Matching for Few-Shot Class Incremental Learning
• [CVPR 2026] Drainage: A Unifying Framework for Addressing Class Uncertainty
• [ICLR 2026] Random Anchors with Low-rank Decorrelated Learning: A Minimalist Pipeline for Class-Incremental Medical Image Classification
• [ECCV 2024] An Incremental Unified Framework for Small Defect Inspection