Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection

Conference: ECCV 2026
Paper: ECCV Official Page
Project: SCL-MGSM
Area: Model Compression
Keywords: class-incremental learning, guided random projection, analytic learning, first-session adaptation, numerical stability

TL;DR

SCL-MGSM uses first-session labels and residuals to select blocks of random projection bases, then freezes the representation and recursively updates a ridge-regression classifier, improving expressivity and numerical stability in exemplar-free class-incremental learning and reaching 77.12% final accuracy on ImageNet-R B-0 Inc-10.

Background & Motivation

Pretrained models already provide useful visual features, so exemplar-free class-incremental learning need not retrain a backbone for every new task. Methods such as RanPAC adapt a pretrained model on the first session, freeze it, expand its features through a random projection layer, and maintain a classifier through analytic updates. With a fixed representation, the contribution of historical data to ridge regression can be compressed into accumulated statistics rather than stored images; inference also requires no task ID.

The random projection layer remains a weak point. Under a substantial gap from the pretraining domain, unguided random directions may fail to expose the features needed for downstream classification. Increasing projection width can improve separability, but may introduce correlated directions, ill-conditioned feature matrices, and amplified errors in finite-precision recursive computation. LoRanPAC adds stabilization for large projections. This paper instead asks whether constructing better bases can reduce reliance on wider layers and subsequent compensation.

First-session adaptation suggests that a small set of initial classes can supply useful inductive bias for later tasks, motivating the extension of data guidance from the backbone to the projection layer. However, directly adopting the unit-wise greedy selection of stochastic configuration networks, SCSM, may over-specialize to the first session. Core Idea: select random bases with a ridge-regularized block-wise residual criterion that accounts for existing bases, allowing candidate groups to jointly explain the remaining targets rather than accumulating arbitrary directions or optimizing each direction greedily.

Method

Overall Architecture

The input is a sequence of labeled image datasets with disjoint class sets; only current-session data are accessible at each stage. SCL-MGSM stands for Stochastic Continual Learner with MemoryGuard Supervisory Mechanism. Its central construction takes place only during initialization: optionally perform first-session adaptation, freeze the backbone, and progressively build a random projection layer using first-session features and labels.

After construction, both the backbone and projection layer remain fixed. Later tasks only expand the classifier's class outputs and update its analytic statistics; inference uses the fixed feature mapping to produce scores for all observed classes. MemoryGuard is neither an exemplar buffer nor a mechanism that reselects projection bases at every stage.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Initial["First-session images"] --> Backbone["Optional first-session adaptation<br/>then freeze backbone"]
    Backbone --> Sampling["Multi-scale random candidates"]
    Sampling --> Selection["Block-wise residual selection"]
    Labels["First-session labels"] -.-> Selection
    Selection --> Readout["Readout refitting and stopping"]
    Readout -->|Residual above threshold| Sampling
    Readout -->|Construction complete| Frozen["Freeze random projection layer"]
    Later["Later labeled data"] --> Update["Analytic updates with fixed features"]
    Frozen --> Update
    Update --> Output["Scores for all observed classes"]

The loop operates only during first-session construction; the dashed label edge denotes construction supervision. Later data are encoded by the same frozen backbone and projection layer before analytic updates. Test images use the fixed representation and current classifier without candidate selection or labels.

Key Designs

1. Multi-scale random candidates: change the search distribution, not just the number of directions

Candidate weights and biases are sampled from a Gaussian distribution, and a nonlinear activation converts backbone features into new feature columns. The following standardized notation expresses this mapping:

\[ H = g(ZW + \mathbf{1}b^{\top}),\qquad w_i,b_i\sim\mathcal{N}(0,\xi^2). \]

Here, \(Z\) contains first-session backbone features, and each column of \(H\) is a candidate random basis evaluated on all first-session examples. Each iteration generates up to \(B_{\max}\) candidate blocks, each containing \(s\) units. The algorithm therefore evaluates whether a small group of basis functions jointly contributes useful directions, rather than whether one unit best matches the current residual.

If no block passes selection at the current scale, construction switches to the next \(\xi\) in a discrete scale set and resamples. Changing the scale changes the response regions of the nonlinear features, helping the search escape the restricted function space of a fixed distribution. This does not involve learning candidate weights through gradients or adapting the sampling distribution throughout later tasks: the search belongs to first-session construction, and accepted parameters are subsequently frozen.

2. Block-wise residual selection: measure joint contributions beyond the existing representation

The difference between labels and predictions from the current ridge-regression readout is the multi-output residual. For each candidate block, MGSM evaluates its correlation with that residual and uses a ridge-regularized Schur complement to account for correlations with existing bases. Intuitively, a block correlated with the targets can still add little if existing features already represent nearly the same directions. Selection therefore goes beyond ranking raw target correlations.

Theorem 1 gives a single-output block acceptance condition, applied to each output column for multiclass learning. Provided the condition holds and the ridge readout is recomputed, the authors prove continued residual contraction and convergence to zero as the number of hidden units tends to infinity; the contraction parameter satisfies \(0<r<1\). Each iteration selects the largest improvement among admissible blocks. Unlike SCSM's unit-wise greedy alignment, MGSM can admit directions that are not individually optimal but work well together.

The guarantee concerns fitting the initialization data and depends on continuing to find admissible candidates. It is neither a generalization guarantee for unseen classes nor a universal condition-number bound under arbitrary input distributions. Operators and exponents are missing from cached Eqs. (12) and (15), so this note does not present a guessed Schur complement or acceptance inequality as the authors' exact formula. An implementation must still check the original equations and Appendix A.

3. Readout refitting and stopping: let residual reduction determine projection width

Once a block is accepted, its response columns are appended to the feature matrix, the readout is refitted, and the updated residual determines whether construction continues. The objective is standard multi-output ridge regression, expressed below in normalized notation based on the surrounding text rather than copied from the damaged equation layout:

\[ W_{\beta}^{*}=\operatorname*{arg\,min}_{W_{\beta}} \left\|Y-HW_{\beta}\right\|_F^2+\lambda\left\|W_{\beta}\right\|_F^2. \]

Here, \(Y\) is the one-hot label matrix and \(\lambda\) is the ridge coefficient. Jointly refitting the readout matters because adding directions changes how much of the targets existing directions need to explain; fitting weights only for the new units would not implement the stated procedure. MGSM construction relies on forward computation and analytic solutions, not back-propagation into the random projection weights.

Construction stops when the Frobenius norm of the multi-output residual is at most \(\varepsilon\), yielding a final width \(L^{*}\). Main experiments determine width adaptively per dataset, whereas the mechanism comparison fixes it at 10,000 to eliminate width as a confounder. Dataset-specific final widths are deferred to Appendix C.2, which is absent from the available cache. Consequently, 10,000 must not be reported as the final width of every main configuration.

4. Analytic updates with fixed features: retain statistics rather than historical images

During class-incremental learning, every task uses exactly the same feature mapping. To explain why replay is unnecessary, the paper's recursive ridge regression can equivalently be written as accumulated normal equations:

\[ P_t=\lambda I+\sum_{k=1}^{t}H_k^{\top}H_k,\qquad Q_t=\sum_{k=1}^{t}H_k^{\top}Y_k,\qquad P_tW_{\beta}^{(t)}=Q_t. \]

Here, \(Q_t\) is explanatory notation for the equivalence; the paper describes recursively maintaining \(P_t\) and the preceding classifier. When classes arrive, existing classifier outputs and historical label statistics have zero entries for the new classes, while new examples contribute their statistics. With a fixed representation and exact arithmetic, this matches joint ridge regression over all observed data in the same feature space, not joint fine-tuning of the entire backbone.

The practical implementation of this equivalence is where numerical issues arise: a very wide, collinear projection makes solving the system more sensitive. MGSM seeks more useful bases, avoiding unnecessary width and improving correlations among bases so that updates do not need excessive regularization merely to remain stable. Exemplar-free operation still has a memory cost: \(P_t\) is an \(L^{*}\times L^{*}\) matrix, and the classifier grows with the number of observed classes. No historical examples does not mean no historical state or constant total memory.

A Worked Example

ImageNet-R B-0 Inc-10 divides 200 classes into 20 sessions of 10 classes each. The first session obtains backbone features from its 10 classes; with FSA, the backbone is adapted before freezing. MGSM then repeatedly proposes random blocks, compares their residual improvements, appends accepted blocks, and refits the readout until the stopping criterion is met.

When the second session arrives, the projection layer is not rebuilt and first-session images are not retrieved. Instead, the same mapping encodes the new 10 classes, and statistics and classifier outputs expand to cover the 20 observed classes. Evaluation after the final session covers all 200 classes and gives the \(A_{\mathrm{last}}\) reported in Table 1. This example explains the actual protocol and state transitions; it does not imply that the paper reports the number of accepted blocks in each construction iteration.

Loss & Training

The default backbone is ViT-B/16-IN21K. First-session adaptation and related ablations use AdaptFormer, SSF, and VPT. Projection construction and later classifier updates center on the ridge-regression readout above, whereas optional FSA is an earlier backbone-adaptation stage.

Shared settings in the main text are \(\varepsilon=0.01\), the adaptive scale range \(\xi\in[0.0008,0.004]\), and step size \(\Delta\xi=0.0001\) across the four main benchmarks. Stability analysis illustrates MGSM with \(\lambda=0.1\) against more strongly regularized random initialization, but this does not establish that every main experiment uses that value. Other hyperparameters are deferred to Appendix C.1, absent from the available cache.

Key Experimental Results

Main Results

The main text covers ImageNet-R, ImageNet-A, and ObjectNet, each with 200 classes, plus OmniBenchmark with 300 classes. \(A_{\mathrm{last}}\) is accuracy over all observed classes after the final session; \(A_{\mathrm{avg}}\) averages seen-class test accuracy across sessions. B-0 denotes equal splits, and Inc-10 adds 10 classes per session, giving 20 sessions for the first three datasets and 30 for OmniBenchmark.

The following subset of original Table 1 uses B-0 Inc-10 throughout. Values are percentages, reported as means and standard errors over 3 random seeds. The strongest comparator is selected separately for each metric, so the two comparator methods within a row can differ.

Dataset	Ours, final accuracy	Best comparator, final accuracy	Ours, average accuracy	Best comparator, average accuracy
ImageNet-R	77.12 ± 0.21	APT: 75.07 ± 0.32	82.23 ± 0.18	EASE: 80.38 ± 0.60
ImageNet-A	56.05 ± 0.32	AnaCP: 55.22 ± 0.39	65.29 ± 0.47	LoRanPAC: 64.16 ± 0.55
ObjectNet	66.45 ± 0.45	AnaCP: 64.68 ± 0.17	75.30 ± 2.03	AnaCP: 72.81 ± 0.94
OmniBenchmark	79.90 ± 0.35	AnaCP: 78.81 ± 0.24	86.92 ± 0.73	AnaCP: 86.45 ± 0.71

Final and average accuracy improve by 2.05 and 1.85 percentage points on ImageNet-R, and by 1.77 and 2.49 percentage points on ObjectNet. Higher means do not establish statistical significance for every comparison, especially given the larger standard error of average accuracy on ObjectNet.

Ablation Study

Original Table 2 removes FSA and fixes all three projection constructions at a final width of 10,000, with recursive ridge regression in subsequent sessions. This isolates the selection rule more directly than the cross-method main comparison. The table below retains final accuracy; subtracting these values from differently configured Table 1 results would not isolate the contribution of FSA.

Dataset	MGSM final accuracy	SCSM final accuracy	Random initialization RI final accuracy
ImageNet-R	70.09 ± 0.13	64.78 ± 0.09	63.00 ± 0.22
ImageNet-A	53.63 ± 0.74	44.85 ± 1.47	47.31 ± 0.43
ObjectNet	59.61 ± 0.30	55.46 ± 0.33	55.91 ± 0.14
OmniBenchmark	77.99 ± 0.17	71.43 ± 0.11	74.91 ± 0.04

On ImageNet-A, MGSM exceeds RI by 6.32 percentage points and SCSM by 8.78 percentage points, while SCSM underperforms RI. Supervised selection of random bases is therefore not sufficient by itself: whether the selection constraint over-specializes to the initial task also matters.

Key Findings

• In the ImageNet-R B-0 Inc-5 analysis of Table 3, fixing \(B_{\max}=10\) and increasing block size from \(s=10\) to \(s=100\) reduces time from 192.21 ± 2.64 seconds to 70.37 ± 1.59 seconds, while average accuracy changes from 75.31 ± 0.21% to 74.86 ± 0.21%. This is a local trade-off of approximately 2.73 times faster computation for 0.45 percentage points, not the same speedup for the entire training pipeline.
• In Figure 8, MGSM has a condition number comparable to RI and clearly better than SCSM; the evidence does not show that MGSM also substantially outperforms RI on this measure. Basis similarities in Figure 7 and statistic norms in Figure 6 offer related mechanism evidence, not a stability theorem for all future tasks.
• With DINO-v2, Table 4 reports ImageNet-A Inc-5 final accuracy of 71.60 ± 0.42%, above AnaCP's 71.05 ± 0.72%, but average accuracy of 78.66 ± 0.90%, below AnaCP's 79.45 ± 0.72%. Cross-backbone benefits should not be summarized as winning every metric.

Highlights & Insights

• First-session adaptation is extended from the backbone to random feature construction. A frozen representation need not be unguided: labels can select a feature space that subsequently remains fixed for long-term use.
• Block-wise selection allows multiple directions to explain residuals jointly. It avoids the local preference of unit-wise greedy selection, while Table 3 demonstrates the practical construction-time benefit of larger blocks.
• Expressivity and analytic-solver stability are treated as a shared design problem. For systems that accumulate statistics over time, feature quality affects both classification boundaries and the reliability of subsequent linear algebra.

Limitations & Future Work

• First-session guidance depends on shared statistical structure between initial classes and future tasks. Non-overlapping initial categories across three seeds are useful evidence, but do not establish robustness to strong cross-session domain shifts or unusual class orders; deliberately unrepresentative initialization tasks would provide a stronger test.
• Infinite-width residual convergence does not guarantee admissible candidates under a finite budget or generalization to unseen classes. Deployment studies should report sampling counts, the frequency of reaching the threshold, and realized final widths.
• A smaller \(\|P_t\|_F\) alone proves neither a smaller condition number nor a smaller solution error. More direct checks would report spectra, deviations between recursive and batch ridge solutions, and errors across floating-point precisions.
• The cache contains only the main text and references, not the cited Appendices A-C. The exact criterion, full hyperparameters, final widths, and additional forgetting analyses cannot be completely verified from this cache. The conclusion claims seven benchmarks, while the main experimental section explicitly lists four; this note limits its data conclusions to the four inspected benchmarks.

Related Work & Insights

• vs RanPAC / ACIL / G-ACIL: These methods share fixed random features and analytic classifiers. The change here concerns feature-basis initialization, not training a new backbone for every session or retaining replay images.
• vs LoRanPAC: LoRanPAC increases random feature dimensionality and addresses the resulting solver-stability issues, whereas this paper prioritizes more effective bases. The approaches are not theoretically incompatible, but this paper does not establish equal accuracy against every possible extremely wide configuration.
• vs SCSM: Both are data-guided constructive networks; MGSM instead uses a regularized block-wise selection criterion. Table 2 demonstrates an important distinction between initialization-task fitting preferences and cross-session performance.
• Research direction: Use low-order statistics of later tasks to detect mismatch in a fixed projection without storing examples, then explore constrained basis expansion. However, changing the old feature mapping requires maintaining consistency of historical statistics; the joint-equivalence result for fixed representations cannot simply be carried over.

Rating

• Novelty: 4/5. Applies supervised stochastic configuration to pretrained class-incremental learning with redesigned block selection and a focus on stability.
• Experimental Thoroughness: 4/5. Four main-text benchmarks, multiple protocols, fixed-width controls, and DINO-v2 experiments provide substantial coverage; appendix results were not verified in the available cache.
• Writing Quality: 3/5. Motivation and workflow are clear, but cached equations are damaged and the benchmark count differs between the main text and conclusion.
• Value: 4/5. Offers a quality-first alternative to expanding random feature layers, relevant to exemplar-free learning and the cost of analytic updates.