Shared LoRA Subspaces for almost Strict Continual Learning

Conference: ECCV2026
Paper: ECCV Original Paper
Project: https://toshi2k2.github.io/share/
Area: Model Compression
Keywords: continual learning, low-rank adaptation, shared subspaces, analytical reprojection, replay-free learning

TL;DR

Share integrates incoming data or LoRA adapters into an evolving shared basis and replaces separate adapters with lightweight task coefficients, reaching 94.20% final accuracy in replay-free CIFAR-100 class-incremental learning; however, its general formulation retains task coefficients and Share-full permits old-data access, so not all results represent strict continual learning.

Background & Motivation

LoRA factorizes a large model's weight update into two low-rank matrices, reducing the cost of individual fine-tuning runs without automatically solving continual learning. Sequentially updating one adapter can overwrite earlier capabilities, whereas retaining an independent LoRA for each task exchanges forgetting for growing storage and switching overhead. Methods such as O-LoRA exploit orthogonality to reduce interference, but separating knowledge does not necessarily accumulate the structure that different tasks share.

The paper starts from the shared weight subspace hypothesis: LoRA factors for different tasks on the same pretrained backbone may contain reusable principal directions. EigenLoRAx has already demonstrated the value of extracting such a subspace from available adapters. In deployment, however, only one initial adapter may exist, with further data or adapters arriving later. The challenge is therefore not one-shot compression of a complete adapter collection, but updating shared representations without historical data while preventing a change of coordinates from damaging earlier task representations.

Core Idea: represent long-term knowledge through shared principal directions and lightweight task coefficients, briefly train a few directions for each new task, then update the shared basis with SVD and analytically transform old coefficients into the new coordinate system.

Method

Overall Architecture

Share addresses parameter-efficient continual fine-tuning: pretrained weights remain frozen, and the input stream can contain either labeled task data or previously trained LoRA adapters. Each adapted layer maintains two shared bases corresponding to the two LoRA factors; the general task formulation also retains task-specific coefficients in both bases. Thus, a single shared model means sharing the main factors, not eliminating every task-specific parameter in the general setting.

The process has three stages: shared-basis initialization, temporary-direction adaptation, and SVD merging and reprojection. Incoming data first trains new directions and then enters merging; incoming adapters enter merging directly, without gradient updates on old samples. This also enables asynchronous adapter streams: externally learned knowledge enters a common representation through weights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Initial["Initial LoRA or data"] --> Init["Shared-basis initialization"]
    Init --> Adapt["Temporary-direction adaptation"]
    Data["New task data"] --> Adapt
    Adapt --> Merge["SVD merging and reprojection"]
    Adapter["New LoRA adapter"] --> Merge
    Merge --> Output["Updated bases and coefficients"]
    Output -->|Next data task| Adapt

Key Designs

1. Shared-basis initialization: replace full factors with coordinates in shared directions

For a frozen weight layer, ordinary LoRA uses \(B\in\mathbb{R}^{n\times r}\) and \(A\in\mathbb{R}^{r\times d}\) to represent the update. Share stacks the available tasks' \(B\) factors column-wise and arranges the \(A\) factors correspondingly; the main text describes centering followed by separate SVDs, retaining the top \(k\) directions. This operation is performed independently for each layer, rather than imposing one global matrix on the entire network. With available adapters, initialization reads weights without data or gradients; otherwise, a LoRA adapter is first trained on the initial task.

To avoid the corrupted Greek symbols in the text cache, the following uses \(U_B,U_A\) for the shared bases and \(C_{B,t},C_{A,t}\) for task coefficients, expressing the product structure of the paper's Eq. (1):

\[ h_t=W_0x+(U_BC_{B,t})(U_AC_{A,t})^{\top}x, \quad U_B\in\mathbb{R}^{n\times k},\ U_A\in\mathbb{R}^{d\times k},\ C_{B,t},C_{A,t}\in\mathbb{R}^{k\times p}. \]

Here \(k\) is the number of reusable directions, whereas the pseudo-rank \(p\) controls the number of columns in reconstructed factors; they are different hyperparameters. With frozen shared bases, the two coefficient matrices contain \(2kp\) parameters per layer, compared with \(r(n+d)\) for ordinary LoRA factors; this is a dimensional count, not compression of the backbone itself. Because the shared directions encode structure from earlier tasks, new tasks can combine existing knowledge instead of searching an unrestricted weight-update space. However, coefficient-only training remains confined to the existing subspace, motivating a separate mechanism for learning new directions.

2. Temporary-direction adaptation: change the subspace without directly rewriting the long-term bases

When new data arrives, Share copies a small number of leading directions from the existing shared bases to initialize temporary factors, then randomly initializes their coefficients. The notation \(q\) below denotes the number of temporary directions, corresponding to the paper's temporary-expansion hyperparameter; it is smaller than \(k\). Training updates these temporary directions and their coefficients, while retaining the long-term shared bases for the subsequent merge. The temporary path itself forms the new task's low-rank update; it should not be interpreted as permanently adding a complete old adapter to a new adapter.

This stage trains \(q(n+d+2p)\) parameters per layer, counting both temporary factors and both coefficient matrices. It is more flexible than coefficient-only tuning because copied directions can move outside the old subspace to capture previously unavailable task variation. It is cheaper than full LoRA because only a few temporary directions are opened, and a complete additional adapter is not retained permanently after training. If an external LoRA arrives instead of data, knowledge acquisition has already occurred elsewhere, so this gradient-training stage is skipped.

3. SVD merging and reprojection: update old coordinates whenever the basis changes

Before merging, Share reconstructs each previously seen task's two factors using the old shared bases and coefficients, then appends the new task factors to their respective stacked matrices. It applies truncated SVD separately to both stacks, replacing the long-term bases with the new top \(k\) directions. The compressed object is the collection of reconstructed old factors and new task factors, not just the new task, and the operation is not simple weight averaging. Without historical data, knowledge preservation is grounded in parameter-space reconstruction rather than prediction recalibration on earlier samples.

Old coefficients cannot simply remain unchanged after a basis update, because the same coordinates would now refer to different directions. Share therefore updates each task's coefficients through least-squares projection; for the \(B\) side, Eq. (6) can be written as:

\[ C_{B,i}^{\mathrm{new}}=(U_B^{\mathrm{new}})^{\dagger}\widehat B_i =(U_B^{\mathrm{new}})^{\top}\widehat B_i \quad\text{when }(U_B^{\mathrm{new}})^{\top}U_B^{\mathrm{new}}=I. \]

The same operation applies to the \(A\) side; the second equality requires the orthonormal columns supplied by SVD. Shared directions can therefore evolve while approximately reconstructing earlier task factors, and directions discovered through later tasks may also help earlier ones. Truncation still discards information: reprojection minimizes factor reconstruction error within the chosen subspace, not a guarantee of unchanged task predictions. The theoretical argument also relies on task alignment with the shared subspace and assumptions including a Lipschitz and strongly convex loss, so it is not a zero-forgetting guarantee for deep networks.

A Worked Example

In the paper's GLUE configuration, the original LoRA rank is \(r=32\), the shared direction count is \(k=32\), the pseudo-rank is \(p=8\), and each adaptation opens only \(q=4\) temporary directions. Initial data trains a LoRA from which the shared bases are extracted; when the second data task arrives, four leading directions on each side are copied and trained with their coefficients. After training, old coefficients reconstruct earlier factors, new factors join the stacks, 32 shared directions are selected again, and task coordinates are updated analytically. Later tasks repeat this process, permanently retaining shared directions and lightweight coefficients rather than the temporary training branches. These numbers are the reported experimental configuration, not a claim that every backbone should use identical ranks.

Loss & Training

Share primarily changes the parameterization and cross-task update procedure; the main text does not introduce a new composite loss shared across all modalities. New-data adaptation uses the relevant task's fine-tuning objective, while merging and coefficient reprojection require no back-propagation. Coefficients may be fine-tuned further after merging, but the variant that accesses previous task data is explicitly called Share-full and must be separated from replay-free Share.

Class-incremental classification has an additional implementation constraint: the paper keeps initial coefficients fixed, updates principal factors through temporary-factor learning and merging, and deploys one classifier head covering all seen classes. Consequently, selecting coefficients by task in the general formulation should not be substituted for the class-incremental test protocol. The paper also mentions an optional task classifier when separate coefficients are retained, but does not describe that option as the default basis of these single-head results.

The main-text hyperparameter summary reports that directions retaining about 60% explained variance can work, temporary direction counts in \([1,k/4]\) are effective, and \(p=1\) can suffice, with \(p=r/3\) suggested as a starting point. These are author recommendations rather than a complete quantitative ablation: the cache does not contain the cited Appendix Sec. 8.6, so missing curves or optimum values cannot be reconstructed from it.

Key Experimental Results

Main Results

Table 1 evaluates Continual GLUE with RoBERTa-base on a six-task data stream, without a pre-supplied LoRA; initial task data provides initialization. CoLA uses Matthews correlation, STS-B uses Pearson correlation, and the remaining tasks use accuracy; Avg. averages these different task metrics and is not uniformly classification accuracy. The configuration is \(r=32,k=32,p=8,q=4\); temporary adaptation grows to 450K trainable parameters, so the final coefficient count is not the peak training count.

Method (paper Table 1)	Parameters (reported convention)	Size / MB	Avg.	Comparison setting
Upper Bound	125M	500	83.90	Non-continual upper bound
LoRA (non-CL)	1.2M × 6	81.6	83.43	Six independent adapters
O-LoRA (CL)	1.2M × 6	81.6	57.50	Continual learning baseline
Share (CL)	0.012M	0.29	78.69	No historical-data replay
Share-full	0.012M	0.29	83.44	Fine-tuning with relaxed old-data access

The claimed 100-fold parameter saving compares one LoRA's 1.2M parameters with 0.012M coefficient parameters; the roughly 281-fold storage saving compares the table's 81.6 MB and 0.29 MB. These comparisons use different accounting bases and represent neither whole-model compression nor measured reductions in peak GPU memory.

Ablation Study

The following is a variant analysis extracted from Table 1, not an appendix component ablation; Share-full also relaxes data-access conditions, so the gain cannot be attributed entirely to analytical merging.

Metric (paper Table 1)	Share (CL)	Share-full	Difference (metric points)
CoLA Matthews	55.99	59.81	+3.82
MRPC accuracy	68.38	86.99	+18.61
STS-B Pearson	88.91	90.80	+1.89
Six-task Avg.	78.69	83.44	+4.75

Vision experiments provide another analytical perspective: Table 2 uses ImageNet-21K-pretrained ViT-B/16, restricts each dataset to 100 classes, and splits them into 10 tasks of 10 classes under replay-free single-head class-incremental learning. Share uses \(k=10,p=1,q=2\), trains for 10–30 epochs per task, and reports 0.10M parameters versus DAP's 0.19M. The table preserves the reported uncertainty; Acc. is final accuracy after all tasks, and lower Forg. is better.

Dataset (paper Table 2)	DAP Acc.	Share Acc.	DAP Forg.	Share Forg.
CIFAR-100	94.05 ± 1.2	94.20 ± 0.9	0.41 ± 0.1	0.40 ± 0.4
Food-100	88.37 ± 0.6	90.10 ± 1.2	0.92 ± 0.1	0.7 ± 0.9
Caltech-100	97.23 ± 0.3	97.70 ± 1.1	2.52 ± 0.8	2.18 ± 2.4
Flowers-100	96.49 ± 0.1	97.90 ± 0.7	2.28 ± 1.0	2.33 ± 1.2

Key Findings

• Share's CIFAR-100 point estimate equals the Table 2 upper bound of 94.20, without implying identical individual runs; its Flowers-100 forgetting exceeds DAP's, so the prose claim of lowest forgetting on every dataset is too broad.
• In Table 3's OOD class-incremental experiments, Share reaches final average accuracy of 81.51 on ImageNet-R and 78.76 on DomainNet, versus 75.26 and 72.82 for SD-LoRA; retained adaptation parameters are 0.10M and 0.37M, respectively.
• Table 5 uses Flux with a four-task hybrid data/adapter stream and \(r=32,k=32,p=8\), averaging 3 random seeds; final T3 CLIP score is 24.436 versus independent LoRA's 24.079, but T2 reaches 22.400 versus 22.985, showing that transfer is not uniformly positive.

Highlights & Insights

• Transforming old task coordinates whenever the basis changes is more flexible than preserving parameters unchanged. Shared representations can evolve without arbitrarily rewriting old tasks merely through a coordinate-system change.
• Data streams and adapter streams use the same merging interface. Previously trained LoRAs from distributed teams can contribute knowledge through weights without requiring their training datasets again.
• Temporary learning cost is treated separately from long-term storage. The transferable design is to briefly permit sufficient adaptation freedom and then compress into a shared representation, rather than minimizing permanent parameter counts alone.

Limitations & Future Work

• Almost strict is a meaningful qualifier: the general formulation retains lightweight coefficients that grow with task count and introduces temporary training expansion; Share-full additionally relaxes historical-data access.
• Shared-subspace assumptions have limits. If a new task mainly requires directions outside the existing basis, truncation can discard important components; the theoretical discussion explicitly acknowledges the resulting singular-value tail error.
• Replay-free GLUE's 78.69 trails Share-full's 83.44 by 4.75 average metric points, with a particularly large MRPC gap; claims of matching independent training need a variant-specific qualifier.
• Factor reconstruction is not function preservation, and sequential compression may accumulate errors. Longer and more heterogeneous streams, task-order sensitivity tests, and full peak-memory and runtime measurements are needed to establish long-term deployment benefits.
• The cache contains the main paper and references, not its cited appendix. Quantitative hyperparameter ablations, full per-timestep results, and some implementation details cannot be checked from this source; learning from scratch remains a stated future direction.

Related Work & Insights

• vs LoRA / O-LoRA: Independent LoRA stores full task factors, while O-LoRA emphasizes orthogonal subspaces to reduce interference; Share explicitly discovers and continually updates shared principal directions, expressing differences through task coefficients.
• vs EigenLoRAx: Both exploit principal subspaces of available adapters, but Share focuses on incremental updates before the entire adapter collection exists and on reconstructing earlier tasks after a basis change.
• vs EASE / prompt-based learning: These are important visual continual learning references; Share contributes cross-task factor sharing and merging, rather than a new diffusion backbone specifically for image generation.
• Practical implication: Budget shared bases, per-task coefficients, temporary training factors, and the frozen backbone separately, rather than interpreting one component's compression ratio as the entire service's resource saving.

Rating

• Novelty: 4/5. Shared subspaces are not new, but continual basis updates and analytical reprojection of earlier tasks form a clear incremental learning mechanism.
• Experimental Thoroughness: 4/5. Evaluation spans modalities and protocols, although some key evidence resides in the unavailable appendix and strict versus relaxed variants require separate interpretation.
• Writing Quality: 3/5. The main procedure is understandable, but parameter accounting and some broad superiority claims need correction through the tables.
• Value: 4/5. Relevant to multi-adapter maintenance and replay-free continual fine-tuning; deployment value still depends on task relatedness and end-to-end resource measurements.