Skip to content

LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning

Conference: ECCV 2026
Paper: ECCV page
Code: https://github.com/mapolinario94/LANCE
Area: Model Compression
Keywords: activation compression, HOSVD low-rank decomposition, on-device continual learning, catastrophic forgetting, null-space projection

TL;DR

LANCE runs a single higher-order singular value decomposition (HOSVD) before fine-tuning to calibrate a fixed low-rank projection subspace per layer activation; during training it stores only the low-rank core tensor and reprojects it in the backward pass with the same orthonormal factors, cutting peak activation memory to 1/25–1/380 of full backpropagation, and the same fixed subspaces can be partitioned across tasks into mutually orthogonal null spaces at offline calibration time, enabling on-device continual learning without replay buffers or resident task-specific matrices.

Background & Motivation

On-device learning β€” local fine-tuning on phones, IoT nodes, and microcontrollers β€” is the common precondition for privacy, personalization, and long-lifetime adaptation, and real deployments rarely need just one task: they need a model that keeps absorbing new tasks from a streaming data distribution without forgetting the old ones. Both requirements hit the same wall: backpropagation must keep the forward-pass intermediate activations in memory. For modern convolutional networks, activation memory is often 5–10Γ— the parameter memory, while a microcontroller has only a few hundred KB of SRAM β€” not enough for the weights, let alone the multiplied activations. Existing workarounds all carry a visible price: checkpointing trades memory for recomputation, reversible networks demand specially designed architectures, quantization and activation pruning introduce approximation error or hardware overhead, and bio-inspired substitutes such as forward-forward methods and local learning rules generally lose accuracy.

The low-rank line is the one that best matches the fact that activations are highly redundant β€” tensor-decomposition methods (HOSVD) can compress activation representations substantially. But they almost always recompute the decomposition during training: every step re-solves an SVD or Tucker decomposition, which is compute-heavy and hardware-unfriendly. More importantly, these methods were never considered for continual learning; re-factorizing per task leaves behind large task-specific matrices, which directly conflicts with on-device storage budgets. On the other side, the gradient-projection methods for continual learning (GPM, TRGP, SGP, CODE-CL) have demonstrated that "activations live in a stable low-rank subspace," but they must keep the per-layer memory matrices {M^(l)} resident in SRAM and project gradients online at every optimization step β€” eating back exactly the memory they just saved on activations.

This paper targets the obvious gap between the two lines: if the low-rank subspace containing the activations is stable enough to be calibrated once and reused, then forgetting mitigation does not have to happen inside every training step β€” it can be encoded once, offline, into the subspace itself, by forcing each new task into the orthogonal complement of the directions previous tasks already used. Core idea: calibrate a reusable fixed activation subspace with a one-shot HOSVD, store only the low-rank core tensor in the forward pass and reproject with the orthonormal factors in the backward pass so that gradient directions are preserved (hence updates remain valid descent directions), and extend to continual learning by constraining the mode-d factors of each new task to the null space of the previous task's memory matrix.

Method

Overall Architecture

LANCE does not change the model; it changes what backpropagation caches. In full BP, the activation tensor \(X^{(l)}\) of layer \(l\) must be kept verbatim in SRAM for the backward pass; LANCE replaces it with a much smaller core tensor \(G^{(l)}\) and projects the information back only when needed, using a set of fixed orthonormal factor matrices. The method runs in two stages. One stage runs offline before training: it estimates the covariance of calibration activations per layer, performs a single decomposition, and lets an energy threshold decide how much rank to keep per mode. The other stage runs on the target device: the forward pass projects activations into core tensors and stores them, and the backward pass reprojects the core tensors with the same factors to obtain the gradient contribution β€” note that only "which forward tensors are stored" is compressed; the backward pass itself is still computed by automatic differentiation as usual, and every layer's weights are fully updated. Continual learning adds an offline epilogue: after each task finishes, the method estimates the principal directions of the activations along the last mode (the input dimension of the layer's weights), removes the directions already occupied by previous tasks via a complementary projection, and merges the remainder into the memory matrix so that the next task's calibration can avoid them.

In one sentence, the data flow is: minibatch in, updated weights \(\theta\) out, and the only compressed quantity in between is the activation cache β€” weights, gradients, and optimizer state are untouched. Per-layer activation storage drops from \(O(\prod_{i=1}^{d} n_i)\) to \(O(\prod_{i=1}^{d} r_i + \sum_{i=1}^{d} n_i r_i)\): the first term is the core tensor, the second the factor matrices that must remain available. The factors are computed once and shared across the whole fine-tuning run, so their cost is fully amortized.

Key Designs

1. One-shot HOSVD subspace calibration: moving the per-step decomposition to before training

Methods such as ASI and HOSVD tensor decomposition already showed that activations can be represented at low rank, but they re-solve the decomposition at every training step. That overhead is unacceptable on the edge: the paper measures that recomputing the decomposition every step pushes training compute for MCUNet from essentially zero to 1.93 TFLOPs, and runs 2.1×–4.4Γ— slower than LANCE on a Raspberry Pi. LANCE moves the decomposition out of the training loop entirely: it passes \(N\) calibration minibatches through the pretrained model, maintains a recursive covariance estimate of the mode-\(i\) unfolding of each layer's activations, and finally performs one SVD of that covariance, keeping the top \(r_i\) singular vectors as the factor matrix \(U_i\) for that mode.

The rank is not set by hand; an energy threshold picks it automatically as the smallest \(r_i\) satisfying

\[\frac{\sum_{j=1}^{r_i}\sigma_j}{\sum_{j=1}^{n_i}\sigma_j} \;\ge\; \varepsilon\]

where \(\sigma_j\) is the \(j\)-th singular value of the covariance and \(\varepsilon\) defaults to 0.7. The only input to this step is a second-order statistic β€” no labels, no backward pass β€” so it can be done offline with forward passes alone, and its one-time cost is negligible once amortized over the whole fine-tuning run. What makes the design actually work is the robustness of the calibration: the paper finds that \(N=2\) calibration batches already yield a stable subspace, with accuracy nearly flat over a wide range of \(N\). The activation subspace of a pretrained model is simply very stable, which is what turns "calibrate once, reuse throughout" from a strained approximation into a reasonable engineering choice.

2. Store only the core tensor, reproject with orthonormal factors: compress activations while keeping the gradient direction

In the forward pass, each activation tensor is projected into a low-rank core tensor:

\[G^{(l)} \;=\; X^{(l)} \times_1 U_1^{(l)\top} \times_2 \cdots \times_d U_d^{(l)\top}\]

Only \(G^{(l)}\) and the fixed factors are retained; in the backward pass the weight gradient is computed from the contribution reprojected out of the core tensor. The justification comes from orthonormality: since each \(U_i\) has orthonormal columns, \(P^{(l)} := U_1U_1^\top \otimes \cdots \otimes U_dU_d^\top\) is an orthogonal projector, so the LANCE weight gradient is exactly the right-projection of the full gradient onto that subspace, i.e. \(\nabla_W \mathcal{L}_{\text{LANCE}} = \nabla_W \mathcal{L}_{\text{full}} P^{(l)}\). Two consequences follow directly: the inner product \(\langle \nabla_W \mathcal{L}_{\text{LANCE}}, \nabla_W \mathcal{L}_{\text{full}} \rangle = \lVert \nabla_W \mathcal{L}_{\text{LANCE}} \rVert_F^2 \ge 0\), so \(-\nabla_W \mathcal{L}_{\text{LANCE}}\) is a descent direction (Theorem 1); and under an \(L\)-smooth loss, the loss decreases monotonically and iterates converge to a projected stationary point whose residual gap is governed solely by the energy threshold, \(\lVert \nabla \mathcal{L}(\theta^\star) \rVert_2 \le C\sqrt{1-\varepsilon}\) (Theorem 2 and Proposition 1). That last point is what makes the compression knob usable: higher \(\varepsilon\) gives solutions closer to a true stationary point, lower \(\varepsilon\) trades accuracy for efficiency with a quantified bound. This is exactly where LANCE diverges from progressive low-rank methods such as ASI: ASI lets the subspace drift during training and keeps correcting it, whereas LANCE gives up that adaptability in exchange for zero decomposition overhead and edge-affordable latency.

3. Null-space constrained calibration and memory update: letting the fixed subspace carry forgetting mitigation

Applying LANCE to continual learning requires answering how the activation directions a previous task occupied can be kept out of the next task's way. The constraint is imposed only on the last mode (mode-\(d\), the input dimension of the layer's weights), because that is where interference enters β€” the weights multiply that mode directly. After task \(t\) finishes, the method estimates the mode-\(d\) activation covariance \(B_d^t\) from \(N_{CL}\) calibration batches and keeps only its component orthogonal to the existing memory:

\[\hat{B}_d^t \;=\; B_d^t - M^{t-1} M^{t-1\top} B_d^t\]

From the projected covariance it retains the top \(r_d\) principal directions satisfying the energy threshold \(\varepsilon_{CL}\), then orthonormalizes and concatenates them with the previous memory to form \(M^t\) (\(M^0\) is empty). When calibrating for the next task, the mode-\(d\) factor matrix is simply replaced by these null-space directions while the other modes follow Design 1 as usual. New-task activations therefore land only in the subspace orthogonal to previous tasks along mode-\(d\), and interference in that direction is removed structurally rather than suppressed softly by a loss regularizer.

The difference from the gradient-projection family is not whether orthogonal subspaces are used but where orthogonality is enforced. GPM/TRGP/CODE-CL project gradients, which must happen online, so their memory matrices stay resident in SRAM throughout training. LANCE projects activations and touches \(M\) only during offline calibration β€” updating the memory, computing the null space, producing the factors β€” while the online phase is ordinary low-rank activation BP, with no memory matrix in training memory and no replay buffer. The price is equally clear: forgetting mitigation covers only the directions expressible in the mode-\(d\) subspace; the other modes, the nonlinearities, and BN statistics remain unconstrained.

Loss & Training

LANCE introduces no additional loss term; the training objective remains the standard classification loss, and what changes is the cached object and the subspace constraint. Calibration hyper-parameters: \(N=100\) calibration minibatches and energy threshold \(\varepsilon=0.7\) for fine-tuning, with 50 fine-tuning epochs; the continual learning setting has its own calibration batch count \(N_{CL}\) and threshold \(\varepsilon_{CL}\). The continual learning protocol is: the first task is trained unconstrained with full BP; each subsequent task is initialized from the previous task's model and trained under LANCE's subspace constraint, using multi-head networks and no replay samples at all. Both memory update and null-space calibration are performed offline, so they do not affect the cost of on-device fine-tuning.

Key Experimental Results

Main Results

Continual learning results are below (ACC is average final accuracy over all tasks, BWT quantifies forgetting of earlier tasks as new ones arrive, Mem is activation storage; mean over five trials). Multitask denotes the upper bound where all tasks are trained jointly.

Method Split CIFAR-100 ACC / BWT / Mem (MB) Split MiniImageNet ACC / BWT / Mem (MB) 5-Datasets ACC / BWT / Mem (MB)
Multitask (upper bound) 79.58 Β± 0.54 / βˆ’ / βˆ’ 69.46 Β± 0.62 / βˆ’ / βˆ’ 91.54 Β± 0.28 / βˆ’ / βˆ’
EWC 68.80 Β± 0.88 / βˆ’2 Β± 1 / βˆ’ 52.01 Β± 2.53 / βˆ’12 Β± 3 / βˆ’ 88.64 Β± 0.26 / βˆ’4 Β± 1 / βˆ’
HAT 72.06 Β± 0.50 / 0 Β± 0 / βˆ’ 59.78 Β± 0.57 / βˆ’3 Β± 0 / βˆ’ 91.32 Β± 0.18 / βˆ’1 Β± 0 / βˆ’
A-GEM 63.98 Β± 1.22 / βˆ’15 Β± 2 / βˆ’ 57.24 Β± 0.72 / βˆ’12 Β± 1 / βˆ’ 84.04 Β± 0.33 / βˆ’12 Β± 1 / βˆ’
ER_Res 71.73 Β± 0.63 / βˆ’6 Β± 1 / βˆ’ 58.94 Β± 0.85 / βˆ’7 Β± 1 / βˆ’ 80.31 Β± 0.22 / βˆ’4 Β± 0 / βˆ’
GPM 72.48 Β± 0.54 / βˆ’0.9 / 22.27 60.41 Β± 0.61 / βˆ’0.9 / 127.37 91.22 Β± 0.20 / βˆ’1.0 / 70.33
TRGP 74.64 Β± 0.32 / βˆ’0.9 Β± 0.01 / 78.86 61.78 Β± 0.60 / βˆ’0.5 Β± 0.60 / 543.22 93.56 Β± 0.10 / βˆ’0.04 Β± 0.01 / 181.57
CUBER 75.54 Β± 0.22 / +0.13 Β± 0.08 / 326.79 62.67 Β± 0.74 / +0.23 Β± 0.15 / 604.61 93.48 Β± 0.10 / βˆ’0.00 Β± 0.02 / 196.91
SGP 76.05 Β± 0.43 / βˆ’1 / 22.27 62.83 Β± 0.33 / βˆ’1 / 127.37 βˆ’
CODE-CL 77.21 Β± 0.32 / βˆ’1.1 Β± 0.28 / 33.80 71.16 Β± 0.32 / βˆ’1.1 Β± 0.3 / 283.82 93.51 Β± 0.13 / βˆ’0.11 Β± 0.01 / 116.72
LANCE 71.52 Β± 0.27 / βˆ’0.17 Β± 0.34 / 2.32 59.68 Β± 1.17 / βˆ’0.99 Β± 0.78 / 32.96 90.76 Β± 0.31 / βˆ’1.02 Β± 0.15 / 34.96

On the single-task side, the paper fine-tunes the last 2 or 4 layers of several models on CIFAR-100 and compares against full BP, iterative HOSVD, and ASI (Acc / peak activation storage / training TFLOPs):

Model Method (layers) Acc ↑ Mem (MB) ↓ TFLOPs ↓
MCUNet BP (2) 52.47 11.71 0.000
MCUNet HOSVD (2) 52.34 0.10 2.590
MCUNet LANCE (2) 47.57 0.04 0.000
MCUNet LANCE (4) 52.59 0.09 0.000
MobileNetV2 BP (4) 74.69 57.42 0.01
MobileNetV2 HOSVD (4) 69.33 0.15 22.86
MobileNetV2 ASI (4) 66.18 0.63 0.06
MobileNetV2 LANCE (4) 68.79 0.17 0.01
ResNet18 BP (4) 77.07 61.25 0.09
ResNet18 HOSVD (4) 75.35 1.21 15.56
ResNet18 ASI (4) 71.93 1.61 0.06
ResNet18 LANCE (4) 75.70 1.16 0.05
ResNet34 BP (4) 77.50 49.00 0.12
ResNet34 HOSVD (4) 76.01 0.95 12.25
ResNet34 ASI (4) 70.60 1.00 0.07
ResNet34 LANCE (4) 75.94 1.02 0.06

⚠️ In the original Tab. 2 the ResNet18/CIFAR-100 LANCE 4-layer accuracy is typeset as "75.7 0"; it is treated here as 75.70 β€” refer to the original paper.

Ablation Study

The energy threshold \(\varepsilon\) is the paper's single main control knob. The table below uses the ImageNet-scale experiment (pretrain on 500 classes, fine-tune on the remaining 500; ResNet18/34) to show the accuracy–memory trade-off between \(\varepsilon=0.8\) and \(\varepsilon=0.9\):

Config ResNet18 2 layers Acc / Mem ResNet18 4 layers Acc / Mem ResNet34 2 layers Acc / Mem ResNet34 4 layers Acc / Mem
BP 60.70 / 24.50 MB 64.28 / 61.25 MB 60.77 / 24.50 MB 64.21 / 49.00 MB
LANCE (Ξ΅=0.8) 57.04 / 2.48 MB 59.76 / 5.79 MB 56.57 / 1.58 MB 60.56 / 3.87 MB
LANCE (Ξ΅=0.9) 59.18 / 7.12 MB 62.20 / 16.59 MB 58.59 / 4.49 MB 62.66 / 10.79 MB

(Memory is in MB; going from Ξ΅=0.8 to Ξ΅=0.9 grows memory roughly 3–4Γ— and recovers 1.5–2 accuracy points.)

The other ablation varies the number of calibration batches \(N\) (CIFAR-100, Ρ=0.7): accuracy stays stable over a wide range of \(N\) while memory grows with \(N\), and \(N=2\) already yields a stable subspace. The paper reports this only as curves in Figure 3 without per-point numbers, ⚠️ refer to the original paper for exact values.

Key Findings

  • The benefit of one-shot calibration is measurable on real hardware, not just in paper FLOPs. On a Raspberry Pi 3B+ (ARM Cortex-A53) fine-tuning MCUNet (CIFAR-10, batch size 128, 1–10 trainable layers, mean over five trials), LANCE achieves the lowest forward, backward, and total training latency: BP is 3–9% slower, ASI is 19–76% slower, and iterative HOSVD is 2.1×–4.4Γ— slower. This result directly supports the central claim that repeated decompositions are unnecessary.
  • Gradient fidelity gives Theorem 1 an empirical face: the angle between LANCE and full-BP gradients stays within ~70Β° and stabilizes as training progresses; more aggressive compression (smaller Ξ΅) increases the angle, but the degradation is smooth and controlled, and directions remain usable even when activation storage is cut by two orders of magnitude. This matches the theoretical characterization of the LANCE gradient as an orthogonal projection of the full gradient.
  • Compression ratios depend heavily on architecture, and reporting only the best number would be misleading. On MCUNet, 11.71 MB β†’ 0.04 MB (about 293Γ—); on ResNet18/ResNet34, about 50Γ—; on MobileNetV2, 57.42 MB β†’ 0.17 MB (about 338Γ—). The abstract and conclusion quote a 25×–380Γ— range, while Section 3.1 and Figure 1c say up to 250Γ—; the two are inconsistent, ⚠️ refer to the original paper.
  • On continual learning, LANCE's BWT is actually more stable than same-family baselines (βˆ’0.17 vs GPM's βˆ’0.9 on Split CIFAR-100), but its ACC lags: 71.52 vs GPM 72.48 and CODE-CL 77.21, with a clear gap to the Multitask upper bound of 79.58. On Split MiniImageNet the gap widens to 59.68 vs CODE-CL's 71.16. The paper's "competitive with gradient projection methods" claim holds against GPM/SGP but is optimistic against CODE-CL.
  • The memory advantage is unambiguous: 2.32 MB on Split CIFAR-100, about 1/10 of GPM (22.27 MB), 1/34 of TRGP (78.86 MB), and 1/141 of CUBER (326.79 MB). On MiniImageNet and 5-Datasets the advantage narrows to roughly 4Γ— and 2Γ—, so the relative memory benefit shrinks as tasks become harder.
  • The low-rank constraint occasionally acts as a regularizer: MobileNetV2 fine-tuned with LANCE on Oxford-IIIT Pets actually beats full BP (90.40 vs 90.32 at 4 layers), while on CUB-200 it is clearly worse (61.21 vs 65.86) β€” a task-dependent effect rather than a stable phenomenon.
  • The "trade layers for accuracy" result is practically useful: at ImageNet scale, LANCE fine-tuning 4 layers of ResNet34 reaches 62.66, already above full BP fine-tuning 2 layers (60.77), while needing only 10.79 MB versus 24.50 MB. Under a strict SRAM budget, training more layers with LANCE beats training fewer with BP.

Highlights & Insights

  • Moving forgetting mitigation out of the training loop and into offline calibration is the paper's best move. Gradient-projection methods must project orthogonally at every step with the memory matrix resident in SRAM; LANCE encodes orthogonality into fixed activation subspaces and pays almost nothing online. The idea transfers to any setting that is "stable subspace + sequential tasks + tight memory," e.g. local personalization on federated clients or long-horizon skill accumulation in robotics.
  • Changing what you store beats changing what you compute. Others compress weights, gradients, or adapters; this paper only changes the forward cache object and leaves the backward pass's functional form untouched, which makes it naturally orthogonal to checkpointing, quantization, and system-level methods like TinyTL β€” they can be stacked.
  • Calibrating from a second-order statistic is nearly free, and the observation that \(N=2\) calibration batches suffice is valuable in itself: the activation subspace of a pretrained model is very stable, so the approximation cost of a fixed subspace has been systematically underestimated.
  • Choosing rank per layer and per mode via an energy threshold lets the rank adapt to how important each layer is, removing per-layer tuning. The trick transfers directly to any layer-wise low-rank compression method.
  • The convergence analysis is not decoration: Theorems 1–2 and Proposition 1 tie together "the gradient is an orthogonal projection," "the loss decreases monotonically," and "the residual is bounded by Ξ΅," turning Ξ΅ from an experimental knob into a hyper-parameter with a bound-level meaning.

Limitations & Future Work

  • Continual learning accuracy is genuinely below the strongest same-family baselines: 71.52 vs CODE-CL's 77.21 on Split CIFAR-100, and 59.68 vs 71.16 on Split MiniImageNet β€” gaps well beyond noise. When the paper says "competitive," it is mainly comparing against the GPM tier, and readers should keep that framing in mind.
  • The mechanism's scope is narrow. The orthogonality constraint is applied only to mode-\(d\) (the input dimension of the weights); the other modes' factors, the nonlinearities, and BN statistics are unconstrained, so tasks can still interfere through those paths. As tasks accumulate, the memory matrix \(M\) keeps growing and the available null space keeps shrinking; the paper gives no saturation or degradation analysis for long task sequences (e.g. 20+ tasks).
  • Each task still requires an offline covariance calibration and SVD at the end. This does not enter online training cost, but it assumes calibration data for that task is available and that the process can pause for an offline step β€” an implicit precondition for streaming, non-pausable deployments.
  • Validation is confined to CNNs and visual classification: the models are MCUNet, MobileNetV2, ResNet18/34, and AlexNet, and the tasks are all image classification. There is no Transformer/ViT activation, no detection or segmentation, and no language-model validation β€” yet the redundancy structure of Transformer activations (especially along the sequence dimension) differs substantially from that of CNNs, so transferability is unverified.
  • Efficiency reporting has inconsistencies and typesetting issues: the 25×–380Γ— range in the abstract and conclusion conflicts with the "up to 250Γ—" in Section 3.1 and Figure 1c; the TFLOPs column of Tab. 2 records LANCE as 0.000/0.01 versus HOSVD's 1.93/22.86, suggesting different accounting for decomposition cost; and the ResNet18/CIFAR-100 LANCE 4-layer accuracy is typeset as "75.7 0." All ⚠️ refer to the original paper.
  • Concrete improvement directions: extend the null-space constraint from mode-\(d\) to all modes (maintaining a memory and complementary projection per mode); replace per-task offline recalibration with incremental/online covariance updates to remove the "must pause" precondition; and make Ξ΅ layer-adaptive (high Ξ΅ for important layers, low for the rest) to buy accuracy at the same memory budget.
  • vs ASI and HOSVD tensor activation compression (Nguyen et al.): They likewise exploit the low-rank structure of activations but incrementally update or recompute the subspace during training; LANCE makes the decomposition one-shot and reuses it. The price is a subspace that no longer drifts with training and thus has fixed expressive power; the payoff is a large drop in training compute and hardware overhead (1.93 β†’ 0.000 TFLOPs on MCUNet, 2.1×–4.4Γ— latency difference on the Raspberry Pi).
  • vs GPM / TRGP / SGP / CODE-CL: All are "orthogonal subspace against forgetting," and the essential difference is what is projected and when β€” they project gradients, which must be done online with the memory matrix resident in SRAM; LANCE projects activations at offline calibration time, so the memory occupies storage but not training memory (2.32 MB vs GPM's 22.27 MB and CUBER's 326.79 MB on Split CIFAR-100). The trade-off is slightly lower accuracy, most visibly against CODE-CL on MiniImageNet.
  • vs LoRA-style low-rank adaptation (e.g. Inflora): LoRA-family methods make the weight update \(\Delta W\) low-rank, saving trainable parameters and optimizer state while keeping a full-precision forward pass; LANCE makes the activation cache low-rank, saving peak training SRAM while updating all weights, adding no parameters, and changing no architecture. They optimize different resources and can in principle be combined (low-rank adapters plus a low-rank activation cache).
  • vs TinyTL / MCUNet / the 256KB training engine: System-level methods fit SRAM budgets by freezing layers, restricting the update scope, or pruning gradient paths, sacrificing trainable capacity. LANCE does not shrink the optimization scope β€” every layer still gets full backpropagation β€” and only makes the cache smaller, so the two are complementary.
  • vs replay-based (ER, A-GEM) and regularization-based (EWC, HAT) methods: The former must store exemplars and the latter Fisher information or attention masks, both consuming extra device memory. LANCE stores no samples and changes no loss function; its forgetting mitigation comes from a structural constraint imposed at calibration time, which suits severely storage-constrained settings.

Rating

  • Novelty: ⭐⭐⭐⭐ One-shot HOSVD reuse is a clean engineering simplification, but pairing it with null-space constrained calibration so that activation compression itself carries forgetting mitigation is a distinctive new combination.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Four models Γ— five fine-tuning datasets plus three continual learning benchmarks and real edge-hardware latency, with convergence analysis to back the design; points deducted for missing Transformer/long-sequence validation and inconsistent efficiency accounting.
  • Writing Quality: ⭐⭐⭐⭐ Stage decomposition and theory are clear, but formulas are corrupted in the cached text and the tables contain typesetting issues, so readers should verify against the original tables.
  • Value: ⭐⭐⭐⭐ The activation-memory bottleneck of on-device full BP is a real and widespread problem, the method is simple, deployable, and orthogonal to system-level approaches; continual learning accuracy is not best-in-class, but the memory advantage is explicit, making it a practical option for trading a little accuracy for a lot of memory.