Breaking the Model Forgetting Cycle in Long-Incremental 3D Object Detection¶

Conference: ECCV2026
Paper: https://eccv.ecva.net/virtual/2026/poster/5691
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/12290.pdf
Code: https://github.com/qianpeisheng/LDMR
Project: https://qianpeisheng.github.io/LDMR-project-page/
Area: Model Compression
Keywords: continual learning, memory replay, learning dynamics, catastrophic forgetting, 3D object detection

TL;DR¶

LDMR turns changes in historical-scene recall into decisions about which samples need more review now and which deserve storage for the next stage, reaching 19.38 [email protected] on the ten-stage SUN RGB-D protocol with TR3D, 3.84 points above the equal-budget SDCoT++ random-memory baseline.

Background & Motivation¶

Indoor 3D detection does not necessarily receive its full category set at once: a system might first learn common furniture and encounter rarer objects in later updates. Training scenes at each stage provide annotations only for the current novel classes, leaving old-class objects potentially unlabeled even when they remain in the point cloud. Direct fine-tuning therefore changes the data distribution and can also treat old objects as background. Methods such as SDCoT and SDCoT++ use teachers to supply old-class pseudo labels, mitigating forgetting over short sequences, but those teachers themselves come from models that may already have forgotten earlier knowledge.

Over longer sequences, this becomes a feedback loop: novel-class supervision degrades old-class detection, the degraded model produces poorer old-class pseudo labels, and subsequent updates learn from those unreliable labels. The paper examines this process through old-class training-loss and pseudo-label-quality trajectories and derives a forgetting bound involving distribution distance and gradient magnitude, among other factors. These analyses support the diagnosis; they do not prove that LDMR eliminates forgetting. Random historical samples already help substantially, so the relevant question is how to use reliable old annotations more effectively than random replay under a fixed storage budget.

Rather than asking whether a scene is statically difficult, the method asks whether it was ever learned, how much was subsequently forgotten, and whether further exposure can help. Core Idea: use measurable learning dynamics for both short-timescale adaptive review and long-timescale memory selection, prioritizing scarce historical annotations for knowledge that is recoverable but not yet stably retained.

Method¶

Overall Architecture¶

Learning-Dynamics-driven Memory and Review (LDMR) takes scenes annotated for the current novel classes, the previous-stage detector, and a fixed-size historical memory bank. It produces a detector for all seen classes and a memory bank for the next stage. Intra-stage review first monitors old-class recall at training checkpoints and increases sampling probabilities for historical scenes that have just suffered forgetting. After the stage, learnability scoring and diversity-aware memory updates determine which scenes to retain.

Memory entries contain actual scenes and the historical-class annotations available when they were acquired, rather than only features or teacher predictions. The method primarily changes training-sample selection and historical-data retention rather than introducing a new 3D detection backbone. The paper evaluates this training framework with TR3D and VoteNet.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["New-stage data<br/>Old model and memory"] --> B["Intra-stage review"]
    B -->|Checkpoint recall feedback| B
    B --> C["Learnability scoring"]
    C --> D["Diversity-aware memory updates"]
    B --> E["Updated detector"]
    D --> F["Next-stage memory bank"]

Key Designs¶

1. Intra-stage review: give recently forgotten scenes more training opportunities

The historical memory bank holds a fixed budget of annotated scenes and starts empty. At subsequent stages, it joins the new-stage data in a shared sampling pool. Each stage is divided into sub-stages, with scene–class recall recorded at checkpoints. Learning dynamics here means neither training loss nor model confidence: it is detection recall measured against reliable historical annotations. The experiments use IoU 0.5 for this measurement, which must not be confused with the reported [email protected]. A class dropping from higher to lower recall provides direct evidence of a current review need.

For each old scene, positive recall drops between adjacent checkpoints are summed over old classes and weighted by the logarithm of their ground-truth object counts. Equation (4) and its accompanying definitions give the following mechanism:

\[ F_s^{i\to i+1}=\sum_{c\in\mathcal{C}_{<t}}\log(1+n_{s,c})\max\left(0,R_t^i(s,c)-R_t^{i+1}(s,c)\right). \]

Here, \(R_t^i(s,c)\) denotes recall at checkpoint \(i\) for scene \(s\) and class \(c\), \(n_{s,c}\) is its ground-truth instance count, and \(\mathcal{C}_{<t}\) is the old-class set. Counting only drops prevents improvements in one class from canceling forgetting in another. Logarithmic count weighting also prevents object-dense rooms from monopolizing review. New scenes keep sampling weight 1, while old scenes receive \(1+\eta F_s^{i\to i+1}\); weights are then normalized over the mixed pool. Every scene retains a baseline weight, so this is neither hardest-example-only training nor a suspension of novel-class learning.

The measured drop guides the next sub-stage; it does not assume foreknowledge of future forgetting. Re-evaluation after each training segment can shift sampling emphasis from one type of historical scene to another, closing the train–evaluate–review loop. Reliable historical annotations provide supervision independent of old-model pseudo labels, but the budget covers only part of the history. This does not imply that all missing old-class annotations are recovered.

2. Learnability scoring: retain knowledge that can improve but has not stayed learned

Intra-stage review decides what to sample during the next training segment, whereas cross-stage retention must consider longer-term value. At stage end, recall-trajectory statistics are extended to the union of new data and old memory, covering both novel and old classes. For each class, scene recalls are averaged at each checkpoint, and the stage peak and final value are recorded. Low final recall indicates incomplete learning, a peak-to-final drop indicates that previously acquired knowledge was lost, and a very low peak indicates little evidence of learnability under the current training process. The class under-learning weight \(\omega_t(c)\) combines the first two needs and modulates them by peak recall, avoiding persistent memory allocation solely because a class remains difficult.

Scene-level evidence of learnability is the sum of all positive recall increments between consecutive checkpoints, denoted \(G_t(s,c)\). This is not final recall minus initial recall: recall can rise and later fall, yielding little net change even though the improvement demonstrates potential benefit from renewed exposure. The scene score combines class-level need, object count, and this observed learning gain:

\[ \mathrm{lrn}_t(s)=\sum_c\omega_t(c)\log(1+n_{s,c})G_t(s,c). \]

A high score therefore does not simply mean many current errors. A scene with persistently low recall and no improvement may be less useful for replay than one that previously improved detection before being forgotten. Conversely, stably mastered classes should not permanently occupy scarce memory. This explanation preserves the interpretable score components without reconstructing the complete under-learning-weight formula, whose operators are damaged in the local text extraction.

3. Diversity-aware memory updates: avoid spending the budget on identical learning patterns

Ranking solely by learnability can repeatedly select scenes with similar behavior. LDMR constructs a two-dimensional representation for each scene: one dimension aggregates positive learning gains weighted by class under-learning weights; the other aggregates, with the same weights, scene–class peak recall minus final recall. After L2 normalization, candidates are compared with selected scenes in terms of learning-dynamics patterns rather than point-cloud geometry or visual features. Diversity thus concerns differences in how the model learns samples, not a direct guarantee of room-type or object-category coverage.

The described update process uses \(\lambda\) to trade off learnability and diversity, adds a valuable candidate from the current data, and replaces a less valuable old-memory scene while preserving the global budget. Candidate checks produce the next-stage bank. An important reproducibility boundary remains: Equation (13) in the local cache appears as an average of nonnegative inner products but is described as increasing with diversity, conflicting with the usual direction of similarity. Combination operators in Equation (14) are also damaged by extraction. This note therefore does not silently substitute a distance formula or a definite weighted sum; the exact score direction and replacement conditions require checking the PDF or code. What is clear is the joint use of both signals and the fixed-budget mechanism of admitting current scenes while evicting old ones.

A Worked Example¶

Consider the ten-stage SUN RGB-D protocol with TR3D: each stage introduces 4 classes, and incremental stages train for 15 epochs split into 5 sub-stages. Suppose a memory scene contains previously learned classes, while current scenes provide annotations for this stage's novel classes. Both enter the mixed pool. If a checkpoint reveals lower old-class recall in the memory scene, its sampling probability increases for the next sub-stage. If recall later recovers, subsequent checks do not permanently retain that earlier drop as a high weight.

At stage end, the algorithm does not simply retain every scene that was sampled most often. It reassesses learnability across multiple checkpoints for old and new candidates and then considers diversity in their learning dynamics. A scene urgently needing review within the stage need not be the best long-term memory entry. The retained real scenes and annotations accompany the detector into the next stage. This example illustrates execution order rather than a measured trajectory for an individual scene in the paper.

Loss & Training¶

The detector retains classification and bounding-box regression losses, optimized under the adaptive sampling distribution. The method description introduces no additional network for predicting learning dynamics. TR3D trains for 90 epochs at the base stage and 15 epochs at each subsequent stage. VoteNet follows SDCoT settings, so the TR3D epoch schedule should not be transferred to it without verification. Optimizers and data augmentation use the corresponding backbone defaults.

The fixed global memory budget is 10% of all training scenes, with \(I=5\) sub-stages, review strength \(\eta=3.0\), and memory-selection trade-off \(\lambda=0.5\). This is neither 10% per class nor unrestricted accumulation of stage data. The method's premise does not hold when privacy or storage constraints prohibit retaining historical scenes altogether.

Key Experimental Results¶

Main Results¶

SUN RGB-D uses 40 classes, split as 20+10+10 for three stages, 8 classes per stage for five stages, and 4 per stage for ten stages. ScanNetV2 uses 35 classes, with 15+10+10, 7 classes per stage, and 3–4 classes per stage, respectively. Classes are ordered by descending occurrence frequency, introducing rarer categories later. The following selection from Table 1 reports final-stage [email protected] over all seen classes; higher is better. Gains are absolute mAP points, not relative percentages.

Dataset	Backbone	Stages	SDCoT++	SDCoT++ + Random Memory	LDMR	Gain over random memory
SUN RGB-D	TR3D	3	22.92	25.18	29.12	+3.94
SUN RGB-D	TR3D	5	14.28	20.02	25.10	+5.08
SUN RGB-D	TR3D	10	8.45	15.54	19.38	+3.84
ScanNetV2	TR3D	3	35.15	36.86	39.00	+2.14
ScanNetV2	TR3D	5	20.61	23.67	28.38	+4.71
ScanNetV2	TR3D	10	7.06	14.86	17.82	+2.96
SUN RGB-D	VoteNet	10	1.67	3.53	6.20	+2.67
ScanNetV2	VoteNet	10	3.38	7.25	9.76	+2.51

The random-memory baseline has the same memory budget as LDMR, making it more informative than replay-free SDCoT++ for assessing the incremental value of the dynamic mechanisms. All-class joint-training references with TR3D are 33.26 on SUN RGB-D and 51.52 on ScanNetV2, but they access all class training data simultaneously and are not equal-resource incremental methods.

The paper also reports Average Forgetting (AF), for which lower is better: for every old class, subtract its current-stage [email protected] from its [email protected] when first introduced, then average across old classes. This is not a historical-best-AP definition of forgetting. This note does not estimate exact AF values from plotted curves.

Ablation Study¶

The following results come from Table 2 on SUN RGB-D with TR3D. The metric is final [email protected], with higher being better. HR denotes intra-stage review and ME denotes cross-stage memory updates.

Config	3 stages	5 stages	10 stages	10-stage drop from full model
LDMR full model	29.12	25.10	19.38	0.00
Without HR	26.67	23.47	17.51	1.87
Without ME	27.44	23.17	18.70	0.68
Without HR and ME	25.18	20.02	15.54	3.84

Key Findings¶

Equal-budget memory substantially improves long-sequence performance, while dynamic mechanisms add further gains. On SUN RGB-D with TR3D over ten stages, results progress from 8.45 for memory-free SDCoT++ to 15.54 with random memory and 19.38 with LDMR. Adding historical supervision and using it more effectively are separate contributions.
The two mechanisms are complementary, but their ablation drops should not simply be added. Removing HR hurts more than removing ME over ten stages; over five stages, the drops are 1.63 and 1.93 points, respectively. Their relative contribution is not identical across protocols.
In Table 4, simplifying the under-learning weight to low recall alone yields 18.45 over ten stages, below the full design's 19.38. This supports the distinction between difficulty and review value.
The authors' hyperparameter analysis shows initial benefits from more review checkpoints, followed by saturation when monitoring becomes too dense. Learnability–diversity trade-offs around \(\lambda=0.3\sim0.6\) perform well. This is evidence within the tested protocols, not a universally optimal interval.

Highlights & Insights¶

Evaluation signals directly drive data decisions. Recall drops are not merely retrospective forgetting curves: they determine sampling emphasis for the next sub-stage, closing the loop between monitoring and training.
Review needs and storage value are treated separately. The former measures recent deterioration, while the latter considers improvement, retention, and redundancy over the whole stage, avoiding permanent priority based on a single difficult episode.
Diversity is defined in learning-dynamics space rather than a high-dimensional feature space. This connects selection more directly to continual learning, but two-dimensional trajectory differences should not be interpreted as guaranteed semantic diversity.

Limitations & Future Work¶

Replaying real historical scenes is a core requirement. Although the storage budget is fixed at 10% of training scenes, a scene-count budget does not resolve data-retention privacy costs or differences in byte size between scenes.
The main paper covers two datasets, two backbones, and multiple stage counts, but all class orders follow frequency. Alternative orders, random-seed variance, and the time overhead of checkpoint evaluation lack systematic quantification in the main text read here, limiting conclusions about stability and deployment cost.
Long-sequence forgetting remains. Ten-stage TR3D scores of 19.38 and 17.82 remain well below the corresponding all-class joint-training references of 33.26 and 51.52.
The local extraction damages the theoretical bound and several scoring formulas, particularly leaving an inconsistency between the prose and visible formula for diversity-score direction. Reproduction should check the original PDF and code; this mechanism explanation is not an authoritative transcription of damaged equations.

vs SDCoT / SDCoT++: These methods sustain old-class supervision through static and dynamic teachers and pseudo labels; SDCoT++ further consolidates teacher labels and calibrates class probabilities. LDMR's key addition is dynamic review and selection of reliable historical scenes. The equal-budget random-memory comparison is more persuasive than gains over methods without memory alone.
vs AIC3DOD: Its emphasis is point-transformer representations and room-layout constraints, whereas LDMR innovates in the cross-stage learning process rather than detector structure. Evaluation with two backbones shows that the mechanism is not tied to a single backbone, but does not establish coverage of all 3D detector architectures.
vs general continual-learning replay: Keeping a few old samples is not itself new. The distinctive contribution is using class- and scene-level recall trajectories to control replay under missing old-object annotations and multi-class scene co-occurrence, rather than relying only on random or static hard-example selection.

Rating¶

Novelty: 4/5. Learning dynamics jointly control intra-stage review and cross-stage memory, with clear adaptations to long-incremental 3D detection, although memory replay has extensive precedent.
Experimental Thoroughness: 4/5. Two datasets, two backbones, multiple stage counts, and equal-budget baselines are covered; cost, variance, and class-order analyses remain limited.
Writing Quality: 3/5. The diagnosis and method narrative are clear, but damaged formulas and score-direction ambiguity in the local text increase the cost of reproducibility checks.
Value: 4/5. The work provides an empirically supported training strategy for continual 3D perception when a small historical-scene buffer is allowed, but is not a complete solution to long-term forgetting.