Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/xxx
Area: Object Detection
Keywords: domain-incremental object detection, orthogonal knowledge refreshing, low-rank subspace expansion, topology-aware consistency, parameter-efficient fine-tuning

TL;DR

To tackle catastrophic forgetting and inter-domain interference in exemplar-free domain-incremental object detection, this paper proposes Orthogonal Knowledge Refreshing (OKR), which incrementally constructs isolated low-rank branches for new domains, projects gradient updates onto the orthogonal complement of historical subspaces, and enforces topology-aware consistency to align cross-domain semantic prototypes without any domain routing at inference.

Background & Motivation

Deep learning object detectors generally rely on the assumption that training and test data share identical distributions, leading to severe performance drops when encountering sequential environmental shifts in the real world, such as transitioning from sunny to foggy and rainy conditions in autonomous driving. While conventional domain adaptation addresses one-off shifts from source to target, Domain-Incremental Object Detection (DIOD) requires continuous adaptation across non-stationary domains without storing historical data. The primary obstacle is catastrophic forgetting: rehearsal-based strategies violate strict data privacy regulations and incur heavy storage burdens; knowledge distillation is constrained by the static capacity of a fixed backbone; and dynamic architecture expansion methods incur computational and memory overheads that scale linearly with the number of tasks.

Parameter-efficient fine-tuning (PEFT) has emerged as a promising direction, yet existing implementations still suffer from acute inter-domain interference. Prompt-tuning and bias-tuning methods merely apply shallow modulations to input embeddings or neuron biases, lacking the representational plasticity required to accommodate profound visual distribution gaps. Furthermore, joint optimization over a shared parameter space inevitably corrupts feature statistics critical to earlier domains, precipitating catastrophic forgetting. In addition, existing methods relying on domain selection or routing (e.g., S-Prompt or SOYO) suffer from redundant multi-pass forward evaluations and brittle failure modes whenever domain classification errs.

Diagnostic investigations into class-agnostic object recall (oRecall) reveal that tuning the backbone alone matches the plasticity of full-model tuning, whereas forgetting primarily stems from destructive parameter overwriting in shared spaces. Consequently, DIOD demands non-interfering feature accumulation rather than naive parameter sharing. Core idea: construct dedicated low-rank adaptation subspaces for each new domain, exploit LoRA's linear additivity for seamless routing-free inference fusion, project new-domain gradient updates onto the orthogonal complement of historical feature subspaces, and align semantic category prototypes via topology-aware consistency.

Method

Overall Architecture

OKR is built upon the ViTDet detector with a Vision Transformer backbone, comprising a transformer-based feature extractor and a linear prediction head for bounding-box regression and classification. During the initial base domain session (Session 1), the entire detector is optimized with standard detection losses. In subsequent domain-incremental sessions (\(t \ge 2\)), the base backbone is frozen, and adaptation proceeds in an exemplar-free manner. OKR coordinates three core mechanisms: Low-Rank Subspace Expansion (SE), Gradient-based Orthogonal Refreshing (GOR), and Topology-Aware Consistency (TAC).

For each incremental domain, the detector injects a dedicated LoRA branch while freezing all preceding branches. Thanks to the linear additivity of low-rank matrices, all branch parameters can be directly fused into the primary weight matrices during forward inference, completely removing the need for domain routing or multiple forward passes. To eliminate the implicit coupling and latent representation drift caused by weight fusion, GOR projects backward gradients onto the orthogonal complement of historical subspaces. Concurrently, TAC anchors emerging domain prototypes to the base domain semantic topology.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input New Domain Stream D_t"] --> B["Low-Rank Subspace Expansion<br/>Inject dedicated branch A_t, B_t & freeze past branches"]
    B --> C["Linear Weight Fusion<br/>W_t = W_0 + accumulated low-rank branches"]
    C --> D["Topology-Aware Consistency<br/>Align cross-domain class prototype topologies"]
    D --> E["Gradient-Based Orthogonal Refreshing<br/>SVD historical subspace M_t & apply orthogonal gradient projection"]
    E --> F["Output Interference-Free Detector<br/>Single forward pass & routing-free inference"]

Key Designs

1. Low-Rank Subspace Expansion: conflict-free capacity expansion via linear additivity

Optimizing a shared parameter set across sequential domains inevitably induces negative transfer and representation overwriting. To expand model capacity without growing architecture complexity, OKR injects dedicated low-rank adaptation matrices \(A_t \in \mathbb{R}^{r \times d}\) and \(B_t \in \mathbb{R}^{d \times r}\) (\(r \ll d\)) into transformer attention projections (Query/Value) and MLP layers for each incoming domain \(t\). During session \(t\), only the current parameters \(\{A_t, B_t\}\) are updated, while all historical modules \(\{A_{1:t-1}, B_{1:t-1}\}\) remain frozen.

Leveraging the linear superposition property of low-rank matrix decomposition, the composite weight matrix at session \(t\) is expressed as the direct sum:

\[W_t = W_0 + \sum_{i=1}^{t} B_i A_i = W_{t-1} + B_t A_t\]

This design bypasses the memory footprint of keeping separate task networks and eliminates the inference latency of domain routing. At inference time, all learned low-rank parameters are folded directly into the primary weights, allowing a single forward pass to serve all domains with minimal parameter expansion (only \(\approx 1.8\%\)).

2. Gradient-Based Orthogonal Refreshing: projecting updates onto orthogonal complements

Although freezing historical branches avoids explicit parameter overwriting, all branches are unified to modulate the same hidden representations, meaning gradients computed on new domain data can still perturb the latent feature manifold vital to earlier domains. Continual learning theory indicates that if parameter updates remain strictly orthogonal to the subspace spanned by historical features, gradient descent induces near-zero interference on previously acquired representations.

Because the gradient of a linear layer lies within the span of its input representations, OKR feeds domain \(D_t\) samples through the model parameterized by the accumulated historical weights \(W_{t-1}\) to collect layer-wise feature matrices \(R_t \in \mathbb{R}^{m \times n}\) prior to session \(t\), followed by Singular Value Decomposition (SVD):

\[\text{SVD}(R_t) = U_t \Sigma_t V_t^T\]

Applying an energy threshold \(\epsilon\) based on the Frobenius norm (\(\|(R_t)_k\|_F^2 \ge \epsilon \|R_t\|_F^2\)), the top-\(k\) left singular vectors form the historical subspace basis \(\mathcal{M}_t = [u_t^1, u_t^2, \dots, u_t^k]\). During back-propagation, raw gradients \(\nabla_w L_t\) on parameters \(w=\{A_t, B_t\}\) are projected onto \(\mathcal{M}_t\) and subtracted, yielding the orthogonalized gradient:

\[\nabla_w L'_t = \nabla_w L_t - (\nabla_w L_t) \mathcal{M}_t \mathcal{M}_t^T\]

This operation restricts updates to the residual orthogonal complement \(\mathcal{M}_t^\perp\), preserving prior feature distributions while forcing the model to seek novel, complementary domain representations.

3. Topology-Aware Consistency: cross-domain prototype alignment to prevent semantic fragmentation

Because each low-rank branch is optimized in an isolated parameter subspace, separate modules can overfit to domain-specific visual styles, causing representations of the same class to drift apart across incremental sessions. This semantic fragmentation undermines generalizability. To preserve categorical coherence, OKR introduces a topology-aware consistency objective.

Upon completing the base domain session, global class prototypes \(\mu_1^c = \frac{1}{N_c} \sum_{i=1}^{N_c} p_i^c \cdot f_\Phi(x_i^c)\) are computed using detector classification confidence weights and anchored permanently as a canonical reference. In subsequent session \(t\), current prototypes \(\mu_t^i\) are computed and aligned against the base prototypes via soft similarity matching:

\[L_{tac} = \frac{1}{N} \sum_{i=1}^N \sum_{j=1}^N w_t^{ij} \cdot \left(1 - \cos(\mu_t^i, \mu_1^j)\right), \quad \text{where } w_t^{ij} = \frac{\exp(\mu_t^i \cdot \mu_1^j)}{\sum_{k=1}^N \exp(\mu_t^i \cdot \mu_1^k)}\]

Minimizing \(L_{tac}\) pulls matching class prototypes closer in cosine distance while maintaining inter-class separation, anchoring new feature distributions to the foundational semantic geometry and mitigating intra-class divergence.

Loss & Training

During the base domain session (\(t=1\)), the model is trained with standard detection loss \(L_{det}\) over all base parameters. For incremental sessions (\(t \ge 2\)), the backbone is frozen, and the combined objective is defined as:

\[L_{final} = L_{det} + L_{tac}\]

Back-propagated gradients on \(\{A_t, B_t\}\) are redirected via the orthogonal projection formula before performing AdamW parameter updates (learning rate \(2 \times 10^{-4}\), weight decay \(0.1\), LoRA rank \(r=16\)). The SVD step runs only once at the start of each session, adding merely \(0.02\%\) training time overhead while maintaining optimization stability.

Key Experimental Results

Main Results

OKR is evaluated on Pascal VOC series (VOC 2007 \(\to\) Clipart \(\to\) Watercolor \(\to\) Comic) and the autonomous driving BDD100K series (BDD100K \(\to\) Cityscape \(\to\) Rainy Cityscape). The metric is mean Average Precision (mAP in %) at an IoU threshold of 0.5 across all observed domains.

Method	Type	Exemplar/domain	VOC S2	VOC S3	VOC S4	VOC \(\Delta\)mAP	BDD S2	BDD S3	BDD \(\Delta\)mAP
Upper-bound	Joint Training	-	72.6	69.4	67.7	-	58.7	59.1	-
TP-DIOD-B	Rehearsal	150/200	65.8	62.1	57.5	-7.7	53.4	51.5	-6.7
FT-Seq	Sequential FT	0	57.5	52.6	49.5	-15.7	51.6	43.6	-14.6
IRG	Source-Free DA	0	51.5	43.7	33.2	-32.0	49.3	38.7	-19.5
LwF	Distillation	0	60.4	53.6	53.2	-12.0	52.1	44.1	-14.1
L2P	Prompt Pool	0	59.9	55.2	45.5	-19.7	51.5	47.7	-10.5
S-Prompt	Domain Prompt + KNN	0	59.4	54.3	45.0	-20.2	51.6	49.4	-8.8
LDB	Bias Tuning	0	68.1	64.2	56.8	-8.4	52.3	51.1	-7.1
LDB+SOYO	Bias + Selector	0	69.2	65.3	59.6	-5.6	52.4	51.7	-6.5
OKR (Ours)	Orthogonal Subspace	0	73.0	67.3	65.2	-0.0	57.6	58.2	-0.0

Per-domain breakdown at the final incremental session highlights OKR's balance between retaining historical knowledge and absorbing new domains:

Method	VOC 2007 (Old)	Clipart (Old)	Watercolor (Old)	Comic (New)	VOC Avg mAP	BDD100K (Old)	Cityscape (Old)	Rainy (New)	BDD Avg mAP
L2P	78.2	32.5	43.3	27.8	45.6	49.7	48.8	44.8	47.7
S-Prompt	80.8	33.9	45.2	20.1	45.0	51.6	52.0	44.7	49.4
LDB	82.4	50.1	57.5	37.0	56.8	50.3	52.7	50.2	51.1
OKR (Ours)	85.2	54.8	66.0	54.6	65.2	50.7	61.8	62.2	58.2

Ablation Study

Component-wise ablation on the Pascal VOC benchmark validates the progressive contributions of each proposed design:

Exp ID	FT-Back	SE (Subspace Exp.)	GOR (Orthogonal)	TAC (Topology)	Session 2 mAP	Session 3 mAP	Session 4 mAP	Note
#1	-	-	-	-	59.1	54.4	44.2	Full fine-tuning baseline with severe catastrophic forgetting
#2	✓	-	-	-	67.7	57.9	49.5	Tuning backbone alone improves stability over classification head
#3	✓	✓	-	-	72.4	64.4	60.9	Subspace expansion isolates parameters, giving +11.4% gain
#4	✓	✓	✓	-	72.4	66.9	64.9	Orthogonal projection removes latent gradient interference
#5	✓	✓	-	✓	72.6	66.0	63.3	Topology consistency regularizes semantic prototype drift
#6	✓	-	-	✓	67.7	59.8	60.2	Standalone TAC alleviates cross-domain semantic degradation
#7 (Full)	✓	✓	✓	✓	73.0	67.3	65.2	Full OKR system achieving state-of-the-art DIOD performance

Key Findings

• Crucial synergy between subspace isolation and orthogonal projection: Injecting independent low-rank branches (SE) raises Session 4 mAP from 49.5% to 60.9% (+11.4%), while adding gradient-based orthogonal refreshing (GOR) further propels accuracy to 64.9% (+4.0%). This demonstrates that physical parameter freezing alone cannot stop latent feature corruption; dynamic gradient redirection is essential.
• Superior adaptability under radical domain shifts: On the stylistically extreme Comic benchmark, OKR achieves 54.6% mAP compared to 37.0% for previous SOTA LDB—a remarkable +17.6% improvement, confirming that weight-level low-rank modulation offers far greater expressiveness than channel bias shifts.
• High throughput and minimal operational footprint: On an RTX-3090, OKR achieves an inference speed of 0.1348 s/sample with 5.1 GB memory, outperforming LDB (0.2836 s/sample, 6.0 GB). Because low-rank matrices are statically merged, inference involves zero domain routing or multi-branch evaluation overhead.

Highlights & Insights

• Zero-cost inference fusion via LoRA linearity: Folding incremental low-rank matrices directly into primary weights eliminates runtime domain classification errors and multi-branch forward passes.
• Efficient input-spanned orthogonal gradient projection: Avoiding second-order Hessian or Fisher approximations, the framework extracts historical feature principal bases via single-pass SVD, achieving non-interfering updates with negligible computational cost.
• Versatile plug-and-play continual tuning paradigm: Combining PEFT with geometric orthogonal projection offers a generalizable blueprint for lifelong adaptation across other dense prediction and multimodal foundation model domains.

Limitations & Future Work

• Subspace capacity saturation in prolonged streams: As the number of tasks \(T\) scales indefinitely, historical bases consume increasingly higher rank dimensions, progressively shrinking the available orthogonal complement \(\mathcal{M}_t^\perp\) and potentially capping plasticity for later domains.
• Static truncation energy threshold: The energy retention threshold \(\epsilon\) remains fixed across all sessions; designing an adaptive mechanism calibrated to domain distance could further balance capacity.
• Reliance on the base domain prototype distribution: Prototype consistency is anchored to Session 1 representations, which may propagate inductive biases if the base domain lacks categorical diversity.

Related Work & Insights

• vs LDB / LDB+SOYO: LDB adapts only bias terms, which severely restricts expressive plasticity during intense domain shifts; SOYO relies on a domain selector network. OKR directly modulates attention weights via low-rank expansion and fuses them linearly, cutting inference latency in half while improving final mAP by +5.6% to +6.5%.
• vs S-Prompt / L2P: Prompt-tuning prepends virtual tokens at the input layer, exerting shallow modulation and risking routing failures during inference. OKR enforces deep geometric orthogonality directly within weight updates without domain routing.
• vs GPM / OWM: Traditional gradient projection memory methods target full-parameter CNNs, incurring massive activation covariance storage and high-dimensional SVD costs. OKR confines orthogonal projection to PEFT low-rank spaces, making the approach practical and lightweight.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Combines linear LoRA fusion with gradient orthogonalization in an elegant, routing-free design)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across Pascal VOC, autonomous driving BDD100K, and 16 corruption domains in VOC-C with detailed ablations)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear logical progression from diagnostic motivation experiments to mathematical formulation and validation)
• Value: ⭐⭐⭐⭐⭐ (Resolves the critical inference latency and forgetting bottlenecks in DIOD, providing high practical utility)