Online Versatile Incremental Learning: Towards Class and Domain-Agnostic Adaptation at Any Time

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/KU-VGI/Online-VIL
Area: Continual Learning / Online Incremental Learning
Keywords: Online Continual Learning, Versatile Incremental Learning, Geodesic Flow Kernel, Topology Preservation, Anytime Inference

TL;DR

To address the realistic scenario of concurrent, continuous, and boundary-free shifts in both classes and domains, the paper formalizes Online Versatile Incremental Learning (Online VIL) and introduces TopFlow, which couples Domain-agnostic Flow Matching (DFM) with Global Topology Preservation (GTP) to enable state-of-the-art anytime and final adaptation without replay memory.

Background & Motivation

Continual learning aims to enable artificial visual perception systems to continuously assimilate novel knowledge while preserving established competencies against catastrophic forgetting. However, the vast majority of prior paradigms artificially compartmentalize distribution shifts: Class Incremental Learning (CIL) assumes stationary sensory domains with sequentially arriving categories, whereas Domain Incremental Learning (DIL) presumes static semantic taxonomies under varying environmental conditions. Even recently introduced Versatile Incremental Learning (VIL) frameworks still rely on discrete task increments and explicit transition demarcations, affording models offline access to localized datasets. This rigid isolation fails to replicate real-world embodied deployment—such as autonomous driving platforms or edge robotics—where streaming data arrive via strict single-pass observations, and environmental shifts in illumination, geography, or weather concurrently collide with the unpredictable emergence of novel obstacles and object classes.

Operating within an online stream devoid of task boundaries introduces acute representational vulnerability under severe single-pass and minimal-batch constraints. When each arriving mini-batch exposes only a partial subset of active categories alongside transient domain attributes, model updates are highly prone to overfitting spurious domain styles, causing irreversible distortion of underlying semantic boundaries. Through a systematic layer-wise empirical probe of frozen pre-trained Vision Transformers (ViTs), the authors uncover a fundamental structural hierarchy: intermediate layers predominantly encode domain-specific variances such as low-level textures and ambient illumination, while deep representations converge sharply on abstract, class-discriminative manifolds. Simultaneously, when observed mini-batches miss substantial numbers of classes, empirical covariance matrices suffer from severe rank deficiency, causing unregularized feature spaces to artificially contract and eradicate the representational capacity reserved for absent classes.

These findings highlight that successful adaptation under continuous and unpredictable shifts requires explicitly decoupling domain noise while anchoring global topological geometry in a memory-free manner. Core idea: disentangle online streaming representation learning into geometry-guided cross-layer flow matching and global relational manifold anchoring, embodied by the TopFlow framework, which leverages Domain-agnostic Flow Matching (DFM) to suppress intermediate domain biases and Global Topology Preservation (GTP) via non-parametric clustering to stabilize global feature structures without storing past exemplars.

Method

Overall Architecture

TopFlow is engineered for replay-free, single-pass streaming adaptation over a frozen pre-trained Vision Transformer backbone equipped with lightweight learnable visual prompts and classification heads. The pipeline processes continuous input batches without knowledge of task identity or distribution shift type. Forward features from intermediate layers and the final output layer are extracted simultaneously. The Domain-agnostic Flow Matching (DFM) module extracts a common orthogonal basis via singular value decomposition to capture cross-layer transport while contrasting transformed intermediate representations against final semantic targets and domain-biased negatives. Concurrently, the Global Topology Preservation (GTP) module dynamically clusters output features with FINCH to derive batch-specific prototypes, aligns them to moving-average global prototypes via Hungarian matching, and preserves relative geometric configurations across updates using a dedicated relational projection network.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Streaming Input Batch $x_t$<br/>(Concurrent class and domain shifts)"] --> B["Frozen Pre-trained ViT Encoder<br/>Multi-layer feature extraction"]
    B --> C["Domain-agnostic Flow Matching DFM<br/>Shared subspace $U$ projection and normalization"]
    B --> D["Global Topology Preservation GTP<br/>FINCH clustering for batch prototypes $p^b$"]
    C --> E["Semantic Pull & Domain-biased Push<br/>Eliminating intermediate domain sensitivity"]
    D --> F["Hungarian Matching & EMA Update<br/>Relational mapping $\phi$ for geometric stability"]
    E --> G["Unified Optimization $L_{\text{total}}$<br/>Updating visual prompts and classifier heads"]
    F --> G

Key Designs

1. Domain-agnostic Flow Matching (DFM): Layer-Wise Subspace Projection and Geodesic Flow Alignment Traditional Geodesic Flow Kernel (GFK) techniques construct smooth paths along Grassmann manifolds between stationary source and target distributions, an approach fundamentally infeasible in streaming environments with unsegmented and heterogeneous shifts. DFM repurposes the flow mechanism by taking network depth as the geometric transition trajectory, exploiting the empirical discovery that intermediate layers capture domain factors while the final layer reflects class semantics. Given intermediate features \(h_n\) and final features \(h_l\), the module constructs the concatenated matrix \(H = [h_n^T, h_l^T]^T\) and performs singular value decomposition \(H = U \Sigma V^T\) to isolate the shared orthogonal subspace basis \(U\). DFM projects adapted representations \(h_n'\) onto \(U\) to filter out local batch variance. By defining final-layer features of the corresponding instance as the positive anchor \(h_+^{(m)}\) and using frozen intermediate features alongside distinct final instances as domain-biased negative sets \(H^-(m)\), DFM optimizes an explicit contrastive loss: $\(L_{\text{DFM}} = -\frac{1}{M} \sum_{m=1}^M \log \frac{\exp(\cos(h_n'^{(m)} U, h_+^{(m)} U) / \tau)}{\sum_{z \in \{h_+^{(m)}\} \cup H^-(m)} \exp(\cos(h_n'^{(m)} U, z U) / \tau)}\)$ where \(\tau\) denotes temperature scaling and cosine similarity cancels amplitude drift. This formulation pushes representations away from transient domain directions and pulls them along the semantic geodesic, guaranteeing domain-agnostic feature accumulation.

2. Global Topology Preservation (GTP): Memory-Free Manifold Anchor via Prototype Relational Vectors Because any online batch \(B_t\) typically contains only \(k < C\) classes, empirical covariance matrices incur rank deficiency, exerting biased gradients that contract feature spaces around current classes and erase absent semantic regions. Rather than relying on memory buffers of raw historical samples, GTP maintains a coarse-grained surrogate manifold represented by \(k\) global prototypes \(\{\bar{p}_j^g\}_{j=1}^k\). For each batch, parameter-free FINCH hierarchical clustering extracts batch prototypes \(\{p_i^b\}_{i=1}^k\) from the final layer. An optimal assignment \(\varsigma^*\) is determined via the Hungarian algorithm by minimizing pairwise Euclidean distances against global prototypes, followed by an exponential moving average (EMA) update with momentum \(\varrho = 0.99\): $\(\bar{p}_{\varsigma^*(i),\text{new}}^g = (1 - \varrho)\bar{p}_{\varsigma^*(i),\text{old}}^g + \varrho p_i^b\)$ To capture higher-order relative geometry beyond isolated point centroids, GTP introduces a learnable 2-layer MLP relational mapping \(\phi: \mathbb{R}^{2d} \to \mathbb{R}^m\), generating pairwise relation vectors \(r_{i,j}^b = \phi([p_i^b; p_j^b])\) and \(\bar{r}_{\varsigma^*(i), \varsigma^*(j)}^g = \phi([\bar{p}_{\varsigma^*(i)}^g; \bar{p}_{\varsigma^*(j)}^g])\). The topological loss enforces structural consistency over cosine distance metric \(D\): $\(L_{\text{GTP}} = \sum_{i=1}^k \sum_{j \neq i}^k D\left(r_{i,j}^b, \bar{r}_{\varsigma^*(i), \varsigma^*(j)}^g\right)\)$ This pairwise relational constraint prevents spatial collapse of unobserved categories, functioning as an agile, computationally efficient short-term topological memory.

Loss & Training

The overall training objective combines supervised cross-entropy with the dual geometric regularizers: $\(L_{\text{total}} = L_{\text{CE}} + \lambda_{\text{DFM}} L_{\text{DFM}} + \lambda_{\text{GTP}} L_{\text{GTP}}\)$ Cross-entropy \(L_{\text{CE}}\) operates with logit masking to avoid penalizing unobserved output heads. The ViT backbone remains completely frozen; only injected visual prompt parameters and the linear classification head receive gradient updates. The framework uses the Adam optimizer with a learning rate of \(5 \times 10^{-3}\) and batch size 64. The relational projector \(\phi\) employs hidden dimensions of 64 on CORe50 and 32 on CLEAR100, mapping concatenated prototype vectors into \(m = 10\)-dimensional relational embeddings.

Key Experimental Results

Main Results

Empirical validation was performed across three dynamic benchmarks: iDigits (5 tasks), CORe50 (10 tasks), and CLEAR100 (10 tasks). Evaluation tracked both Average Area Under the Curve (AAUC) to gauge anytime inference reliability across training, and final accuracy (ALast). The table below summarizes comparative performance under the strict replay-free setting (original paper Table 1):

Method / Benchmark	iDigits AAUC (%)	iDigits ALast (%)	CORe50 AAUC (%)	CORe50 ALast (%)	CLEAR100 AAUC (%)	CLEAR100 ALast (%)
Lower-bound (Sequential FT)	13.45 ± 0.62	12.71 ± 3.34	3.39 ± 0.10	3.24 ± 1.09	2.38 ± 0.32	2.66 ± 0.55
EWC [21]	20.07 ± 2.84	14.67 ± 3.61	18.60 ± 4.64	16.07 ± 1.56	23.61 ± 3.11	19.93 ± 1.33
LwF [30]	19.61 ± 3.50	15.38 ± 1.04	25.27 ± 3.77	21.97 ± 4.22	23.80 ± 2.24	21.70 ± 4.19
CODA-P [46]	23.96 ± 4.74	20.62 ± 3.47	54.06 ± 5.32	48.88 ± 2.92	28.82 ± 5.77	25.61 ± 3.52
SLCA [61]	35.81 ± 3.98	24.88 ± 2.82	33.49 ± 4.47	27.36 ± 2.06	32.16 ± 2.28	31.70 ± 1.13
PEC [60]	34.77 ± 3.02	28.01 ± 2.69	51.35 ± 4.39	46.95 ± 2.28	53.93 ± 3.20	51.66 ± 2.09
ICON [42]	33.60 ± 2.16	30.63 ± 2.77	49.42 ± 3.29	45.15 ± 2.94	59.38 ± 2.46	58.60 ± 2.57
S6MOD [32]	31.18 ± 1.62	30.42 ± 2.55	52.33 ± 2.74	47.20 ± 3.15	59.60 ± 2.79	57.47 ± 2.48
DUCT [63]	36.46 ± 3.48	30.94 ± 2.41	53.66 ± 4.37	47.24 ± 1.28	66.79 ± 2.94	65.46 ± 3.40
MVP [38] (Strongest Baseline)	38.29 ± 5.74	31.05 ± 3.15	58.30 ± 4.48	52.84 ± 1.17	79.73 ± 3.59	77.11 ± 2.33
TopFlow (Ours)	48.52 ± 1.25	32.18 ± 1.01	64.51 ± 2.50	66.20 ± 4.18	87.12 ± 0.01	80.64 ± 2.67
Upper-bound (Joint Supervised)	-	87.18 ± 0.13	-	91.66 ± 0.25	-	94.36 ± 0.28

When augmented with rehearsal replay buffers (Buffer size 500 / 2000, Table 2 in original paper), TopFlow establishes consistent dominance. At buffer size 500 on CORe50, TopFlow delivers 85.16% AAUC / 91.14% ALast, exceeding MVP-R (83.26% / 80.16%) by 10.98% in final accuracy and approaching the joint training ceiling (91.66%).

Ablation Study

The individual and combined impacts of DFM and GTP were analyzed on CORe50 (original paper Table 3):

Configuration	DFM Included	GTP Included	AAUC (%)	ALast (%)	Analysis / Observation
MVP Baseline	-	-	58.30	52.84	Prompt tuning baseline
+ DFM	✓	-	64.13	64.57	Removes domain bias; ALast jumps +11.73%
+ GTP	-	✓	63.95	65.28	Stabilizes topology; ALast jumps +12.44%
TopFlow (Full Model)	✓	✓	64.51	66.20	Optimal synergy; total ALast gain +13.36%

In the layer selection ablation for DFM pairs \((n, l)\) (original paper Table 5, without GTP): - Early layer pair \((0, 5)\) degraded performance to 49.92% (below the 52.84% baseline), reflecting semantic mismatch; - Later layer pairs \((5, 10)\) and \((6, 11)\) achieved 52.98% and 54.82% respectively, proving that deep intermediate layers strike the optimal trade-off between semantic compatibility and domain decoupling.

Key Findings

• Complementary Dual Mechanisms: Incorporating either DFM or GTP individually accounts for an over 11% absolute jump in final accuracy on CORe50, verifying that domain suppression and global topological preservation address distinct failure modes.
• Superior Anytime Adaptability: TopFlow achieves remarkable gains on the AAUC metric—outperforming previous SOTA by 10.23% on iDigits and 7.39% on CLEAR100—demonstrating smooth representation evolution without sudden catastrophic dips.
• Model-Agnostic Modularity: Integrating DFM and GTP into other prompt architectures such as CODA-P boosted ALast from 49.13% to 54.62% (original paper Table 6), confirming plug-and-play versatility across continual learning pipelines.

Highlights & Insights

• Empirical Uncovering of ViT Layer-Wise Knowledge: By methodically examining feature geometries and linear probing across Transformer layers, the paper reveals that domain traits dominate early layers while semantic classes coalesce in deep layers, laying an empirical foundation for inter-layer flow matching.
• Memory-Free Global Manifold Regularization: GTP avoids complex Riemannian curvature calculations by elegantly combining parameter-free clustering, Hungarian bipartite assignment, and low-dimensional pairwise relation projections, effectively halting feature collapse in streaming settings.
• Realistic Scenario Formulation: Formalizing Online VIL exposes the critical fragility of classical CIL/DIL methods when exposed to concurrently drifting, boundary-free data streams, bridging the gap between theoretical CL setups and dynamic real-world deployment.

Limitations & Future Work

• Stochastic Task Stream Variance: Because category and domain distributions evolve stochastically across random seeds to simulate natural unpredictability, evaluation results display non-negligible variance across sequence runs.
• Fixed Prototype Capacity: GTP employs a predefined prototype pool size derived via FINCH clustering; in streams with sudden massive spikes in category diversity, a static capacity may restrict representational granularity.
• Future Directions: Exploring Bayesian non-parametric dynamic prototype expansion, and extending TopFlow's geometric flow principles to streaming test-time adaptation for multi-modal vision-language foundation models.

Related Work & Insights

• vs MVP [38]: MVP addresses blurry task boundaries using stochastic masks and visual prompts, but lacks explicit cross-layer domain decoupling; TopFlow adds DFM and GTP, elevating replay-free final accuracy on CORe50 by over 13 percentage points.
• vs ICON [42]: ICON pioneered offline Versatile Incremental Learning with discrete task transitions. When transferred to strict single-pass online streams, ICON degenerates significantly (achieving only 45.15% on CORe50), whereas TopFlow demonstrates robust generalizability across both online and offline regimes.
• vs Conventional GFK [12]: Classical Geodesic Flow Kernel algorithms demand static, fully accessible source and target domains on Grassmann manifolds; DFM reinterprets the depth dimension of neural networks as the geodesic trajectory, enabling online alignment without offline dataset access.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering formulation of Online VIL and novel integration of cross-layer flow matching with relational topology preservation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous multi-benchmark testing, comprehensive buffer-free vs buffer-aware evaluations, layer-wise ablations, and t-SNE topological analyses]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivations, tight problem-solution alignment motivated by empirical ViT probes, and well-structured expositions]
• Value: ⭐⭐⭐⭐⭐ [Significantly advances continual learning towards realistic, boundary-free streaming deployment without costly memory storage]

Related Papers

• [ICML 2025] Autoencoder-Based Hybrid Replay for Class-Incremental Learning
• [ECCV 2026] Hierarchical Style Aggregation for Versatile Chinese Handwriting Generation
• [ECCV 2026] Source-Agnostic Image Translation Based on Latent Aware Adaptive Masking
• [ECCV 2026] Bridging Online and Offline Handwriting via Differentiable Physical Rendering
• [AAAI 2026] ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning