Audio-Visual Continual Test-Time Adaptation without Forgetting

Conference: ECCV 2026
Paper: Official paper page · Paper PDF
Project: AVReCAP
Area: Model Compression (Continual Learning / Test-Time Adaptation)
Keywords: continual test-time adaptation, audio-visual fusion, parameter retrieval, catastrophic forgetting, source-free adaptation

TL;DR

AVReCAP retrieves historical fusion-layer parameters using input statistics instead of continually overwriting one parameter state, achieving 43.51% mean accuracy on VGGSound-2C, 6.28 percentage points above the source model, while substantially reducing source-domain forgetting.

Background & Motivation

Audio-visual deployment shifts need not affect both modalities equally: video may become blurred while audio remains clear, or both may deteriorate together. Test-time adaptation uses current unlabeled inputs to repair the cross-modal associations learned on source data. However, independent adaptation to one domain does not capture the accumulated errors of long-running deployment. Across a sequence of unknown domains, earlier mistakes become the initialization for later updates, amplifying prediction bias and modality imbalance. Even READ, which restricts adaptation to the fusion layer, can suffer this progressive drift.

The paper considers online adaptation without source data or known task boundaries, rather than resetting the source model whenever the domain changes. Replaying past samples violates its storage constraint, but keeping only the latest weights discards fusion configurations that previously worked well. The motivating experiment adapts fusion parameters on one corruption and then freezes them for evaluation on other corruptions. Their transfer often remains competitive with, or better than, the source model, suggesting that historical states can be useful starting points rather than obsolete checkpoints.

AVReCAP turns this observation into a parameter memory indexed by raw-input statistics. Its contribution is continual-state selection and maintenance, not a new audio encoder or recognition task; following the project taxonomy, it belongs to the continual-learning branch of Model Compression. Core idea: initialize each online adaptation step from a historical fusion state whose associated input distribution resembles the current batch, and preserve these reusable states in a bounded buffer.

Method

Overall Architecture

Inputs are paired video frames and audio spectrograms; outputs are predictions within a fixed class vocabulary. The CAV-MAE backbone encodes audio and video separately, concatenates their tokens, processes them through a joint encoder, and applies a classification head. Only the query, key, and value projection matrices of the joint attention layer are adapted; both unimodal encoders and the classifier remain frozen.

The method consists of fusion-parameter snapshots, bimodal statistical retrieval, and bounded buffer maintenance. Each snapshot associates fusion weights with the mean and diagonal covariance of its audio and visual inputs. A retrieval hit supplies historical weights for one adaptation step; a miss keeps the previous step's weights as the initialization. The method then updates the matched entry or inserts a new one, merging similar entries when the buffer reaches its budget without replaying stored samples.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current frames and spectrograms"] --> B["Fusion-parameter snapshots"]
    B --> C["Bimodal statistical retrieval"]
    C -->|"Hit: load historical weights"| D["Frozen encoders and classifier<br/>Adapt fusion layer only"]
    C -->|"Miss: retain current weights"| D
    D --> E["Bounded buffer maintenance"]
    E -->|"Available to later batches"| C
    D --> F["Class prediction"]

The snapshot node denotes the maintained state representation, not an assumption that a populated memory exists at startup. The buffer begins empty: the first batch updates the source fusion layer and supplies the first statistics–parameter entry. Prediction and adaptation belong to the same online batch-processing step; the diagram does not require an additional post-update forward pass on that batch.

Key Designs

1. Fusion-parameter snapshots: retain transferable interactions rather than whole models

CAV-MAE has 11 attention blocks in each unimodal encoder and one attention block in its joint encoder. Following READ, AVReCAP stores and updates only the joint attention projections \(W_Q,W_K,W_V\), rather than copying entire encoders or adapting every normalization layer. The argument is not that fewer trainable parameters automatically prevent forgetting. Figure 3 instead supplies empirical evidence that these adapted fusion matrices retain value on unseen corruptions. For example, parameters adapted to Gaussian Noise transfer to Shot Noise and Impulse Noise; within-category transfer is often stronger, but transfer across categories is also observed.

Each entry pairs these matrices with a compact description of the inputs that produced them. Entries are not experts trained for known domain labels such as rain or noise, because neither domain labels nor boundaries are provided online. They are statistical anchors created during adaptation, and an anchor may support inputs from more than one actual domain. Freezing the backbone limits representation changes, but the additional protection over READ comes from retaining alternative historical starting points rather than merely reducing the number of updated layers.

2. Bimodal statistical retrieval: select a starting state without trusting model confidence

Prediction confidence can become misleading as adaptation drifts, so AVReCAP builds its retrieval descriptor directly from raw inputs. For video, it computes channel-wise means and variances over the batch and spatial dimensions. For audio, it computes per-frequency statistics over the batch and time dimensions of the spectrogram. Each modality is approximated by a Gaussian with diagonal covariance, avoiding raw-sample storage and a separate full-model evaluation for every candidate state. These descriptors capture low-order statistics, not semantics or a complete joint audio-visual distribution.

For every stored entry, the method adds the KL divergence of the current audio distribution from its stored counterpart to the analogous visual divergence. The following expression summarizes the distance explicitly defined in the text, with \(P_u^t\) denoting the current Gaussian approximation for modality \(u\) and \(P_u^n\) the approximation in entry \(n\):

\[ g^*(t,n)=D_{\mathrm{KL}}(P_a^t\Vert P_a^n)+D_{\mathrm{KL}}(P_v^t\Vert P_v^n). \]

The nearest entry is retrieved only if its distance is below threshold \(\tau\). Its three projection matrices replace the current fusion weights before adaptation. If every entry has distance at least \(\tau\), the method continues from the previous step's parameters instead of forcing a historical match. Thus, the threshold controls both reuse and memory expansion; it does not select an expert by measuring its accuracy.

With visual-only corruption, the audio distance may be small and the visual term may drive retrieval more strongly; under bimodal corruption, both terms contribute. Nevertheless, low statistical distance does not guarantee matching semantics or optimal weights: it is an empirical proxy, not a proven selection rule. Equation (5) is corrupted in the local extraction, so this note follows the adjacent prose describing the nearest entry within the threshold rather than reproducing the damaged expression.

3. Bounded buffer maintenance: smooth known states and retain anchors for new inputs

After a retrieval hit and one adaptation step, the selected entry's parameters and input statistics receive exponential moving average updates. The paper gives a typical smoothing factor of \(\beta=0.99\), intended to absorb new observations without allowing one batch to abruptly replace the stored state. The statistics update is described as moment-preserving, so it should not be reduced to blindly averaging variances. Equations (6)–(8) are incomplete in the cached extraction; their exact covariance update is therefore omitted rather than reconstructed.

After a miss, the model first adapts from its current weights and then inserts the new input statistics with the resulting fusion parameters. At the buffer budget \(\eta\), it finds the closest pair of entries using the same bimodal statistical distance and merges their corresponding contents by averaging. This compresses similar states into one representative and leaves capacity for new variation, rather than evicting entries by age or access frequency. Whether weight averaging harms particular domains remains empirical; the experiments show that some finite budgets preserve or improve mean accuracy, not that merging is universally safe.

Storage therefore scales with the number of snapshots rather than the complete sample history, but each snapshot still contains three full projection matrices. “No stored samples” does not mean “only a few scalars” or imply differential privacy. Deployment requires accounting for the entry budget, numerical precision, and pairwise comparison overhead.

A Worked Example

Consider a stream that first contains visual Gaussian noise and later another noise type; this is an illustrative walkthrough, not an additional measured experiment. With an empty buffer, the first batch updates only the fusion layer and creates a statistics–parameter entry. A later batch sufficiently close to that entry retrieves its historical fusion weights rather than inheriting a more recent state that may have drifted under unrelated corruptions. If both modalities are statistically far from every stored entry, adaptation instead continues from the current state and creates another anchor afterward. When capacity is reached, the closest pair of anchors is merged without replaying their original video or audio. The loop needs only the current unlabeled batch, frozen backbone, and parameter buffer—not corruption labels or known transition times.

Loss & Training

The adaptation objective is inherited from READ, not introduced as a new supervised signal. Its confidence term depends on the maximum predicted probability of each sample, as specified by Equation (10):

\[ \mathcal L_{\mathrm{conf}}=\mathbb E_i[-p_{i,\max}\log p_{i,\max}]. \]

This is not the full per-sample class entropy and should not be silently replaced with TENT's objective. A negative-entropy term based on batch-aggregated predictions is added to discourage concentration on too few classes. The class indexing and normalization in the cached Equation (11) are insufficiently clear, so its role is explained without supplying a supposedly repaired formula.

Online adaptation uses Adam with learning rate \(10^{-4}\) and batch size 32. The threshold is \(\tau=0.005\) for unimodal corruption and \(\tau=0.01\) for bimodal corruption. Each batch is used once, and domain changes do not trigger a source-model reset; loading a historical snapshot is different from resetting to source parameters. The source model is trained on the corresponding dataset beforehand, but source samples and target labels are unavailable during adaptation. The main text does not clearly specify how Adam momentum states are handled when weights are switched, leaving an implementation detail that should not be assumed immaterial.

Key Experimental Results

Main Results

Experiments use the Kinetics50 and VGGSound test sets; all reported accuracies are higher-is-better. The unimodal settings contain 15 visual or six audio corruptions, while the bimodal setting uses 15 corruptions from AVRobustBench, all at severity 5. Each corruption is applied to the entire test set and forms one sequential task. The table reports mean accuracy across corrupted tasks, not clean source-test accuracy.

Dataset and corruption	AVReCAP budget	AVReCAP accuracy (%)	Reference and accuracy (%)	Difference (percentage points)	Source
Kinetics50-C, visual	50	62.75	SuMi: 61.10	+1.65	Table 1, upper block
VGGSound-C, visual	300	56.20	SOURCE / SAR: 55.94	+0.26	Table 1, lower block
VGGSound-C, audio	400	30.61	PTA: 29.26	+1.35	Table 2, lower block
VGGSound-2C, bimodal	Unbounded	43.51	SOURCE: 37.23	+6.28	Figure 5b

The final row intentionally compares against the unadapted SOURCE model, not a claimed strongest adaptation baseline; SuMi reaches 37.24% in Figure 5b. Finite and unbounded budgets are different configurations, and these means do not imply wins on every corruption. BriMPR* removes source-data access from BriMPR, so its results do not represent the original method under its original conditions; the comparison concerns the paper's source-free setting.

Ablation Study

The following selection from Table 3 changes the retrieval distance rather than the classifier. Metrics are mean accuracy in percent, higher being better; K50-C denotes visual corruption and K50-2C bimodal corruption. The full-method values correspond to the unbounded-buffer results.

Retrieval distance	K50-C visual	K50-2C bimodal	Question tested
Sum of Euclidean distances between means and standard deviations	49.47	48.79	Statistical comparison without KL
Visual KL only	61.53	49.28	Ignoring audio statistics
Audio KL only	60.52	49.70	Ignoring visual statistics
Audio KL + visual KL	62.37	50.29	Full bimodal retrieval

Bimodal KL exceeds the Euclidean alternative by 12.90 percentage points on K50-C, indicating that storing historical parameters alone is insufficient: routing matters. This does not establish that KL is optimal for every task, because distance scale, thresholds, and the statistical model jointly influence selection.

Key Findings

• The task-dominant modality matters. Kinetics50 is visually dominant and VGGSound audio dominant; gains tend to be larger when the dominant modality is corrupted and can be small when it remains clean.
• Reduced forgetting is not zero forgetting. Figure 7 evaluates the final adapted state on the source test set: after VGGSound-2C adaptation, AVReCAP loses 2.9 percentage points of source accuracy, versus 27.9 for READ.
• A larger buffer is not always better. Figure 8b reports 43.90% on VGGSound-2C at budget 200, compared with 43.51% for an unbounded buffer, consistent with merging sometimes providing regularization.
• The threshold explanation is internally inconsistent. Section 5.2 associates larger thresholds with larger buffers, whereas Section 4.3 inserts an entry only when every distance is at least the threshold; holding statistics fixed, increasing the threshold should reduce insertion opportunities.

Highlights & Insights

• Changing the adaptation starting point can matter more than another loss modification. The loss is inherited from READ; the new mechanism escapes a single continually overwritten parameter trajectory by preserving selectable states.
• Statistical indexing and parameter memory have distinct roles. Cheap input descriptors route retrieval, while heavier fusion matrices retain past adaptations, avoiding a complete model evaluation for each candidate.
• One distance supports retrieval and capacity control. The same criterion guides reuse, insertion, and merging, simplifying the method while exposing multiple stages to any descriptor bias.

Limitations & Future Work

• Author-stated modeling assumptions: diagonal-Gaussian descriptors can be noisy at small batch sizes. Batch-size experiments examine this issue, but do not establish reliable single-sample operation.
• Scope of the authors' setting: the main evidence uses CAV-MAE, two corruption benchmarks, a fixed class space, and a sequence of non-repeated domains—not a comprehensive test of emerging classes or long-running natural drift.
• Reader assessment: similar pixel or frequency statistics need not imply similar semantics, and the descriptors do not model complete cross-modal dependence. A false match can select an unsuitable parameter state.
• Reader assessment: finite capacity does not automatically mean low memory or latency. End-to-end timing, parameter-memory costs, and variation over randomized domain orders are not sufficiently characterized in the main text.
• Evidence boundary: the local paper contains the full method and main experiments but no appendix. Several extracted equations are damaged, and the threshold explanation conflicts with the insertion rule; unverifiable formulas are omitted, and “without Forgetting” is not treated as a literal guarantee.

Related Work & Insights

• Versus READ: both adapt fusion projections with the same loss; AVReCAP adds statistics-indexed historical-state retrieval and maintenance, rather than a new recognition backbone.
• Versus TENT, EATA, and SAR: these emphasize normalization parameters, entropy objectives, or stable updates, while AVReCAP emphasizes reusable cross-modal fusion states. These experiments do not imply that the other methods fail on all vision-only tasks.
• Versus SuMi and PTA: SuMi emphasizes information sharing and PTA prediction bias; AVReCAP targets parameter-trajectory drift across time, addressing a different source of error.
• Versus sample replay: what is reused is an adaptable weight initialization, not past training examples. Sample-replay guarantees about forgetting or privacy therefore do not transfer automatically.

Rating

• Novelty: 3.5/5 — A clear combination of fusion-state memory and statistical retrieval, built on existing adaptation layers and losses.
• Experimental Thoroughness: 3.5/5 — Covers unimodal/bimodal corruption, distance, budget, and forgetting, but natural drift and resource costs remain underexplored.
• Writing Quality: 3/5 — The main mechanism is understandable, although the threshold explanation and some implementation details need clarification.
• Value: 4/5 — Provides a concrete, testable parameter-reuse strategy for source-free audio-visual continual adaptation.