Forget, Anticipate and Adapt: Test Time Training for Long Videos¶

Conference: ECCV 2026
arXiv: 2606.26515
Code: https://github.com/rajatmodi62/ffn (project page)
Area: Semantic Segmentation
Keywords: Test-Time Training, Long Videos, Semantic Segmentation, Adaptive Window, Surprise

TL;DR¶

This paper proposes the Frame Forgetting Network (FFN). By adopting a "forget-anticipate-adapt" mechanism that processes only the leaving and entering frames of a sliding window, FFN reduces the computational complexity of test-time training (TTT) for long videos from \(O(k)\) to a constant \(O(1)\). Moreover, it dynamically determines when to execute TTT based on a "surprise" metric, achieving a superior accuracy-efficiency trade-off across dense segmentation, depth estimation, and action classification.

Background & Motivation¶

Test-Time Training (TTT) provides an effective unsupervised path for model adaptation during the inference phase: the model continuously updates its weights at test time through self-supervised tasks (such as image reconstruction or masked autoencoding) to adapt to the shifting scene distributions in video streams. However, deploying TTT in the video domain has been severely bottlenecked by computational overhead. Existing methods invariably employ a fixed-size sliding window containing the past \(k\) frames, computing the self-supervised loss and backpropagating gradients over all frames within the window at each slide step. This means that the computational cost scales linearly with the window size. For a two-hour video (around 7200 frames), recomputing the entire window at each step would take over 8 hours to complete, making it highly impractical for offline scenarios like UAV disaster relief or edge device deployment. Worse still, even when adjacent frames are virtually identical (e.g., static surveillance), the model wastes computation on redundant adaptation, targeting repetitive content that will not reappear in the future.

The root cause of this redundancy lies in a long-overlooked observation: when the sliding window advances from \(W_t\) to \(W_{t+1}\), only two frames out of the \(k\) frames actually change—frame \(x_{t-k}\) leaves the window, while frame \(x_{t+1}\) enters, leaving the remaining \(k-2\) frames completely identical. Although prior works like TTT-Online adhere to the "locality principle" by resetting weights (so that earlier video frames do not bias later adaptation), the practice of recomputing window-by-window is essentially reprocessing already-adapted frames. This creates a difficult trade-off between computational efficiency and accuracy gains. Therefore, similar to sliding window techniques in data structures (such as Kadane's algorithm), can we process only the leaving and entering frames while maintaining an implicit running state to preserve temporal context?

From this data structure perspective, this paper proposes the Frame Forgetting Network (FFN). FFN consists of two core modules: the Memory Restoration Mechanism (MRM), which actively "forgets" the adaptive influence of the leaving frame by pulling the backbone weights back to their state prior to adapting to that frame; and the Adaptive Window Algorithm (AWA), which dynamically determines whether the current frame requires adaptation based on "surprise"—specifically, the model predicts the next frame and compares it with the ground truth to measure the "amount of new information" from both pixel-space and latent-space levels, executing TTT only when the gap exceeds a dynamic threshold. Core Idea: By utilizing the "two-frame change" invariance of sliding windows, the computing paradigm of long video TTT is transformed from \(O(k)\) recomputation of the entire window to \(O(1)\) processing of only three frames (leaving, entering, and current). Active forgetting combined with surprise-driven adaptive decision-making achieves a 5.8x speedup while substantially improving dense segmentation accuracy.

Method¶

Overall Architecture¶

The core insight of FFN is that only two frames change when the sliding window advances. Therefore, it only needs to process three frames: the leaving frame \(x_{t-k}\) (responsible for forgetting), the current frame \(x_t\) (responsible for decision-making), and the entering frame \(x_{t+1}\) (responsible for validation). The entire framework runs in a streaming fashion. At each step, MRM first forgets the adaptive impact of the leaving frame, and then AWA computes the surprise of the current frame to decide whether to execute TTT. If the surprise exceeds a dynamic threshold, a single gradient update is performed on the backbone and SSL heads; otherwise, it only undergoes a single forward inference pass before directly advancing to the next time step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Streaming Video Input<br/>Only past and current frames visible at each step"] --> B["Window slides W_t → W_{t+1}<br/>Only two frames change"]
    B --> C["Memory Restoration<br/>Mechanism: Forget leaving frame"]
    B --> D["Adaptive Window<br/>Algorithm: Compute surprise"]
    D --> E{"S_t ≥ Dynamic Threshold τ_t?"}
    E -->|No| F["Inference Only<br/>Single forward pass"]
    E -->|Yes| G["Execute TTT<br/>1-step gradient update for backbone + SSL head"]
    C --> H["Advance to next time step"]
    F --> H
    G --> H

Key Designs¶

1. Memory Restoration Mechanism (MRM): Actively Forgetting the Leaving Frame's Adaptation to Maintain Locality

Since TTT continuously updates the backbone weights, when frame \(x_{t-k}\) leaves the window, the model's current weights \(f_t\) still "remember" the adaptation to that frame. If subsequent frames are processed with this adaptation, the influence of old frames will continue to bias current predictions, violating the locality principle. The idea of MRM is: during the training phase, the pre-adaptation features \(f_{t-k-1}(x_{t-k})\) of \(x_{t-k}\) when it just entered the window are cached. When this frame leaves the window, the backbone weights are "pulled back" by minimizing the distance between the current features and the pre-adaptation features. Here, an \(L_2\) loss is used to calculate the difference, and a single gradient backpropagation step is performed to adjust the weights of \(f_t\). To distinguish the same frame at different time steps (since the pre-adaptation features differ when the same pixel content enters the window at different times), MRM also introduces a temporal module using a three-layer MLP: the time step \(t-k\) is encoded into a 1D sinusoidal position embedding and injected into the backbone, which essentially tells the backbone "which frame's feature this is." This time-conditional forgetting is more delicate than directly resetting weights—it only erases the local influence of the leaving frame while retaining the useful knowledge learned by the backbone when adapting to other frames, thereby maintaining the internal representation continuity of the model.

2. Adaptive Window Algorithm (AWA): Dynamically Determining When to Adapt Based on "Surprise"

Executing TTT at every step of the sliding window is clearly suboptimal—adjacent frames are often highly similar, or they may exhibit large pixel differences due to camera shake while remaining semantically unchanged. The core of AWA is to let the model self-evaluate whether adaptation is needed by "predicting the next frame." It combines two complementary signals. The first is the visual-space difference \(v_{\text{visual}}\): the current frame \(x_t\) is fed into a prediction head (sharing parameters with the SSL head) to generate the next-frame prediction \(x'_{t+1}\), and the pixel-wise normalized \(L_2\) distance is calculated. Using \(v_{\text{visual}}\) alone can misjudge camera shake—where pixels change dramatically but the scene does not. Therefore, a second signal is introduced, namely, the latent-space similarity \(A\): a memory buffer \(B\) is maintained to record the backbone latent features of all recent frames where TTT was executed, and the cosine similarity between the current frame's latent features and those of the most recently adapted frame is calculated. When the scene remains unchanged, even if the pixel difference is large, the latent features remain highly similar (\(A \approx 1\)). Combining these two yields the surprise metric:

\[S_t = [\log(1+v_{\text{visual}}(t))] \times [1-A]\]

When \(S_t\) exceeds the dynamic threshold \(\tau_t\), TTT is triggered. \(\tau_t\) is defined as the sum of the mean and standard deviation of surprise over the most recent \(W\) frames (buffer size of 50): \(\tau_t = \mu_t + \sigma_t\). This allows it to adaptively adjust to video pacing changes: the threshold automatically rises for slow-moving videos and adapts for scene cuts. At the same time, this decision-making mechanism naturally handles three typical scenarios: static frames (\(v_{\text{visual}} \approx 0, A \approx 1 \rightarrow S_t \approx 0\), no trigger), camera shake (\(v_{\text{visual}} \uparrow\) but \(A \approx 1 \rightarrow S_t\) is low, no trigger), and scene cuts (\(v_{\text{visual}} \uparrow\) and \(A \downarrow \rightarrow S_t\) is high, triggers adaptation).

Loss & Training¶

A standard two-stage TTT training is adopted. Training phase: Jointly train the backbone \(f\), the SSL head \(g\), and the downstream head \(h\). Surprisingly, the SSL task is set to "next frame prediction" instead of the common current frame reconstruction (ablation studies demonstrate that the former introduces a stronger inductive bias). Testing phase: Freeze the downstream head \(h\) and only update \(f\) and \(g\), with at most one gradient update per frame using the Adam optimizer and a memory buffer size of \(W=50\). During the cold-start phase, adaptation is forced for the first 60 frames to fill the buffer (introducing an initial latency of about 42 seconds), after which AWA takes over the adaptive decision-making.

Key Experimental Results¶

Main Results¶

Dataset	Metric	Online TTT-MAE (s.w.)	TTT-Online (n.o. s.w.)	FFN (Ours)	Gain
COCO-Videos Inst.	AP	37.6	35.3	45.1	+7.5
COCO-Videos Pan.	PQ	21.7	20.8	29.6	+7.9
KITTI-STEP Val.	mIoU	55.4	48.1	57.3	+1.9
KITTI-STEP Test	mIoU	54.3	51.7	59.5	+5.2
EpicTours Sem.	mIoU	42.8	—	49.3	+6.5
EpicTours Inst.	AP	31.4	—	36.7	+5.3

Method	Time (s/frame)	Notes
Online TTT-MAE (s.w.)	4.1	Recompute entire window per step
Self-Train	6.6	Pseudo-label self-training
FFN (Ours)	0.7	Process only 3 frames
Pure inference (no TTT)	1.8	Single forward pass

Ablation Study¶

Configuration	COCO Inst. AP	KITTI Val. mIoU	Description
Full FFN (Absolute encoding + MBO + Cosine)	45.1	57.3	Full model
+ RoPE positional encoding	46.8	58.9	Rotary positional encoding further improves by +1.7↑
+ FIFO + MBO buffer	45.7	59.1	Best combination of both
+ L1 forgetting loss	43.2	56.1	Inferior to cosine loss
FIFO buffer	43.9	56.4	Only first-in-first-out
Remove temporal condition	42.7	—	Do not feed time \(t\) to backbone

Key Findings¶

Temporal conditions in MRM are indispensable: Removing the temporal encoding injected into the backbone drops the COCO-Videos instance segmentation AP from 45.1 to 42.7. This indicates that the backbone requires knowledge of "which frame's feature this is" to accurately restore the pre-adaptation state.
Effectiveness of dynamic surprise thresholding: The performance of FFN remains stable or even slightly increases over a 3-hour long video, whereas TTT-Online quickly degrades after 50 minutes. The dynamic decision-making mechanism effectively prevents weight drift caused by continuous over-adaptation.
Moderate buffer capacity is optimal: Increasing the buffer size beyond 50 frames leads to performance degradation, validating the locality principle of "keeping only recent key frames." Excessively large buffers cause old frame statistics to slow down the threshold's responsiveness to scene changes.
Next-frame prediction outperforms current-frame reconstruction: Predicting future frames as the SSL objective equips the model with a "sense of direction" regarding scene dynamics. This yields additional gains in action classification tasks that require distinguishing directions (e.g., "moving up/down") on Something-Something v2.

Highlights & Insights¶

Introducing data structure concepts of sliding windows to neural network adaptation: Concepts like "processing only leaving/entering frames" (e.g., Kadane's algorithm) are classic in algorithms, but implementing them in a TTT weight update environment requires sophisticated forgetting and restoration mechanisms. This work demonstrates that this concept is highly viable in neural networks.
"Forgetting as restoration" rather than "forgetting as discarding": The elegance of MRM lies in not discarding the knowledge from the leaving frame. Instead, it utilizes an \(L_2\) loss to "pull back" the backbone to its pre-adaptation state. This simultaneously erases local bias while preserving global learning achievements, offering higher flexibility than hard weight resets.
Dual-signal design of surprise \(S_t\): The combination of visual discrepancy and latent-space similarity circumvents the pitfalls of pure pixel-based differences. The dynamic threshold \(\tau_t = \mu_t + \sigma_t\) allows the system to autonomously adapt to various video tempos (e.g., slow surveillance vs. rapid urban drive) without pre-defined constants.
EpicTours dataset fills a evaluation gap in long video TTT: Previously, the longest video datasets were only around 5 minutes. EpicTours provides real-world urban roaming videos up to 3 hours long with dense annotations, offering a highly realistic evaluation platform for subsequent research.

Limitations & Future Work¶

Cold-start latency: The model requires forced adaptation on the first 60 frames (approx. 42 seconds) to initialize the memory buffer, during which the computational savings of AWA cannot be enjoyed. For short videos with frequent scene switches, the impact of this delay is more pronounced.
Backpropagation remains the primary bottleneck: Even with only one gradient update step per frame, backpropagation itself is still significantly more expensive than pure inference. The authors discuss backpropagation-free alternatives (like forward-forward algorithms and target propagation), but these cannot currently substitute backpropagation in large-scale vision tasks.
Inefficiency in long-term static scenes: When a security camera captures no scene changes for a long time, FFN remains idle. The authors suggest that self-training via Langevin dynamics sampling could be executed during idle periods, but the concrete implementation remains unexplored.
Limited category granularity in EpicTours: Currently, only 30 COCO-subset classes are annotated. Future work can extend this to fine-grained categories and more diverse environments.

vs. TTT-Online (Wang et al., JMLR 2025): A direct baseline that recomputes the entire sliding window at each step (\(O(k)\) complexity) and resets weights. FFN achieves \(O(1)\) complexity per step while maintaining weight continuity via MRM+AWA, outperforming TTT-Online by 7.5 points in COCO-Videos instance segmentation AP while being 5.8x faster.
vs. Online Model Distillation (Mullapudi et al., 2018): Student-teacher dual-model architectures require deploying the teacher on a remote server, which is unsuitable for offline/edge scenarios. FFN runs fully locally using a single model.
vs. Tent (Wang et al., 2020): Tent leverages entropy minimization for test-time adaptation, but it only updates normalization layers and does so for every frame. FFN dynamically decides when to adapt and updates all backbone parameters, leading to a substantial lead in dense segmentation.
vs. Surprise in Video Compression: The philosophy of deciding compression bitrate based on inter-frame differences in video coding is similar to AWA. However, FFN implements this through a learnable prediction head and compresses all knowledge into model weights rather than coded files.

Rating¶

Novelty: ⭐⭐⭐⭐ MRM introduces the forgetting-restoration concept to TTT scenarios, and AWA's dual-signal surprise mechanism is cleverly designed. However, the overall scheme is inspired by classic sliding-window data structures and is not a disruptive innovation.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated across three tasks (segmentation, depth estimation, and action classification) over 11 datasets. The authors also constructed the EpicTours long video evaluation set, with ablation studies thoroughly covering every design choice of each module.
Writing Quality: ⭐⭐⭐⭐ Progressive reasoning from problem insight \(\rightarrow\) sliding window invariance \(\rightarrow\) three-frame mechanism. The figures are clear, and the discussion section is honest and inspiring.
Value: ⭐⭐⭐⭐⭐ The computational bottleneck of TTT on long videos is a critical obstacle to its practical adoption. FFN provides a validated, constant-complexity solution that directly accelerates edge deployment and real-time applications.