ECCV 2026 Self-Supervised Learning test-time training linear-complexity vision backbone visual sequence modeling gradient-updated hidden state high-resolution efficiency

Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/imKQv/Vision-TTT
Area: Self-Supervised Learning
Keywords: test-time training; linear-complexity vision backbone; visual sequence modeling; gradient-updated hidden state; high-resolution efficiency

TL;DR¶

The paper brings Test-Time Training (TTT) from NLP into vision: it treats an image's token sequence as a dataset and updates a per-head hidden state online by gradient descent on a self-supervised reconstruction loss, then adds a dual-dataset strategy and Conv2d dataset preprocessing to turn the originally unidirectional TTT into a 2D-aware vision backbone — Vittt-T/S/B reach 77.7/81.8/82.7 Top-1 on ImageNet and, at 1280×1280, save 79.4% FLOPs, run 4.72× faster and use 88.9% less memory than DeiT-T.

Background & Motivation¶

Visual representation learning has long traded expressiveness against efficiency. CNNs build spatial hierarchies cheaply, but their convolution kernels are static once trained, which caps how far performance scales; Vision Transformers model the visual sequence with patch-wise self-attention and have demonstrated far better scalability in large-scale experiments, making them the default backbone in both academia and industry. The price is quadratic complexity in the number of tokens — acceptable for 224×224 inputs with 196 tokens, but at high resolution (1280×1280 means 6400 tokens) both compute and memory explode at inference time. The community therefore turned to RNN-style linear-complexity sequence models: the Vision Mamba family uses selective-scan state space models (SSMs) with 2D scan paths, Vision-RWKV introduces temporal decay, and ViG adopts gated linear attention. These bring complexity back to linear, but their hidden-state update rules are predefined — an SSM forgets multiplicatively through a structured (typically diagonal-sparse) decay matrix, while linear attention accumulates history into a matrix product through a fixed kernel function. Their "memory" is written into the formula before training ever starts, and cannot adapt to the content of the particular image being processed.

TTT offers a different route: it treats a token sequence X=[x₁,…,x_T] directly as a dataset, maintains a hidden state W, and for every incoming token computes a gradient on a self-supervised task, takes one gradient-descent step on W, and then applies W to the current token to produce the output. The hidden state is thus rewritten not by a predefined rule but by gradients — what to learn and what to forget is decided by the data itself. The authors exploit exactly this to draw a Gradient Magnitude Map and advertise it as a visual explanation tool on par with attention maps. But vanilla TTT was designed for language: it is a unidirectional causal sequence model with an inherent temporal dependence, and porting it to images discards 2D spatial locality — neighbouring patches are forced onto a one-dimensional chain, and global modelling degrades.

The core idea of this paper is therefore to turn TTT from a 1D language sequence model into a 2D vision learner: keep the mechanism that gives it expressiveness (a hidden state updated online by self-supervised gradients), restore bidirectionality through a dual-dataset strategy, augment the dataset itself in 2D with a depth-wise convolution, and use multi-head hidden states plus mini-batch gradient descent to convert sequential computation into GPU-friendly small matrix multiplications — obtaining linear complexity and a global receptive field within a single backbone.

Method¶

Overall Architecture¶

Vision-TTT is a plain (non-hierarchical) vision backbone with three stages: ① Patchification — the image is split into 16×16 patches, linearly projected into a token sequence and given positional encodings; ② the Vision-TTT Encoder — L hybrid blocks in series, each a Vittt block (residual) followed by a SwiGLU MLP (residual), both with pre-LN; ③ Task Adapters — for classification, LayerNorm plus mean pooling plus a linear head on the last layer's output; for downstream detection/segmentation, the backbone is frozen and only feature adapters and task heads are trained. Everything novel lives inside the Vittt block: instead of aggregating information by "weighting all keys against a query" as self-attention does, it views this layer's token sequence as a small dataset, runs a few steps of mini-batch gradient descent on a self-supervised reconstruction task to compress the sequence semantics into a small per-head hidden-state matrix W, and then queries the current token with the updated W. In other words, the TTT layer is a linear layer whose weights change online with the input, both at training and at inference time — which is literally what "test-time training" means.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input image"] --> B["Patchification<br/>patches + positional encoding"]
    B --> C["Token sequence as a dataset<br/>self-supervised gradients update the state"]
    C --> D["Dual dataset strategy<br/>forward / backward routes in parallel"]
    D --> E["Conv2d dataset preprocessing<br/>depth-wise conv injects 2D locality"]
    E --> F["Multi-head state + mini-batch descent<br/>small matmuls mapped to Tensor Cores"]
    F --> G["SwiGLU MLP + task adapters"]
    G --> H["Classification / detection / segmentation"]

Key Designs¶

1. Treating the token sequence as a dataset: the hidden state is updated online by self-supervised reconstruction gradients

This is the foundation of the whole paper. Given a visual sequence X=[x₁,…,x_T], the model first projects it into key, query and value views as attention would: X^K=θ_K X, X^Q=θ_Q X, X^V=θ_V X. It then sets up a deliberately simple self-supervised task — reconstruct the value view from the key view — with an L2 reconstruction loss:

\[\ell(\mathrm{W}_{t-1};x_t)=\lVert \mathrm{W}_{t-1}x_t^{K}-x_t^{V}\rVert^{2}\]

The token pairs (X^K, X^V) are therefore the "training set" (X^V the label view, X^K the input view), while X^Q is the "test set": the output token is produced by querying with the updated hidden state, \(z_t=\mathrm{W}_t x_t^{Q}\). What is being trained is not the network but a per-head d×d hidden-state matrix W, which acts as an associative memory — it writes the (key, value) relations already seen into itself online and then reads the current token out of that updated memory. The most essential difference from SSMs and linear attention hides in this reconstruction task: an SSM evolves its state multiplicatively, h_t=Ā_{t:s}B̄s x_s, and causal linear attention accumulates outer products, s_t=Σφ(K_s)V_sᵀ, whereas the TTT update explicitly contains a subtractive discrepancy term (V_s − WK_s) — the model writes into the state only how much the current token still differs from the global memory. The authors argue that this string of W_{s−1}K_s terms acts as a sequence of relative-position anchors that inherently preserve locality, so TTT is not merely "the same thing viewed through gradients" but a position-aware vision learner.

Differentiating the reconstruction loss with respect to W_{t−1} gives the gradient at that timestep, \(G_t = 2(\mathrm{W}_{t-1}x_t^{K}-x_t^{V})(x_t^{K})^{\mathsf T}\), whose magnitude directly reflects how much "new information" in this token the memory fails to explain. Visualising G_t over the patch grid (Figs. 3 and 9 of the paper) shows larger gradients in semantically salient regions and a sharper distribution after training, which is why the authors claim the Gradient Magnitude Map is a built-in, patch-level interpretability tool for Vision-TTT, comparable to ViT attention maps and Vision Mamba activation maps.

2. Dual dataset strategy: restoring 2D bidirectionality with forward and backward datasets

The update above is strictly causal: the t-th token only sees the tokens before it. That is a sound inductive bias for text, but for images it forces information flow into a single chain, so patches to the right and below can never influence the ones at the top left. The authors' fix is straightforward — since a TTT "dataset" is just a set of token pairs, build a second dataset in the reverse direction, (X'^K, X'^V)={(x'^K_t, x'^V_t)}, t=T,…,1, let the same self-supervised task run simultaneously along a forward route (Z_forth) and a backward route (Z_back), each maintaining its own hidden state, and finally flip the backward output back and gate-fuse it with the forward one:

\[\mathrm{Z}_{\text{Dual}}=\sigma(\texttt{Linear}(X_{\text{Conv2d}}))\odot\bigl(\mathrm{Z}_{\text{forth}}\ \|\ \texttt{flip}(\mathrm{Z}_{\text{back}})\bigr)\]

The gating follows the vanilla TTT block (a sigmoid gate computed from the input itself modulates the output), so going bidirectional still amounts to a per-channel soft selection rather than a plain sum. The idea is borrowed from bidirectional RNNs, but in the TTT setting it buys something extra: the backward route covers the other scan direction of the 2D image. In the ablation it contributes +1.9% classification accuracy, +1.1% APb, +1.0% APm and +1.5% mIoU — the single largest gain on the design route — at the cost of a 38.2% drop in FPS, which is exactly why the next-but-one design had to win efficiency back.

3. Conv2d dataset preprocessing: injecting 2D locality into the dataset itself

Two directions are still not enough. The original TTT block applies only a Conv1d preprocessing step before feeding tokens into the self-supervised task (X_Conv1d), which amounts to admitting that the input is a 1D sequence: patches have no notion of neighbourhood, and pixels that are adjacent in the image but not adjacent in scan order (the end of one row and the start of the next) never meet. This paper replaces that step with a depth-wise convolution: X_Conv2d = DWConv(X), which first lays the patch sequence back onto a 2D grid, applies a local convolution, and unfolds it again — so what enters the dataset is no longer an isolated token but a token carrying its neighbourhood context. Depth-wise separable convolution is chosen over a dense one to keep the parameter cost at 0.02M / 0.04M / 0.08M for T/S/B while still adding 0.1–0.2% on every metric. The numbers look small, but the Effective Receptive Field (ERF) picture reveals its structural role: vanilla TTT has a banded/fan-shaped ERF around the central pixel, the "+ Dual" version covers the whole image but with a directional bias, and only with Conv2d preprocessing does the ERF become a globally radial, isotropic blob matching the global coverage of DeiT and Vim. The first two designs determine how far the model can see; this one determines whether it sees isotropically.

4. Multi-head hidden states plus mini-batch gradient descent: mapping small matrix multiplications onto Tensor Cores

Vanilla TTT has an engineering flaw: one gradient step per token means step t must wait for step t−1, a fully serial computation that leaves GPU parallelism idle. The authors make two changes. First, multi-head: the original D×D hidden state is split into nh matrices of d×d (D=nh×d), each head keeping its own memory, which shrinks individual matrices and raises parallelism. Second, the granularity of gradient descent goes from 1 to 16: instead of updating per token, the sequence is split into mini-batches of size b=16, with the causal form preserved inside a batch (the state at step t within the batch is W₀ minus the sum of the first t gradients, \(\mathrm{W}_t=\mathrm{W}_0-\eta\sum_{s=1}^{t}G_s\)) while batches run in parallel. The value 16 is not arbitrary: 16×16 is the native granularity of Tensor Core small-matrix multiplication, and the authors hand-write the forward and backward kernels in Triton, borrowing Mamba's kernel fusion and recomputation to cut the memory footprint from a sum of terms down to O(BTD). This step is what turns "linear complexity" from a theoretical statement into measured linear throughput. The paper's complexity comparison is Ω(ViT)=4TD²+2T²D, Ω(Vim)=6TD²+18T(2D)N and Ω(Vittt)=6TD²+6TDd+4bTD (b=16, d=64, N=16 being Mamba's state expansion factor): ViT is quadratic in sequence length, Vim and Vittt are both linear. The price is that Vittt's constant factor carries an extra term tied to the embedding dimension D, which explains why Vittt-S/B have slightly more theoretical FLOPs than Vim-S/B; in wall-clock terms Vittt still wins across the board, because Vim's selective scan runs on CUDA Cores whereas Vittt's small matmuls feed the Tensor Cores directly.

A Worked Example¶

Take Vittt-T processing a 224×224 image. Patchifying into 16×16 pieces yields 196 tokens, which after embedding and positional encoding enter the first Vittt block. Inside that block the 196 tokens are first augmented in 2D by the depth-wise convolution and then projected into K/Q/V views; the dataset is duplicated into a forward (X^K,X^V) and a backward (X'^K,X'^V) copy, each starting from its own initial W₀. With b=16, each route takes ⌈196/16⌉=13 mini-batch gradient-descent steps, so a single layer performs roughly 26 memory writes for this image. Each head's memory is only 64×64; after reading all 196 tokens it holds a compressed semantic summary of the image. The two routes are then flipped, gated and fused, passed through the SwiGLU MLP, and handed to the next block. For comparison, the concurrent baseline ViT3 performs a single gradient-descent step over the whole sequence (mathematically equivalent to gated linear attention), whereas Vittt performs T/b steps — which is precisely where its lead on long-sequence tasks such as detection and segmentation comes from.

Loss & Training¶

The internal self-supervised loss is the L2 reconstruction term above; it needs no extra labels and keeps acting during both training and inference. The outer supervised training follows the DeiT and Swin recipes (full hyper-parameters in Appendix D of the paper): ImageNet-1K pretraining uses mean pooling plus a linear classification head with a cross-entropy objective. For downstream adaptation, COCO2017 detection/instance segmentation uses a ViT-Adapter with a Mask R-CNN head at 1333×800, and ADE20K semantic segmentation uses a UperNet head at 512×512; in both cases the encoder backbone is frozen and only the feature extractors and task heads are trained. The initial hidden state W₀ can either be initialised once before training or made a learnable parameter trained under supervision — an option ablated below.

Key Experimental Results¶

Main Results¶

ImageNet-1K classification (224² inputs, against linear-complexity / TTT-style baselines):

Method	#Param.	FLOPs	Top-1 (%)
DeiT-T	6M	1.3G	72.2
VRWKV-T	6M	1.2G	75.1
Vim-T	7M	1.5G	76.1
ViT3-T	6M	1.2G	76.5
Vittt-T	7M	1.4G	77.7
DeiT-S	22M	4.6G	79.8
Vim-S	26M	5.2G	80.3
ViT3-S	24M	4.8G	81.6
Vittt-S	26M	5.3G	81.8
DeiT-B	86M	17.6G	81.8
VRWKV-B	94M	18.2G	82.0
Vim-B	98M	18.8G	81.9
ViT3-B	90M	18.0G	82.6
Vittt-B	102M	20.3G	82.7

At matched scale Vittt-T/S beat Vim-T/S by +1.6% / +1.5%, and Vittt-B beats DeiT-B, VRWKV-B and Vim-B by +0.7–0.9%; all three sizes also surpass ViT3, the other TTT-based vision model. Note that ViT3 uses fewer parameters and fewer FLOPs, and Vittt-B's 20.3G is the highest in this tier — so these are same-scale rather than iso-FLOPs comparisons. (The full table including hierarchical baselines such as RegNet, ConvNeXt, Swin and VMamba is Tab. 2 of the paper.)

Downstream dense prediction (COCO2017 detection reporting APb / APm with FLOPs at 1333×800; ADE20K segmentation reporting mIoU with FLOPs at 512×512):

Method	#Param.	Det. FLOPs	APb	APm	Seg. FLOPs	mIoU
ViT-T	8M	147.1G	41.6	37.9	20.9G	42.6
Vim-T	9M	76.7G	41.8	38.2	18.4G	43.4
ViT3-T	8M	69.2G	42.0	38.3	16.7G	43.6
Vittt-T	9M	74.3G	42.9	38.7	17.8G	44.3
Vim-S	32M	203.6G	44.9	40.2	49.7G	47.0
ViT3-S	30M	195.6G	45.4	40.7	47.7G	47.6
Vittt-S	32M	206.3G	46.3	41.4	50.3G	48.4
Vim-B	111M	611.5G	46.6	41.6	149.2G	48.9
VRWKV-B	107M	599.0G	46.8	41.7	146.0G	49.2
ViT3-B	103M	597.1G	47.4	42.1	145.7G	49.4
Vittt-B	115M	646.7G	48.6	43.0	157.7G	50.3

Vittt-T beats Vim-T by +1.1 APb / +0.5 APm / +0.9 mIoU, the margin widens for Vittt-S over Vim-S (+1.4 / +1.2 / +1.4), and Vittt-B still holds +2.0 / +1.3 / +1.1 over VRWKV-B. Notably ViT3 has already caught up with Vittt on classification but falls clearly behind on dense prediction, which the authors attribute to its single gradient-descent step; when the task sequence grows from 512×512 (1024 tokens) to 1333×800 (4200 tokens) Vittt's advantage does not shrink, suggesting multi-step adaptation pays off more on longer sequences.

Efficiency (1280×1280, batch 64, L40S): Vittt-T/S/B save 79.4% / 66.3% / 48.9% FLOPs, run 4.72× / 4.23× / 3.88× faster and use roughly 89.0% less memory than DeiT-T/S/B. The linearly growing FLOPs curve matches the theoretical complexity analysis; against Vim, Vittt-T uses fewer FLOPs while Vittt-S/B use slightly more (larger constant factor), yet all three beat Vim in measured FPS.

Ablation Study¶

Design route (Vittt-T, accumulated step by step; FPS measured at batch 64):

Config	#Param.	FLOPs	FPS	Top-1	APb	APm	mIoU
TTT layer	5.87M	1.14G	3607	74.2	40.4	36.8	41.5
+ shared Q/K	5.43M	1.07G	3892	74.0	40.1	36.5	41.2
+ gating	5.89M	1.15G	3565	75.2	41.2	37.1	42.0
+ Conv1d (vanilla TTT)	5.90M	1.16G	3346	75.6	41.6	37.6	42.6
+ dual dataset strategy	6.96M	1.42G	2068	77.5	42.7	38.6	44.1
+ Conv2d dataset preprocessing (full)	6.98M	1.44G	2029	77.7	42.9	38.7	44.3

Classification feature extraction (Vittt-T):

Strategy	#Param.	FLOPs	Top-1	APb	APm	mIoU
HeadClassTok	6.98M	1.44G	76.6	41.6	37.8	43.0
MidClassTok	6.98M	1.44G	76.8	41.7	37.9	43.0
DoubleClassTok	6.98M	1.45G	75.4	41.2	37.2	42.5
MaxPool	6.98M	1.44G	77.2	42.2	38.4	43.8
MeanPool	6.98M	1.44G	77.7	42.9	38.7	44.3

Hyper-parameters: increasing the mini-batch b from 4 (49 gradient-descent steps) to 196 (a single batch gradient step) makes Top-1 drop sharply at first and then recover slightly, with the worst point in the middle at b=64 — neither enough update steps nor a sufficiently global gradient view; the 4–16 region containing b=16 is the sweet spot between accuracy and efficiency. For the hidden-state initialisation, Vittt-W0⁰ (no learnable state at all) is worst, sharing one learnable initial state for the two routes (Vittt-W1⁰) is better, and giving each route its own (Vittt-W2⁰) is best — separate initialisations reduce interference between the two self-supervised datasets.

Key Findings¶

The single largest gain comes from the dual dataset strategy (+1.9% Top-1, +1.5 mIoU), and so does the largest cost (FPS drops from 3346 to 2068, −38.2%); Conv2d preprocessing adds only +0.1–0.2 points but is what turns the ERF from directionally biased into globally radial — a contribution to the shape of the representation rather than to the score.
Sharing the Q/K projection costs essentially nothing (74.2 → 74.0) while saving 0.44M parameters, an almost free design; gating plus Conv1d together lift vanilla TTT by 1.6% accuracy at the price of 14.1% throughput.
Pooling heads clearly beat class-token heads, with mean pooling best; the authors attribute this to TTT's gradient-driven representation being a compression of the whole sequence rather than a feature at one position.
Multi-step gradient descent matters more on dense prediction: on classification ViT3 ties with Vittt, yet on detection/segmentation Vittt leads clearly, and the lead holds as the sequence grows.

Highlights & Insights¶

Selling "a hidden state updated by gradients rather than by a fixed rule" as a visual inductive bias is the paper's real "aha": an SSM's decay matrix and linear attention's kernel are frozen before training, whereas TTT asks, for every token, "how much did I mispredict?" and writes the residual into memory. This is nearly unexplored in vision, and this paper is the first to turn it into a general-purpose vision backbone.
Using a single hyper-parameter, b=16, to solve three problems at once — more gradient steps, preserved GPU parallelism, and alignment with Tensor Core granularity — is an engineering-flavoured but decisive design; it converts "linear complexity" from a derivation into measured throughput and is the concrete improvement over vanilla TTT.
The Gradient Magnitude Map is a transferable interpretability tool: any model whose state is updated by gradients of a self-supervised loss (TTT variants, memory networks, some meta-learning models) can plot a similar token-importance map, usable for weakly supervised localisation or data selection.
The dual dataset strategy is a general recipe for making a unidirectional sequence model bidirectional without touching its internal update rule, as long as a second dataset can be constructed — directly reusable for video, point clouds or long documents.

Limitations & Future Work¶

The backbone is still pretrained with supervised ImageNet training; the paper does not answer whether the approach holds under large-scale self-supervised pretraining (the MAE / DINOv2 regime). Since much of ViT's dominance comes from that paradigm, this is the biggest gap in the scaling argument.
Only three sizes (T/S/B) are trained, with no larger model and no comparison against ViT-L/H; the "ViT-L/16 76.5" entry in the table is an old low-resolution recipe and does not support a scalability claim.
The efficiency-accuracy trade-off is hard-wired into the design route: the 38.2% FPS cost of the dual dataset strategy is paid for in accuracy, and no finer-grained compromise (for instance using the backward route only in some layers) is explored.
The explanation for the worst point at b=64 (not enough steps and not enough global view) is qualitative, with no theoretical characterisation and no account of why accuracy recovers at b=196 — the evidence here is weaker than the conclusion.
Downstream evaluation is limited to COCO detection and ADE20K segmentation with a frozen backbone and light adapters, so tasks that need end-to-end fine-tuning (video, 3D) remain untested; the GMM and ERF analyses are also purely visual, without quantitative metrics or downstream validation.

vs ViT / DeiT: they weight the whole sequence by softmax self-attention at O(T²) cost, so compute and memory explode at high resolution; this paper replaces attention with gradient-updated hidden states and achieves linear complexity, saving 79.4% FLOPs and about 89% memory versus DeiT-T at 1280×1280. The trade-off is that DeiT is slightly faster at 224², so Vittt's advantage only appears as resolution grows.
vs Vim / VMamba (SSM family): also RNN-style linear-complexity models, but SSMs forget multiplicatively through a structured decay matrix with a predefined state evolution; TTT writes the prediction residual into the state through the subtractive term (V_t − W_{t−1}K_t) as relative-position anchors, and its state is a full d×d matrix rather than a diagonal structure. In engineering terms Vittt feeds Tensor Cores while Vim's selective scan runs on CUDA Cores, so Vittt is faster at comparable FLOPs.
vs ViT3: the closest competitor, also bringing TTT to vision, but it takes only a single gradient step, which is mathematically equivalent to gated linear attention. This paper observes that sequence-aware multi-step (T/b) updates are the real source of expressiveness — the two are close on classification, but once long-sequence dense prediction is involved the advantage of multi-step adaptation shows, with Vittt-B leading ViT3-B by +1.2 APb and +0.9 mIoU.
vs vanilla TTT [Sun et al., 2024]: this paper keeps its core form (L2 self-supervised reconstruction plus the output rule z_t=W_t x_t^Q) but points out that its 1D causal design does not fit 2D vision; the dual dataset strategy and Conv2d preprocessing exist precisely to make the dataset itself two-dimensional, with the ERF turning from banded/fan-shaped into globally radial as the clearest evidence.

Rating¶

Novelty: ⭐⭐⭐⭐☆ First to turn TTT into a general vision backbone and to restore 2D inductive bias via the dual dataset plus Conv2d preprocessing; the direction is clear, though "porting a sequence model to vision" is a familiar paradigm shift and the core update rule is inherited from NLP TTT.
Experimental Thoroughness: ⭐⭐⭐⭐☆ ImageNet, COCO, ADE20K, a multi-resolution efficiency analysis and four ablations form a complete picture with measured FPS/FLOPs/memory; large-scale self-supervised pretraining and a larger model size are missing.
Writing Quality: ⭐⭐⭐⭐☆ The design route (Tab. 4 plus the ERF figure) is clearly told and the step-by-step "from TTT layer to Vittt block" narrative is convincing; complexity and kernel details are pushed to the appendix, leaving the main-text implementation description a little thin.
Value: ⭐⭐⭐⭐☆ It opens a new technical route for linear-complexity vision backbones, and the engineering work (Triton kernels aligned with Tensor Cores) is directly reusable; the value would be higher still if validated under large-scale self-supervised pretraining.