DnA: Denoising Attention for Visual Tasks¶

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/rjccv/DnA
Area: Video Understanding / Visual Representation Learning
Keywords: denoising attention, softmin, dual value subspaces, space-time attention, visual probes

TL;DR¶

DnA uses softmax/softmin queries over shared keys to retrieve complementary interactions, then combines separate value projections with a learned per-head coefficient, raising ViT-B ImageNet-1K Top-1 from 81.1% to 81.9% and improving video classification and visual-probe question answering at additional parameter and computational cost.

Background & Motivation¶

Visual attention can mistake frequent co-occurrence for relevance to the current task. A breastplate may appear with a person and helmet; an action may occur beside a visually prominent but irrelevant object. Standard softmax emphasizes large similarity scores and drives smaller, especially negative, scores toward zero. This makes it effective at selecting strong associations, but gives potentially discriminative low-score interactions little room for independent expression. Here, negative interactions are learned internal relationships, not manually annotated background regions.

Differential Attention already attempts to remove common noise by subtracting two attention distributions. Both branches, however, aggregate the same value representation. If both attend to useful regions, subtraction can remove useful contributions alongside noise. The question is therefore not only whether attention can express negative contributions, but also where each branch encodes the information it retrieves. Changing weights over shared values and changing both selection and representation are different interventions.

The authors draw motivation from the relationship between principal angles and error bounds in subspace classification, then test more separated branch representations in image models, video models, and a vision-language adapter. Core idea: preserve high-score interactions while explicitly learning low-score interactions, project the two streams through separate value mappings, and combine them without directly canceling potentially useful signals in a shared representation.

Method¶

Overall Architecture¶

DnA replaces the attention operator; it is neither a diffusion model nor a sparse-attention method that removes tokens. Given image or video tokens, it produces two queries, one shared key representation, and two value representations. The positive branch uses softmax and the negative branch uses softmin; each aggregates its own values before a per-head weighted combination. Positional embeddings, the classification token, MLPs, and the rest of the ViT retain their existing structure, and the output remains contextualized features compatible with the original Transformer.

Video evaluation uses two distinct insertion points. TimeSformer replaces both spatial and temporal self-attention. The video LLM setup instead replaces cross-attention in VisCoP visual probes, not attention throughout the language model. Task integration in the diagram denotes alternative experimental settings rather than a single model sequentially executing every task.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    IN["Visual tokens<br/>or probes and visual features"] --> Q["Complementary dual-query selection<br/>Shared keys"]
    Q -->|Positive query: softmax| VP["Separate value projection<br/>Positive branch"]
    Q -->|Negative query: softmin| VN["Separate value projection<br/>Negative branch"]
    VP --> F["Dual-value subspace fusion<br/>Learned per-head coefficient"]
    VN --> F
    F --> T["Task integration<br/>Classification or visual probes"]
    T --> OUT["Image and video classification<br/>or video question answering"]

Key Designs¶

1. Complementary dual-query selection: give suppressed interactions their own retrieval channel

Each head learns separate positive and negative queries, both of which form scaled dot products with the same keys. Shared keys preserve a common set of candidate tokens, while independent queries let the branches learn different selection criteria. The positive branch retains ordinary softmax. The negative branch uses \(\operatorname{softmin}(z)=\operatorname{softmax}(-z)\), assigning larger normalized weights to lower scores. Because the queries differ, the negative distribution is neither the complement nor the direct negation of the positive distribution.

Softmin weights remain nonnegative and sum to one. The word negative describes a preference for low-score interactions, not negative probabilities. The authors motivate softmax and softmin as maximum-entropy distributions under expected-value constraints with opposite signs. This offers a reason to inspect both extremes, but does not establish that every low-score token is noise. Whether a branch learns object, contextual, or distracting features still depends on end-to-end task supervision.

2. Dual-value subspace fusion: stop forcing both aggregations to share one value representation

DnA learns independent value projections for the branches, producing \(V_h^+\) and \(V_h^-\). A learned scalar \(\alpha_h\) controls the negative branch in each head. The single-head computation, written from the paper's method equation, is:

\[ A_h=\operatorname{softmax}\!\left(\frac{Q_h^+K_h^\top}{\sqrt d}\right)V_h^+ +\alpha_h\operatorname{softmin}\!\left(\frac{Q_h^-K_h^\top}{\sqrt d}\right)V_h^-. \]

Here \(d\) is the head dimension and \(\alpha_h\in\mathbb{R}\) is learned, not constrained to remain negative. Compared with subtracting two distributions before multiplying a shared \(V\), separate values allow the same token to carry different feature directions in the two branches. Even when both attend to a region, their contributions need not cancel along the same representation direction. Head outputs are then concatenated according to the multi-head organization for downstream processing.

Why might independent projections help? The authors relate useful and distracting interactions to classes concentrated near different subspaces: larger principal angles improve an error bound in the cited low-noise classification model. DnA does not impose an explicit orthogonality constraint at every training step. Instead, it studies approximate orthogonality under assumptions about initialization, boundedness, and the distributions of training iterates, and reports post-training subspace statistics. The practical goal is complementary representations learned with the ordinary task loss rather than an additional constrained optimization procedure.

This motivation should not be confused with an unconditional guarantee. Separate projections do not automatically become orthogonal, and assumptions that iterated weights remain independent, identically distributed, and sub-Gaussian need not hold under arbitrary training. The empirical similarity distribution has a substantial right tail, meaning some heads still overlap. The defensible conclusion is that the measured representations are more separated, not that every head is strictly orthogonal or that useful signals can never be lost.

3. Task integration: distinguish space-time self-attention from probe cross-attention

In TimeSformer, spatial attention relates patches within a frame, while temporal attention relates information across frames. DnA replaces the original operator in both locations, allowing spatial aggregation to reduce irrelevant object cues and temporal aggregation to handle interactions unhelpful for action recognition. The experiments use 8 frames at 224×224 resolution. There is no new frame selector and no claim of reducing the number of input tokens.

In VideoLLaMA3 with VisCoP, queries come from learnable visual probes, whereas keys and values come from intermediate visual-encoder features. Through layerwise cross-attention, the probes extract compact domain-specific cues that reinforce frozen visual-encoder embeddings before they reach the language model. DnA therefore changes how probes read visual evidence. The result should not be described as retraining the entire visual backbone or replacing all language self-attention with DnA.

Introducing a branch into a pretrained model also requires protecting its existing behavior. Positive projections inherit pretrained cross-attention weights; negative query and value projections copy their positive counterparts and are scaled by \(10^{-4}\). The authors describe this as avoiding symmetry from equivalent branch initialization. This differs from random initialization when training ViT from scratch and should also be distinguished from the theory's independent-random-weight assumptions.

A Worked Example¶

Consider the paper's clog image with a nearby cat. The target is the shoe, but the cat's salient appearance may dominate ordinary attention. Positive queries learn to aggregate shoe-relevant interactions, while negative queries use a separate scoring function to retrieve complementary low-score interactions. These regions pass through independent value projections and are fused to support classification. A patch attended to by both branches can have different encoded directions instead of necessarily losing its shared contribution through subtraction in one value space.

This is a qualitative explanation, not a guarantee that the positive branch always selects only the shoe and the negative branch only the cat. The paper provides gradient-based relevance visualizations, not per-token weights or branch-level annotations for this example, so no numerical attention assignments are inferred.

Loss & Training¶

DnA retains the task-specific training objectives, collectively denoted by the task loss in the paper. It does not require explicit orthogonality regularization or object/background labels for the two branches. Several theoretical equations are damaged in the cached PDF text extraction; this note retains only the clearly recoverable attention computation and does not reconstruct the classification bound or regularizer details.

Image classification trains ViT-B from scratch using a DeiT recipe: 12 layers, 12 heads per layer, patch size 16, head dimension 64, 224×224 ImageNet-1K inputs, and 300 epochs on an NVIDIA H100. Video classification initializes from these image-trained weights, pretrains on Kinetics400, and then fine-tunes on the target action datasets; the main text reports 15 epochs for target training.

The video LLM experiment uses approximately 46K egocentric video QA pairs from EgoExo4D. It trains interaction modules, the visual-probe projector, and the vision-embedding projector, with LoRA applied to the LLM. Training lasts 3 epochs at an initial learning rate of \(10^{-5}\), with cosine scheduling for the projectors and LLM. Evaluation temperature is 0. The main text does not specify every implementation hyperparameter or complete loss weighting, so missing settings are not inferred.

Key Experimental Results¶

Main Results¶

Accuracy entries are percentages and gains are percentage points. ImageNet numbers come from the Test Set Top-1 column of original Table 1. Toyota uses mean-class accuracy, NTU60 uses Top-1, and Ego-in-Exo uses average accuracy over four task categories. Baselines and metrics differ across rows, so these gains should not be averaged into a single performance claim.

Dataset / protocol	Baseline	Baseline result	Differential Attention	DnA	Gain over baseline
ImageNet-1K, Test Top-1	ViT-B	81.1	81.5	81.9	+0.8
Toyota Smarthome, CS mCA	TimeSformer	67.5	66.6	68.8	+1.3
Toyota Smarthome, CV2 mCA	TimeSformer	59.5	59.4	63.5	+4.0
NTU60, CS Top-1	TimeSformer	81.2	81.5	82.4	+1.2
NTU60, CV Top-1	TimeSformer	88.6	89.1	89.2	+0.6
Ego-in-Exo PerceptionMCQ, average	VisCoP	78.1	76.1	78.6	+0.5

Cog Attention also scores 81.5% in the ImageNet test column, so DnA's test gain over either alternative attention is 0.4 points. The corresponding validation-column gap is 0.5 points. For video QA, the proper comparison is VisCoP, not the unaugmented VideoLLaMA3 result of 72.8%; otherwise the contribution of visual probing would be incorrectly attributed to DnA.

Ablation Study¶

The available cache contains the main paper and references only. It points to supplementary Table 12 for parameter-matched controls and supplementary Section C.5 and Table 13 for component and initialization ablations, but does not include their numerical tables. The following reports verifiable mechanism analyses rather than invented module-removal results. These statistics alone cannot isolate how much softmin, dual queries, and dual values each contribute.

Main-text analysis, ViT-B / ImageNet-1K	Differential Attention	DnA	Interpretation and boundary
Average cosine similarity between branch outputs	0.96	0.32	More complementary outputs, not orthogonality in every direction
Peak of intruder-count distribution among Top-10 left singular vectors	6–7	8–9	Peak ranges from original Figure 5, not means
Mean / median normalized Frobenius inner product of value matrices	Not reported	0.22 / 0.18	Original Figure 6; the right tail indicates overlapping heads
Parameters	86.6M	100.7M	Additional projections have a real capacity cost
GFLOPs per image	17.6	21.1	More computation, not a zero-cost replacement

The intruder analysis computes branch-output SVDs for each sample, layer, and head. Among the leading 10 left singular vectors, it counts directions whose cosine similarity to the other branch's vectors falls below \(\cos(\pi/3)\). It uses 5 images per ImageNet-1K class, or 5K images and 720K counts. Value-matrix similarity uses the full validation set, producing 7.2M sample-layer-head records. A normalized Frobenius inner product divides each matrix by its own Frobenius norm before taking their inner product. Values near zero indicate low overall matrix alignment, but do not establish that every subspace principal angle is large.

Key Findings¶

The largest video gain is Toyota CV2, from 59.5% to 63.5%, not a universal 4.0-point improvement. This cross-view result supports the approach's potential, but accuracy alone does not identify which distractions were removed.
Video QA gains concentrate in Action, from 81.8% to 83.1%, and Task, from 86.1% to 87.1%. HOI changes from 79.3% to 79.2%, and Hand stays at 65.1%. A higher average does not mean every category improves.
On ImageNet-A, DnA accuracy is 27.7%, below Differential Attention's 28.1%. Its calibration-error column is better at 24.2 versus 25.8, and AURRA is 40.7 versus 39.7. Better reliability metrics should not be presented as dominance on every robustness measure.
Original Table 1 reports identical throughput of 909.1 img/s for ViT-B and DnA despite higher parameter and GFLOP counts. This is a measurement under the paper's setup, not a guarantee of no overhead on other hardware or batch sizes.

Highlights & Insights¶

Attention denoising depends on the value representation, not just the signs of aggregation weights. Separating what is selected from where it is encoded offers a more informative design perspective than simply adding a subtractive branch.
Softmin preserves a channel for low-score interactions without requiring negative probabilities. It can work with signed value features and learned fusion, keeping negative scores, negative weights, and negative relevance conceptually distinct.
The transfer from self-attention to probe cross-attention is localized. Testing complementary branches where an existing video VLM reads visual evidence is a reusable experimental strategy, although its benefits still need to justify the added cost.

Limitations & Future Work¶

The authors explicitly acknowledge additional parameters and training cost, citing roughly 3% extra parameters in video LLMs to 23% in video transformers. More parameter-efficient implementations are future work, not an achieved property of the current method.
The theory is conditional. Low-noise subspace-classification assumptions and training-weight distribution assumptions do not guarantee orthogonality at every step of a real visual model. Copy-based initialization in video adaptation particularly deserves separate examination.
The verifiable main text lacks numerical component ablations and parameter-matched controls. Although it references supplementary evidence, that reference alone cannot rule out all capacity effects. Replication should separately control softmin, dual queries, dual values, and initialization under matched parameter and compute budgets.
The net video QA improvement is 0.5 points, with no gain in every category. The available main text does not provide the repeated-run variance needed here; confidence intervals, multiple seeds, and broader tasks would help establish stability.
A near-zero matrix inner product can reflect cancellation across directions and is not equivalent to strict subspace orthogonality. Principal-angle spectra, head-level interventions, and controlled background replacement could test whether representational separation causally produces denoising.

Versus Differential Transformer: It takes a weighted difference of two softmax distributions over shared values. DnA uses softmax/softmin, shared keys, and separate values before weighted fusion. Its gains cannot be attributed only to allowing negative attention.
Versus Cog Attention: Cog Attention directly investigates the expressiveness of negative weights, while DnA emphasizes geometric separation of branch representations. ImageNet test results favor DnA, but ImageNet-A accuracy and calibration have different rankings and require metric-specific interpretation.
Versus TimeSformer: DnA retains divided space-time processing and changes the operator at both attention locations. Its contribution is aggregation, not a new video tokenization scheme or long-video sampling policy.
Versus VisCoP: It retains visual probes that read intermediate visual features and replaces their cross-attention. The transferable research idea is to introduce complementary representations at visual-evidence compression and inspect per-category QA results to identify genuine capability gains.

Rating¶

Novelty: 4/5. Combining softmin selection with independent value subspaces clearly differs from shared-value subtraction, while building on signed attention and existing subspace theory.
Experimental Thoroughness: 4/5. Image, video, and video LLM evaluations are supported by mechanism statistics; the current cache lacks key supplementary ablation numbers needed for fine-grained attribution.
Writing Quality: 3/5. The architecture and integration paths are clear, but some theory-to-evidence claims are stronger than warranted, and validation/test gains and calibration terminology require careful reading.
Value: 4/5. A reusable attention modification across visual models, provided deployment decisions account for actual computation and the stability of modest gains.