Skip to content

Towards Robustness against Typographic Attack with Training-free Concept Localization

Conference: ECCV 2026
Paper: ECCV
Code: https://github.com/Liu-524/SamplingTAR
Area: Interpretability / Multimodal VLM
Keywords: Typographic Attack, Mechanistic Interpretability, Training-free Concept Localization, Attention-Head Attribution, VLM Robustness

TL;DR

This paper localizes typographic-attack vulnerability to a small set of "text-reading" attention heads in CLIP ViTs: it samples concept directions in the multi-head subspace, scores each head with a gradient-based text-focus metric (nTAS) to mine those heads, and then applies a training-free test-time intervention (attention reweighting or zero ablation) to them — raising object classification accuracy by more than 6 points on average across five CLIP backbones and four typographic-attack benchmarks while cutting the text confusion rate by 10–36 points.

Background & Motivation

CLIP vision encoders are now the perceptual backbone of large vision-language models (LVLMs) such as LLaVA, Qwen-VL, and InternVL, yet they carry an under-examined failure mode: once text unrelated to the scene appears in an image, the visual representation is pulled toward lexical meaning rather than visual semantics — put the word GOOSE on a picture of a cat and the prediction flips to goose. This is a Typographic Attack (TA). It differs fundamentally from imperceptible adversarial perturbations: the attacker exploits the model's own recognition capacity, which makes it cheap to mount, highly deceptive, and directly relevant to safety-critical deployments such as autonomous driving. Meanwhile, our understanding of CLIP's internals remains limited. Prior interpretability work mostly stays at the level of residual-stream decomposition or hidden-space disentanglement with sparse dictionary learning (SDL), treating layers as indivisible blocks and rarely asking which attention heads look at what, or along which routing path lexical information reaches the task prediction.

The defense side is no better off. Defense-Prefix takes a supervised route, learning a prompt token on annotated data to trigger CLIP's lexical-overlook behavior; the recent training-free line (Dyslexify) turns to interpretable intervention on vulnerable modules, but it relies on a labeled greedy search over noisy attention statistics, which is expensive in both data and iterations. The core tension is that TA vulnerability is highly localized — only a few attention heads do the "reading" — yet pinpointing exactly those heads without any training presupposes having concept directions (a concept dictionary) in hand, and learning that dictionary is precisely the cost this paper wants to avoid. The linear representation hypothesis suggests an escape: if the hidden space is a superposition of concept directions, perhaps that dictionary never needs to be learned at all.

This paper's angle is to reduce concept-direction acquisition to a lottery draw. Randomly sampled vectors are the probes and gradient attribution is the judge; a score that unifies attention gating with concept alignment sifts the text-reading heads out of the population, after which a single training-free test-time operation is applied to them. Core idea: sample concept directions by stochastic lottery inside the low-dimensional MHSA head subspace, compress each sample into the comparable nTAS score via gradient attribution, and use that score to localize and intervene on the typographic reading circuit — no parameter training and no paired supervision, yet the text-hijacked prediction is corrected.

Method

Overall Architecture

The problem can be cast as a sparsity-constrained trade-off: viewing the ViT as a set of nested functional modules, find the smallest subset \(\mathcal{C}_{\text{lex}}\) (the "typographic reading circuit") such that an intervention \(I\) on it keeps the error rate on attacked inputs below \(\epsilon_{\text{robust}}\) while keeping the loss on clean data below \(\epsilon_{\text{benign}}\). ⚠️ Eq.(3) of the original paper is garbled in the retrieved cache; the constraint is restated here from the paper's prose, and the exact formulation should be checked against the original.

The pipeline has four steps, none of which trains any parameter:

Step one, sampling: in a number of later transformer blocks, sample \(K=16\times d_{\text{head}}\) random \(d_{\text{head}}\)-dimensional vectors \(u\) per attention head from a standard normal distribution, treating each as a candidate "concept direction."

Step two, attribution: project the candidate direction \(u\) onto that head's output space to obtain a per-patch concept magnitude, then differentiate with respect to the pre-softmax attention logit to obtain a patch-level attribution map — it answers how sensitive the destination token's concept reading is to each patch's attention routing.

Step three, scoring and selection: convert the attribution map into a text attribution score using the mask of the known text region, then normalize the text-mask and inverse-mask branches into \(\text{nTAS}\in[0,1]\); a head's average nTAS over all sampled vectors is its "text-reading tendency," and a within-layer z-test selects the significantly high ones, chaining them into a sparse cross-layer head set.

Step four, intervention: at test time only those heads are touched — CLS-token attention is redistributed for classification, and the outputs of vulnerable heads are zero-ablated for VQA.

The hyper-parameters are frugal: only the last 20% of transformer blocks are inspected (following the prior observation that concepts emerge late in ViTs); 16 times the hidden dimension is sampled per attention module; the z threshold defaults to 1; the reweighting strength is \(a=1\), i.e. maximal intervention. The one-time mining uses 1,280 images and finishes in under a minute on a single A100 (7.5 s for ViT-B/16, 14.0 s for L/14, 24.0 s for H/14, 32.1 s for g/14, 45.7 s for bigG/14); test-time overhead is near zero because the intervention operates on a fixed set of head indices.

Key Designs

1. Stochastic lottery: obtain concept directions for free inside the MHSA head subspace

The first obstacle to localizing text-reading heads without training a dictionary is where the concept direction \(u\) comes from. SDL-style approaches train a sparse dictionary on attention-layer outputs, at substantial compute and data cost — exactly the expense this paper sets out to remove. Here \(u\) itself becomes a random probe. Writing the value vector of patch \(i\) in some head as \(\mathbf{v}_i\) and decomposing it orthogonally with respect to a target concept \(\mathbf{c}_{\text{target}}\), we get a true concept strength \(\alpha_i\) along \(\mathbf{c}_{\text{target}}\) plus a polysemantic interference term \(\boldsymbol{\xi}_i\) orthogonal to it, so the reading of a random probe splits naturally into signal and interference:

\[ \langle \mathbf{v}_i,\mathbf{u}\rangle = \underbrace{\alpha_i\langle \mathbf{c}_{\text{target}},\mathbf{u}\rangle}_{\text{signal } S_{\mathbf{u},i}} + \underbrace{\langle \boldsymbol{\xi}_i,\mathbf{u}\rangle}_{\text{interference } I_{\mathbf{u},i}} \]

Whether the lottery is usable hinges on interference not drowning the signal, and the paper argues this in two steps. First, a Gaussian tail bound: the maximum interference across all \(N\) patches is bounded by \(\max_i |I_{\mathbf{u},i}| \le \|\boldsymbol{\xi}_{\max}\|\sqrt{\log N / d_{\text{head}}}\), so the worst interference does not blow up with the patch count (the full derivation is deferred to the appendix). Second, a separation condition: as long as the signal at the weakest on-concept patch still dominates the maximum interference by a margin \(\tau\), that single random vector marks all on-concept patches correctly at once and yields a concept-faithful attribution map; the construction of a typographic attack guarantees the injected region carries the lexical concept strongly, so \(\alpha_k\) is bounded well above zero. The probability \(p\) that a single draw succeeds therefore depends only on the signal-to-noise ratio, and to make "at least one winning ticket" happen with probability \(P_{\text{success}}\) the sample size must satisfy \(K \ge \log(1-P_{\text{success}})/\log(1-p)\).

What really matters is where the sampling happens. In the global residual stream the dimension is \(d_{\text{model}}\), and the interference term \(\boldsymbol{\xi}_{\max}\) accumulates components from every head, so its norm is large, \(p\) is small, and \(K\) explodes to infeasibility. Restricting the sampling to a single head's \(d_{\text{head}}\)-dimensional subspace pushes head-excluded concepts into the null space, which shrinks \(\|\boldsymbol{\xi}_{\max}\|\), raises \(p\), and lets a computationally feasible \(K\) reliably draw winning tickets. This is the "advantage of the MHSA bottleneck," and it is what distinguishes this work from random probing in the full hidden space — the same lottery idea simply does not work in the wrong space.

2. Attributing to the QK routing: pick out text-focusing heads via "attention gate × marginal utility"

With candidate concept directions in hand, a comparable score is still needed to answer whether a head is actually reading text. Rather than looking at features, this work looks at the QK routing: aggregating concept strength onto destination token \(i\) gives \(F_i(\mathbf{u})=\langle\mathbf{o}_i,\mathbf{u}\rangle=\sum_j A_{i,j}V_j(\mathbf{u})\), where \(A_{i,j}\) is the post-softmax attention weight and \(V_j(\mathbf{u})=\langle\mathbf{v}_j,\mathbf{u}\rangle\) is the concept projection at patch \(j\). Differentiating with respect to the pre-softmax logit \(s_{i,j}\) and applying the chain rule through the softmax yields:

\[ \frac{\partial F_i(\mathbf{u})}{\partial s_{i,j}} = A_{i,j}\big(V_j(\mathbf{u}) - F_i(\mathbf{u})\big) \]

The two factors have distinct roles: \(A_{i,j}\) is the attention gate this patch receives, while \(V_j(\mathbf{u})-F_i(\mathbf{u})\) is its marginal utility — how much stronger the patch's concept is than the current aggregate. Their product means a patch is attributed as a "source of the text" only if it is both genuinely attended to and conceptually more salient than its surroundings; either factor alone is insufficient, since trusting attention weights alone is misled by globally high-weight patches such as attention sinks, and trusting feature similarity alone discards whether the information actually flowed into the destination token.

For scoring, the mask of the known text location turns the expression above into two quantities. The Mask Attribution Score (MAS) accumulates only positive attributions (consistent with the additive nature of the linear representation hypothesis) and normalizes by mask size; dividing the text mask \(\mathbf{m}^+\) branch by the sum with the inverse mask \(\mathbf{m}^-\) branch (the CLS position is always zeroed) gives the normalized Text Attribution Score:

\[ \text{nTAS}_i(\mathbf{m}^+,\mathbf{m}^-) = \frac{\text{MAS}_i(\mathbf{m}^+)}{\text{MAS}_i(\mathbf{m}^+) + \text{MAS}_i(\mathbf{m}^-)} \]

Lying in \([0,1]\), nTAS captures both "how much the head attends to text" and "how little it attends to the object," so it is directly comparable across heads and backbones. A head's final score is its mean over all sampled vectors and augmented unlabeled images, after which a within-layer z-test picks out the significantly high ones (default \(z=1\); a small calibration set can also be used to choose the threshold). The whole mining procedure needs only unlabeled images plus automatic text-injection augmentation — it is label-free, which is the most direct difference from Dyslexify's labeled greedy search.

3. Test-time intervention: attention reweighting or zero ablation on the localized heads

Once \(\mathcal{C}_{\text{lex}}\) is located, the remaining question is how to act on it. For classification, a ViT has a dedicated CLS token and lexical information reaches it through the attention paid to patch tokens, so the surgery happens directly on the CLS attention distribution: given a control parameter \(a\in[0,1]\), the original attention map \(\mathbf{a}_{\langle\text{cls}\rangle}\) is redistributed as

\[ \mathbf{a}'_{\langle\text{cls}\rangle}[i] = a\cdot\mathbb{I}(i=0) + \mathbf{a}_{\langle\text{cls}\rangle}[i]\cdot\frac{1-a}{\sum_{j=1}^{N-1}\mathbf{a}_{\langle\text{cls}\rangle}[j]}\cdot\mathbb{I}(i\neq 0) \]

That is, a fraction \(a\) of the attention is locked onto the CLS token itself while the remaining \(1-a\) is spread over all patch tokens in their original proportions. The virtue of this design is that it preserves the normalization and the original relative preferences of the attention map and only compresses the patch-side contribution proportionally, so it does not destroy the other visual information the head carries the way hard zeroing would; the experiments set \(a=1\) for maximal intervention.

LVLMs are different: many have no dedicated CLS token at all and rely entirely on patch-token embeddings to carry visual information downstream, leaving attention reweighting with no handle. The paper therefore switches to zero ablation, replacing the output of vulnerable heads with a zero vector to sever this lexical pathway from the visual token stream. Migrating the pipeline to LVLMs also requires answering a prerequisite: without a CLS token, which token does attribution target? The paper takes the first visual token (fixed at the top-left corner) as a CLS surrogate, motivated by the observation that ViTs repurpose redundant patch tokens for global information — in an LVLM that first visual token is highly likely to have spontaneously assumed a CLS-like role. This choice lets the same sampling–attribution–intervention pipeline transfer to the vision encoders of Qwen3-VL, InternVL3.5, and Gemma3 without retraining or architectural change.

A Worked Example

Walking one sample through the pipeline, on ViT-H/14 with RTA-100:

  • Input: a picture of a cat with GOOSE superimposed in white-on-black text.
  • Sampling: inside the last 20% of blocks, each attention head draws \(16\times d_{\text{head}}\) candidate directions from \(\mathcal{N}(0,\frac{1}{d_{\text{head}}}I)\).
  • Attribution: most random vectors are unrelated to the concept and produce a diffuse, noisy attribution map with a low nTAS; when a direction aligned with "lexical shape" is drawn, the map becomes highly concentrated on the strokes of the text (the paper's Fig. 2 reports nTAS = 0.8980 for one such vector), and that head's within-layer z-score stands out accordingly.
  • Selection: a within-layer z-test (\(z=1\)) selects 15 heads on H/14 (loosening to \(z=0.5\) gives 24, tightening to \(z=2.0\) leaves only 4).
  • Intervention: at test time those 15 heads have their CLS attention reweighted with \(a=1\), suppressing the patch-side attention entirely.
  • Outcome: the same image was answered as goose before the intervention and returns to cat after it; over the whole RTA-100 set, H/14's object classification accuracy rises from 53.4% to 76.2% and its text confusion rate falls from 42.0% to 14.4%.

Key Experimental Results

Main Results

Table 1: robustness across backbones on zero-shot classification (higher OCA, lower TCR; arrows show baseline → with intervention)

Model RTA-100 OCA RTA-100 TCR IN-100-Text OCA IN-100-Text TCR
ViT-B/16 56.3 → 68.7 (+12.4) 30.8 → 12.6 (−18.2) 54.6 → 74.2 (+19.6) 33.0 → 7.1 (−25.9)
ViT-L/14 54.6 → 68.9 (+14.3) 39.0 → 21.2 (−17.8) 58.2 → 74.9 (+16.7) 32.9 → 12.1 (−20.8)
ViT-H/14 53.4 → 76.2 (+22.8) 42.0 → 14.4 (−27.6) 56.6 → 79.1 (+22.5) 36.9 → 9.5 (−27.4)
ViT-g/14 50.3 → 68.8 (+18.5) 45.8 → 23.4 (−22.4) 57.0 → 76.4 (+19.4) 36.4 → 12.7 (−23.7)
ViT-bigG/14 61.0 → 75.7 (+14.7) 32.5 → 15.3 (−17.2) 62.3 → 80.6 (+18.3) 31.0 → 8.9 (−22.1)

Table 2: average accuracy against prior defenses (%, IN-100 is clean, unattacked)

Method RTA-100 Disentangling PAINT IN-100-Text IN-100 (clean)
Defense-Prefix (supervised, trained) 63.6 67.8 67.8 70.1 81.4
Dyslexify (training-free, reproduced here) 67.0 67.6 70.1 72.8 81.3
Dyslexify* (as reported in its own paper) 68.5 70.7 72.2 — 81.0
Ours (nTAS) 71.7 78.7 74.7 77.0 81.0

Table 3: VQA accuracy after transfer to LVLMs (RIO-Bench obj-attack split; baseline → with intervention)

Model Easy Medium Hard Overall
Qwen3-VL-4B 68.29 → 69.71 65.67 → 66.52 54.90 → 55.53 62.95 → 63.92 (+0.97)
Qwen3-VL-8B 74.19 → 75.81 71.23 → 73.34 64.91 → 65.92 70.11 → 71.69 (+1.58)
Qwen3-VL-30B-A3B 67.34 → 69.11 66.71 → 66.87 60.90 → 62.35 64.98 → 66.11 (+1.13)
InternVL3.5-8B 63.87 → 63.96 60.77 → 61.18 50.09 → 50.28 58.24 → 58.47 (+0.23)
InternVL3.5-14B 56.95 → 57.01 55.65 → 55.37 46.62 → 46.87 53.07 → 53.08 (+0.01)
Gemma3-4B 47.98 → 49.72 46.46 → 47.98 38.60 → 40.52 44.35 → 46.07 (+1.73)
Gemma3-12B 49.91 → 52.27 48.74 → 49.84 46.08 → 47.22 48.24 → 49.78 (+1.53)

The object-classification benchmarks are RTA-100, Disentangling, and PAINT, plus IN-100-Text, which the authors build on top of ImageNet-100 by injecting a semantically conflicting class name with Qwen-Image-Edit in one of seven rendering styles (the pixel-level edit region above a 20/255 max-channel difference has a mean/median share of 16.3%/11.5%). VQA is evaluated on RIO-Bench, a benchmark built specifically for typographic attacks.

Ablation Study

Table 4: sensitivity to the z threshold (ViT-H/14, RTA-100)

z threshold Heads selected OCA TCR Clean IN-100 accuracy
0.5 24 75.9 14.0 83.2
1.0 (default) 15 76.3 15.4 83.3
2.0 4 66.5 27.1 83.9

Table 5: stability across random seeds (expansion ratio 16, 4 seeds, mean ± std)

Model RTA-100 OCA / TCR Disentangling OCA PAINT OCA
ViT-B/16 68.7 ± 0.0 / 12.6 ± 0.0 88.3 ± 0.0 73.8 ± 0.0
ViT-L/14 68.8 ± 0.1 / 21.3 ± 0.2 68.3 ± 0.0 68.9 ± 0.0
ViT-H/14 76.2 ± 0.1 / 14.1 ± 0.6 83.2 ± 1.9 77.2 ± 2.9
ViT-g/14 68.8 ± 0.0 / 23.4 ± 0.1 83.2 ± 3.1 75.7 ± 0.0
ViT-bigG/14 75.4 ± 0.2 / 15.7 ± 0.6 70.6 ± 2.6 75.0 ± 3.7

Key Findings

  • The intervention works on every backbone and every attack benchmark: OCA gains range from 7.7 to 36.1 points (the smallest is +7.7 for ViT-L/14 on PAINT; the largest is +36.1 for both ViT-B/16 and ViT-H/14 on Disentangling), with TCR drops of 10.7 to 36.1 points. ViT-H/14 is the strongest configuration overall (OCA 75.7–82.2 across the four attacked benchmarks).
  • The gains are not bought by sacrificing clean performance: on clean IN-100 the method scores 81.0 against 81.4 for Defense-Prefix and 81.3 for Dyslexify, with the paper reporting an accuracy trade-off below 1%; on clean VQA the effect ranges from −0.3% to +0.6%.
  • Comparison with trained methods: averaged over the four attacked sets, Ours reaches 75.5, Dyslexify (as reproduced here) 69.4, and Defense-Prefix 67.3 — about 6.1 points above the prior training-free method and 8.2 points above the supervised one. Note that the "Dyslexify*" row is the number self-reported in its own paper (requiring iterative evaluation over the full ImageNet-100 training set) and is not directly comparable with the reproduced figures.
  • Gains on LVLMs are markedly uneven: Qwen3-VL is consistently positive across all three sizes (+0.97 / +1.58 / +1.13 on average), Gemma3 gains the most at both sizes (+1.73 / +1.53), whereas InternVL3.5 barely moves (+0.23 / +0.01, with the 14B model even losing 0.28 on the medium split). The paper offers no explanation for why the InternVL family fails to improve, which bears directly on how general the "first visual token as CLS surrogate" assumption really is.
  • Hyper-parameters are insensitive until the threshold gets too strict: for \(z\in\{0.5,1.0\}\) the method selects 24 and 15 heads with OCA/TCR essentially stable at 75.9/76.3 and 14.0/15.4, and clean accuracy barely moves (83.2/83.3); at \(z=2.0\) only 4 heads remain, OCA collapses to 66.5 and TCR rises to 27.1 — shaving the circuit too aggressively evidently drops heads that genuinely carry lexical reading.
  • Sampling stabilizes at 8 times the hidden dimension: the standard deviation of the module score across random seeds falls as the expansion ratio grows and flattens beyond 8 samples/dim across all five backbones, which justifies the default of 16 times the hidden dimension.
  • Qualitative validation: attributing with an unaligned concept vector yields a diffuse, semantically unfocused map, while an aligned vector produces a map concentrated on the text strokes. Contrasting high-nTAS and low-nTAS cohorts shows nTAS magnitude correlates positively with the degree of lexical focus, supporting its use for mining the lexical circuit.
  • ⚠️ A few numbers differ across tables: ViT-H/14's RTA-100 TCR appears as 14.4 in Table 2, 14.1 ± 0.6 in Table 5, and 15.4 in the z-threshold discussion; ViT-bigG/14's PAINT OCA is 79.6 in Table 2 but 75.0 ± 3.7 in Table 5; ViT-L/14's IN-100-Text TCR is 12.1 vs 12.2. The paper does not say whether these reflect different random-seed conventions or different settings; they are reported here as printed, without reconciliation.

Highlights & Insights

  • Replacing dictionary learning with "stochastic lottery + attribution filtering": this removes the entire cost of training a sparse autoencoder, and the interference bound justifies why the sampling must happen in the MHSA head subspace — the same random probing in the global residual stream accumulates interference norms across heads and makes the required sample size explode. This is the paper's weightiest step, fusing the linear representation hypothesis and the lottery ticket hypothesis into a usable training-free interpretability tool.
  • The attribution identity \(\partial F/\partial s = A\cdot(V-F)\) is the most elegant piece of design: multiplying "is attended to" (the attention gate) by "is conceptually more salient" (the marginal utility) is more faithful than attention weights alone (easily skewed by sink-like high-weight patches) or feature similarity alone (which discards whether information reached the destination token), and it produces a patch-level mask that is inherently visualizable and interpretable.
  • "First visual token = emergent CLS surrogate" is a cheap, reusable trick: it transfers an attribution pipeline designed around the CLS token to LVLMs that have none, with zero modification; the idea (borrowing from the register-token / redundant-patch-token line of work) can be carried over to any interpretability method that needs an aggregation position in a CLS-free model.
  • The intervention operator is extremely cheap: it acts on a fixed set of head indices and adds no test-time compute, and reweighting preserves attention normalization and relative preferences while only compressing the patch-side contribution proportionally — far gentler than zeroing — an idea directly transferable to other inference-time scenarios that need to suppress a class of inputs.

Limitations & Future Work

  • It relies on a known text-location mask: computing MAS/nTAS requires \(\mathbf{m}^+\), which the paper builds through automatic text-injection augmentation. In the wild the text location is unknown and a detection or segmentation step would be needed first; errors there would directly contaminate the attribution scores and hence the circuit mining. The authors do not evaluate degradation under imperfect masks.
  • The reweighting strength is fixed at \(a=1\) (maximal) with no trade-off curve: the paper never systematically reports the robustness-vs-clean-performance frontier as \(a\) varies from 0 to 1. The blanket compression of CLS-side attention may also remove non-lexical visual information carried by the same head, a risk that is higher for heads with severe concept entanglement.
  • LVLM gains are small and uneven: InternVL3.5 shows essentially nothing (+0.01 / +0.23), and the paper attributes this to model differences without a mechanistic explanation. This hints that "the first visual token is an emergent CLS surrogate" does not hold for all architectures and deserves systematic verification (for instance, comparing the concentration of attribution maps at different token positions).
  • Clean-performance evaluation is narrow: it covers only IN-100 classification and the RIO-Bench clean split; side effects on captioning or visual grounding, which depend more on fine-grained visual detail, are untested.
  • The randomness of concept directions leaves a reusability gap: a drawn \(u\) is a "winning-ticket-once" random direction rather than the stable, cross-model-reusable concept basis a sparse dictionary provides. Whether the concept actually drawn is semantically consistent across backbones is supported only by a few qualitative visualizations.
  • Inconsistent numbers across tables (see the last item under Key Findings), which complicates precise citation of individual gains.
  • Directions for improvement: internalize mask acquisition (e.g. unsupervise text-region discovery from the attribution map itself); auto-calibrate \(a\) and \(z\) instead of hand-picking defaults; and give a testable criterion for the "emergent CLS surrogate" so LVLM transfer stops being a matter of luck.
  • vs Dyslexify: both are training-free and both mine circuits over vulnerable modules, but Dyslexify relies on a labeled greedy search over attention statistics, whereas this work models concept alignment explicitly into the attribution score and replaces greedy search with random sampling. The cost is that this method needs a known text-location mask and Dyslexify does not; the claimed 6.1-point average advantage mixes self-reported and reproduced numbers, so a direct head-to-head comparison needs care.
  • vs Defense-Prefix: a supervised prompt-token approach that needs annotated data and provides no mechanistic explanation. This work trains nothing, needs no paired supervision, scores 8.2 points higher on the attacked sets, and gives up only 0.4 points of clean accuracy — while additionally producing an interpretable circuit localization.
  • vs sparse-dictionary interpretability (SDL / CLIP head-and-layer decomposition): those works train dictionaries or run greedy text-description searches and analyze layers as indivisible blocks; this work trains nothing, drops the granularity to attention heads, and explicitly argues that random probing in the global stream is infeasible and must be done in the head subspace. The price is losing the stable concept basis and cross-model reusability that a dictionary offers.
  • vs benchmarks such as RIO-Bench and TypoD: they reveal and quantify how pervasive TA vulnerability is (including multi-image settings and realistic text distractors in self-driving scenes), whereas this paper is the defense-side answer to the same problem — using mechanistic localization to cut the lexical pathway, so the two are complementary.

Rating

  • Novelty: ⭐⭐⭐⭐ Treating random sampling as an interpretability tool and proving with an interference bound that head-subspace sampling is preferable is a weighty mechanistic argument, though the gradient-attribution and circuit-mining framework leans heavily on prior work.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Five backbones × four attack benchmarks plus seven LVLM configurations, with threshold, seed, and sample-size stability studies; missing an \(a\) trade-off curve and evaluation on broader clean tasks.
  • Writing Quality: ⭐⭐⭐ The method and theory are clearly laid out, but several numbers are inconsistent across tables and some symbol definitions require a trip to the appendix.
  • Value: ⭐⭐⭐⭐ A training-free, interpretable, zero-overhead-at-inference defense against typographic attacks that is directly usable in safety-critical VLM deployments; the LVLM gains are modest but the direction is clear.