Attention is Case-Sensitive

Conference: ECCV 2026
Paper: ECCV Official Page · PDF
Area: Multimodal VLM
Keywords: Case sensitivity, attention allocation, prompt intervention, visual grounding, reasoning buffer

TL;DR

Controlled casing interventions across 13 models show that uppercase and alternating-case targets attract attention without guaranteeing better accuracy; in VLMs, stronger local target concentration can coexist with reduced whole-image attention, so attention enhancement should not be equated with better understanding.

Background & Motivation

Uppercase letters stand out against lowercase text for human readers. Language models, however, process discrete tokens rather than directly seeing this typographic contrast, so the relevant question is not whether they possess human visual perception, but whether pretraining makes case an implicit importance cue. Existing approaches such as PASTA and AutoPASTA explicitly rescale attention, while activation steering changes internal representations. This paper asks a more basic question: how does attention change when only the surface form of the input changes, without parameter updates or runtime steering interventions?

Answering that question requires separating internal measurements from final behavior. A correctly highlighted answer receiving more attention does not automatically imply better understanding: an unfamiliar spelling pattern may also make the content harder to process. Vision-language models introduce another ambiguity. Increased concentration inside a target box can accompany reduced attention to the image overall, so a sharper local heatmap need not mean greater use of visual evidence.

The authors therefore frame their work as characterization rather than a deployable accuracy-improvement algorithm. Benchmark ground truth identifies the span to modify, reducing target-selection errors before different casing combinations are compared. Core idea: use target-context casing contrast as a controlled input variable, jointly measuring target attention, textual task accuracy, and the visual modality budget to test whether salience and task utility actually agree.

Method

Overall Architecture

Inputs are text examples with ground-truth answer spans, or images with target labels and ground-truth boxes; the output is not a new model, but attention and performance changes relative to naturally cased inputs. The protocol proceeds through Ground-Truth Target Alignment, Controlled Casing Interventions, frozen-model inference, and Dual-Level Readouts, followed by Cross-Model Boundary Tests. The diagram describes the experimental workflow, not a proposed multi-module neural architecture.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Text or image-text examples<br/>with ground-truth labels"] --> Align["Ground-Truth Target Alignment"]
    Align --> Cases["Controlled Casing Interventions"]
    Cases --> Frozen["Frozen-model inference<br/>naturally cased baseline"]
    Frozen --> Readout["Dual-Level Readouts"]
    Readout --> Boundary["Cross-Model Boundary Tests"]
    Boundary --> Output["Does attention<br/>agree with utility?"]

Key Designs

1. Ground-Truth Target Alignment: remove incorrect target selection as a confound

Textual targets come from benchmark answer annotations. Case-insensitive exact matching covers 98.2% of samples; the remaining 1.8% use Levenshtein alignment with a threshold of at most 2 character edits, checked through manual verification of a 100-sample subset. Transformations affect letters only, retaining punctuation, digits, whitespace, and word order. Inherently mixed-case entities such as iPhone are transformed as complete spans, for example into IPHONE, rather than preserving their original branding.

For vision, the target object label is embedded in a generic description template. That label string is the span whose case changes, while its ground-truth bounding box defines the region to inspect; the image remains unchanged. This makes the effect of formatting on attention comparable across conditions, but ground-truth target selection itself carries answer-related information. The results therefore concern controlled, known-target settings, not a zero-shot gain directly available when the correct answer is unknown.

2. Controlled Casing Interventions: distinguish absolute uppercase from relative contrast

The authors define 5 families containing 15 static rules. U makes the entire input uppercase, lowercase, or title case. TE uppercases the target against lowercase, alternating-case, or natural context. TT uses title-case targets against uppercase, lowercase, or natural context. TA alternates the target's case against uppercase, lowercase, or natural context. ADE reverses the emphasis, lowercasing the target against uppercase, alternating-case, or natural context. This crossed design matters because simply uppercasing the target cannot distinguish its absolute form from its contrast with the background.

TE1 means an uppercase target in lowercase context; TA2 uses an alternating-case target in lowercase context; TE2 places an uppercase target inside alternating-case context. These respectively test conventional emphasis, unnatural salience, and background competition. ADE1 lowercases the target and uppercases the context to examine reversed emphasis. The paper expresses lexical preservation through invariance under lowercasing; the following is an equivalent, readable restatement:

\[ \operatorname{lower}\!\left(T_C(X)\right)=\operatorname{lower}(X). \]

Here \(T_C\) applies only the predetermined casing rule. This preserves the case-insensitive character string, not necessarily the model's token sequence, token count, or semantics in every language. One passage claims token boundaries are preserved, while later passages acknowledge possible changes in token IDs and counts. The defensible reading is that whitespace and lexical structure are not deliberately edited, not that tokenization effects have been eliminated.

3. Dual-Level Readouts: measure attention strength and task utility separately

For text, mean attention to the target span is the internal readout, evaluated alongside frozen-model task accuracy. The methodology describes aggregation across layers and heads, and Figure 1 additionally describes averaging per token before word-level aggregation. However, the local cache lacks the full implementation and appendices, so it does not establish the exact query positions, generation-step aggregation, or denominator details. This note retains the reported metrics and changes rather than inventing a precise-looking extraction formula.

Vision requires two simultaneous measurements: microscopic concentration inside the ground-truth box, and macroscopic allocation to the entire image. The authors interpret the first as alignment of the residual visual budget to the target, and call reduction in the second visual disengagement, meaning a shift from image features toward the text prompt. The two quantities have different baselines and normalization conventions: they should not be added together, and relative target-region attention gains are not grounding accuracy gains. Standard difference notation distinguishes the units used in the paper:

\[ \Delta m_{\mathrm{pp}}=100(m_C-m_0),\qquad \Delta m_{\mathrm{rel}}=100\frac{m_C-m_0}{m_0}\%. \]

Here \(m\) is a consistently defined readout expressed as a proportion. These equations explain units; they do not reconstruct the missing attention-extraction algorithm. Visual disengagement reports a reduction as a positive magnitude, opposite in sign to the whole-image attention change. In particular, an average absolute change across models should not be used directly to recompute an average of their relative percentage changes.

4. Cross-Model Boundary Tests: treat reasoning regime and modality as experimental conditions

The study covers 13 models: 7 non-reasoning LLMs, 2 reasoning LLMs, 2 non-reasoning VLMs, and 2 reasoning VLMs, ranging from 1B to 20B parameters across LLaMA, Gemma, Mistral, Qwen, and GPT families. All receive the same intervention taxonomy without per-model tuning, with changes measured against each model's naturally cased baseline. Agreement across tokenizers weakens an explanation based on an accident of one tokenizer, but does not establish that tokenization plays no role.

The informative boundary is that reasoning models are not uniformly more robust across modalities. Text-only thinking is comparatively insensitive to surface form, motivating the authors' reasoning-buffer interpretation. Visual reasoning models, however, can rely more heavily on textual coherence and further reduce image attention under unusual casing. This is an empirical difference between model groups, not yet a causal mediation result from switching reasoning on and off in the same model.

A Worked Example

Consider the illustrative phrase the red car is beside a tree, with red car selected by ground truth. TE1 produces the RED CAR is beside a tree; TA2 produces the rEd CaR is beside a tree; ADE1 produces THE red car IS BESIDE A TREE. These strings illustrate the rules and are not additional experimental examples or results reported by the paper.

If the phrase describes an image, all 3 inputs use the same image and target box. Evaluation must inspect not only whether the target heatmap becomes brighter, but also whether whole-image attention falls. If local concentration rises while the overall visual budget shrinks, the warranted conclusion is changed cross-modal allocation, not necessarily better vehicle grounding. Text tasks likewise require separately checking the generated answer rather than inferring correctness from higher target attention.

Loss & Training

There is no new loss function, fine-tuning, or parameter update. The intervention occurs before tokenization and the models remain frozen. Text benchmarks are MMLU-Pro, ARC-Challenge, and SQuADv2; the visual benchmark is RefCOCOg. The claim that model access is unnecessary applies to making the casing intervention itself. Measuring internal attention and producing Grad-CAM visualizations still require observability and should not be described as achievable from black-box output alone.

Key Experimental Results

Main Results

The table below uses the main-text analysis and Figure 2 for Qwen2.5-7B-Instruct. Every number is a percentage-point change (pp) from naturally cased input, not a relative growth rate. The main text does not provide the full absolute scores underlying these aggregate readouts.

Scheme	Target / context	Target attention change (pp)	Task accuracy change (pp)
TE1	Uppercase / lowercase	+2.05	+1.85
TA1	Alternating / uppercase	+2.44	-2.88
TA2	Alternating / lowercase	+2.75	-2.60
TA3	Alternating / natural	+2.77	-2.83

The striking result is the reversed ranking, not simply which configuration scores highest: TA3 attracts more attention than TE1 but reduces accuracy. The authors call TA a destructive attractor and TE-style uppercase emphasis a productive attractor. These are behavioral summaries of this experiment, not universal prompting rules.

Ablation Study

There are no removable training modules; controlled casing comparisons and cross-modal analyses serve as the core ablations. The table summarizes main-text RefCOCOg results across 4 VLMs. Visual disengagement is the relative reduction in whole-image attention, not a grounding failure rate.

Intervention aggregate	Target-region attention change (pp)	Whole-image attention change (pp)	Relative visual disengagement
Overall mean in main text	+1.55	-1.85	12.41%
TA family	+2.84	Not given at this point in main text	24.22%
TE family	+1.77	Not given at this point in main text	21.12%
TE2	Not given at this point in main text	-4.11	33.35%

The overall relative target-region attention increase is 23.08%, which uses different units from +1.55 pp. TE2 makes the surrounding context alternating case and causes the largest whole-image drain; Qwen3-VL-2B-Thinking reaches 43.74% relative visual disengagement in that condition. The dominant visual effect can therefore be a shift toward text rather than more effective use of the image.

Key Findings

• The casing effect appears across the evaluated non-reasoning LLMs and tokenizers, but this finite-sample observation is not a theorem about every transformer. The paper reports mean accuracy gains reaching +8.95 pp under uppercase target emphasis, while alternating-case configurations produce a maximum loss of -13.96 pp for LLaMA-3.1-8B-Instruct.
• Text reasoning models usually vary by less than ±0.5 pp in accuracy, whereas visual reasoning models exhibit stronger macroscopic disengagement from images. This is a more precise boundary condition than saying that more thinking always removes formatting sensitivity.
• VLM family-level aggregates retain the TA-over-TE tendency, but individual spatial responses are uneven. Gemma-3-4B-IT peaks at +6.23 pp target-region attention under TA1, which does not establish an identical fine-grained ordering across all 4 VLMs.

Evidence boundary: the local cache ends with the main paper and references, without the cited appendix Tables 9, 13, 14, and others. Accordingly, this note does not supply missing absolute accuracies, full per-model tables, confidence intervals, or normalization details.

Highlights & Insights

• Attention and utility are tested independently. Alternating case gives an intuitive counterexample: stronger attraction need not make information easier to use correctly. Interpretability studies should record task outputs alongside heatmaps.
• Visual attention needs both budget and distribution measurements. Tracking the whole image and the target box reveals disengagement hidden by local concentration. This approach can transfer to VLM audits of prompt formatting, text length, and salient-word interventions.
• A cheap input variable can be diagnostically useful. Casing creates controlled comparisons without retraining and is suitable for formatting-robustness tests. Ground-truth-guided discovery and automatic target selection at deployment must nevertheless be evaluated separately.

Limitations & Future Work

• Author-stated scope: The evaluation focuses on languages with letter case. Non-Latin scripts and broader surface-salience effects require further study, while automatic target selection and causal mediation analysis remain future work.
• Ground-truth dependence limits deployment claims: Highlighting a known answer does not establish that a model can identify what deserves highlighting. Future evaluations should jointly include target-prediction errors, incorrect emphasis, and genuinely unknown-answer settings.
• Causal interpretations must not exceed observations: Casing interventions support an effect of input form on readouts, but do not fully separate tokenization, frequency, pretraining distribution, and reasoning trajectories. An importance signal learned during pretraining remains a mechanistic interpretation, not the uniquely established explanation.
• Lexical preservation is not universal semantic preservation: Case can matter for entity names, abbreviations, and code identifiers. The text results should not be extrapolated directly to all languages or code tasks.
• Evidence of visual task utility is limited: The main text primarily reports attention changes and does not provide a verifiable grounding-accuracy improvement here. Further work should combine grounding outcomes, visual-evidence dependence tests, and explicit attention normalization definitions.

Related Work & Insights

• Versus PASTA / AutoPASTA: These approaches explicitly rescale attention during inference; this paper changes only input case. One directly controls computation, while the other characterizes existing formatting sensitivity, so they are not interchangeable implementations.
• Versus ROME / MEND and activation steering: Weight editing changes knowledge-related behavior through parameters, while activation steering changes internal states. This paper edits neither; its value is prompt-level diagnosis and characterization rather than a demonstrated replacement for knowledge editing.
• Connection to text dominance: The cited work "When Language Overrules" examines textual dominance in multimodal models. This study suggests that even superficial casing changes can alter modality allocation, motivating tests of whether formatting emphasis increases reliance on incorrect textual hints.

Rating

• Novelty: 4/5. Systematic controls characterize an overlooked casing effect and reveal different boundaries for textual and visual reasoning.
• Experimental Thoroughness: 3/5. Coverage spans 13 models and many rules, but ground-truth dependence, main-text measurement details, and limited visual task-utility evidence constrain the claims.
• Writing Quality: 3/5. The attention-performance distinction is clear, although wording about token boundaries, black-box interventions, and internal measurements requires care.
• Value: 4/5. A useful starting point for VLM formatting robustness and attention interpretability, not a ready-made general accuracy-improvement recipe.