Learning from Primitive: Probing Visual Reasoning of LVLMs via Counting¶
Conference: ECCV 2026
Paper: ECCV Official
Area: LLM Reasoning
Keywords: Large Vision-Language Models, Mechanistic Interpretability, Counting Circuits, Visual Reasoning Primitives, Activation Patching
TL;DR¶
This paper establishes visual counting as a minimalist probe to uncover the internal reasoning mechanisms of large vision-language models (LVLMs), introduces Visual Activation Patching (VAP) and HeadLens to identify a multi-stage counting circuit, and shows that lightweight fine-tuning on simple synthetic primitives significantly boosts out-of-distribution counting and general visual reasoning.
Background & Motivation¶
Counting is one of the most foundational yet revealing capabilities of visual intelligence, demanding that a model individuate discrete entities, maintain intermediate representations, and aggregate quantities under architectural constraints without relying on semantic memorization or dataset correlations. Cognitive science shows that human numerosity perception exhibits a profound discontinuity: near-perfect precision on small numbers (subitizing, quantities 1 to 4) and noisy estimation for larger sets. Recent state-of-the-art LVLMs, such as Qwen2.5-VL, demonstrate impressive general visual reasoning performance across benchmarks, yet controlled experiments reveal a glaring failure mode: they struggle to reliably count fewer than ten simple black dots on a clean canvas, even while scoring higher on complex realistic images by exploiting language and contextual biases. This discrepancy raises the fundamental question of whether LVLM counting reflects emergent reasoning or superficial correlational hallucinations.
Existing attempts to improve counting in LVLMs typically treat it as an isolated task, relying on image preprocessing, CLIP-based tuning, or attention interventions to boost leaderboards without explaining the internal circuit-level computations. Meanwhile, concurrent probing methods only examine representations at a coarse behavioral level, failing to isolate specific attention heads or establish causal pathways. Extending mechanistic interpretability methods such as activation patching or the Logit Lens to vision-language models faces two hurdles: real-world images contain entangled visual semantics that prevent clean counterfactual perturbations, and the multi-head self-attention mechanism lacks a direct way to decode individual head projections before they merge into the residual stream.
The authors address this by leveraging highly controllable synthetic image pairs to perform causal activation patching across modalities, while decomposing multi-head attention outputs into interpretable vocabulary tokens via linear additivity. Core idea: use controlled geometric counting as a minimalist causal probe to uncover the internal circuits spanning visual grounding, cross-modal routing, and counting aggregation via Visual Activation Patching and HeadLens, and enhance general visual reasoning by targeted intervention on simple counting primitives.
Method¶
Overall Architecture¶
The proposed methodology consists of two core phases: mechanistic circuit discovery and interpretability-guided targeted intervention. During the discovery phase, the model receives paired synthetic images with identical spatial layouts but differing object counts, allowing Visual Activation Patching (VAP) to measure the causal overwrite rate across layers and heads. Next, a novel tool termed HeadLens projects the isolated outputs of individual attention heads directly into the vocabulary space to decode their semantic properties, categorizing them into four distinct functional classes. During the intervention phase, the identified circuit components are reinforced using just 8,000 synthetic images of black dots and polygons: an Object-focused Attention Regularizer supervises early and middle layer attention maps against a spatial Gaussian prior, while Adaptive Head Temperature Tuning sharpens routing and aggregation heads during inference without requiring additional parameter updates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Controlled Synthetic Image Pairs & Prompt<br/>Clean/Corrupted Dot Layouts"] --> B["Visual Activation Patching & Layer Tracing<br/>Locate Layer 15-22 Cross-Modal Routing Bottleneck"]
B --> C["HeadLens Semantic Decoding<br/>Linear-Additive Projection + Affine Vocabulary Mapping"]
C --> D["Circuit Head Categorization<br/>Grounding / Routing / Aggregation / Awareness"]
D --> E["Object-focused Attention Regularizer<br/>Gaussian Prior Supervision on Middle Layers"]
E --> F["Adaptive Head Temperature Tuning<br/>Importance-based Entropy Reduction at Inference"]
F --> G["Output: Enhanced OOD Counting & Reasoning<br/>Transfer to Realistic Counting & Complex Tasks"]
Key Designs¶
1. Visual Activation Patching and Cross-Modal Routing: Locating the Layerwise Handover
Standard activation patching is designed for discrete text tokens and cannot be applied directly to continuous visual scenes. The authors generate synthetic image pairs where object counts differ (e.g., 3 dots vs. 5 dots) while fixing the random seed so that existing dots preserve exact pixel coordinates and spatial relations. By replacing the target token hidden states \(s_i^l\) at layer \(l\) of a clean forward pass with their corrupted counterparts, the causal overwrite rate measures how effectively the intervention flips the model output. The analysis reveals that across early-to-middle layers (Layers 1–14), counting information is overwhelmingly anchored in All Image Tokens. From Layer 15 onward, the causal impact of image tokens drops steeply while the Last Prompt Token (the "Assistant:" tag) surges, peaking at Layer 23. This identifies Layers 15–22 as the critical bottleneck where spatially distributed visual representations are compressed and handed over to the linguistic stream, with the Logit Lens confirming that ground-truth number tokens emerge in the top predictions between Layers 19–26 and reach Rank-1 by Layer 27.
2. HeadLens: Decoding Semantic Contributions of Individual Attention Heads
To overcome the inability of multi-head self-attention (MHSA) to reveal single-head contributions after projection, HeadLens exploits the linear additivity of attention projections. For the \(i\)-th head output \(h_i(x) \in \mathbb{R}^{d_{\text{head}}}\), an expanded zero-padded vector \(\tilde{h}_i(x) \in \mathbb{R}^{d_{\text{model}}}\) is constructed such that the MHSA projection satisfies \(\hat{x} = \sum_{i=1}^H \tilde{h}_i(x) W_O\). HeadLens applies a learned affine translator \(T(z) = A z + b\) directly to \(\tilde{h}_i(x) W_O\) to map it into the final residual stream, and computes the head logits using the language model unembedding matrix \(U \in \mathbb{R}^{|V| \times d_{\text{model}}}\): $\(\ell_i(x) = U \cdot T\big(\tilde{h}_i(x) W_O\big) + c\)$ By computing the proportion of visual attribute tokens (color, shape) and counting tokens in the top-10 decoded candidates, the authors define the Visual Grounding Score (VGS) and Counting Token Emergence Rate (CTER), uncovering four functional classes: 1. Visual Grounding Heads: Heavily concentrated in Layer 1, these heads focus on object patches and extract low-level geometric attributes, exhibiting the highest VGS; 2. Cross-Modal Routing Heads: Concentrated in Layers 18–24 (e.g., L19H23), combining high attention on image tokens with high CTER to bridge visual spatial points into linguistic numbers; 3. Counting Aggregation Heads: Positioned in deeper layers (Layers 23–28, e.g., L26H26), paying minimal attention to the image but heavily processing text tokens to assemble final predictions; 4. Awareness Heads: Specialized deep heads including L26H8, which serves as an existence detector outputting "1" whenever any entity is present, and L23H19, which decodes difficulty indicators such as "difficult" or "impossible" on larger numerosities.
3. Object-Focused Attention Regularizer: Injecting Spatial Gaussian Priors
To prevent attention dispersion across uniform backgrounds, the method exploits synthetic coordinates \(\{c_k\}_{k=1}^N\) to construct a soft instance prior over the patch grid. The unnormalized score at patch \(p\) is defined as \(u(p) = \sum_{k=1}^N \exp\left(-\frac{\|p - \pi(c_k)\|_2^2}{2\sigma^2}\right)\) with \(\sigma=1\), which is normalized to yield target distribution \(g\) over visual tokens \(\mathcal{V}\). Across supervised layers \(\mathcal{L}\) (e.g., Layer 2 and Layers 18–22 in Qwen2.5-VL-7B), the mean attention across all heads \(q_t^l\) is penalized using cross-entropy against \(g\): $\(\mathcal{L}_{\text{focus}} = \frac{1}{|\mathcal{L}||\mathcal{T}|} \sum_{l \in \mathcal{L}} \sum_{t \in \mathcal{T}} \left( -\sum_{j \in \mathcal{V}} g(j) \log\left(q_t^l(j) + \varepsilon\right) \right)\)$ This regularizer forces routing and grounding heads to focus tightly on discrete physical objects rather than diffusing across empty canvas regions.
4. Adaptive Head Temperature Tuning: Training-Free Inference Entropy Reduction
Recognizing that routing and aggregation heads exhibit sharp, task-specific activation patterns that can be corrupted by background entropy, the authors introduce a training-free inference adjustment. For each identified target head \(h\), an inverse temperature multiplier \(\beta_h = \alpha \times \gamma_h\) is applied to its pre-softmax attention logits, where baseline scale \(\alpha \ge 1\) (set to 1.2) and \(\gamma_h \ge 0\) represents the head importance score from activation patching: $\(A_h = \text{softmax}\left( \beta_h \frac{Q_h K_h^T}{\sqrt{d_k}} \right)\)$ This dynamically sharpens attention focus on high-salience tokens, amplifying the signal-to-noise ratio throughout the counting circuit without modifying model weights.
Loss & Training¶
Fine-tuning optimizes a joint objective combining standard Supervised Fine-Tuning (SFT) and the focus regularizer: $\(\mathcal{L} = \mathcal{L}_{\text{SFT}} + \lambda \mathcal{L}_{\text{focus}}\)$ where \(\mathcal{L}_{\text{SFT}}\) is the cross-entropy loss applied solely to the target number token, and \(\lambda\) is set to 1.0. Fine-tuning uses LoRA (\(r=64\)) on attention projection layers across 8,000 synthetic images from SynDot and SynPoly. Optimization is conducted for 2 epochs on a single NVIDIA H200 GPU using AdamW, BF16 precision, batch size 2, and a learning rate of \(2 \times 10^{-5}\) with linear decay and 3% warmup.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on three LVLM backbones (Qwen2.5-VL-7B, Qwen3-VL-8B, and LLaVA-1.5-7B) across out-of-distribution (OOD) realistic counting benchmarks (SynReal and PixMo-Count) and three complex multimodal reasoning benchmarks (MMMU, RealWorldQA, and MathVista), despite fine-tuning only on synthetic dots and polygons.
| Backbone | Method | SynReal Acc ↑ | SynReal MAE ↓ | PixMo-Count Acc ↑ | PixMo-Count MAE ↓ |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | Baseline | 73.48 | 0.79 | 58.79 | 0.84 |
| Ours | 84.83 | 0.74 | 64.15 | 0.62 | |
| Qwen3-VL-8B | Baseline | 88.79 | 0.15 | 58.75 | 0.73 |
| Ours | 91.21 | 0.11 | 66.98 | 0.60 | |
| LLaVA-1.5-7B | Baseline | 54.27 | 3.59 | 31.88 | 1.81 |
| Ours | 60.11 | 1.56 | 32.64 | 1.61 |
General multimodal reasoning results:
| Backbone | Method | MMMU | RealWorldQA | MathVista | \(\Delta\) (avg.) |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | Baseline | 54.89 | 61.96 | 57.30 | — |
| Ours | 56.33 | 64.14 | 58.30 | +1.54 | |
| Qwen3-VL-8B | Baseline | 58.33 | 70.05 | 66.30 | — |
| Ours | 60.74 | 71.83 | 67.90 | +1.93 | |
| LLaVA-1.5-7B | Baseline | 44.44 | 56.21 | 23.90 | — |
| Ours | 44.56 | 56.48 | 26.30 | +0.93 |
Ablation Study¶
A component ablation on Qwen2.5-VL-7B evaluates the progressive additions of SFT, the attention regularizer (\(\mathcal{L}_{\text{focus}}\)), and adaptive head temperature tuning (\(\beta_h\)):
| Config | SynReal Acc ↑ | PixMo-Count Acc ↑ | MMMU | RealWorldQA | MathVista |
|---|---|---|---|---|---|
| Baseline | 73.48 | 58.79 | 54.89 | 61.96 | 57.30 |
| + SFT | 79.78 | 61.67 | 56.00 | 63.14 | 58.10 |
| + SFT + \(\mathcal{L}_{\text{focus}}\) | 82.34 | 63.32 | 56.11 | 64.05 | 58.20 |
| + SFT + \(\mathcal{L}_{\text{focus}}\) + \(\beta_h\) (Full) | 84.83 | 64.15 | 56.33 | 64.14 | 58.30 |
Key Findings¶
- High Circuit Overlap between Counting and Reasoning: Performing mean ablation over attention heads to extract the top-20 heads reveals that Real Counting shares a Jaccard similarity of 0.38 with Mathematical Reasoning (MathVista) and 0.33 with Spatial Relations (CLEVR), compared to only 0.18 with simple color attribute recognition. This provides direct circuit-level evidence for why enhancing counting primitives generalizes to high-level reasoning.
- Strong OOD Transfer from Primitives: Fine-tuning exclusively on synthetic dots and polygons yields an absolute improvement of +5.36% on the realistic PixMo-Count benchmark and reduces MAE from 0.84 to 0.62 for Qwen2.5-VL, demonstrating that the network acquires generalizable numerosity mechanisms rather than dataset artifacts.
- Orthogonal Synergy Among Components: LoRA-SFT aligns basic numeric instruction compliance, \(\mathcal{L}_{\text{focus}}\) prevents attention drift (+1.65% on PixMo-Count), and training-free \(\beta_h\) temperature scaling sharpens signal routing (+0.83% on PixMo-Count).
Highlights & Insights¶
- Novel HeadLens Tool for Head-Level Semantic Decomposition: By utilizing the linear additivity of multi-head projections prior to residual addition, HeadLens provides a generalizable way to map individual head hidden states to interpretable vocabulary tokens without modifying model architecture.
- Cognitive Science Alignment as a Guiding Principle: Demonstrating that LVLMs exhibit subitizing (precise counting for 1–4) and topological entanglement for larger numbers links neural network mechanistic behaviors directly to human cognitive psychophysics.
- Data-Efficient Primitive Intervention: Training on merely 8,000 synthetic geometric images generated in minutes produces gains across diverse real-world benchmarks, highlighting that targeting foundational cognitive circuits is far more sample-efficient than indiscriminate instruction tuning.
Limitations & Future Work¶
- Model Scale Exploration: Mechanistic analyses and interventions were performed on 7B/8B open-source architectures (Qwen2.5-VL, Qwen3-VL, LLaVA); confirming whether identical circuit structures exist in massive mixture-of-experts (MoE) or proprietary models remains open.
- Numerosity Range Constraints: The synthetic training and probing datasets primarily covered small sets (\(N \in [1, 10]\)). Investigating whether models develop distinct statistical density estimation circuits for very large crowds (\(N > 50\)) requires further study.
- Broader Cognitive Primitives: Future work should explore whether other primitive visual operations, such as spatial occlusion, containment, and relational binding, can be decoded into structured circuits and similarly enhanced.
Related Work & Insights¶
- vs CountScope (Hasani et al., 2025) & [De|Re] (Alghisi et al., 2025): Prior studies relied on behavioral linear probes to observe layerwise accumulation, unable to isolate individual attention head functionalities or provide causal interventions. This work introduces head-level HeadLens decoding and targeted attention regularization.
- vs Text-based Activation Patching (Meng et al., 2022; Heimersheim & Nanda, 2024): While conventional patching operates on clean/corrupted text tokens, this paper successfully adapts causal tracing to multimodal architectures using geometry-aligned counterfactual image pairs.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering mechanistic circuit discovery for multimodal counting with novel VAP and HeadLens interpretability tools.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across synthetic, real-world, and general reasoning benchmarks with rigorous Jaccard head overlap analysis.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear, structured progression connecting cognitive psychology, circuit localization, and intervention.
- Value: ⭐⭐⭐⭐⭐ Proves that enhancing low-level cognitive primitives transfers to high-level visual reasoning, establishing a promising foundation for future LVLM interpretability research.