Cross-token Guidance Transformer for Weakly Supervised Object Localization

Conference: ECCV 2026
Paper: ECCV Official
Area: Object Detection
Keywords: Weakly Supervised Object Localization, Visual Transformer, Cross-token Guidance, Attention Regulation, Guided Filtering

TL;DR

To tackle the severe diffuse activation caused by relying exclusively on location tokens without semantic context, this paper proposes the Cross-token Guidance Transformer (CGTR), which employs an Attention Regulation Module (ARM) and a Filter Regulation Module (FRM) to establish cross-scale semantic guidance between class and location tokens, achieving state-of-the-art localization precision on CUB-200-2011 and ILSVRC.

Background & Motivation

Weakly supervised object localization (WSOL) aims to learn object locators solely from image-level class labels, avoiding the prohibitive cost of collecting dense bounding boxes or pixel-wise mask annotations. Early WSOL methodologies predominantly relied on Convolutional Neural Networks (CNNs) coupled with Class Activation Maps (CAMs), attempting to expand coverage beyond the most discriminative regions via adversarial erasing, data augmentation, or feature mining techniques. However, constrained by the intrinsic limitations of convolution operations—namely restricted receptive fields and insufficient global context perception—CNN-based frameworks often activate only partial discriminative object components (e.g., bird heads or car wheels) rather than the entire object envelope.

The advent of Vision Transformers (ViT) substantially mitigated the partial activation dilemma owing to their powerful long-range dependency modeling. To prevent task-level optimization conflicts between classification and localization objectives, recent prominent paradigms (such as SAT) decoupled the architecture by introducing a dedicated location token (\(T_{loc}\)) separate from the classification token (\(T_{cls}\)). Nonetheless, this isolated design discards the high-level semantic insights inherent to the classification token. In the absence of top-down semantic constraints, the location token frequently entangles co-occurring visual elements in complex scenes—such as {riverbed, diamondback}, {accordion, musician}, or {bushes, chipping sparrow}—leading to severe and uncontrolled diffuse activations across irrelevant background regions.

An essential observation reveals that although the attention map generated by the class token yields relatively coarse object boundaries, it exhibits superior capability in disentangling the target object from co-occurring contextual noise. Hence, completely severing the communication between localization and classification is sub-optimal; an effective cross-token guidance mechanism is critically needed. Core idea: propose the Cross-token Guidance Transformer (CGTR) to leverage class tokens as both global semantic guidance and local filtering templates, systematically refining the spatial activations and structural boundaries of location tokens with zero extra learnable parameters.

Method

Overall Architecture

CGTR builds upon a DeiT-S transformer backbone. Given an input image divided into patch embeddings, a learnable class token \(T_{cls}\) and a location token \(T_{loc}\) are appended together with standard positional embeddings. The input sequence first traverses \(M\) base transformer blocks to capture foundational global representations. The remaining \((L - M)\) blocks function as location transformer blocks employing spatial-query attention. Inside the encoder, the Attention Regulation Module (ARM) injects class token semantics into the location token's attention distribution. Post-encoder, the Filter Regulation Module (FRM) applies semantic guided filtering to smooth and regularize the localization maps. Finally, the attention map from the location token and the regularized map from FRM are averaged to construct the final high-precision localization map.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input image + Patch Tokens<br/>Append class token Tcls and location token Tloc"] --> Base["M Base Transformer Blocks<br/>Global feature encoding and standard self-attention"]
    Base --> ARM["Attention Regulation Module (ARM)<br/>Cross-token interaction between qloc and kcls"]
    ARM --> LocBlocks["(L-M) Location Transformer Blocks<br/>Spatial-query attention to generate M_loc and M_cls"]
    LocBlocks --> FRM["Filter Regulation Module (FRM)<br/>Semantically guided local structure filtering"]
    LocBlocks --> Fuse["Dual-path Fusion & Post-processing<br/>Pixel-wise averaging of M_loc and M_frm"]
    FRM --> Fuse
    Fuse --> Out["Connected Component Analysis<br/>Tight bounding box and mask prediction"]

Key Designs

1. Attention Regulation Module: cross-token semantic injection to suppress contextual bias This design directly addresses the tendency of the location token to over-activate co-occurring background elements during spatial query attention. Rather than allowing the location token to attend to spatial patches in isolation, ARM modulates the value representations using the class token. Within each attention head, the module extracts the location query \(q_{loc}\) along with the class key \(k_{cls}\) and class value \(v_{cls}\). Scaled dot-product attention computes the semantic alignment score \(s_{lc}\): $\(s_{lc} = \text{Softmax}\left(\frac{q_{loc} k_{cls}^\top}{\sqrt{D}}\right)\)$ The spatial value features of the location token are subsequently recalibrated by conditioning on class semantics: $\(\tilde{v}_{loc} = v_{loc} \cdot s_{lc} \cdot v_{cls}\)$ Substituting \(v_{loc}\) with \(\tilde{v}_{loc}\) in the value matrix explicitly transfers the category-specific inductive bias of \(T_{cls}\) into the localization stream. Furthermore, across the final \((L-M)\) blocks, the resulting location attention map \(M^\star_{loc}\) and class attention map \(M^\star_{cls}\) are constrained by an area regularization loss: $\(\mathcal{L}_{arm} = \left| \left( \frac{1}{hw} \sum_{i=1}^h \sum_{j=1}^w M^\star_{loc}(i,j) \cdot M^\star_{cls}(i,j) \right) - \xi \right|\)$ where hyper-parameter \(\xi\) penalizes localization maps that deviate excessively from the semantic footprint defined by the class token.

2. Filter Regulation Module: semantic guided filtering for local structural refinement While ARM achieves global semantic alignment, raw transformer attention maps often suffer from noisy textures, ragged edges, and interior holes. FRM introduces an efficient edge-preserving filtering operator inspired by guided image filtering, treating the coarse yet clean class attention map \(M^\star_{cls}\) as a structural guide to refine the location attention map \(M^\star_{loc}\). Across local \(k \times k\) windows, local means (\(m^{avg}_{loc}\), \(m^{avg}_{cls}\)), variance (\(m^{var}_{cc}\)), and cross-covariance (\(m^{cov}_{lc}\)) are computed: $\(w_{loc} = \frac{m^{cov}_{lc}}{m^{var}_{cc} + \varepsilon}, \quad b_{loc} = m^{avg}_{loc} - w_{loc} \odot m^{avg}_{cls}\)$ After spatial average pooling over linear coefficients to obtain \(w^{avg}_{loc}\) and \(b^{avg}_{loc}\), the guided filtering transformation is applied to \(M^\star_{cls}\) and clamped to \([0, 1]\): \(M_{frm} = \Phi(w^{avg}_{loc} \odot M^\star_{cls} + b^{avg}_{loc})\). This formulation smoothly preserves sharp object contours while suppressing isolated spurious noise. To encourage definitive separation between foreground and background, an entropy regularization loss is imposed: $\(\mathcal{L}_{frm} = -\frac{1}{hw} \sum_{i,j} \left( M_{frm} \log(M_{frm} + \epsilon) + (1 - M_{frm}) \log(1 - M_{frm} + \epsilon) \right)\)$

3. Dual-scale Collaborative Fusion: complementary multi-level localization Instead of relying solely on either the multi-layer aggregated map or the filtered map, CGTR computes an unweighted element-wise mean: \(M = \frac{1}{2}(M^\star_{loc} + M_{frm})\). This collaborative combination leverages the global semantic multi-layer representation of ARM alongside the localized edge sharpness from FRM. Crucially, the entire pipeline operates without introducing any additional trainable parameters or heavy prediction sub-networks, ensuring maximum computational and memory efficiency.

Loss & Training

The overall architecture is trained in an end-to-end multi-task manner with the total objective: $\(\mathcal{L} = \mathcal{L}_{cls} + \mathcal{L}_{arm} + \lambda \mathcal{L}_{frm}\)$ where \(\mathcal{L}_{cls}\) is the standard cross-entropy classification loss, \(\mathcal{L}_{arm}\) is the area regulation loss (\(\xi = 0.37\)), and \(\mathcal{L}_{frm}\) is the entropy regularization loss balanced by \(\lambda = 0.45\). Using DeiT-S pre-trained on ILSVRC with input crops of \(224 \times 224\), the model is optimized via AdamW (\(\beta_1=0.9, \beta_2=0.99\), weight decay \(5 \times 10^{-4}\)). CUB-200-2011 is fine-tuned for 30 epochs with an initial learning rate of \(1 \times 10^{-4}\) (batch size 256), whereas ILSVRC requires only 1 epoch with an initial learning rate of \(1.5 \times 10^{-5}\) (batch size 512) on A800 GPUs.

Key Experimental Results

Main Results

CGTR was rigorously benchmarked against premier CNN- and Transformer-based WSOL frameworks on the fine-grained CUB-200-2011 and large-scale ILSVRC benchmarks.

Dataset	Method	Backbone	Top-1 Loc (%)	Top-5 Loc (%)	GT-known Loc (%)	Top-1 Cls (%)
CUB-200-2011	TS-CAM [ICCV21]	DeiT-S	71.30	83.80	87.70	80.30
CUB-200-2011	LCTR [AAAI22]	DeiT-S	79.20	89.90	92.40	85.00
CUB-200-2011	SAT [ICCV23]	DeiT-S	80.96	94.13	98.45	-
CUB-200-2011	CDTR [TPAMI25]	DeiT-S	81.33	94.06	96.89	83.87
CUB-200-2011	CGTR (Ours)	DeiT-S	81.80	94.40	98.68	82.29
ILSVRC	TS-CAM [ICCV21]	DeiT-S	53.40	64.30	67.60	74.30
ILSVRC	SCM [ECCV22]	DeiT-S	56.10	66.40	68.80	76.70
ILSVRC	SAT [ICCV23]	DeiT-S	60.15	70.52	73.13	78.41
ILSVRC	CIAT [IJCAI24]	DeiT-S	59.80	69.90	72.10	78.60
ILSVRC	CGTR (Ours)	DeiT-S	60.35	70.68	73.33	78.39

In mask fidelity (PxAP) on CUB-200-2011, CGTR attained 90.22%, substantially outperforming TS-CAM (81.49%) and SAT (89.87%). Under the threshold-independent MaxBoxAccV2 metric on ILSVRC, CGTR achieved 84.66% at \(\delta=0.3\), 73.33% at \(\delta=0.5\), and an average accuracy of 71.50%, establishing the top mark across all evaluated models.

Ablation Study

Component-wise ablations validate the distinct contributions of ARM and FRM to localization precision across both benchmarks.

Dataset	Configuration	ARM	FRM	Top-1 Loc (%)	Top-5 Loc (%)	GT-known Loc (%)
ILSVRC	(a) Vanilla Baseline	-	-	56.86	66.56	69.10
ILSVRC	(b) Baseline + ARM	✓	-	58.57	68.39	70.91
ILSVRC	(c) Baseline + FRM	-	✓	58.22	68.11	70.58
ILSVRC	CGTR (Full Model)	✓	✓	60.35	70.68	73.33
CUB-200-2011	(d) Vanilla Baseline	-	-	76.25	87.87	91.77
CUB-200-2011	(e) Baseline + ARM	✓	-	79.43	91.89	95.96
CUB-200-2011	(f) Baseline + FRM	-	✓	78.36	90.46	94.39
CUB-200-2011	CGTR (Full Model)	✓	✓	81.80	94.40	98.68

Parameter sensitivity evaluations demonstrate: - Loss balancing factor \(\lambda\) produces highly robust performance across \([0.40, 0.55]\), peaking at \(\lambda = 0.45\). - The area target parameter \(\xi = 0.37\) provides the ideal trade-off between coverage and under-activation. - Filter kernel size \(k = 3\) yields optimal structural coherence; larger windows (\(k \ge 5\)) introduce excessive smoothing and degrade accuracy.

Key Findings

• On ILSVRC, isolated integration of ARM and FRM improves GT-known Loc by +1.81% and +1.48% respectively, while their combined deployment yields a +4.23% gain, confirming strong positive synergy between global semantic guidance and local boundary filtering.
• Visual inspection demonstrates that while SAT mistakenly activates background regions like riverbeds, foliage, or diving equipment, CGTR confines activations strictly to object boundaries, directly resolving diffuse contextual activations.

Highlights & Insights

• Rethinking Token Decoupling: While previous works advocated strict isolation between classification and localization tokens to avoid gradient conflict, CGTR insightfully demonstrates that one-way semantic guidance from the class token is indispensable to prevent spatial tokens from wandering into contextual noise.
• Parameter-free Hybrid Filtering: Integrating the closed-form formulation of Guided Image Filtering directly into deep Transformer attention maps provides a clean, zero-parameter mechanism for edge-preserving feature regularization.
• Broad Transferability: The dual-scale cross-token regulation scheme can be readily adapted to weakly supervised semantic segmentation (WSSS) and open-vocabulary referring expression grounding to eliminate co-occurrence context artifacts.

Limitations & Future Work

• Single-label Object Bias: The formulation assumes single dominant category semantics per image, which aligns well with standard WSOL benchmarks but requires architectural extensions to handle dense multi-class co-occurrences (e.g., in MS COCO).
• Fixed Window Prior: The guided filter currently uses a fixed pooling kernel (\(k=3\)), which may not dynamically scale across vastly different object aspect ratios and sizes (e.g., elongated snakes vs. compact birds).
• Future Directions: Developing adaptive window sizing and extending cross-token guidance to multi-modal vision-language foundation models (such as CLIP or SAM) are promising avenues.

Related Work & Insights

• vs TS-CAM (ICCV 2021): TS-CAM couples semantic tokens and patch attention maps via simple multiplication without dedicated localization tokens, resulting in blurry boundaries; CGTR introduces independent tokens governed by directed cross-token guidance, outperforming TS-CAM by nearly 7% Top-1 Loc on ILSVRC.
• vs SAT (ICCV 2023): SAT isolates spatial and class tokens completely, making it vulnerable to co-occurring context bias; CGTR restores directed cross-token guidance, eliminating diffuse activations.
• vs Guided Filter (TPAMI 2012): Guided Image Filtering was originally formulated for pixel-level color and depth maps; CGTR adapts its closed-form formulation to latent attention maps inside deep vision transformers.

Rating

• Novelty: ⭐⭐⭐⭐☆ Insightful revisit of token isolation, pairing elegant cross-token attention modulation with parameter-free guided filtering.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across CUB and ILSVRC benchmarks, multi-threshold IoU metrics, mask PxAP, and detailed ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, lucid motivation, coherent figures and self-consistent formulations.
• Value: ⭐⭐⭐⭐☆ Offers an efficient, highly practical blueprint for transformer-based weakly supervised visual perception.