Token-Based Affordance Grounding with Large Vision-Language Models¶
Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/DrFirstLee/tokAG
Area: Multimodal VLM / Robotics & Embodied AI
Keywords: Affordance Grounding / Large Vision-Language Models / Zero-Shot Learning / Cross-Attention Mechanism / Spatial-Aware Token Selection
TL;DR¶
TokAG is a training-free zero-shot affordance grounding framework that aggregates cross-attention activations across all layers and heads of an LVLM and applies an object-aware spatial constraint to select the single dominant token, converting implicit action semantics into high-fidelity affordance heatmaps.
Background & Motivation¶
Affordance grounding seeks to localize specific functional regions on visual objects that support intended human physical actions, serving as an indispensable foundation for embodied perception, physical intelligence, and robotic manipulation. Historically, the dominant paradigm has been weakly supervised learning, which transfers visual knowledge from exocentric images with action-level labels to localize actionable regions in target egocentric views. However, existing methodologies are constrained by two fundamental bottlenecks. First, exocentric datasets suffer from severe action co-occurrence ambiguity; empirical audits show that 31.7% of exocentric images in AGD20K are reused across different action classes (such as holding and brushing with a toothbrush simultaneously). This visual conflation prevents models from isolating action-specific evidence, often yielding identical heatmaps for distinct actions. Second, existing frameworks rely on brief action phrases that lack the descriptive granularity needed to differentiate semantically similar actions on the identical object (such as distinguishing "sit on" from "push" on a motorcycle, where each requires grounding completely separate components like the seat versus the handle).
Meanwhile, modern large vision-language models (LVLMs, e.g., Qwen3-VL and InternVL3) have accumulated vast open-vocabulary action semantics and commonsense knowledge. When presented with visual scenes alongside action-oriented conversational prompts, their autoregressively decoded text sequences inherently reflect deep spatial and semantic grounding within the input imagery. However, standard LVLMs are primarily optimized for conversational text generation rather than explicit coordinate or heatmap outputs. Naively extracting internal cross-attention responses fails because autoregressive generation suffers from attention sinks and diffuse background activations, producing chaotic attention maps that cannot be directly deployed for actionable spatial grounding.
To resolve this dilemma, this work bypasses external supervision and complex cross-view distillation pipelines, exploring instead the latent cross-modal alignment within frozen, off-the-shelf LVLMs. Core idea: develop TokAG, a training-free zero-shot affordance grounding framework that aggregates cross-attention activations across all architectural layers and attention heads, and employs an object-level spatial mask to pick the single output token whose activation is most concentrated on the target object, translating implicit action semantics into sharp affordance heatmaps.
Method¶
Overall Architecture¶
TokAG takes an egocentric input image \(I\), a target object class \(O\), and a queried action \(A\), outputting an affordance heatmap \(H\) in a completely zero-shot and training-free manner. The pipeline integrates a frozen conversational LVLM with an off-the-shelf open-vocabulary segmentation model: first, a structured, action-conditioned prompt is queried to prompt the LVLM into generating an explanatory sentence; second, the cross-modal attention maps between generated output tokens and visual patches are comprehensively aggregated across all Transformer layers and multi-head subspaces; third, an object-level binary spatial prior is extracted via a pre-trained open-vocabulary segmenter; finally, an object-region overlap score evaluates each token to isolate the single Top-1 dominant token, whose masked attention map is bilinearly upsampled to formulate the final affordance heatmap.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Egocentric Image I + Action A + Object O"] --> B["Intention-Inducing Prompt Generation<br/>'When people perform A with O, which part...'"]
B --> C["Frozen LVLM Autoregressive Decoding<br/>Generates Token Sequence T = {t1, t2, ..., tN}"]
C --> D["Cross-Layer & Multi-Head Aggregation<br/>Aggregate all layers and heads into M_ti"]
A --> E["Open-Vocabulary Object Mask Extraction<br/>CLIPSeg extracts spatial prior M_obj"]
D --> F["Spatial-Aware Token Selection<br/>Hadamard overlap scoring S_i = sum(M_ti ⊙ M_obj)"]
E --> F
F --> G["Top-1 Dominant Token Identification<br/>Select maximum activation token t* = argmax S_i"]
G --> H["Bilinear Upsampling & Heatmap Generation<br/>Derive final affordance heatmap H"]
Key Designs¶
1. Intention-Inducing Prompting and Full-Layer Multi-Head Attention Aggregation: Resolving Representation Dispersion
Inputting isolated action phrases into an LVLM fails to trigger deep functional reasoning paths. TokAG utilizes a structured query prompt that explicitly directs causal functional attribution: "When people perform \(A\) with \(O\), which part of the \(O\) is used for '\(A\)'? Answer in one sentence." As the LVLM autoregressively decodes the token sequence \(T = \{t_1, t_2, \dots, t_N\}\), rich cross-modal attention weights link each generated token with input visual patches. However, across different linguistic layers (early layers capturing low-level textures versus deeper layers encoding abstract semantics) and individual heads, spatial activations are highly dispersed and frequently drift into irrelevant background context. While prior visual grounding literature frequently relies on heuristic layer/head selection or supervised pruning, determining optimal configurations under a zero-shot setting is intractable. TokAG eliminates heuristic sub-selection by uniformly averaging attention responses across all Transformer layers and attention heads, yielding a unified, robust cross-modal attention map \(\mathcal{M}_{t_i} \in \mathbb{R}^{H \times W}\) for each token \(t_i\) that balances local edge precision with global conceptual focus.
2. Object-Aware Spatial Masking: Suppressing Background Attention Sinks
Due to the intrinsic autoregressive decoding mechanics of multimodal LLMs, strong visual attention sinks inevitably emerge, redirecting substantial attention mass toward non-informative background regions, image borders, or punctuation. To filter out these background distractors without introducing manual annotations or task-specific fine-tuning, TokAG incorporates an object-level spatial constraint. An off-the-shelf open-vocabulary segmentation model (such as CLIPSeg or SAM 3) extracts a binary object mask \(M_{\text{obj}} \in \{0, 1\}^{H \times W}\) for the target object \(O\). Performing a pixel-wise Hadamard product restricts the aggregated attention map strictly within the object's physical contour:
This step physically extinguishes noise outside the object boundary, guaranteeing that subsequent scoring strictly measures functional interactions on the target entity.
3. Spatial-Aware Token Scoring and Top-1 Salience Selection: Surpassing Positional and Syntactic Heuristics
Previous generative grounding works typically adopt hardcoded positional heuristics (such as inspecting only the first decoded token or the terminal punctuation) or restrict candidates to noun tokens. Empirical analysis reveals that autoregressive generation routinely anticipates subsequent functional nouns by focusing spatial attention on preceding grammatical articles (such as "a" or "the" before "knife"). Rigid syntactic filtering thereby discards the most potent spatial cues. TokAG resolves this by measuring the cumulative spatial energy concentrated inside the object boundaries. For every output token \(t_i\), its overlap score is computed as:
The framework then designates the single token \(t^*\) with the highest object-confined activation as the primary grounding carrier:
The final continuous affordance heatmap \(H\) is directly derived by bilinearly upsampling the selected masked attention map: \(H = \operatorname{UP}(\widetilde{\mathcal{M}}_{t^*})\).
Token Interpretability and Semantic Emergence¶
Decoding the selected Top-1 tokens \(t^*\) back into human-readable text reveals remarkable physical interpretability: the model autonomously picks "blade" for "Cut with knife", "mattress" for "Lie on bed", "seat" for "Sit on bicycle", "handle" for "Push motorcycle", and "cork" for "Open bottle". Quantitative semantic analysis shows that for the 2B backbone, 54.3% of the Top-1 tokens correspond directly to meaningful actions, spatial regions, or functional components. When scaling to the 32B backbone, this ratio jumps to 62.4%, with explicit functional component nouns doubling from 21.5% to 40.4%. This empirical trend confirms that scaling LVLM backbones directly enhances the fidelity and precision of grounded affordance semantics.
Key Experimental Results¶
Main Results¶
TokAG is evaluated on the standard AGD20K benchmark (split into Set 1 for seen action-object pairs and Set 2 for unseen cross-object generalization) as well as the human-object interaction benchmark HICO-IIF. Evaluation metrics include Kullback-Leibler Divergence (KLD, lower is better), Similarity (SIM, higher is better), and Normalized Scanpath Saliency (NSS, higher is better).
| Dataset | Method | Supervision Mode | KLD ↓ | SIM ↑ | NSS ↑ |
|---|---|---|---|---|---|
| AGD20K (Set 1, Seen) | Cross-View-AG+ (CVPR'22) | Weakly Supervised (Exocentric) | 1.489 | 0.342 | 0.981 |
| LOCATE (CVPR'23) | Weakly Supervised (Exocentric) | 1.226 | 0.401 | 1.177 | |
| INTRA (ECCV'24) | Weakly Supervised (Exocentric) | 1.199 | 0.407 | 1.239 | |
| WSMA (AAAI'24) | Weakly Supervised (Exo + Text) | 1.176 | 0.416 | 1.247 | |
| LoopTrans (ICCV'25) | Weakly Supervised (Closed-Loop) | 1.088 | 0.445 | 1.322 | |
| Moon et al. (ICCV'25) | Weakly Supervised (CLIP Contrastive) | 1.124 | 0.433 | 1.280 | |
| TokAG (2B, Ours) | Zero-Shot (Training-Free) | 1.020 | 0.459 | 1.406 | |
| TokAG (32B, Ours) | Zero-Shot (Training-Free) | 1.010 | 0.458 | 1.424 | |
| AGD20K (Set 2, Unseen) | Cross-View-AG+ (CVPR'22) | Weakly Supervised (Exocentric) | 1.765 | 0.279 | 0.882 |
| LOCATE (CVPR'23) | Weakly Supervised (Exocentric) | 1.405 | 0.372 | 1.157 | |
| INTRA (ECCV'24) | Weakly Supervised (Exocentric) | 1.365 | 0.375 | 1.209 | |
| WSMA (AAAI'24) | Weakly Supervised (Exo + Text) | 1.335 | 0.382 | 1.220 | |
| LoopTrans (ICCV'25) | Weakly Supervised (Closed-Loop) | 1.247 | 0.403 | 1.315 | |
| Moon et al. (ICCV'25) | Weakly Supervised (CLIP Contrastive) | 1.243 | 0.405 | 1.368 | |
| TokAG (2B, Ours) | Zero-Shot (Training-Free) | 1.060 | 0.455 | 1.514 | |
| TokAG (32B, Ours) | Zero-Shot (Training-Free) | 1.043 | 0.455 | 1.549 | |
| HICO-IIF | Cross-View-AG+ (CVPR'22) | Weakly Supervised (Exocentric) | 1.779 | 0.263 | 0.946 |
| LOCATE (CVPR'23) | Weakly Supervised (Exocentric) | 1.593 | 0.327 | 0.966 | |
| WSMA (AAAI'24) | Weakly Supervised (Exo + Text) | 1.465 | 0.358 | 1.012 | |
| LoopTrans (ICCV'25) | Weakly Supervised (Closed-Loop) | 1.399 | 0.379 | 1.226 | |
| Moon et al. (ICCV'25) | Weakly Supervised (CLIP Contrastive) | 1.358 | 0.378 | 1.234 | |
| TokAG (2B, Ours) | Zero-Shot (Training-Free) | 1.056 | 0.459 | 1.600 | |
| TokAG (32B, Ours) | Zero-Shot (Training-Free) | 1.032 | 0.449 | 1.655 |
Ablation Study¶
Table 1: Token Selection Strategies (AGD20K Set 2, 2B Backbone)¶
Assessing the impact of the spatial-aware Top-1 token selection against heuristic choices:
| Token Selection Strategy | Description | KLD ↓ | SIM ↑ | NSS ↑ |
|---|---|---|---|---|
| Average All Tokens | Direct average of attention maps across all output tokens | 1.111 | 0.435 | 1.457 |
| First Output Token | Fixate on the first decoded token in the output sequence | 1.078 | 0.444 | 1.493 |
| Last Output Token | Fixate on the terminal token (e.g., period or EOS token) | 1.358 | 0.364 | 1.116 |
| Noun Tokens Only | Constrain selection strictly to tokens classified as nouns | 1.112 | 0.435 | 1.461 |
| Top-5 Tokens | Average the top-5 tokens with highest object-area activation | 1.084 | 0.444 | 1.482 |
| Ours (Spatial Top-1) | Select the single token with highest object-area activation | 1.060 | 0.455 | 1.514 |
Table 2: Layer & Head Aggregation Strategies (AGD20K Set 2, 2B Backbone)¶
Comparing isolated layer/head sampling against comprehensive global aggregation:
| Aggregation Configuration | Selected Architectural Components | KLD ↓ | SIM ↑ | NSS ↑ |
|---|---|---|---|---|
| L25-H5 | Layer 25, Head 5 only | 1.132 | 0.441 | 1.431 |
| L25 | All heads within Layer 25 | 1.090 | 0.450 | 1.471 |
| H5 | Head 5 aggregated across all layers | 1.062 | 0.454 | 1.510 |
| L16-H15 | Layer 16, Head 15 only | 1.135 | 0.446 | 1.459 |
| L16 | All heads within Layer 16 | 1.100 | 0.461 | 1.466 |
| H15 | Head 15 aggregated across all layers | 1.100 | 0.450 | 1.476 |
| Ours (All Layers & Heads) | Uniform aggregation across all layers and attention heads | 1.060 | 0.455 | 1.514 |
Key Findings¶
- Zero-Shot Dominance over Supervised Baselines: Without using a single training image from AGD20K, TokAG-2B achieves an NSS of 1.514 on the Unseen split (Set 2), outperforming the prior state-of-the-art weakly supervised method (Moon et al., 1.368 NSS) by 10.7%. On HICO-IIF, the gain reaches 29.7% (1.655 vs 1.234 NSS). This indicates that the latent cross-modal alignment in pretrained LVLMs provides vastly superior semantic generalizability compared to distillation over noisy exocentric datasets.
- Why Noun Restriction Underperforms: In autoregressive generation (e.g., "cut with a knife"), anticipating the functional noun causes attention on actionable parts (the blade) to burst early during the preceding article ("a"). Restricting selection to nouns discards these early aligned spatial signals, leading to degraded grounding accuracy.
- Backbone Scaling and Functional Emergence: Moving from 2B to 32B parameters steadily boosts grounding metrics while doubling the probability that the selected token explicitly names the functional part (40.4% vs 21.5%), verifying that scaling enhances grounded physical reasoning.
Highlights & Insights¶
- Zero-Shot Geometric Projection without Task Tuning: TokAG demonstrates that off-the-shelf LVLMs can be turned into high-precision spatial affordance grounders purely through attention aggregation and binary mask filtering, requiring zero gradient steps or prompt optimization.
- Inherent Multi-Instance and Multi-Part Coverage: Even though only a single Top-1 token is selected, the underlying cross-modal attention map naturally highlights multiple instances or symmetrical functional components (e.g., all seats on a couch or multiple forks) that offer equivalent action affordances.
- Superiority over Cascaded Foundation Models: Prompting LVLMs to predict explicit coordinates (bounding boxes or center points) fed to SAM 3 produces subpar NSS (1.085 and 0.675). Generic foundation models lack discrete vocabulary for non-rigid sub-object functional zones, whereas TokAG's continuous attention maps bypass the coordinate bottleneck entirely.
Limitations & Future Work¶
- Single-Token Limitation for Compositional Affordances: The framework currently assumes that an affordance maps onto a single dominant semantic token. This breaks down for bimanual manipulation (e.g., holding a jar with one hand while twisting the lid with the other) or compositional tasks (e.g., pouring requiring both handle and spout). Extending toward multi-token clustering is a necessary next step.
- Inference Latency: Because it relies on autoregressive LVLM decoding and full multi-head attention extraction, TokAG operates at 0.59 FPS (2B) and 0.20 FPS (32B) on an RTX 3090 / A100 GPU, trailing lightweight task-specific heads (e.g., WSMA at 20.4 FPS) and posing challenges for real-time robotic servoing.
- Distraction by Salient Visual Artifacts: In failure cases (48 of 77 failure images caused by token selection error), attention maps are frequently distracted by high-contrast surface textures, brand logos, or printed text (such as the brand logo on a drum surface) rather than true actionable physical regions.
Related Work & Insights¶
- vs Cross-View-AG [28] & LOCATE [23]: Early weakly supervised frameworks depend on exocentric prototypes, suffering severely from action co-occurrence noise in 31.7% of training images. TokAG completely sidesteps exocentric datasets by directly probing the egocentric view with language models.
- vs WSMA [42] & Moon et al. [32]: Prior methods use short action phrases and CLIP contrastive alignment, which cannot disambiguate nuanced actions like "push" vs "sit on" on the same vehicle. TokAG elicits descriptive causal sentences, isolating fine-grained component-level attention.
- vs Heuristic Head Pruning (e.g., Kang et al. [17]): While prior visual grounding literature searches for sparse "golden heads" via supervision, TokAG shows that unweighted all-layer all-head aggregation provides superior noise cancellation and maximum semantic robustness in a zero-shot setting.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Replaces exocentric weak supervision with an elegant, training-free token attention selection mechanism]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across AGD20K Seen/Unseen, HICO-IIF, and EPIC-Aff, supplemented with deep token interpretability analysis and foundation baseline comparisons]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and coherent structural design]
- Value: ⭐⭐⭐⭐ [Highly valuable reference for embodied AI manipulation and post-hoc visual interpretability of multimodal LLMs]