A4-Agent: An Agentic Framework for Zero-Shot Affordance Reasoning¶
Conference: ECCV2026
Paper: ECCV Official Page · PDF
Area: VLM Reasoning
Keywords: affordance prediction, visual imagination, decoupled reasoning and grounding, zero-shot generalization, model coordination
TL;DR¶
A4-Agent assigns interaction imagination, functional-part selection, and pixel-level localization to specialized pretrained models, reaching 70.52 gIoU on ReasonAff without affordance-specific fine-tuning, at approximately 32 seconds per inference for the full pipeline.
Background & Motivation¶
Recognizing a knife is not enough to decide which region matters for a task. Cutting calls for the blade, whereas safely picking it up calls for the handle. Affordance prediction therefore combines two different problems: interpreting the functional intent of a language instruction and locating the relevant part precisely in the image. A detector may recognize parts without understanding an implicit request; a capable vision-language model may name the correct part but produce inaccurate coordinates that mislead downstream segmentation.
Methods such as AffordanceLLM and Affordance-R1 address this problem by training a unified model to connect reasoning with spatial outputs. The paper points to uneven cross-dataset results: Affordance-R1 obtains 62.72 cIoU on ReasonAff but 33.4 on RAGNet-3DOI. The authors partly attribute such patterns to specialization-induced forgetting, although cross-dataset scores alone do not establish that causal mechanism. The more directly testable question is whether existing foundation models can cooperate effectively without retraining one model to master both semantic interpretation and precise localization.
The proposed system also introduces a generated intermediate image: visualize the object performing the requested interaction, use the resulting contact or motion cues to identify its functional part, and then locate that part in the original observation. Core idea: use imagination to help decide what to interact with, and a separate grounding pipeline to determine where it is, connecting the two through an explicit textual part description.
Method¶
Overall Architecture¶
The input consists of an RGB image and a natural-language task. The primary output is an affordance mask in the original image, accompanied by bounding boxes and keypoints. Despite the name A4-Agent, the paper specifies three roles, not four agents: Dreamer imagines the interaction, Thinker selects the actionable part, and Spotter localizes it.
The default configuration uses GPT-4o for prompt construction and semantic reasoning, Qwen-Image-Editing for image editing, Rex-Omni for boxes and points, and SAM2-Large for mask refinement. The imagined image is evidence for reasoning, not the coordinate frame of the prediction. The original image remains available to Thinker and Spotter so that a plausible edited interaction is not mistaken for the current visual state.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Original image and task"] --> Dreamer["Dreamer: Interaction Imagination<br/>Prompt construction and editing"]
Dreamer --> Thinker["Thinker: Part Selection<br/>Reason over both images"]
Input --> Thinker
Thinker -->|Part description JSON| Spotter["Spotter: Spatial Grounding<br/>Rex-Omni prompts and SAM2"]
Input --> Spotter
Spotter --> Output["Boxes, points, and masks<br/>in the original image"]
This is a fixed sequence of specialized model calls with explicit interfaces. The cached paper does not describe a dynamic planner, an error-retry controller, or an iterative verification loop, so the word "agentic" should not be taken as evidence that those capabilities are present.
Key Designs¶
1. Dreamer: Interaction Imagination
A static observation shows an object's appearance but may not expose which part becomes functionally important during an action. Dreamer first asks a VLM to produce a short image-editing instruction grounded in the image and task. The instruction should name the target object and a visible functional part, specify the minimal interaction and contact configuration, and avoid unsupported attributes. Qwen-Image-Editing then edits the original image into a plausible interaction state. Conditioning on the observation matters: the objective is to provide task-relevant cues about this object, not merely to generate an unrelated example of the same category.
An interaction region is not necessarily a grasp region. For pouring, the relevant region may be the spout or outlet rather than the handle; for cutting, it may be a blade. Accordingly, Dreamer can depict hand-object or object-object interactions instead of always adding a hand holding the object. The generated image externalizes interaction priors that may help a smaller VLM, but it is still a hypothesis rather than a validated physical simulation. Because the editing prompt already requests a functional part, this stage also depends on an initial semantic judgment; it is not an independent source of ground truth.
2. Thinker: Part Selection
Thinker receives the original image, the imagined image, and the task. Its instruction asks it to recognize candidate components in the original observation, consult the imagined contact or motion cues, and identify an actionable part that is actually present in the original image. Keeping both images is important because editing may move objects, alter geometry, or introduce unsupported structures. Imagination should help interpret function without supplying coordinates for a different visual scene.
The response has a Thinking section and an Output section. Only Output is parsed downstream, as JSON containing task, object_name, and object_part. The part field follows a phrase such as "the blade of the shears," binding the part to its parent object instead of passing an ambiguous noun such as "blade." This description has no coordinates and does not ask the VLM to generate a mask. It converts a potentially indirect task into the kind of short semantic query an open-vocabulary detector can use. The interface also permits replacing the reasoning model without retraining the spatial modules. Exact prompts are referred to an appendix, but the available cache ends with the references and does not provide that appendix or JSON parsing failure handling.
3. Spotter: Spatial Grounding
Once a part has been selected, Spotter translates its textual description into geometric prompts in the original image. Rex-Omni first predicts one or more bounding boxes enclosing the relevant regions and keypoints providing representative spatial anchors. These boxes and points are passed to a promptable segmentation model to obtain precise masks. The default implementation uses SAM2-Large; references to "SAM" in the method describe this segmentation step more generally.
The two spatial stages solve different interface problems. A segmentation model can delineate boundaries but needs a reliable indication of what to segment; an open-vocabulary detector can turn a part phrase into a region but need not produce its exact contour. Spotter therefore bridges text to coarse geometry and then coarse geometry to a detailed mask, rather than repeating the earlier functional reasoning. Each predicted region retains its box, point, and mask. The paper does not specify a detailed candidate-conflict resolution or mask-ranking algorithm, so a confidence-fusion procedure should not be added to the description.
A Worked Example¶
Consider the paper's example instruction, "open the refrigerator." Dreamer first constructs a minimal interaction description such as a hand grasping the visible vertical refrigerator handle, then edits the observation to depict that interaction. This makes the relation between opening and contact with a particular part explicit, instead of merely identifying the entire refrigerator.
Thinker compares the observation with the edited image and records the task, object, and refrigerator-handle description in the three JSON fields. Rex-Omni then locates the handle in the original image, and SAM2 refines the resulting box and point prompts into a handle mask. Even if the imagined hand occludes the handle in the edited image, the output should not become a hand mask: grounding is performed in the original observation. This walkthrough illustrates the reported mechanism and is not an additional measured experiment.
Loss & Training¶
There is no new task-specific loss or optimization phase. The models are not trained or fine-tuned on ReasonAff, RAGNet, or UMD for this system. Here, zero-shot means absence of affordance-task adaptation, not a guarantee that the foundation models have never encountered related objects or internet images during pretraining.
Full inference consists of editing-prompt construction, image editing, semantic part reasoning, detection, and segmentation. The Lite variant removes Dreamer and uses Qwen-2.5-VL-3B as Thinker while retaining the spatial grounding path. The authors report averaging all results over three independent runs, with every metric's standard deviation below 1.0, but do not provide per-entry error bars. The cached text also lacks a complete set of reproduction settings such as image-editing inference steps.
Key Experimental Results¶
Main Results¶
ReasonAff contributes 600 image-task pairs. The RAGNet 3DOI and HANDAL subsets contain 3,018 pairs in total. UMD evaluation samples one tenth of the frames, producing 1,922 images covering 17 object categories and seven affordance types. Additional PhysToolBench and web images are used for qualitative open-world evaluation only.
The following results use gIoU and cIoU from Tables 1-3. Higher is better, and gains are score points rather than relative percentages. Under the usual segmentation convention, gIoU averages per-example IoU, while cIoU divides accumulated intersection by accumulated union. The cache does not spell out their implementation or empty-mask rules; gIoU here should not be confused with the Generalized IoU detection loss.
| Test set | Comparator | Comparator gIoU / cIoU | A4-Agent gIoU / cIoU | Gain |
|---|---|---|---|---|
| ReasonAff | Affordance-R1 | 67.41 / 62.72 | 70.52 / 64.62 | +3.11 / +1.90 |
| RAGNet-3DOI | AffordanceVLM | 38.1 / 39.4 | 63.9 / 58.3 | +25.8 / +18.9 |
| RAGNet-HANDAL-easy | AffordanceVLM | 58.3 / 58.1 | 61.1 / 61.7 | +2.8 / +3.6 |
| RAGNet-HANDAL-hard | AffordanceVLM | 58.2 / 57.8 | 61.0 / 59.6 | +2.8 / +1.8 |
| UMD | Affordance-R1 | 49.85 / 42.24 | 65.38 / 59.81 | +15.53 / +17.57 |
The RAGNet-3DOI ablation table reports the more precise 63.94 gIoU, whereas the main table rounds to one decimal place. The overview graphic lists UMD gIoU as 65.58, but Table 3 gives 65.38, consistent with the text's 15.53-point gain; this note follows Table 3. The ordering of P50 and P50-95 in the ReasonAff table is questionable, and other threshold results also contain inconsistencies with the usual threshold relationship. They are therefore not used to draw threshold-precision conclusions here.
Ablation Study¶
These RAGNet-3DOI results come from Tables 4-5. In the experiment described as removing Spotter, SAM still segments from boxes predicted directly by Thinker; this is not the removal of segmentation altogether.
| Configuration | gIoU | cIoU | Interpretation |
|---|---|---|---|
| Full pipeline, GPT-4o | 63.94 | 58.30 | Default configuration |
| GPT-4o, no Dreamer | 62.30 | 54.43 | Imagination adds 1.64 / 3.87 points |
| Full pipeline, Qwen-2.5-VL-7B | 63.02 | 49.87 | Alternative Thinker |
| Qwen-2.5-VL-7B, no Dreamer | 58.48 | 49.26 | Imagination adds 4.54 / 0.61 points |
| Spotter only | 45.91 | 39.82 | Both Dreamer and Thinker removed |
| Dreamer + GPT-4o boxes, followed by SAM | 6.55 | 3.89 | No dedicated localization model |
| Full pipeline, GPT-5.2 | 64.65 | 59.21 | Thinker upgrade in Table 4 |
The full GPT-4o system exceeds Spotter alone by 18.03 gIoU and 18.48 cIoU. Because this comparison removes two roles together, the entire difference cannot be attributed to Thinker alone. Conversely, the 6.55 gIoU result from direct GPT-4o boxes illustrates how correct high-level semantics can fail to translate into accurate localization.
Key Findings¶
Dreamer helps the smaller model more on gIoU, but not on every metric: its cIoU gain is only 0.61 for Qwen versus 3.87 for GPT-4o. Qwen with Dreamer reaches 63.02 gIoU, exceeding GPT-4o without Dreamer at 62.30, yet its cIoU of 49.87 remains below 54.43. The result supports a metric-specific compensation effect, not an across-the-board replacement of the stronger model.
Table 6 exposes the cost of the full system. The following approximate times and memory figures are reported using an A40. Full inference includes remote GPT-4o calls, whose server-side parameters and resources are not represented by local GPU memory.
| Configuration | Inference time | GPU memory | 3DOI gIoU / cIoU |
|---|---|---|---|
| Affordance-R1 | About 7 s | About 19 GB | 39.04 / 33.39 |
| AffordanceVLM | About 4 s | About 17 GB | 38.10 / 39.40 |
| A4-Agent Lite | About 4 s | About 16 GB | 52.91 / 41.82 |
| A4-Agent Full | About 32 s | About 30 GB | 63.94 / 58.30 |
Highlights & Insights¶
- A semantic interface is more reliable than direct coordinate generation. Restricting the VLM to functional-part selection preserves its task understanding while delegating precision to specialists. The direct-box ablation provides concrete support for that division.
- Generated imagery can serve as intermediate reasoning evidence. Contact and motion cues supplement the static observation, while the original image anchors the final answer. This distinction prevents the task from silently changing into segmentation of an imagined scene.
- Individual capabilities can improve without affordance retraining. Component substitutions in Table 4 demonstrate that generation, reasoning, and grounding backbones are replaceable. Better off-the-shelf modules provide an alternative source of progress to collecting more task labels.
Limitations & Future Work¶
- Accuracy and real-time operation remain in tension. The authors offer Lite to address efficiency, but approximately 32 seconds for Full is unsuitable for a high-frequency control loop. No robot execution success rate is reported.
- Imagination may reinforce an early semantic error. The editing prompt already involves functional-part identification, so a mistaken selection can shape the generated evidence. The paper does not quantify robustness to misleading edits, wrong-part prompts, or conflicts between the two images; this is an evidence gap identified in this reading.
- Open-world evidence is qualitative. Strong standard-benchmark results and selected novel-object examples do not establish broad out-of-distribution reliability or physically executable interactions.
- Reproduction and comparison remain incomplete. The cache lacks the referenced prompt appendix and some inference settings, and threshold metrics contain ambiguities. Closed-model versions and service costs also complicate controlled reproduction and resource comparisons.
- Pretraining exposure is not ruled out. Task-level zero-shot evaluation does not guarantee absence of pretraining overlap. Controlled novel-object tests, corrupted-imagination tests, and physical execution evaluation would strengthen the conclusions beyond additional successful demonstrations.
Related Work & Insights¶
- Versus AffordanceLLM and LISA: These approaches learn connections between language representations and segmentation outputs. A4-Agent links existing models through part phrases, geometric prompts, and masks at inference time, avoiding task adaptation while adding model-call overhead.
- Versus Affordance-R1 and VisionReasoner: These systems improve a unified model's reasoning and localization through training, including reinforcement learning. A4-Agent instead specializes components, allowing independent upgrades but not a single compact checkpoint for the complete computation.
- Versus open-vocabulary grounding alone: A detector can locate a named part without knowing which part an implicit task requires. The contribution is the separation of task-to-part reasoning from part-to-pixel grounding, not merely the use of a stronger detector.
Rating¶
- Novelty: 4/5. Imagination-assisted reasoning and explicit grounding separation address the task well, although the central contribution is coordination of existing foundation models.
- Experimental Thoroughness: 4/5. Multiple datasets, component substitutions, removal ablations, and efficiency measurements are included; physical execution and systematic generation-failure analysis are missing.
- Writing Quality: 3/5. The three-stage responsibilities are clear, but metric inconsistencies and the absent appendix limit verification of the cached version.
- Value: 4/5. A strong zero-shot baseline with replaceable interfaces, whose deployment value still depends on latency, service cost, and error propagation.