Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale¶

Conference: ECCV 2026
arXiv: 2506.12009
Project Page: https://junha-l.github.io/affogato/
Dataset: HuggingFace
Code: None
Area: Robotics/Embodied AI
Keywords: affordance grounding, open-vocabulary annotation, automated data generation, 3D visual understanding, embodied AI

TL;DR¶

Affogato establishes a fully automated annotation pipeline by cascading three foundation models (Gemma3, Molmo, and MobileSAM), generating a scale of 750K open-vocabulary affordance heatmap dataset (Affogato-750K) from 150K 3D assets in Objaverse. It designs a unified cross-modal architecture, Espresso-3D/2D, to validate the effectiveness of this dataset as a general supervision signal—demonstrating consistent improvements across 3D and 2D affordance grounding benchmarks after pre-training, with the largest gains observed on unseen categories.

Background & Motivation¶

Background: Affordance grounding aims to locate regions on an object where specific interactions can be performed (e.g., the "holding" region of a cup), which forms the fundamental capability of embodied agents to understand "where to manipulate." Existing works span both 3D point clouds and 2D images, each developing independent baselines and evaluation benchmarks.

Limitations of Prior Work: All existing affordance datasets are constrained by the scale and cost of manual annotation. 3D datasets exclusively share 23 object categories from PartNet, containing at most 37 affordance types; 2D datasets cover up to 304 object categories and 73 affordance types. These closed category sets are far from sufficient to cover the open and diverse objects and interactions in the real world—affordance is inherently an open-vocabulary concept, but manual annotation cannot scale to match this openness.

Key Challenge: Affordance grounding naturally requires open-vocabulary understanding capabilities (the interaction modes of an object cannot be exhausted by a finite set), but the closed category labels of all existing datasets fundamentally limit the generalization capability of models. This contradiction of "tasks requiring open vocabulary vs data restricted to closed sets" is fundamentally caused by the cost bottleneck of manual annotation.

Goal: To design a fully automated annotation pipeline to generate open-vocabulary affordance supervision signals from large-scale 3D asset libraries, breaking the scale ceiling of manual annotation, and utilizing a unified model architecture to validate the transfer value of the dataset.

Key Insight: Instead of training any new model for annotation, the capabilities of multiple off-the-shelf foundation models (Gemma3/Molmo/MobileSAM) are cascaded, leveraging their respective strengths to form an automated pipeline of "query generation -> point localization -> mask segmentation -> 3D aggregation."

Core Idea: A VLM chain (Gemma3 for generating open queries + Molmo for localizing interaction points + MobileSAM for generating masks + multi-view voting for aggregating to 3D) is employed to automatically construct the Affogato-750K dataset containing 750K query-heatmap pairs. Combined with a text-conditioned cross-modal unified model, Espresso, this validates that the supervision signals produced by this automated pipeline can consistently improve the affordance grounding performance of various architectures on both seen and unseen categories.

Method¶

Overall Architecture¶

The core mechanism of Affogato is to orchestrate the zero-shot capabilities of several foundation models into an automated annotation pipeline, then pre-train a minimized unified model with the generated dataset to validate data quality. The entire framework consists of two main parts: the first half is data generation (starting from Objaverse 3D assets, yielding 3D affordance heatmaps through a three-stage pipeline), and the second half is model validation (pre-training Espresso-3D/2D on Affogato-750K, and evaluating on benchmarks such as LASO and AGD20K). The data pipeline is the core contribution of this work—fully automatically generating affordance annotations with open-vocabulary coverage and reliable spatial localization without training any model, utilizing the complementary strengths of three off-the-shelf VLMs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Objaverse 3D Assets<br/>Multi-view Rendering (25 views)"] --> B["CoT-guided Query Generation<br/>Gemma3 infers semantics -> 5 affordance queries"]
    B --> C["Molmo Interaction Point Prediction<br/>Each query x each view -> pixel coordinates"]
    C --> D["Multi-view Mask Voting Aggregation<br/>MobileSAM points -> masks -> 3D projection voting"]
    D --> E["Affogato-750K<br/>150K objects x 5 queries = 750K heatmap pairs"]
    E --> F["Text-conditioned Unified Decoder<br/>Text embeddings as queries -> cross-attention -> heatmap prediction"]
    F --> G["3D / 2D Affordance Heatmap Output"]

Key Designs¶

1. CoT Guided Query Generation: Enabling VLMs to Reason about Object Function Rather than Relying Solely on Shape

Directly presenting object images to a VLM and prompting it to generate affordance queries suffers from a typical failure mode: the model often guesses interaction modes solely based on visual shape, ignoring the true function of the object—for example, misidentifying a hat rack as a chair to generate a "sit" query, or proposing "hold to carry" for a door. Affogato resolves this issue using a Chain-of-Thought (CoT) prompting strategy: first, Gemma3 is prompted to identify the semantic category of the object ("What object is this?"), then it reasons about its typical functions based on this category ("What interactions typically occur with this object?"), and finally, it generates formatted, function-oriented queries. This "identify-then-reason" two-stage pipeline forces the model to leverage its pre-trained knowledge regarding object functions instead of solely relying on the visual features of the current image. Each object is rendered from 5 views sampled out of 25 uniformly distributed views to ensure the model sees different sides of the object, ultimately generating 5 diverse affordance queries per object—these queries are open-vocabulary, unrestricted by any predefined category set.

2. Multi-view Mask Voting Aggregation: Robust Mapping from 2D Point Predictions to 3D Heatmaps

Single-view 2D affordance localization is inherently unreliable—occlusions, perspective bias, and VLM randomness all lead to inconsistent predictions. Affogato lifts 2D noisy signals into robust 3D annotations through a four-step pipeline: "point -> mask -> projection -> voting". First, Molmo receives the image of each view and the corresponding affordance query to predict the pixel coordinates where the interaction is most likely to occur; Molmo performs exceptionally in precise pixel localization, but occasionally makes errors or points to different locations across views. Second, MobileSAM generates 2D segmentation masks prompted by these points. A key design choice is made here: SAM returns three mask candidates of different granularities (small to large) for each point prompt, and Affogato consistently selects the smallest mask. The reason is that larger masks tend to map to the whole-object silhouette, whereas smaller masks precisely isolate part-level interaction regions (e.g., handles, buttons, lids), perfectly matching the part-level granularity of affordance. Third, the 2D masks of each view are converted into \([0,1]\) probability heatmaps via sigmoid and projected onto the 3D object surface using known camera parameters and depth information. Fourth, multi-view voting: 3D surface points consistently identified as affordance regions across multiple views receive high confidence, while noisy predictions appearing in only a few views are naturally diluted. This step is the core of the pipeline's robustness—even if Molmo misidentifies locations or MobileSAM segments incompletely in some views, the consensus voting delivers clean 3D heatmaps as long as the majority of views produce correct predictions. Once 3D heatmaps are generated, they can also be back-projected onto 2D, selecting views with the highest visibility to serve as high-quality 2D annotations.

3. Text-conditioned Unified Decoder: Cross-modal Architecture using Text Embeddings instead of Learnable Queries

To provide a "clean" validation of the data quality of Affogato-750K, the authors require a model architecture uncorrupted by modality-specific designs. Existing 3D and 2D methods have accumulated distinct design traditions; directly using them as cross-modal baselines would conflate architectural differences with the dataset's contribution. Therefore, the designs of Espresso-3D and Espresso-2D are kept highly restrained: their architectures are fully symmetrical, centered around a text-conditioned heatmap decoder—unlike standard transformer mask decoders, its cross-attention queries are not a set of learnable fixed vectors, but are directly the affordance query embeddings output by a text encoder. This design inherently supports open-vocabulary: any free-form natural language query can directly drive the decoder, removing the need for predefined category lists or learning independent query tokens for each category. Espresso-3D uses PartField as the 3D visual encoder (pre-trained on large-scale 3D data to capture general part concepts) and Recap-CLIP as the text encoder, fine-tuning both; Espresso-2D uses DINOv2 as the 2D visual encoder and CLIP as the text encoder, freezing both to preserve pre-trained representations.

Loss & Training¶

Espresso-3D: The loss function is an equally weighted sum of BCE Loss (handling point-wise affordance binary classification) and Dice Loss (improving region-level alignment), consistent with LASO to ensure a fair comparison. Both models are trained for 50 epochs on LASO and Affogato-750K with a batch size of 64, utilizing 8 NVIDIA RTX A6000 GPUs. 3D point clouds use a high resolution of 16,384 points (significantly higher than LASO's 2,048 points), allowing the model to capture fine-grained geometric details.

Espresso-2D: Uses only BCE Loss. Pre-trained for 52,000 iterations on Affogato-750K with a learning rate of 0.001 and a batch size of 512 (using 7 RTX 3090 GPUs). Since the rendered images of Affogato-750K lack real-world backgrounds, background replacement augmentation is performed during pre-training (randomly sampling backgrounds from a background dataset), followed by random cropping from 256 to 224 and horizontal flipping. Fine-tuning on AGD20K reduces the learning rate to 0.0001 and requires only 400 iterations—demonstrating that pre-training on Affogato-750K provides a strong initialization.

Key Experimental Results¶

Main Results¶

3D affordance grounding on LASO (Table 2): Affogato-750K pre-training consistently improves all methods under both seen and unseen settings of LASO, with the unseen setting yielding the most significant gains:

Method	Seen aIoU	Seen SIM	Unseen aIoU	Unseen SIM
OpenAD	14.2	53.3	14.6	51.8
OpenAD + Affogato-750K pre-training	16.1 (+1.9)	53.9 (+0.6)	15.5 (+0.9)	53.4 (+1.6)
PointRefer	20.8	62.9	14.6	50.7
PointRefer + Affogato-750K pre-training	20.2 (-0.6)	60.0 (-2.9)	18.6 (+4.0)	56.1 (+5.4)
Espresso-3D	20.4	63.3	18.7	60.0
Espresso-3D + Affogato-750K pre-training	21.9 (+1.5)	63.7 (+0.4)	20.8 (+2.1)	61.4 (+1.4)

2D affordance grounding on AGD20K zero-shot (Table 4a): Without any training on AGD20K, relying solely on Affogato-750K pre-training, Espresso-2D substantially outperforms all zero-shot baselines on both seen and unseen settings:

Method	Seen KLD↓	Seen SIM↑	Unseen KLD↓	Unseen SIM↑
LISA-7B	1.627	29.6	1.830	25.6
Molmo+SAM2	1.804	26.1	1.953	22.6
AffordanceNet	1.729	26.6	1.932	22.6
Espresso-2D (Affogato-750K pre-trained)	1.426	40.2	1.571	37.6

Under the supervised learning setting of AGD20K (Table 4b), Affogato-750K pre-training also brings consistent improvements: Espresso-2D improves from 1.034 KLD / 50.3 SIM to 0.974 KLD / 51.9 SIM under full supervision, achieving optimal performance across weak, 1-shot, and full supervision levels.

Ablation Study¶

Analysis Item	Setting	Key Metrics	Note
Held-out Category Filtering	Full Pre-training	LASO unseen 20.8 aIoU	Full Affogato-750K
	Filter Overlapping Categories	LASO unseen 20.2 aIoU	Removing 5.46% of samples that potentially overlap with LASO unseen leads to only a 0.6 aIoU drop, still significantly outperforming from-scratch (18.7); gains stem from scale diversity rather than category leakage.
Data Scale Scaling	50K Pre-training	LASO unseen 18.7 aIoU	Comparable to from-scratch
	150K Pre-training	LASO unseen 19.4 aIoU
	400K Pre-training	LASO unseen 19.6 aIoU
	750K Full Scale	LASO unseen 20.8 aIoU	Monotonic improvement, confirming scale-driven nature; on the 2D side, KLD decreases from 1.064 -> 1.007 -> 0.974 -> 0.974, reaching saturation at 400K.
CoT Prompting	w/o CoT	Shape-oriented Queries	Without CoT, the model guesses solely based on shape (hat rack -> "sit", door -> "carry"). CoT forces the model to reason about categories before generating functional queries.
SAM Mask Size	Largest Mask	Covers whole object	SAM large masks tend to produce object-level silhouette segmentation.
	Smallest Mask (Ours)	Part-level Accuracy	The smallest mask isolates interactable parts such as handles/buttons, and works with multi-view voting to compensate for omissions in individual views.
Background Augmentation (2D)	w/o Augmentation	Seen SIM 35.5	Sim-to-real transfer is significantly worse without background augmentation.
	w/ Augmentation	Seen SIM 40.2	Random background replacement significantly boosts zero-shot generalization (+4.7 SIM).

Key Findings¶

Affogato-750K pre-training yields the most significant gains on unseen categories, which is consistent across architectures: PointRefer receives the largest improvement (+4.0 aIoU) on LASO unseen, while experiencing a slight decrease (20.8 -> 20.2) in the seen setting. This indicates that the core value of the dataset lies in providing supervision signals that cover a wide range of category distributions, helping models generalize to unseen object-affordance combinations. This gain holds true across three different architectures: OpenAD, PointRefer, and Espresso.
Severe asymmetry exists in cross-domain transfer: Daily-Used -> Furnitures (18.2 aIoU) significantly outperforms Furnitures -> Daily-Used (4.6 aIoU). Ablation studies controlling the training volume (Appendix Table A4) indicate that the difference in data volume (121K vs 8.8K objects) is the dominant factor, with the gap narrowing by half (13.6 -> 6.1) after control. The remaining gap stems from category coverage: the 515 categories in Daily-Used cover the test distribution of the Furniture category better than the 122 categories in Furniture.
Synthetic rendered images generalize zero-shot to real-world robotic scenarios: Espresso-2D, pre-trained solely on clean Objaverse rendered images, still correctly localizes affordances in the real-world, cluttered robot manipulation scenes of Open X-Embodiment. This validates that the affordance representations learned from synthetic 3D data possess cross-domain transfer capabilities and do not overfit to the visual features of real-world photos.
Human verification pass rate of the data pipeline is 84.8%: Among 5K human-verified samples, the primary failure modes are Molmo point prediction errors (34.9% of failure cases) and Gemma3 query errors (27.6%), with the remainder including SAM mask boundary deviations (5.2%) and insufficient camera coverage (10.5%).

Highlights & Insights¶

A "Zero-Training Annotation" Paradigm via VLM Chaining: Cascading three off-the-shelf VLMs—Gemma3 (query generation), Molmo (point localization), and MobileSAM (mask segmentation)—into an automated annotation pipeline where each model performs its specific task with zero fine-tuning. This concept of "combining the zero-shot capabilities of multiple models into a system that far outperforms individual use" essentially treats VLMs as combinable functional modules rather than standalone tools. It can be easily migrated to any scenario requiring spatial-language joint annotation (e.g., part segmentation, grasping region detection).
Multi-view Consensus Voting as an Elegant Noise-Suppression Mechanism: Single-view VLM predictions inherently contain random noise, but Affogato demonstrates that simple multi-view voting is sufficient to suppress noise to negligible levels. This requires no post-processing models or adversarial training, relying purely on the consensus mechanism where "the majority of views are correct." This forms the cornerstone of the pipeline's robustness and can be generalized to any multi-view 2D-to-3D knowledge distillation task.
A Minimalist Design of Directly Using Text Embeddings as Decoder Queries: The decoder of Espresso bypasses learnable query tokens entirely, directly using the output embeddings from the text encoder as query vectors for cross-attention. This design is cleaner and more general than learnable query schemes, naturally supporting arbitrary natural language inputs and essentially acting as the "minimum viable architecture" for referring segmentation. Serving as an evaluation tool to validate data quality, this restrained design is more persuasive than complex architectures.
An Efficient Data Strategy of "Machine Labeling + Human Spot-checking": Affogato first fully automatically generates 750K annotations and then extracts only 5K samples for human verification and correction—focusing human labor on quality evaluation rather than repetitive annotation. This "coarse-label-then-fine-inspect" workflow changes the role of manual annotators from "producers" to "inspectors," enormously increasing human-AI collaboration efficiency and serving as a highly referable strategy for building large-scale AI datasets.

Limitations & Future Work¶

Lack of Background Information Limits Transfer Performance boundary in Real-world Scenarios: Affogato-750K is based on isolated object rendering from Objaverse, lacking real-world backgrounds. Although random background augmentation partially mitigates the sim-to-real gap (boosting SIM from 35.5 to 40.2), performance still degrades in cluttered, multi-object occluded real-world scenarios. The authors suggest that the data engine could be extended to indoor/outdoor scene data to tackle navigation and other tasks.
Only Localizing Contact Areas, without Modeling Interaction Dynamics: Current methods can only address the question of "where to manipulate", leaving "how much force to apply" or "the temporal dynamics of manipulation" unaddressed. The authors explicitly identify fine-grained physical attributes (e.g., force requirements) as a future challenge.
Annotation Pipeline Still Has an ~15% Error Rate: The primary sources of failure are fine-grained localization errors from Molmo and object misidentifications from Gemma3. This implies that the upper bound of the pipeline's quality is limited by the capabilities of the VLMs utilized—advancements in VLMs themselves will directly improve the output quality of the Affogato pipeline, representing a "riding on the coattails" limitation.
2D Transfer Benefits Saturate at 400K: Unlike 3D tasks where performance continues to climb up to the full 750K scale, the 2D KLD reaches its lowest value of 0.974 at 400K, showing diminishing marginal returns for real-world image transfer with further synthetic data. The root cause is the visual domain gap between synthetic renderings and natural images, which cannot be bridged merely by scaling data volume.
Extremely Weak Cross-domain Generalization from Furniture to Daily Objects (4.6 aIoU): Performance drops sharply when transferring from a narrow training domain to a wide test domain, exposing the insufficiency of uniform sampling strategies in constructing a balanced dataset and highlighting the need for smarter category coverage strategies.

vs LASO (PointRefer): LASO is the first 3D affordance grounding benchmark introducing free-text queries, but its data is constructed by converting PartNet's closed-set labels into text descriptions, rendering it inherently closed-set. Affogato utilizes a fully automated pipeline to generate 750K annotations from 150K objects in Objaverse, achieving an order of magnitude larger category scale (>450 object classes, >350 affordance types) compared to LASO's 23 classes. More importantly, pre-training on Affogato-750K directly boosts PointRefer itself (+4.0 unseen aIoU)—this "new data improves old models" validation style demonstrates the independent value of data quality and generalizability.
vs 3D AffordanceLLM / SeqAfford: These methods attempt to generate diverse affordance query text using LLMs to increase text-side diversity, yet the 3D objects still originate from the same set of 23 categories in PartNet. Affogato breaks through the category limitation from the source—not only are the queries open-vocabulary, but the objects themselves are drawn from 150K instances in Objaverse. Moreover, query generation is conditioned on visual input rather than object category labels, allowing automatic adaptation to intra-class variations (such as chairs with different designs).
vs 2D Large-scale Datasets (2HandedAfforder/RAGNet): These works similarly explore automated annotation of large-scale 2D affordance data, but their annotations rely solely on a single view—if a VLM prediction fails in a specific view, the error is directly written into the dataset. Affogato's multi-view aggregation avoids this issue by design: the path of "3D consensus -> 2D projection" naturally possesses view consistency, equivalent to validating 2D with 3D rather than flatly extending 2D with 2D. This dimension-reduction advantage is likely one of the reasons why Affogato-750K dramatically outperforms similar automated dataset baselines (22-27 SIM) in 2D zero-shot (40.2 SIM).
vs General 2D-to-3D Knowledge Distillation (ULIP/PartField): These works distill the semantic knowledge ("what this is") of 2D foundation models into 3D encoders. Affogato extends this paradigm to functional understanding ("what can be done here"), pushing further in the depth of knowledge from "semantics to functionality" and providing a complete cognitive chain of "what the object is -> how to interact here" for embodied AI.

Rating¶

Novelty: 4/5 [The systems engineering approach of using a VLM chain for fully automated open-vocabulary affordance annotation is novel and practical, but each single link (Gemma3/Molmo/MobileSAM/multi-view aggregation) relies on existing techniques; the core contribution lies in orchestration and integration rather than algorithmic innovation.]
Experimental Thoroughness: 5/5 [Highly comprehensive coverage spanning 3D+2D dual modalities, three settings of seen+unseen+cross-domain, four supervision levels of zero-shot+weak/1-shot/full, multiple baselines + pre-training gains, and five ablations on data scale/held-out filtering/CoT/SAM mask size/background augmentation, alongside synthetic-to-real generalization. The appendix further supplements in-depth analyses such as resolution, cross-domain asymmetry, and failure cases.]
Writing Quality: 4/5 [Clear structure, high-quality figures and tables, and deep experimental analysis, though the main experimental tables are dense and the appendix is lengthy (18 pages), placing a heavier cognitive load on readers.]
Value: 5/5 [The 750K-scale open-vocabulary affordance dataset fills a crucial gap in the field—prior to this, the combined sum of all datasets was nowhere near this scale and diversity. The consistent pre-training gains across modalities and architectures validate its practical value for the entire embodied AI community, and its public release on HuggingFace further lowers the barrier to entry.]