Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/hq-King/DAG
Area: Robotics & Embodied AI / 3D Vision
Keywords: affordance grounding, diffusion priors, human-object interaction, cross-modal learning, few-shot generalization

TL;DR¶

DAG uses a frozen text-to-image diffusion model to extract human-object interaction features and combines them with action text and point-cloud geometry to predict 3D affordance masks, reaching 16.09 mIoU against GREAT's 12.05 on PIAD2 unseen affordances, although its few-shot results do not establish superiority over all fully trained baselines.

Background & Motivation¶

Recognizing an object does not identify which part supports a particular action. For example, a cup's handle and body belong to the same object, but a grasping query requires localization of an actionable region. 3D affordance grounding turns this functional judgment into dense point-cloud predictions, connecting embodied perception to manipulation without constituting a complete robot controller. Methods such as IAGNet and MIFAG learn these associations from reference human-object interaction images, while GREAT introduces affordance-centric descriptions. However, visual correlations learned from limited paired data may not transfer to new objects, actions, or combinations, and extra bounding boxes or detailed descriptions increase supervision costs.

The paper draws on the large-scale pretraining experience of text-to-image diffusion models. The authors observe that these models can synthesize semantically plausible human-object interactions and highlight some action-related regions internally, suggesting reusable interaction priors. Yet image generation does not itself provide point-cloud supervision: 2D features have one spatial organization, 3D points have another, and the action word changes the target region on the same object. Even useful pretrained knowledge therefore requires mechanisms for extraction, action-specific selection, and integration with 3D geometry. The attention maps and incorrect-instruction examples in Figure 2 motivate this hypothesis, but visualization alone cannot establish physical causal understanding.

DAG consequently neither learns every interaction regularity from scratch nor asks a diffusion model to produce 3D masks directly. It preserves frozen generative representations and connects them to the 3D task through trainable semantic adaptation, fusion, and decoding components. Labeled affordance training data remain necessary; open-world capability is evaluated primarily through specified unseen splits and few-shot settings. Core Idea: turn implicit interaction knowledge in diffusion models into action-conditioned features, then constrain those features with global and local point-cloud geometry to locate actionable regions in 3D.

Method¶

Overall Architecture¶

DAG takes a target point cloud, a reference RGB human-object interaction image, and affordance text as input, and outputs an affordance mask over the points. Section 3.1 represents the point cloud as \(P\in\mathbb{R}^{N\times3}\); the image supplies an interaction example, the text specifies the action, and the geometry branch describes the target 3D object. The image first undergoes Implicit Semantic Encoding to form a condition for the diffusion U-Net, followed by Diffusion Prior Aggregation to obtain a unified visual interaction representation. Action-Conditioned Fusion combines this representation with explicit action text, and Multi-Source Geometric Decoding uses a frozen point encoder's global token and dense point features for localization. The diffusion backbone performs a single feature-extraction forward pass, not a complete multistep image-generation process or iterative denoising of a 3D mask.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Image["Reference interaction image"] --> ASE["Implicit Semantic Encoding"]
    ASE --> Prior["Diffusion Prior Aggregation"]
    Image --> Prior
    Prior --> Fusion["Action-Conditioned Fusion"]
    Text["Action text<br/>Frozen CLIP encoding"] --> Fusion
    Fusion --> Decoder["Multi-Source Geometric Decoding"]
    Points["Point cloud<br/>Frozen encoding and propagation"] --> Decoder
    Decoder --> Mask["Per-point affordance mask"]
    Labels["Training labels"] -.->|BCE and Dice supervision| Mask

Training labels participate only in loss computation, not as test-time inputs. The point-cloud branch contains standard geometric encoding and feature propagation, whose role is explained together with multi-source decoding. Reference images and action text are still required at test time, so the results should not be interpreted as zero-shot discovery of every affordance from an arbitrary point cloud alone.

Key Designs¶

1. Implicit Semantic Encoding: condition diffusion on the interaction image

A pretrained text-to-image U-Net normally expects textual conditioning, but task data may lack sentences that adequately describe contact patterns and spatial relations. A single action verb omits interaction context, while generating a natural-language caption can lose details or introduce errors. The Affordance Semantics Encoder (ASE) constructs an implicit condition directly from the image without first expressing everything in natural language. Specifically, a frozen CLIP image encoder supplies features from several late layers, which are separately linearly projected and combined through weighted summation. A learned MLP then maps the aggregate into the U-Net's conditioning space to form an implicit affordance-knowledge representation. The main text does not provide a complete list of selected CLIP layers, so it does not justify inventing a fixed layer count or exact tensor dimensions.

This conditioning pathway is distinct from the user-supplied action word. ASE extracts interaction context from the entire image, whereas the explicit action word specifies the function to localize in later affordance blocks. The method therefore cannot be reduced to encoding "grasp" with CLIP and feeding it into diffusion. ASE connects existing image content to the diffusion conditioning interface; Table 3 tests whether it improves on empty text, verbs, or generated captions.

2. Diffusion Prior Aggregation: read multiscale interaction cues from a frozen generator

The reference image and the ASE-derived condition jointly enter the frozen Stable Diffusion U-Net, from which multiple internal visual features are extracted. Unlike a single global image vector, these intermediate representations retain spatial information at different scales and can express both object semantics and local interaction regions. Following ODISE, the implementation extracts features every three U-Net blocks and resizes them into a feature pyramid. An aggregation network learns mixing weights across layers, allowing the downstream task to select useful representations instead of assuming one layer is optimal for affordance grounding. The core of the paper's Equation (3) is a weighted sum over layer features adapted into a compatible representation:

\[ A_g=\sum_{l=1}^{L}w_l A_{v,l}. \]

Here, \(A_{v,l}\) denotes visual features from layer \(l\), \(w_l\) is a learned mixing weight, and \(A_g\) is the aggregated affordance prior. The text does not state that weights must be nonnegative or normalized, so they should not be reinterpreted as layer-selection probabilities. The default diffusion timestep is \(t=0\), preserving spatial detail without introducing multistep sampling computation and noise. The timestep ablation in Table 6 also shows worse localization metrics at 50 and 100. Generative pretraining supplies the representations; downstream training does not add an image-reconstruction or denoising objective. The method thus transfers generative representations rather than synthesizing new images to expand its training set.

3. Action-Conditioned Fusion: turn general interaction priors into query-specific features

Visual priors can contain several interaction cues, and the same object can support multiple actions, making the current query essential. A frozen CLIP text encoder embeds the affordance text, which enters the Affordance Blocks alongside the aggregated visual prior. According to Section 3.4, the module first performs a scaling operation and self-attention fusion, followed by residual connections and layer normalization. It then applies cross-attention conditioned on text features, followed by a feed-forward network, residual connections, layer normalization, and average pooling to obtain affordance embeddings. The text does not specify the initial scaling operator in detail, so this note does not replace it with an unverified elementwise product or fixed-temperature scaling.

The purpose is not simply to concatenate modalities but to organize visual functional cues around the requested action. A grasping query, for example, should emphasize usable holding regions rather than the most visually salient person or the entire object. The pooled representation transfers action-related knowledge to the geometry branch, while final spatial localization still depends on point features. Explicit text acts as a query condition here, whereas the implicit ASE representation has already influenced visual feature extraction; their roles are distinct and complementary. Table 4 removes the Affordance Block and global token separately to test the utility of this fusion rather than assuming effectiveness from the architecture alone.

4. Multi-Source Geometric Decoding: use global shape to guide local affordance prediction

An interaction region in an image cannot directly serve as a label for 3D points because the representations have no inherent shared indexing. DAG uses a pretrained frozen point encoder called Uni3D in the main text to obtain a global [CLS] token and point-cloud features. Geometry-guided upsampling and feature propagation convert the encoded representation into dense point features. The main paper delegates propagation details to supplementary material, while the supplied full text contains only the main paper and references; interpolation rules and neighborhood sizes are therefore unavailable. The decoder first uses the global [CLS] token as the query and action-related affordance embeddings as keys and values in cross-attention. This ordering lets the target object's overall geometry select interaction knowledge instead of directly projecting a reference-image region.

After residual connections and a feed-forward network, the fused representation becomes the query for a second fusion with point features. An Affordance Net, implemented as an MLP, then outputs the affordance mask. Section 3.6 specifies the query, key, and value flow but does not fully explain the tensor organization from global queries to per-point outputs. It supports the design interpretation of combining global shape with action knowledge before dense geometry, but does not specify every decoder dimension needed for reproduction. The model learns semantic-geometric associations without relying on known pixel-to-point registration that the paper does not declare. Its output is a potential interaction region, not a grasp pose, collision check, force analysis, or action trajectory.

A Worked Example¶

Consider the cup-grasping example in the introduction: the inputs are an image of a person holding a cup, a target cup point cloud, and the action text "grasp". ASE extracts implicit interaction context from the reference image, and the diffusion U-Net produces multiscale features in one forward pass at \(t=0\). The aggregation network combines these features, and the affordance blocks use the text embedding of "grasp" to select holding-related information. The global point token connects this knowledge to the target cup's overall shape, while dense geometry supports per-point predictions of functional regions such as the handle. This is a mechanism illustration, not a reported per-point probability example, and it does not imply that every cup can only be grasped by its handle. During training, the predicted mask is compared with annotations to update adaptation components; testing follows the same feature pathway without labels.

Loss & Training¶

Mask supervision uses binary cross-entropy (BCE) and Dice losses, with no additional diffusion-denoising loss reported. BCE constrains individual point predictions against labels, while Dice emphasizes overlap between predicted and annotated regions. Frozen components include the diffusion model, CLIP encoders, and pretrained point encoder; trainable components perform semantic projection, aggregation, fusion, and decoding. Main experiments train on one A100 for 80 epochs with a learning rate of \(10^{-4}\). The text names Adam as the optimizer, but its cited reference [31] is Decoupled Weight Decay Regularization; this note does not silently change it to AdamW. Equations (1), (8), and (10) contain missing symbols in the extracted text, so the noise step, attention flow, and loss components are explained from neighboring prose rather than presenting guessed equations as exact author formulations.

Key Experimental Results¶

Main Results¶

PIAD1 Seen retains similar object and affordance distributions across training and testing, while Unseen reserves some object-affordance pairings for testing. PIAD2 further separates unseen objects from unseen affordances; these difficulties should not be merged into a single open-world accuracy figure. The following comparison of GREAT and DAG comes from Table 2, page 11; higher mIoU, AUC, and SIM are better, while lower MAE is better. AUC measures discrimination across thresholds, mIoU measures region overlap, SIM measures distributional similarity between prediction and target maps, and MAE measures mean absolute error. Values retain the paper's scales and DAG's reported uncertainty terms; the main text does not adequately define the uncertainty statistic or number of repetitions.

PIAD2 setting	Method	mIoU ↑	AUC ↑	SIM ↑	MAE ↓
Seen	GREAT	38.03	91.99	0.676	0.067
Seen	DAG	47.19 ± 0.3	94.71 ± 0.4	0.779 ± 0.02	0.062 ± 0.02
Unseen Obj	GREAT	20.16	79.57	0.402	0.109
Unseen Obj	DAG	28.93 ± 0.7	85.41 ± 0.3	0.592 ± 0.05	0.102 ± 0.02
Unseen Aff	GREAT	12.05	69.81	0.290	0.127
Unseen Aff	DAG	16.09 ± 0.4	75.23 ± 0.3	0.372 ± 0.05	0.123 ± 0.02

The mIoU gain is 4.04 percentage points for unseen affordances and 8.77 points for unseen objects, supporting both types of transfer while leaving substantial localization error. In PIAD1 Seen, Table 1 on page 10 reports 24.84 mIoU for DAG against 22.72 for GREAT; the Unseen values are 9.730 and 8.820, respectively.

Ablation Study¶

The following entries come from Table 3 on page 12 and Table 5 on page 13, comparing conditioning strategies and feature extractors in the PIAD1 Seen setting. The two ablation groups share the full-model row but do not represent jointly replacing several components in a single model.

Ablation group	Config	mIoU ↑	AUC ↑	SIM ↑	MAE ↓
Conditioning, Table 3	Empty	13.9	80.5	0.462	0.116
Conditioning, Table 3	BLIP	18.4	85.5	0.542	0.108
Conditioning, Table 3	Verb	20.2	86.7	0.578	0.094
Feature extractor, Table 5	ResNet	13.7	82.3	0.514	0.128
Feature extractor, Table 5	CLIP	21.2	86.2	0.573	0.091
Feature extractor, Table 5	DINOv2	22.3	87.5	0.582	0.084
Full model, Tables 3/5	DAG, ASE + Diffusion	24.84 ± 0.5	90.16 ± 0.3	0.637 ± 0.03	0.078 ± 0.01

ASE improves mIoU over verb conditioning by 4.64 percentage points, while diffusion features improve over DINOv2 by 2.54 points, separately supporting conditioning adaptation and representation choice. For Unseen, Table 4 on page 13 reports 8.45 mIoU without both the Affordance Block and [CLS], compared with 9.730 for the full model. The middle Seen rows of Table 4 are merged in text extraction, so their component assignments are not treated as reliably verified data here.

Key Findings¶

Few-shot results come from Table 7 on page 15; this reduces labeled examples per affordance category and is not another version of the PIAD2 unseen-affordance main table. The fully trained reference values match PIAD1 Seen main results, but Table 7 itself does not provide separate Seen and Unseen columns.

Table 7 config	Labeled-data description	mIoU ↑	AUC ↑	SIM ↑	MAE ↓
DAG, 1 shot	Less than 1%	12.98	79.42	0.4952	0.125
DAG, 3 shots	1%	14.34	80.22	0.4955	0.124
DAG, 5 shots	2%	15.74	82.04	0.4869	0.121
PFU	All data	12.31	77.50	0.432	0.135
IAGNet	All data	20.51	84.85	0.545	0.098

One-shot DAG exceeds fully trained PFU in mIoU, but even five-shot DAG remains below fully trained IAGNet's 20.51; this is not superiority over every fully trained baseline. SIM falls from 0.4955 at three shots to 0.4869 at five shots, so continuous improvement with more examples does not hold for every metric. Table 6 on page 13 reports 24.85 mIoU at \(t=0\), whereas the main table reports 24.84; this note preserves the respective table values rather than silently reconciling them.

Highlights & Insights¶

Generative priors serve as transferable dense representations rather than a data-generation tool. Single-pass extraction also avoids inserting a full sampling chain into localization.
Implicit image conditioning and explicit action queries have distinct roles. The former improves extraction context, while the latter selects the interaction function to predict.
Global shape and local geometry enter decoding separately. A reusable principle is to select cross-modal knowledge with target geometry before using local features for spatial precision.

Limitations & Future Work¶

The authors plan further knowledge transfer with limited 3D data; current few-shot results do not replace adequate supervision or establish reliable generalization to arbitrary new actions.
Partial-point-cloud evidence is primarily qualitative in Figure 5, not a measured real-robot success rate or systematic noise-robustness evaluation.
Plausible generations and attention examples in Figure 2 support representation usefulness but do not disentangle physical understanding, visual co-occurrence, and pretraining coverage.
Verifiable supplementary implementation details are unavailable, and the point-encoder name has an uncertain correspondence with the detection paper listed as reference [68]; reproduction requires confirming the actual encoder version and propagation implementation.
The main text lacks complete inference latency, peak memory, and matched-scale pretraining comparisons, so improved diffusion features do not directly imply better deployment efficiency.

vs IAGNet / MIFAG: These methods also use 2D interactions for 3D grounding; DAG mainly introduces frozen generative representations and semantic adaptation, rather than originating image-to-point affordance transfer.
vs GREAT: GREAT uses affordance-related descriptions for geometric and intention inference; DAG uses image-derived implicit conditioning and action-text fusion to exploit knowledge already stored in a generative backbone.
vs ODISE: DAG borrows diffusion feature extraction and implicit conditioning ideas, but targets point-cloud interaction regions, requiring additional geometric encoding and multi-source decoding.
Research direction: Controlling pretrained model scale and data coverage could clarify the contributions of implicit conditions, explicit descriptions, and actual contact evidence; this is a reader proposal, not a completed experiment in the paper.

Rating¶

Novelty: 4/5. A coherent integration of diffusion internals with 3D affordances, although several components inherit prior cross-modal and diffusion-feature ideas.
Experimental Thoroughness: 4/5. Two datasets, unseen splits, and multiple ablations are included, while few-shot and physical-interaction evidence remain limited.
Writing Quality: 3/5. The overall flow is understandable, but implementation details, some numerical conventions, and the scope of several claims need greater precision.
Value: 4/5. A reusable generative-prior transfer approach for embodied perception research, not yet a deployment-ready manipulation system.