title: >- [Paper Note] RelAfford6D: Relational 6D Affordance Graphs for Constraint-Driven Robotic Manipulation description: >- [ECCV 2026][robotics][Relational Affordance] Introduces Relational 6D Affordance Graphs that formulate robotic manipulation as part-conditioned SE(3) kinematic constraint manifold tracking, achieving superior training-free zero-shot execution and closed-loop robustness across articulated and rigid objects. tags: - ECCV 2026 - robotics - Relational Affordance - Articulated Manipulation - Vision Foundation Models date: 2026-09-19 content_hash: 76f0967b6a12cf4c

RelAfford6D: Relational 6D Affordance Graphs for Constraint-Driven Robotic Manipulation¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Robotics & Embodied AI
Keywords: robotic manipulation, relational affordance, vision foundation models, articulated object manipulation, kinematic constraint manifold

TL;DR¶

RelAfford6D proposes a training-free robotic manipulation framework centered on Relational 6D Affordance Graphs, transforming open-world language instructions into part-level SE(3) kinematic constraint manifolds and closed-loop visual replanning to achieve superior zero-shot manipulation on both articulated and rigid objects.

Background & Motivation¶

The long-standing goal of open-world robotic manipulation is to enable robots to interpret free-form natural language instructions and reliably manipulate physically diverse, previously unseen objects. However, translating high-dimensional multi-modal visual observations into low-level, high-precision physical control commands is fundamentally impeded by the perception-action gap. While recent data-driven visuomotor policies and foundation model paradigms attempt to overcome this gap via large-scale demonstration datasets, they face a critical dilemma between representational expressiveness and strict kinematic physical constraints.

This representational bottleneck is particularly severe for complex articulated objects like microwaves, drawers, and laptops. Traditional 2D pixel-level affordance heatmaps lack true 3D metric depth, preventing accurate physical execution. Methods based on sparse 3D point clouds or isolated keypoints provide contact coordinates but sever the continuous relative geometry between interacting parts, making them unable to explicitly capture rotation axes or sliding directions. Meanwhile, dense 3D descriptor fields treat manipulation as gradient descent over implicit energy fields, failing to enforce the hard kinematic joint constraints essential for rigid articulated mechanisms. In practice, opening a microwave door is not merely a matter of reaching its handle; it is strictly governed by the rigid-body kinematic relationship between the door and the appliance body.

The key insight of this work is that articulated manipulation is fundamentally governed by the relative rigid-body geometry between object parts rather than isolated contact points. The authors decompose interaction into a spatial relational pair comprising a primary interacting part and a physical anchor part. Core idea: introduce the Relational 6D Affordance Graph to deduce part-level semantic topology from language, elevate topological nodes into metric SE(3) poses via vision foundation models, and analytically formulate action execution as continuous kinematic constraint manifold tracking with real-time closed-loop replanning.

Method¶

Overall Architecture¶

RelAfford6D bridges high-level semantic reasoning and low-level physical control through three modular stages: 1. Semantic Topology Generation: Combines LLM reasoning with a PartNet-Mobility-derived kinematic knowledge base via retrieval-augmented generation (RAG) to deduce an underlying kinematic chain $((s_p, s_a), a)$, identifying the primary interacting part $s_p$, physical anchor part $s_a$, and relational constraint type $a$ (e.g., revolute or prismatic). 2. Metric Visual Grounding: Leverages an open-world segmentation foundation model (SAM3) to extract part-level 2D binary masks $\{M_p, M_a\}$, which are passed to a category-agnostic pose estimator (FoundationPose) along with depth observations to instantiate topological nodes as absolute metric SE(3) poses $\mathbf{T}_P^*$ and $\mathbf{T}_A^*$, yielding the spatial graph $\Psi$. 3. Constraint-Driven Kinematic Execution: Analytically constructs 1D kinematic manifolds ($\mathcal{M}_{\text{rev}}$ or $\mathcal{M}_{\text{pris}}$) from the relative pose and constraint type, executes collision-free reaching via RRT-Connect followed by precise manifold tracking, and maintains closed-loop visual feedback at 5 Hz to trigger dynamic replanning upon disturbances.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Language Instruction + RGB-D Observation"] --> B["Semantic Topology Generation<br/>Kinematic KB Retrieval + LLM Reasoning (sp, sa, a)"]
    B --> C["Metric Visual Grounding<br/>SAM3 Segmentation + FoundationPose SE(3) Estimation"]
    C --> D["Kinematic Constraint Manifold Derivation<br/>Analytic screw axis and manifold from relative poses"]
    D --> E["Two-Stage Trajectory Execution<br/>RRT-Connect reaching + 1D manifold tracking"]
    E --> F{"Closed-Loop Tracking & Deviation Detection<br/>Anchor pose deviation > 5 cm?"}
    F -->|Yes| G["Dynamic Manifold Replanning<br/>Re-instantiate manifold and re-synthesize remaining path"]
    F -->|No| H["Execution Completed at Target Joint State"]
    G --> H

Key Designs¶

1. Semantic Topology Generation: Deducing physically valid kinematic topologies from open language

While large language models excel at semantic decomposition, unconstrained generation frequently hallucinates physically invalid actions or omits essential grounding structures. To address this issue, the authors construct a structured kinematic knowledge base $\mathcal{K} = \{((s_{pi}, s_{ai}), a_i)\}_{i=1}^N$ from PartNet-Mobility. Given an input query, the LLM refines it into a compact query $q$, which is encoded to retrieve the top-$k$ nearest kinematic chains using cosine similarity. These candidates serve as structured geometric priors injected into the LLM prompt. The LLM then outputs the validated relational tuple $((s_p, s_a), a)$, where $s_p$ designates the active part, $s_a$ the anchor part, and $a$ serves as the Kinematic Manifold Selector, preventing common reasoning failures such as attempting to pull a revolute hinge along a straight line or manipulating a door without anchoring to the cabinet body.

2. Metric Visual Grounding: Cascaded foundation models for part-level SE(3) instantiation

Transforming semantic graph nodes into physically executable targets requires high-fidelity metric coordinates, yet conventional pose estimators treat entire objects as monolithic rigid bodies. RelAfford6D resolves this via a decoupled two-stage grounding function $g_{\text{percept}} = f_{\text{geo}} \circ f_{\text{sem}}$. First, SAM3 generates independent binary masks $\{M_p, M_a\}$ conditioned on the relational part queries $(s_p, s_a)$. Next, category-agnostic FoundationPose processes the masked RGB-D regions with reference-view-reconstructed meshes (without requiring CAD models) to estimate independent 6D rigid transformations $\mathbf{T}_P^*, \mathbf{T}_A^* \in \text{SE}(3)$ in the world frame. By restricting pose estimation strictly within part masks, the module provides reliable, millimeter-level relative boundary conditions between interactive components.

3. Constraint-Driven Kinematic Execution: Formulating manipulation as analytic manifold tracking and replanning

Rather than learning black-box trajectory distributions that fail under distribution shifts, action execution is formulated as an analytical SE(3) kinematic constraint satisfaction problem. The end-effector establishes a rigid grasp with the primary part at an initial relative offset $\mathbf{T}_{\text{offset}} = (\mathbf{T}_P^*)^{-1}\mathbf{T}_{\text{ee}}(0)$. Downstream motion is then analytically confined to a 1D manifold parameterized by scalar joint state $q$: - Prismatic Constraint Manifold ($a = \text{pull}$): Translates along a dominant screw axis $\xi_{\text{pris}} = (v, 0)$ defined in the anchor frame: $$\mathcal{M}_{\text{pris}}(q) = \mathbf{T}_A^* \cdot \mathbf{Trans}(qv) \cdot (\mathbf{T}_A^*)^{-1} \mathbf{T}_P^* \cdot \mathbf{T}_{\text{offset}}$$ - Revolute Constraint Manifold ($a = \text{rotate}$): Rotates about a hinge axis $\xi_{\text{rev}} = (u, p \times u)$ anchored at $\mathbf{T}_A^*$: $$\mathcal{M}_{\text{rev}}(q) = \mathbf{T}_A^* \cdot \mathbf{Rot}(u, q) \cdot (\mathbf{T}_A^*)^{-1} \mathbf{T}_P^* \cdot \mathbf{T}_{\text{offset}}$$

Trajectory execution decouples global collision-free reaching via RRT-Connect from local execution, where $N$ waypoints $\tau^* = \{\mathcal{M}_a(q_k)\}_{k=1}^N$ are sampled across $[0, q_{\text{target}}]$. During execution, FoundationPose tracks the live anchor pose $\hat{\mathbf{T}}_A(t)$ at 5 Hz. If external physical disturbance shifts the anchor beyond a threshold $\delta = 5\text{ cm}$, the system dynamically updates boundary conditions and re-synthesizes the remaining waypoints over $[q_t, q_{\text{target}}]$ in under 0.05 s without heuristic restarts.

Key Experimental Results¶

Main Results¶

Evaluated in the SAPIEN physics simulator using a Franka Panda robotic arm across 10 articulated object categories and 3 classical rigid object tasks. Learning-based baselines are trained on 10,000 trajectories across 20 articulated categories or 5,000 expert demonstrations for rigid tasks. RelAfford6D operates in a completely zero-shot, training-free manner. Articulated manipulation success is defined as joint displacement $|\theta_{\text{after}} - \theta_{\text{before}}| \ge 0.1$ (rad or m).

Method	Training Type	Articulated Objects Avg Success (%)	PickCube (Rigid)	StackCube (Rigid)	PushCube (Rigid)
Where2act	Data-driven (10k trajectories)	27.0	-	-	-
FlowBot3D	3D Flow field (10k trajectories)	40.0	-	-	-
ManipLLM	Embodied MLLM (10k trajectories)	62.0	-	-	-
RoboMamba	Embodied Mamba (10k trajectories)	68.0	-	-	-
RDT-1B	Diffusion Foundation Model (5k demos)	-	77.2 ± 0.48	74.0 ± 0.30	100.0 ± 0.00
OpenVLA	VLA Foundation Model (5k demos)	-	8.0 ± 0.00	8.0 ± 0.00	8.0 ± 0.00
Diffusion-Policy	Action Diffusion Policy (5k demos)	-	40.0 ± 0.00	80.0 ± 0.00	88.0 ± 0.00
RelAfford6D (Ours)	Completely Training-free (Zero-Shot)	74.0	100.0 ± 0.00	100.0 ± 0.00	100.0 ± 0.00

In physical real-world experiments on a Realman RM75 manipulator equipped with an Intel RealSense D435 camera across 7 task categories (Table 6), RelAfford6D achieves superior zero-shot transfer: Open Microwave 8/10, Pull Drawer 8/10, Press Dispenser 8/10, Close Laptop 8/10, Screw Bottle 9/10, Stack Cubes 10/10, and Pick Spatula / Pour 9/10, significantly outperforming ManipLLM (~60%) and FlowBot3D (~30%).

Ablation Study¶

Systematic ablations on 10 manipulation tasks (30 trials per task, reported in Table 4 and Table 5) quantify component contributions and perceptual robustness:

Variant / Configuration	Success Rate (%)	Mechanism & Performance Impact
Ours (Full Model)	74.4	Complete system: Kinematic RAG + SAM3 + Anchor Constraints + 5 Hz Replanning
No Kinematic Knowledge Retrieval	65.0	Raw LLM prompt inference without priors causes anchor omissions or axis hallucinations (-9.4%)
Replace VFMs (Grounding DINO / T-Rex2)	68.5	Bounding box overlap degrades thin-part pose estimation (-5.9%)
*Anchor-Free (Track $\mathbf{T}_P^$ only)**	48.5	No anchor frame leads to incorrect linear trajectories on hinges; performance collapses (-25.9%)
No Dynamic Manifold Replanning	35.0	Open-loop execution fails catastrophically under external shifts (-39.4%)
Visual Robustness - Clean Scene	78.0 / 82.0	Baseline performance on Open Microwave / Pull Drawer
Visual Robustness - 30% Occlusion	72.0 / 74.0	Graceful degradation of only 6-8% despite severe occlusion

Key Findings¶

Dual-part anchor relationships are fundamental for articulated interaction: Removing the physical anchor constraint (Anchor-Free) drops the success rate from 74.4% to 48.5%. Tracking only the handle lacks knowledge of the rotation axis origin and orientation, causing straight-line pulls that jam hinges or slip grippers.
Closed-loop replanning provides vital dynamic resilience: Under physical disturbances during execution, open-loop execution succeeds in only 35.0% of trials, whereas continuous tracking of the anchor pose and dynamic replanning restores success to 74.4%.
Kinematic constraints mitigate perceptual degradation: Even when replacing SAM3 with coarser detectors or occluding 30% of target parts, success rates remain resilient because the 1D physical manifold analytically restricts valid motion directions.

Highlights & Insights¶

Decoupling relational semantics from rigid-body physics: Instead of forcing neural networks to memorize complex articulation mechanics end-to-end, RelAfford6D delegates semantic topology deduction to LLMs and grounds physical trajectories on analytical SE(3) manifolds, achieving both open-vocabulary reasoning and kinematic rigor.
Zero-shot out-of-the-box deployment: By leveraging off-the-shelf vision foundation models (SAM3 and FoundationPose), the framework operates directly without requiring robot trajectory datasets, yet surpasses state-of-the-art baselines trained on 10,000 domain demonstrations.
Asymmetric latency budget: Heavy multi-modal grounding occurs only during one-time initialization (~20.7 s), while online closed-loop tracking runs at 5 Hz, manifold replanning takes less than 0.05 s, and low-level joint control runs in under 0.01 s.

Limitations & Future Work¶

Constraint scope limited to rigid 1-DoF mechanisms: Current formulations model revolute and prismatic joints. While helical or multi-DoF joints can theoretically be composed, non-rigid deformable objects (ropes, cloth) violate rigid SE(3) assumptions.
Sensitivity to catastrophic initial perception failures: Severe initial occlusions or intense specular highlights that completely break SAM3 part segmentation prevent pose initialization, as the system currently lacks active exploratory recovery.
Future directions: Integrating temporal 3D tracking, active multi-view camera repositioning for cluttered scenes, and extending relational manifold formulations to multi-finger dexterous hands and non-rigid manipulation.

vs Where2act / FlowBot3D: Where2act and FlowBot3D predict isolated contact points or instantaneous 3D flow vectors, lacking global rotation center constraints over long-horizon motions; RelAfford6D enforces continuous rigid-body SE(3) manifold consistency between parts.
vs ReKep / VoxPoser: ReKep uses keypoint relational optimization and VoxPoser relies on language-guided 3D value maps. While effective for free-space navigation, they incur high optimization overhead at rigid mechanical interfaces; RelAfford6D collapses the search space to an exact 1D manifold.
vs End-to-End VLA Models (OpenVLA / ManipLLM): VLA models suffer from distributional shifts and out-of-distribution hallucinations; RelAfford6D combines neural perception with symbolic kinematics for reliable zero-shot cross-category generalization.

Rating¶

Novelty: ⭐⭐⭐⭐☆ (Combines relational affordance graphs with analytical SE(3) kinematic manifold tracking in a clean, principled formulation)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across articulated and rigid benchmarks in SAPIEN simulation and validated on physical Realman RM75 manipulator)
Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation, cohesive narrative bridging semantic reasoning, perceptual grounding, and control theory)
Value: ⭐⭐⭐⭐⭐ (Offers a practical training-free paradigm for embodied manipulation, highlighting the value of geometric physics priors over pure imitation learning)