Panoramic Affordance Prediction¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Robotics
Keywords: Panoramic Vision, Affordance Prediction, Embodied AI, Visual Grid Prompting, Adaptive Gaze

TL;DR¶

Addressing the narrow Field-of-View and fragmented observations inherent to pinhole cameras in embodied AI, this paper pioneers the task of Panoramic Affordance Prediction, introduces PAP-12K—the first benchmark featuring 1,003 native 12K panoramic images and over 13k QA pairs—and proposes PAP, a training-free, human fovea-inspired pipeline that robustly handles severe geometric distortion, boundary discontinuity, and extreme scale variations.

Background & Motivation¶

In embodied AI, affordance prediction bridges visual perception and physical execution by specifying which parts of the environment afford interaction and how an agent can interact with them. Despite its importance in high-level task planning and low-level robotic manipulation, existing affordance prediction frameworks remain predominantly anchored to conventional pinhole camera models. Because pinhole cameras possess a severely restricted Field-of-View (FoV), embodied agents suffer from acute "tunnel vision," perceiving the 3D environment only as fragmented, disjointed observations. To locate interaction targets or gather necessary contextual cues in practical deployments, robots are forced to repeatedly reorient themselves, incurring heavy movement overhead and computational burdens. Crucially, targets situated in peripheral or rear regions are frequently overlooked, leading to suboptimal planning or outright failure.

Adopting \(360^\circ\) panoramic cameras provides a natural remedy by capturing global spatial relationships and holistic scene geometry in a single shot. However, standard panoramic imagery represented in Equirectangular Projection (ERP) introduces three severe physical and geometric challenges: severe geometric distortion, where objects stretch dramatically near the poles; extreme scale variations, where target sizes fluctuate across orders of magnitude (e.g., from tiny sensors occupying 0.0038% of pixels to large curtains occupying 39%); and boundary discontinuity, where continuous objects wrap around and get severed across the image's left and right borders. When existing state-of-the-art affordance models tailored for standard perspective images are applied directly to ERP images, they experience catastrophic degradation due to acute domain shift and attention dilution across ultra-high resolutions.

The angle of attack in this work is inspired by the human foveal visual system: rather than processing wide visual fields with uniform high acuity, humans first scan broad scenes with low-resolution peripheral vision to locate areas of interest, then adjust their gaze to align the fovea with the target for detailed parsing. Core idea: decouple panoramic affordance prediction into a training-free fovea-inspired pipeline that recursively routes coarse target locations on downsampled ERP images via visual grid prompting, applies spherical tangent-plane adaptive gaze to eliminate distortion and boundary seams, and grounds fine-grained masks using 2D foundation models before inverse-projecting back to the panorama.

Method¶

Overall Architecture¶

PAP is an entirely training-free, coarse-to-fine framework organized into three sequential stages: Recursive Visual Routing via Grid Prompting for coarse semantic-spatial isolation, Adaptive Gaze for geometric rectification onto a distortion-free perspective plane, and Cascaded Affordance Grounding for precise pixel-level mask extraction and re-projection into equirectangular space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Input: 12K Panoramic Equirectangular (ERP) Image I_ERP<br/>and Task Description T"] --> RVR["Recursive Visual Routing via Grid Prompting<br/>4×3 Discrete Grid Overlay + Scale-Aware Dynamic Downsampling"]
    RVR --> AG["Adaptive Gaze Projection<br/>Latitude/Longitude Alignment + Adaptive FoV Spherical Tangent Projection"]
    AG --> CAG["Cascaded Affordance Grounding & Inverse Mapping<br/>Rex-Omni Open-Vocabulary Detection + SAM-2 Segmentation<br/>+ Inverse Perspective-to-Spherical Projection"]
    CAG --> Output["Output: High-Precision Panoramic Affordance Mask M_ERP"]

Key Designs¶

1. Recursive Visual Routing via Grid Prompting: Transforming continuous coordinates to discrete choices with scale-aware zoom

While modern Vision-Language Models (VLMs) excel at semantic reasoning to deduce what tools are needed for complex instructions, they struggle with direct continuous coordinate regression in ultra-high-resolution panoramas due to token constraints and attention dispersion. To resolve this grounding bottleneck, this module overlays a \(4 \times 3\) numerical grid (indexed 1 through 12) onto the ERP image. The VLM is prompted to generate an explicit object description \(T_{obj}\) and select the discrete grid index enclosing the target, casting continuous spatial localization into a discrete multimodal multiple-choice formulation. To cope with extreme scale variations, a recursive routing policy dynamically zooms in: if the target spans multiple grids, recursion terminates as resolution is already sufficient; if confined within a single grid, the sub-scale region is cropped and overlaid with a new grid for an iterative pass. Crucially, computational efficiency is guaranteed by aggressive downsampling: the full panorama is downsampled to \(2000 \times 1000\) (a \(\sim 1/6\) scaling ratio) in the initial pass, while the sub-scale cropped patch is downsampled to \(1500 \times 1000\) (a \(\sim 1/2\) ratio), ensuring progressive visual fidelity without exceeding token capacity.

2. Adaptive Gaze: Spherical tangent-plane projection for training-free domain adaptation

Performing segmentation directly on cropped ERP patches suffers from severe domain gaps because 2D foundation models are pre-trained strictly on rectilinear perspective imagery, whereas ERP formats suffer from latitude-dependent curvature and split boundaries. Mimicking the human behavior of turning one's head to center gaze upon an object, Adaptive Gaze aligns the virtual camera's principal point with the grid center's latitude and longitude \((\phi_c, \theta_c)\) and dynamically calibrates the Field of View (FoV) according to the grid dimensions. By projecting this spherical surface patch onto the local tangent plane, it synthesizes a distortion-free perspective image \(I_{persp}\). Because this projection operates intrinsically on the continuous spherical manifold, geometric stretching is completely rectified, minute targets are magnified to standard resolutions, and objects severed at the 2D ERP boundaries are naturally unified, enabling seamless zero-shot transfer of standard 2D vision models.

3. Cascaded Affordance Grounding: Decoupled open-vocabulary detection and instance segmentation

Given the rectified perspective patch \(I_{persp}\), the system conducts fine-grained pixel-level segmentation via a two-stage cascaded architecture. First, an Open-Vocabulary Detector (Rex-Omni) takes the explicit target name \(T_{obj}\) (deduced during recursive routing) and detects bounding box \(B\) and keypoints \(P\). Because the search space is confined to the target patch and guided by an explicit object name rather than a vague instruction, the detector is freed from wide-scene background clutter and semantic ambiguity. Next, bounding box \(B\) and keypoints \(P\) serve as dense spatial prompts for SAM-2-Large to produce an accurate instance-level mask \(M_{persp}\). Finally, an analytical inverse perspective-to-spherical transformation maps \(M_{persp}\) back onto the original \(11904 \times 5952\) ERP coordinate space, generating the full panoramic affordance mask \(M_{ERP}\).

Loss & Training¶

PAP operates entirely training-free. During inference, Qwen3-VL-32B serves as the visual-language backbone, Rex-Omni acts as the open-vocabulary detector, and SAM-2-Large performs zero-shot instance segmentation. No model weights undergo parameter updates or gradient backpropagation.

Key Experimental Results¶

Main Results¶

The PAP-12K benchmark comprises 1,003 native \(11904 \times 5952\) resolution panoramic images, 6,103 annotated instances, and 13,493 reasoning-driven QA pairs. Evaluation relies on four complementary metrics: generalized IoU (gIoU), cumulative IoU (cIoU), precision at IoU=0.5 (\(P_{50}\)), and mean precision across IoU thresholds 0.5 to 0.95 (\(P_{50-95}\)).

Method	gIoU (%) ↑	cIoU (%) ↑	\(P_{50}\) (%) ↑	\(P_{50-95}\) (%) ↑	Inference Time
OV-Seg	29.48	17.85	32.00	18.80	\(\sim\)8s
LISA	15.21	16.34	13.66	8.30	\(\sim\)7s
VisionReasoner	49.33	44.64	51.06	38.06	\(\sim\)12s
AffordanceVLM	9.66	13.11	8.96	5.41	\(\sim\)7.8s
Affordance-R1	51.80	50.32	55.47	40.70	\(\sim\)10.4s
A4-Agent (2nd)	62.55	49.97	67.09	54.28	\(\sim\)11.8s
PAP (Ours)	71.56	62.30	75.49	64.97	\(\sim\)10s

Under difficulty partitioning (Hard subset denotes objects occupying \(>30\%\) or \(<0.1\%\) of total pixels, or split across boundary edges, representing \(\sim 30\%\) of the benchmark):

Difficulty Subset	Method	gIoU (%) ↑	cIoU (%) ↑	\(P_{50}\) (%) ↑	\(P_{50-95}\) (%) ↑
PAP-12K-Hard	A4-Agent (2nd)	42.75	36.42	46.48	30.38
	PAP (Ours)	60.35	52.59	63.82	52.17
	Gain (\(\Delta\))	+17.60	+16.17	+17.34	+21.79
PAP-12K-Normal	A4-Agent (2nd)	70.91	54.94	75.79	64.36
	PAP (Ours)	76.40	65.20	80.52	70.49
	Gain (\(\Delta\))	+5.49	+10.26	+4.73	+6.13

Ablation Study¶

Ablations on a randomly sampled 10% subset of PAP-12K analyze prompt strategies, recursive visual routing, and adaptive gaze rectification.

Ablation on Prompt Style (Chain-of-Thought CoT vs. Visual Grid Prompting VGP):

CoT	VGP	gIoU (%) ↑	cIoU (%) ↑	\(P_{50}\) (%) ↑	\(P_{50-95}\) (%) ↑
✗	✗	57.70	57.68	60.76	50.52
✗	✓	69.56	61.05	74.02	61.90
✓	✗	67.22	58.75	70.61	60.51
✓	✓	71.56	62.30	75.49	64.97

Ablation on Adaptive Gaze (AG) and Recursive Visual Routing (RVR):

Configuration	Evaluation Scope	gIoU (%) ↑	cIoU (%) ↑	\(P_{50}\) (%) ↑	\(P_{50-95}\) (%) ↑
w/o Adaptive Gaze	All	64.99	55.43	68.43	56.37
w/ Adaptive Gaze	All	71.56	62.30	75.49	64.97
w/o RVR (Single-step Grid)	All	70.56	57.05	74.43	63.23
w/ RVR (Recursive Zoom)	All	72.69	63.85	76.29	66.13
w/o RVR	Hard Subset	57.37	34.81	61.00	47.65
w/ RVR	Hard Subset	62.40	47.11	64.89	53.64

Key Findings¶

Adaptive Gaze bridges the geometric domain gap: Eliminating Adaptive Gaze and running OVD plus SAM directly on equirectangular crops drops gIoU by 6.57% and fine-grained precision \(P_{50-95}\) by 8.60%, confirming that pre-trained 2D priors degrade severely under unrectified spherical curvature.
Recursive Visual Routing is decisive for hard edge cases: While RVR improves overall cIoU by +6.80%, its impact on the Hard subset reaches an extraordinary +12.30% gain (jumping from 34.81% to 47.11%), demonstrating that dynamic multi-scale zooming is vital for resolving minute, sub-scale affordance targets.
Visual anchors outperform verbal spatial prompts: Replacing explicit visual grid overlays with verbal descriptions of image splits results in an 11.86% drop in gIoU without CoT and a 4.34% drop with CoT, validating visual grid anchors as essential spatial grounding anchors for high-resolution VLMs.

Highlights & Insights¶

Zero-shot bridging of 2D foundation models to \(360^\circ\) vision: By exploiting spherical tangent projections to eliminate ERP distortions, the method successfully leverages existing 2D detectors and segmenters without requiring any expensive panoramic fine-tuning or specialized architectural overhauls.
Discretizing continuous spatial localization: Converting complex coordinate predictions into discrete grid selection through visual grid prompting sidesteps VLM grounding weaknesses while enabling adaptive multi-scale recursion.
Pioneering PAP-12K benchmark: Built with real-world 12K cameras across hundreds of varied indoor environments, PAP-12K captures authentic geometric distortions, scale disparities, and boundary wraps, filling an important gap in embodied perception research.

Limitations & Future Work¶

Multi-model inference latency: Cascading Qwen3-VL-32B, Rex-Omni, and SAM-2 yields an overall inference time of \(\sim\)10s per query, which remains too slow for real-time, closed-loop robot manipulation.
Focus on 2D pixel masks rather than 3D grasp poses: The benchmark and framework output 2D equirectangular masks; mapping affordance masks to actionable 6-DoF end-effector grasp poses requires integrating external depth and pose estimators.
Future directions: Developing native, lightweight end-to-end panoramic affordance models and extending panoramic affordance grounding into 3D metric point clouds.

vs A4-Agent [49]: While A4-Agent pioneered decoupled reasoning and grounding for conventional perspective images, PAP is the first to tackle omnidirectional \(360^\circ\) panoramas, introducing recursive grid routing and spherical tangent gaze projection to overcome severe ERP distortion and boundary discontinuity.
vs Affordance-R1 [36] & RAGNet [38]: These methods rely on reinforcement learning or fine-tuning VLMs on perspective images. In 12K panoramic settings, their spatial grounding fails due to extreme resolution downsampling and severe polar distortion. PAP demonstrates that training-free foveal decomposition substantially outperforms fine-tuned perspective baselines.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneering work establishing the panoramic affordance prediction task, a 12K native benchmark, and an elegant fovea-inspired zero-shot framework.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking against perspective baselines, difficulty split analyses, and meticulous module ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with intuitive biological motivation and rigorous geometric formulations.
Value: ⭐⭐⭐⭐⭐ Provides a foundational benchmark and an effective paradigm for expanding embodied agent perception to omnidirectional scenes.