OpenPanoD: Aligning Multimodal Prompts and Spherical Representations for Open-Vocabulary Panoramic Detection¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/hangxu-hdu/OpenPanoD
Area: Object Detection
Keywords: Panoramic Object Detection, Open-Vocabulary Detection, Multimodal Prompting, Spherical Representation, GeoFormer

TL;DR¶

To overcome semantic ambiguity from contextual co-occurrence in text prompts and severe geometric distortion from equirectangular projections in panoramic scenes, OpenPanoD introduces a multimodal prompt encoder uniting descriptive text with visual exemplars, paired with a geometry-aware spherical Transformer (GeoFormer) built on a subdivided icosahedral grid for zero-shot 360-degree object detection.

Background & Motivation¶

Panoramic object detection equips robotic systems, autonomous platforms, and immersive virtual reality with comprehensive omnidirectional scene perception. Nevertheless, mainstream panoramic detectors remain strictly confined to closed-set paradigms where models only locate predefined object categories. Extending open-vocabulary detection (OVD) to omnidirectional imagery is essential for navigating unpredictable real-world environments, but the core foundation of open-vocabulary success—reliable cross-modal alignment between user queries and visual features—breaks down in 360-degree settings due to two compounding factors.

On the query side, standard open-vocabulary frameworks rely heavily on language-only prompts passed through vision-language pre-trained models such as CLIP. In dense panoramic scenes, rich visual context and frequent entity co-occurrence cause language prompts to latch onto spurious correlations—for example, a human holding a basketball is frequently misclassified with extreme confidence as basketball due to dominant co-occurrence patterns. While visual prompts offer fine-grained visual specificity to disambiguate such instances, they lack semantic generalization across diverse appearances. On the image side, equirectangular projection (ERP) introduces severe nonlinear stretching near the poles and artificial boundary discontinuity at horizontal borders. While base categories can partially memorize distorted geometric appearances during training, novel zero-shot categories lack prior distortion exposure, making the alignment between undistorted semantic concepts and warped image features fail catastrophically.

Consequently, open-vocabulary panoramic detection requires bridging both semantic ambiguity on the query side and projection corruption on the image side. Core idea: pair a multimodal prompt encoder that merges generic textual descriptions with specific visual exemplars on the query side, with a geometry-aware spherical encoder (GeoFormer) that operates on a quasi-uniform subdivided icosahedral grid on the image side, achieving robust zero-shot panoramic detection via a unified cross-modal detection head.

Method¶

Overall Architecture¶

OpenPanoD receives an equirectangular panoramic image alongside candidate category prompts, predicting bounding field of view (BFoV) or oriented rotated bounding field of view (RBFoV) parameters. The framework operates across three key stages: First, in multimodal prompt generation, category descriptions generated by an LLM and visual exemplars retrieved from ImageNet-21K are encoded via frozen CLIP backbones and projected into a 256-dimensional detection-aligned prompt space via distinct MLPs. Second, in the panorama-aware image encoder (GeoFormer), ERP features are resampled onto a quasi-uniform spherical icosahedral grid, enriched with 3D Cartesian positional embeddings, and processed through stacked spherical self-attention layers. Third, a cross-modal Transformer decoder expands averaged category prompts with learnable instance offsets into object queries, interacts with spherical image features via cross-attention, and predicts bounding boxes while computing multi-modal max-similarity classification scores under supervised contrastive learning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Panoramic Image & Multimodal Inputs"] --> B["Multimodal Prompt Generation<br/>Text descriptions + visual exemplars via offline extraction & MLP"]
    A --> C["Spherical Grid Construction<br/>Subdivided icosahedron sampling + 3D coordinate injection"]
    C --> D["GeoFormer Spherical Attention<br/>Distortion-free spherical token exchange with circular padding"]
    B --> E["Unified Detection Head & Decoder<br/>Prompt-conditioned queries + cross-modal attention"]
    D --> E
    E --> F["Decoupled Prediction & Multi-task Loss<br/>BFoV/RBFoV regression + max-similarity classification + contrastive loss"]

Key Designs¶

1. Multimodal Prompt Generation: Balancing Semantic Generalization and Visual Specificity Text-only prompts are prone to contextual bias in dense indoor panoramas, whereas single visual exemplars struggle with abstract conceptual coverage. OpenPanoD implements an offline dual-modality library construction pipeline. For the text modality, an LLM (DeepSeek) generates $K_t=10$ descriptive phrases per class spanning brief names to functional descriptions; these phrases are processed by the frozen CLIP text encoder and averaged into a canonical text embedding: $$e_t = \frac{1}{K_t} \sum_{i=1}^{K_t} \text{CLIP}_{\text{text}}(\text{phrase}_i)$$ For the visual modality, to prevent any contamination from panoramic benchmarks, candidate images are retrieved from ImageNet-21K and ranked by ImageNet-1K ResNet-50 classifier confidence or CLIP image-text similarity, retaining the top $K_v=5$ exemplars. Their frozen CLIP image features are averaged into a prototypical visual embedding $e_v$. Both embeddings are then mapped into a shared 256-dimensional detection space using two independent two-layer MLPs: $$p_c^t = \text{MLP}_{\text{text}}(e_c^t), \quad p_c^v = \text{MLP}_{\text{vision}}(e_c^v)$$ This strategy provides category-level compact guidance for cross-modal query construction while preserving fine-grained exemplar embeddings for downstream classification.

2. Spherical Grid Construction and GeoFormer: Eliminating ERP Distortion and Boundary Seams Applying standard Transformer patches directly onto planar ERP maps suffers from non-uniform solid angle sampling near the poles and feature disconnection at the $360^\circ$ azimuth wrap-around boundary. GeoFormer builds a quasi-uniform spherical discretization through successive subdivisions of a regular 20-face icosahedron. For each 3D vertex $s_i = (x_i, y_i, z_i)$ on the unit sphere, spherical coordinates are computed and projected back to continuous ERP coordinates $(u_i, \nu_i)$: $$\theta_i = \operatorname{atan2}(y_i, x_i), \quad \phi_i = \arcsin(z_i)$$ $$u_i = \frac{\theta_i + \pi}{2\pi} W, \quad \nu_i = \left(\frac{\phi_i + \pi/2}{\pi}\right) H$$ Features are sampled from the ERP feature map using bilinear interpolation equipped with horizontal circular padding, preserving seamless azimuthal continuity. To inform the network of true 3D spatial configuration, vertex Cartesian coordinates $[x_i, y_i, z_i]$ are projected by an MLP and added to initialize the spherical tokens $x_i^{\text{in}}$. Stacked spherical multi-head self-attention layers then conduct distortion-free global reasoning across the entire sphere.

3. Unified Detection Head with Max-Similarity Classification: Decoupling Queries and Prototypes To handle variable numbers of object instances per category, the model constructs prompt-conditioned object queries by combining the average category prompt $\bar{p}_c = \frac{1}{|\mathcal{P}_c|}\sum_{p\in\mathcal{P}_c} p$ with $K$ learnable instance offset vectors $o_k$: $$\mathbf{P}_{c,k}^0 = \bar{\mathbf{p}}_c + \mathbf{o}_k, \quad c=1,\dots,N_c, \; k=1,\dots,K$$ These queries enter an $L$-layer Transformer decoder to attend over GeoFormer visual tokens via multi-head cross-attention (MHCA) and interact among themselves via self-attention (MHSA). At the classification stage, rather than matching solely against the pooled prompt, the final detection feature $z_i$ computes scaled cosine similarities across all individual prompt variants in $\mathcal{P}_c = \{p_c^t, p_c^v\}$ and takes the maximum: $$s_{i,c} = \max_{j: \text{label}(p_j)=c} \left[ \alpha \cdot \operatorname{sim}(\mathbf{z}_i, \mathbf{p}_j) + \beta \right]$$ where $\alpha, \beta$ are learnable scale and shift parameters. This max-pooling scheme allows an instance to activate strongly whenever it matches either clear semantic descriptions or visual exemplar appearances.

Loss & Training¶

The framework is trained end-to-end solely on base category annotations with frozen CLIP weights, optimized using AdamW under cosine scheduling: $$\mathcal{L} = \lambda_{\text{cls}} \mathcal{L}_{\text{cls}} + \lambda_{\text{ct}} \mathcal{L}_{\text{ct}} + \lambda_{\text{box}} \mathcal{L}_{\text{box}}$$ 1. Classification Loss $\mathcal{L}_{\text{cls}}$: Standard Focal Loss applied to max-similarity logit scores $s_{i,c}$, managing severe background-foreground class imbalance. 2. Supervised Contrastive Loss $\mathcal{L}_{\text{ct}}$: Pulls matched prediction embeddings $z_i$ closer to all valid positive category prompts $\{p^+\}$ while repelling prompts from all negative classes $\{p^-\}$ with temperature $\tau=0.07$: $$\mathcal{L}_{\text{ct}} = \sum_{i \in \mathcal{M}} -\log \frac{\sum_{p^+\in\mathcal{P}_i} \exp(\operatorname{sim}(\mathbf{z}_i, \mathbf{p}^+)/\tau)}{\sum_{p^+\in\mathcal{P}_i} \exp(\operatorname{sim}(\mathbf{z}_i, \mathbf{p}^+)/\tau) + \sum_{p^-\in\mathcal{N}_i} \exp(\operatorname{sim}(\mathbf{z}_i, \mathbf{p}^-)/\tau)}$$ 3. Bounding Box Loss $\mathcal{L}_{\text{box}}$: Linear combination of smooth L1 loss and Gaussian Label Distribution Learning (GLDL) loss to handle spherical angle ambiguity and orientation uncertainty in BFoV/RBFoV bounding boxes.

Key Experimental Results¶

Main Results¶

Evaluations are conducted on two standard benchmarks: 360-Indoor (BFoV horizontal boxes, 30 base / 7 novel classes) and PANDORA (RBFoV oriented boxes, 40 base / 7 novel classes). Metrics follow the Sphere-IoU=0.5 threshold protocol, measuring base mAP, novel mAP, and their Harmonic Mean (HM).

Dataset	Method	Prompt Type	Backbone	$\text{mAP}_{\text{base}}$	$\text{mAP}_{\text{novel}}$	HM
360-Indoor	ViLD	Text Prompt	ResNet-50	17.4	13.8	15.4
	GroundingDINO	Text Prompt	ResNet-50	26.5	20.8	23.3
	YOLO-World	Text Prompt	YOLOv8-M	27.9	26.6	27.2
	Prompt-DINO	Text Prompt	ViT-L	28.3	26.8	27.5
	DINOv	Visual Prompt	Swin-L	22.3	20.7	21.5
	T-Rex2	Visual Prompt	Swin-L	21.4	19.6	20.5
	OpenPanoD (Ours)	Text Prompt	GeoFormer	30.5	29.8	30.1
	OpenPanoD (Ours)	Visual Prompt	GeoFormer	25.9	25.5	25.7
	OpenPanoD (Ours)	Multimodal Prompt	GeoFormer	35.3	33.9	34.6
PANDORA	ViLD	Text Prompt	ResNet-50	14.5	11.1	12.6
	GroundingDINO	Text Prompt	ResNet-50	23.9	18.8	21.0
	YOLO-World	Text Prompt	YOLOv8-M	26.8	25.3	26.0
	Prompt-DINO	Text Prompt	ViT-L	27.1	25.5	26.3
	DINOv	Visual Prompt	Swin-L	20.8	18.9	19.8
	OpenPanoD (Ours)	Text Prompt	GeoFormer	29.6	28.7	29.1
	OpenPanoD (Ours)	Visual Prompt	GeoFormer	25.4	24.9	25.1
	OpenPanoD (Ours)	Multimodal Prompt	GeoFormer	34.4	31.3	32.8

Ablation Study¶

1. Component Analysis on PANDORA

Config	Text Prompts	Visual Prompts	Contrastive Loss $\mathcal{L}_{\text{ct}}$	Spherical Encoder	$\text{mAP}_{\text{base}}$	$\text{mAP}_{\text{novel}}$	HM	Note
(1)	✓				27.6	24.9	26.2	Baseline planar ERP text detector
(2)	✓	✓			29.7	26.1	27.8	Complementing visual exemplars (+1.2% novel)
(3)	✓	✓	✓		31.5	29.6	30.5	Adding contrastive supervision (+3.5% novel)
(4)	✓	✓	✓	✓	34.4	31.3	32.8	Full model with GeoFormer (+1.7% novel)

2. Comparison Across Image Encoders on PANDORA

Image Encoder	Geometric Treatment	$\text{mAP}_{\text{base}}$	$\text{mAP}_{\text{novel}}$
Standard ViT	Regular grid on planar ERP image	31.5	29.6
SphereUFormer	Spherical U-Net with local attention	33.1	30.2
GeoFormer (Ours)	Subdivided icosahedral grid + global spherical attention	34.4	31.3

Key Findings¶

Multimodal Synergy Outperforms Unimodal Inputs: Unimodal text prompts achieve 29.8% novel mAP on 360-Indoor, while visual prompts alone yield 25.5%. Combining both pushes performance to 33.9% (+4.1 points), validating that visual grounding curbs text contextual bias while linguistic semantics maintain conceptual breadth.
Distortion Alleviation Is Essential for Zero-Shot Generalization: Replacing a standard planar ViT with GeoFormer yields a +1.7% boost in novel mAP. Novel categories benefit disproportionately because they lack training-time visual exposure to distorted appearances, relying crucially on true spherical representation.
Prompt Count Saturation Profiles: Ablation over prompt quantity shows text prompts saturate around $K_t=10$ phrases, while visual exemplars peak at $K_v=5$ images. Exceeding 5 visual exemplars introduces noisy non-canonical instances that dilute prototypical representation.

Highlights & Insights¶

Dual-Sided Co-Design for Open-Vocabulary Alignment: Rather than naively plugging CLIP into panoramic networks, OpenPanoD directly addresses the twin root causes of failure—linguistic contextual bias on the query side and polar stretching on the image side.
Leak-Free Prototype Retrieval with Max-Pooling Match: The visual library is built strictly from ImageNet-21K without exposing target panoramic scenes, and classification utilizes max-similarity matching across modality variants to preserve fine-grained discriminative power.
Geometric Continuity via Circular Padding: Coupling icosahedral spherical coordinates with horizontal circular padding in ERP space seamlessly bridges the $360^\circ$ stitch seam, preventing boundary splitting.

Limitations & Future Work¶

Quadratic Complexity with Grid Subdivision: Global self-attention over fine icosahedral vertices induces notable memory overhead when scaling up to detect tiny objects; future iterations could adopt spherical linear attention or deformable sparse sampling.
Reliance on Curated External Repositories: Visual prompt retrieval relies on ImageNet-21K synset alignment, which can falter for highly specialized industrial objects or novel compound nouns without clean image exemplars.
Broader 360-Degree Vision Tasks: While demonstrated on 2D BFoV and oriented RBFoV detection, extending the framework to 3D bounding box estimation, omnidirectional tracking, and panoptic segmentation remains promising.

vs GroundingDINO / Prompt-DINO: Prominent planar open-vocabulary detectors that falter under panoramic distortion and rely on single-modality or planar prompting. OpenPanoD outperforms Prompt-DINO by 7.1 and 5.8 novel mAP on 360-Indoor and PANDORA, respectively.
vs SphereUFormer / SphereNet: Prior spherical networks focus on closed-set tasks using local operators without vision-language semantic alignment. OpenPanoD introduces open-vocabulary capability to spherical geometry.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates open-vocabulary panoramic detection through joint multimodal prompt and spherical grid alignment.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across both horizontal and rotated spherical bounding boxes against nine competing baselines.
Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, well-defined problem decomposition, and mathematically sound formulations.
Value: ⭐⭐⭐⭐⭐ Provides an extensible zero-shot foundational detector for autonomous robots and VR applications.