Skip to content

title: >- [Paper Note] Seek to Segment: Active Perception for Panoramic Referring Segmentation description: >- [ECCV 2026][segmentation][Active Perception] To overcome the passive assumption of static referring segmentation, this paper introduces Active Panoramic Referring Segmentation (APRS) and PanoSeeker, leveraging explicit spatial visual memory EgoSphere and GRPO reinforcement learning to achieve 75.4% SR and 0.57 SPL. tags: - ECCV 2026 - segmentation - active_perception - panoramic_vision - embodied_ai date: 2026-09-19 content_hash: 4c663678c8fab489

Seek to Segment: Active Perception for Panoramic Referring Segmentation

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://henghuiding.com/APRS/
Area: Segmentation
Keywords: panoramic vision, active perception, referring image segmentation, spatial visual memory, embodied AI

TL;DR

Addressing the bottleneck of static referring image segmentation, this paper proposes the Active Panoramic Referring Segmentation (APRS) task along with PanoSeeker—an embodied agent utilizing explicit spherical visual memory (EgoSphere) and GRPO reinforcement learning to achieve efficient non-redundant search and accurate target alignment.

Background & Motivation

Referring Image Segmentation (RIS) aims to ground and segment target objects specified by natural language expressions. Although recent vision-language models (e.g., LISA, VisionReasoner) have substantially advanced reasoning segmentation performance, they inherently suffer from a passive perception paradigm: they assume the target object is already captured within a static, pre-cropped perspective image. In practical Embodied AI and real-world robotics, an autonomous agent operates in continuous \(360^\circ\) panoramic spaces where targets are frequently out of the initial field of view, requiring multi-step cross-view reasoning based on spatial relations (e.g., "the floor cabinet opposite the bed in the room").

Transitioning to active target search in \(360^\circ\) environments poses a fundamental technical dilemma. On one hand, explicit 3D representations (such as 3D point clouds or Gaussian Splatting) capture complete spatial layouts but demand extensive online multi-view scanning and heavy 3D reconstruction pipelines, rendering them computationally prohibitive for real-time mobile agents. On the other hand, feeding flattened equirectangular panorama (ERP) images directly into conventional 2D vision models introduces severe spherical distortions and boundary wraparound artifacts; conversely, partitioning the scene into isolated perspective crops destroys global spatial consistency, causing agents to wander aimlessly or fall into repetitive exploration dead-ends.

This paper addresses this challenge by utilizing easily accessible high-resolution panoramic images to simulate continuous active perception without heavy 3D reconstruction overhead. The core idea is to introduce PanoSeeker, an active vision-language agent that progressively projects sequential local observations onto an explicit 360° visual memory canvas (EgoSphere), converting active target localization into a structured visual canvas completion task and optimizing trajectory efficiency via two-stage SFT and GRPO reinforcement learning.

Method

Overall Architecture

The operational pipeline of PanoSeeker integrates sequential environment observation, dynamic visual memory updating, path planning, and terminal active alignment with segmentation. Operating under a realistic field of view (\(\psi_h = 120^\circ, \psi_v = 90^\circ\)), the agent starts from an initial orientation and continuously obtains perspective views. Each local observation is projected back onto a unified equirectangular memory canvas (EgoSphere) via inverse gnomonic projection. A Vision-Language Model (Qwen3-VL-8B) jointly attends to the high-resolution local view, the global spatial memory canvas, and the natural language instruction to predict next-step angular adjustments \((\Delta\theta_t, \Delta\phi_t)\) or output a termination signal [STOP]. Upon termination, an active viewpoint alignment module centers the target in the optical axis to prevent boundary truncation before invoking SAM-3 for pixel-level mask extraction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Natural Language Instruction + Initial Perspective View"] --> B["EgoSphere Spatial Visual Memory<br/>Inverse Gnomonic Projection to ERP Canvas"]
    B --> C["Spatial-Aware Visual Prompting<br/>Crosshair + Lat-Long Grid + Directed Trajectory Paths"]
    C --> D["PanoSeeker Policy Network<br/>Joint Local View and Global Canvas Reasoning"]
    D -->|Continue Search| E["Execute Viewpoint Shift (Δθ, Δϕ)"]
    E --> B
    D -->|Target Found [STOP]| F["Active Alignment and Segmentation<br/>Center-Seeking Target Re-alignment + SAM-3 Mask"]
    F --> G["Output: High-Quality Pixel-Level Mask"]

Key Designs

1. EgoSphere: Explicit Spatial Visual Memory Eliminating Redundant Search Under constrained fields of view, conventional agents relying on rolling text history or isolated image buffers suffer from catastrophic memory forgetting and context length bloat, repeatedly revisiting scanned regions. To overcome this, EgoSphere instantiates visual memory as a fixed-resolution (\(1024 \times 512\)) equirectangular panorama canvas \(M \in \mathbb{R}^{H \times W \times 3}\), initialized to zeros. At step \(t\), given the perspective observation \(V_t\) at orientation \((\theta_t, \phi_t)\), the agent projects \(V_t\) back onto a unit sphere and updates the ERP canvas: $\(M_t = M_{t-1} \oplus \text{Proj}^{-1}(V_t, \theta_t, \phi_t, \psi)\)$ Zero-valued regions naturally represent unexplored frontiers. By condensing unbounded temporal histories into a geometrically grounded 2D canvas of constant size, EgoSphere guarantees constant context length regardless of exploration trajectory length.

2. Spatial-Aware Visual Prompting for Canvas Completion Reasoning Even with accumulated ERP canvases, general-purpose VLMs struggle to estimate absolute angular displacements and link egocentric viewpoints to allocentric environments. To bridge this semantic gap, EgoSphere overlays three explicit geometric prompts directly onto the canvas: (1) a prominent central crosshair highlighting the optical axis of the current perspective view to ground egocentric-allocentric correspondence; (2) a latitude-longitude grid spaced at \(30^\circ\) intervals to serve as an explicit metric scale for rotational angle estimation; and (3) directed trajectory lines tracking visited viewpoints \(\{(\theta_1, \phi_1), \dots, (\theta_{t-1}, \phi_{t-1})\}\). These visual augmentations transform active 3D exploration into an intuitive visual "fill-in-the-blank" game, enabling the policy to infer targets in unexplored frontiers accurately.

3. Two-Stage Policy Optimization with GRPO for Geodesic Path Efficiency While pure imitation learning is prone to out-of-distribution drift, direct reinforcement learning struggles with vast action spaces and sparse terminal rewards. PanoSeeker resolves this via a two-stage training scheme: first, Supervised Fine-Tuning (SFT) over expert-annotated exploration trajectories optimizes the negative log-likelihood of expert actions, instilling semantic instruction following; second, critic-free Group Relative Policy Optimization (GRPO) refines trajectory optimality. During exploration, a dense geodesic distance efficiency reward \(R_{\text{eff}}\) is evaluated at each step: $\(R_{\text{eff}} = \gamma \left[ \text{dist}(V_{t-1}, V^*) - \text{dist}(V_t, V^*) \right] - \eta\)$ Approaching the target orientation yields positive rewards while the step penalty \(\eta\) penalizes redundant wandering. Upon emitting [STOP], a terminal reward \(R_{\text{term}}\) dispenses a success bonus weighted by mask IoU if within angular threshold \(\tau\), or imposes a heavy penalty \(-R_{\text{pen}}\) otherwise, training the agent to follow near-optimal exploration paths.

4. Active Alignment and Segmentation for Boundary-Free Mask Generation When stopping near a target, the object frequently falls near the FOV perimeter, leading to boundary truncation and degraded mask evaluation. To solve this, PanoSeeker predicts an initial bounding box \(B\) alongside the [STOP] token. An active alignment module computes the angular centroid offset \((\Delta\theta^*, \Delta\phi^*)\) and commands a final centering adjustment. With the target repositioned squarely at the optical center, SAM-3 predicts a complete, non-truncated segmentation mask \(S\).

Loss & Training

During the SFT stage, PanoSeeker optimizes action prediction across sampled expert trajectories \(\mathcal{D}_{\text{expert}}\): $\(\mathcal{L}_{\text{SFT}} = - \mathbb{E}_{\mathcal{T} \sim \mathcal{D}_{\text{expert}}} \sum_{t=1}^{T} \log p_{\theta}(a_t \mid V_t, M_t, I)\)$ In the subsequent RL phase, trajectory rewards are normalized across sampled groups of size \(G=8\) to derive advantage values \(A_i\). The clipped surrogate objective with KL divergence regularization regularizes policy updates over LoRA parameters (\(r=64, \alpha=64\)), preventing policy collapse.

Key Experimental Results

Main Results

Evaluation was conducted on the APRS benchmark comprising 7,420 samples across 4,971 panoramic scenes, comparing static baselines, heuristic scanning, and active VLM agents.

Table 1: Quantitative comparison on the APRS benchmark (corresponds to Table 2 in original paper) | Method Category | Model | Memory Mechanism | SR (%) ↑ | AS ↓ | SPL ↑ | mIoU (%) ↑ | |---|---|---|---|---|---|---| | Static Method | VLT (ICCV'21) | ✗ | 46.7 | 1.0 | - | 34.7 | | Static Method | CRIS (CVPR'22) | ✗ | 55.4 | 1.0 | - | 39.2 | | Static Method | LISA (CVPR'24) | ✗ | 64.1 | 1.0 | - | 44.5 | | Static Method | SAM4MLLM (ECCV'24) | ✗ | 62.3 | 1.0 | - | 46.2 | | Static Method | VisionReasoner (ICLR'26) | ✗ | 66.2 | 1.0 | - | 47.7 | | Heuristic Scanning | VLT (ICCV'21) | ✗ | 28.4 | 4.6 | 0.14 | 22.1 | | Heuristic Scanning | LISA (CVPR'24) | ✗ | 42.3 | 5.2 | 0.22 | 34.1 | | Heuristic Scanning | VisionReasoner (ICLR'26) | ✗ | 49.5 | 4.9 | 0.26 | 39.9 | | Active VLM Agent | Qwen3-VL-30B-Thinking | Text Log | 62.9 | 8.0 | 0.29 | 51.0 | | Active VLM Agent | Gemini-3-Flash | Text Log | 64.9 | 6.8 | 0.31 | 51.4 | | Active VLM Agent | GPT-5.2 | Text Log | 69.1 | 6.2 | 0.36 | 53.2 | | Active VLM Agent | PanoSeeker (Ours) | EgoSphere | 75.4 | 4.8 | 0.57 | 55.8* |

*Note: Static methods process the entire panorama in a single forward pass (AS=1.0); SPL is not applicable (-).

Ablation Study

Component-wise ablations validate the substantial contributions of spatial visual memory, supervised demonstration, and policy optimization.

Table 2: Ablation of PanoSeeker components (corresponds to Table 3 in original paper) | Variant | Training Scheme | EgoSphere Memory | GRPO RL | SR (%) ↑ | AS ↓ | SPL ↑ | mIoU (%) ↑ | Note | |---|---|---|---|---|---|---|---|---| | (a) Zero-shot Baseline | None | ✗ | ✗ | 41.6 | 12.4 | 0.14 | 28.5 | High wandering rate and dead loops | | (b) + Memory | None | ✓ | ✗ | 56.5 | 8.7 | 0.26 | 39.9 | Visual map prevents re-scanning (+14.9% SR) | | (c) + SFT | Expert SFT | ✓ | ✗ | 70.3 | 6.2 | 0.40 | 50.7 | Learns action semantics (+13.8% SR) | | (d) Full Model (Ours) | SFT + RL | ✓ | ✓ | 75.4 | 4.8 | 0.57 | 55.8 | GRPO cuts redundant steps (SPL +0.17) |

Table 3: Comparison of memory architectures under zero-shot setting (corresponds to Table 4 in original paper) | Memory Variant | Text Log | Visual History | SR (%) ↑ | AS ↓ | SPL ↑ | mIoU (%) ↑ | Note | |---|---|---|---|---|---|---|---| | Baseline (Zero-shot) | ✗ | ✗ | 41.6 | 12.4 | 0.14 | 28.5 | Random exploratory behavior | | Textual Log | ✓ | ✗ | 50.3 | 9.9 | 0.21 | 35.7 | Text lacks explicit spatial geometry | | Visual Buffer | ✗ | 5 perspective frames | 46.8 | 10.5 | 0.17 | 32.0 | Frame sequence breaks global context | | Hybrid Buffer | ✓ | 5 perspective frames | 43.6 | 11.0 | 0.15 | 30.6 | Multimodal cross-cue interference | | EgoSphere (Ours) | ✗ | 1 ERP canvas | 56.5 | 8.7 | 0.26 | 39.9 | Continuous spherical projection is best |

Key Findings

  • EgoSphere fundamentally breaks the dead-loop failure mode: In zero-shot testing, integrating EgoSphere immediately elevates SR from 41.6% to 56.5% and drops AS from 12.4 to 8.7 steps. Unlike discrete frame buffers that fragment geometric continuity, the unified ERP canvas maintains constant spatial context.
  • GRPO policy optimization drives exploration efficiency: Adding GRPO on top of SFT elevates SPL from 0.40 to 0.57 (a 42.5% relative gain) and reduces average steps from 6.2 to 4.8. The dense geodesic step reward successfully purges sub-optimal detours present in human expert annotations.
  • Active search dramatically outperforms static single-pass models: Processing distorted 2D panoramas directly causes state-of-the-art models like VisionReasoner to stall at 47.7% mIoU, whereas PanoSeeker's active orientation search achieves 55.8% mIoU with higher localization robustness.

Highlights & Insights

  • Spherical projection transforms 3D perception into 2D canvas completion: Rather than constructing resource-heavy 3D point clouds or Gaussian splats, PanoSeeker maps active camera rotation into an equirectangular canvas, retaining geometric continuity under a bounded, constant context length.
  • Explicit visual prompting grounds physical rotations: By rendering a central crosshair, latitude-longitude grid lines, and directed trajectory arrows, the model translates continuous camera poses into visually measurable navigational cues for general VLMs.
  • Generalizable embodied active perception: The EgoSphere mechanism—local FOV capture, spherical accumulation, and geodesic distance reward shaping—can be broadly applied to robotic inspection, active object search, and embodied visual navigation.

Limitations & Future Work

  • Lack of camera translational motion: The current formulation focuses purely on rotational adjustments (\(\Delta\theta, \Delta\phi\)) from a fixed panoramic standpoint, omitting 3D translational navigation (e.g., navigating across multi-room layouts).
  • Polar geometric distortion in equirectangular projection: Gnomonic re-projection onto ERP introduces non-linear stretching at extreme elevation angles (zenith and nadir), which may confuse vision-language models when targets are positioned directly overhead or on the floor.
  • Future Directions: Extending the ERP memory to support 6-DoF continuous movement via topological panoramic scene graphs, augmented with learned depth priors.
  • vs LISA / VisionReasoner (Static Referring Segmentation): Static methods assume targets are fully observable in static viewframes; PanoSeeker empowers agents to actively search \(360^\circ\) spaces and refine viewpoints to segment out-of-view targets.
  • vs MemGPT / VisMem (Agent Memory Systems): Prior memory frameworks rely on unbounded dialogue logs or frame queues that trigger severe context bloat and query latency; EgoSphere maps observations into a fixed-resolution 2D ERP canvas, maintaining constant token overhead.
  • vs Refer360 / REVERIE (Embodied Panoramic Navigation): Refer360 relies on dense step-by-step guidance and REVERIE evaluates passive multi-choice view selection; APRS tests true autonomous exploratory reasoning and fine-grained pixel segmentation from high-level linguistic expressions.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the APRS task and introduces EgoSphere—a constant-context, geometrically continuous spherical visual memory.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4,971 panoramic scenes, multi-difficulty referring expressions, and rigorous ablation across memory types and training stages.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulation, sharp motivational framing, and high-quality pipeline illustrations.
  • Value: ⭐⭐⭐⭐☆ Offers an efficient, low-overhead benchmark and architecture for active perception in embodied robotics.