Skip to content

Sentinel: Embodied Cooperative Spatial Reasoning and Planning

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/UMass-Embodied-AGI/Sentinel
Area: Multimodal VLM / Robotics & Embodied AI
Keywords: embodied AI, multi-agent cooperation, spatial reasoning, route replanning, dynamic obstacle avoidance

TL;DR

To overcome the failure of embodied multi-agent coordination in large outdoor domains with dynamic threats, this work introduces the Sentinel Challenge benchmark and CoSaR framework, coupling structured spatial memory with VLM-guided route refinement to achieve rapid, hazard-free gathering.

Background & Motivation

Multi-agent cooperation is an essential cornerstone for embodied artificial intelligence operating in open-world physical environments. Humans navigate and coordinate across city-scale domains intuitively: agreeing on safe meeting locations via conversation, exchanging real-time updates regarding blocked streets or dynamic hazards, and continually updating route commitments as conditions evolve. However, existing communicative embodied systems remain confined to static indoor settings (such as household rearrangement in CoELA). Scaling multi-agent coordination to city-scale outdoor environments introduces vastly larger spatial extents, severe perceptual occlusion, and dynamically moving hazards that render indoor navigation paradigms completely ineffective.

In city-scale multi-agent coordination, the primary challenge stems from the stark mismatch between coarse global priors and dynamic local hazards. On one hand, global navigation tools typically provide only coarse point-to-point waypoints without awareness of dynamic obstacles; on the other hand, outdoor environments contain active patrolling sentinels that detect and capture agents. Operating purely with local first-person sensory inputs, decentralized agents struggle to converge on mutually convenient gathering places through language, while risking catastrophic capture en route. Furthermore, end-to-end multi-agent reinforcement learning (such as MAT) fails to converge over such massive joint state-action spaces.

To bridge the gap between static indoor tasks and city-scale outdoor navigation, this paper investigates Cooperative Spatial Intelligence. Core idea: develop an embodied cooperative spatial reasoning and planning framework (CoSaR) that integrates multimodal perception, natural language communication, and classical graph navigation through a structured spatial memory, leveraging vision-language models (VLMs) on schematic spatial maps for zero-shot route refinement and safe rendezvous.

Method

Overall Architecture

CoSaR formulates the decentralized multi-agent gathering task under dynamic spatial threats as a Decentralized Partially Observable Markov Decision Process with Communication (DEC-POMDP-COM). The architecture is structured around five coordinated components: Perception Module, Communication Module, Spatial Memory, Spatial-aware Reasoning Module, and Execution / Navigation Module. At each decision step, an agent ingests egocentric RGB-D observations, map tool responses, and incoming messages from teammates. These signals are parsed into objects, sentinel warning tags, teammate poses, and estimated arrival times (ETAs), which update a unified spatial memory. When the reasoning module detects an impending hazard along the current route, it triggers multimodal route refinement followed by low-level collision-free control.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Environment Input<br/>RGB-D / Teammate Messages / Map Tool Response"] --> B["Multimodal Perception and Semantic Parsing<br/>Grounding DINO + SAM 2 + Text Extraction"]
    B --> C["Structured Spatial Memory Maintenance<br/>Occupancy Grid / Danger Zones / Pose Registry / ETA Map"]
    C --> D["Spatial-aware Reasoning and Decision Making<br/>Query / Communicate / Navigate / Wait Actions"]
    D -->|Path Safe| E["Low-level Local Path Tracking<br/>A* Local Planning and Emergency Avoidance"]
    D -->|Hazard Encountered| F["Route Refinement via Multimodal Grounding<br/>VLM Schematic Detour + Map Tool Snapping"]
    F --> E
    E --> G["Execute Motor Actions and Broadcast Updates"]

Key Designs

1. Structured Spatial Memory Maintenance: Grounding Heterogeneous Knowledge into Unified Geometry
Prior communicative embodied agents often concatenate unconstrained dialogue logs into raw text prompts, causing severe loss of geometric constraints and spatial hallucinations. CoSaR addresses this by establishing a hierarchical, persistent spatial memory that decouples metric geometry from cooperative semantics. At the geometric foundation, the agent constructs a voxel grid to maintain a ground-plane traversability occupancy map and dilates detected sentinel coordinates into circular danger zones. At the cooperative level, the agent maintains a teammate pose registry and an ETA map across candidate gathering places. Whenever incoming messages arrive, an LLM parses the dialogue into structured spatial tuples, directly grounding natural language discussions into spatial geometric priors.

2. Spatial-aware Reasoning and Decision Making: Event-driven Deliberation with Minimal Cognitive Overhead
Decentralized agents must continually balance individual movement with collective coordination. CoSaR employs an LLM as a central reasoning engine prompted with the structured spatial memory summary to arbitrate among four macro-action types: querying the map tool for spatial clarity (Query), broadcasting alignment updates (Communicate), advancing toward the target (Navigate), and pausing to synchronize arrival timing (Wait). To eliminate prohibitive computational latency and prevent motion stuttering caused by step-by-step LLM querying, CoSaR introduces an event-driven activation mechanism: the reasoning engine awakens only when novel messages arrive, a sentinel is newly sighted, a disagreement regarding the meeting place is detected, or a 120-second fallback timer expires.

3. Route Refinement via Multimodal Grounding: Bridging VLM Geometric Hallucination with Graph Feasibility
When an agent's global path intersects a dynamic sentinel danger zone, naive grid-level search frequently stalls in local minima, whereas text-only LLMs cannot reliably predict physically viable waypoints. CoSaR resolves this via a multi-stage route refinement pipeline: the agent queries the map tool for a coarse top-down 2D occupancy image and renders an augmented schematic spatial map depicting its ego-pose, the invalid reference trajectory, and active danger zones. A VLM then inspects this visual representation to infer safe detour waypoints in continuous image space. To eliminate VLM spatial jitter and hallucinated off-road coordinates, the coarse waypoints are passed through QueryRefinedRoute to snap onto valid topological road graph nodes, with a fallback pruning heuristic in case of repeated VLM retries.

A Worked Example

Consider three agents (Alice, Bob, and Charlie) randomly initialized across an 800m × 800m urban scene. Alice queries the map tool for distances to landmarks and broadcasts via natural language: "My ETA to the Plaza is 12 minutes; I propose we gather there." Bob parses the message, updates his ETA map, confirms an ETA of 10 minutes, and broadcasts his agreement. Meanwhile, Charlie encounters a patrolling sentinel near a street intersection, flags the coordinates with Grounding DINO, and broadcasts a danger alert. Alice's planned route intersects this newly designated danger zone. CoSaR automatically triggers route refinement: generating a schematic top-down map with the sentinel danger zone highlighted in red and prompting the VLM to propose detour waypoints. The map tool snaps the detour to adjacent navigable avenues, allowing all three agents to rendezvous safely without sentinel detection.

Key Experimental Results

Main Results

The benchmark is evaluated on Virtual Community across 24 realistic 800m × 800m city scenes (each containing 50–150 candidate places, up to 15 agents, and up to 20 sentinels). The table below summarizes performance under the challenging 5-agent, 10-sentinel configuration evaluated over 14 test scenes across 6 runs within a 1500-step horizon.

Method Sentinel Pattern Success Rate (%) ↑ Caught Rate (%) ↓ Detect. Rate (%) ↓ Time Cost (Steps) ↓ Distance Traveled (m) ↓
Oracle Centered 10 Stationary 7.14 51.43 1.80 1429.86 1800.07
Oracle Centered w/ DZ 10 Stationary 11.90 39.29 1.86 1401.40 1598.50
MCTS Planner 10 Stationary 10.71 47.38 2.21 1371.26 2206.20
RoCo [29] 10 Stationary 26.19 26.90 1.13 1255.81 1333.27
CoELA [65] 10 Stationary 16.67 25.24 1.08 1340.77 1522.53
MAT [58] 10 Stationary 0.00 35.71 1.13 1500.00 1453.82
CoSaR (Ours) 10 Stationary 32.14 21.43 1.27 1194.45 1552.45
Oracle Centered 10 Patrolling 14.29 28.57 1.50 1367.07 1401.93
Oracle Centered w/ DZ 10 Patrolling 29.76 22.14 1.66 1228.89 1381.91
MCTS Planner 10 Patrolling 15.48 40.48 2.02 1338.67 2226.39
RoCo [29] 10 Patrolling 29.76 28.57 1.73 1306.00 1480.76
CoELA [65] 10 Patrolling 26.19 17.62 0.94 1304.94 1436.53
MAT [58] 10 Patrolling 7.14 20.24 0.78 1447.17 1269.10
CoSaR (Ours) 10 Patrolling 32.14 27.62 2.22 1246.86 1577.99

Under the Oracle Perception setting, CoSaR achieves a 53.57% success rate and a 9.05% caught rate against 10 stationary sentinels, markedly outperforming RoCo (46.43%) and CoELA (34.52%).

Ablation Study

The contribution of each individual module is quantified below in the 5-agent, 10-stationary sentinel setting:

Config Success Rate (%) ↑ Caught Rate (%) ↓ Detect. Rate (%) ↓ Time Cost (Steps) ↓ Distance Traveled (m) ↓ Note
CoSaR Full Model 32.14 21.43 1.27 1194.45 1552.45 Complete coordination and replanning system
w/o Route Refinement 26.19 28.80 1.18 1243.31 1446.25 Refinement retry set to 0; cannot detour around sentinels
w/o Message Analyzer 19.26 31.38 1.29 1310.20 1488.76 Incoming messages ignored; ETA and pose registry empty
w/o Spatial Memory 21.42 31.19 1.27 1283.70 1489.28 Reasoning module operates without access to spatial memory
w/o Emergency Avoidance 24.05 31.95 1.15 1268.58 1513.40 Bolder navigation; no local reflex upon sudden confrontation

Key Findings

  • Structured memory is vital for communication efficacy: Disabling the message analyzer causes the largest performance drop, decreasing the success rate by 12.88 percentage points (from 32.14% to 19.26%) while increasing the caught rate from 21.43% to 31.38%. Without structured grounding, natural language dialogue fails to produce coordinated physical maneuvers.
  • Unpacking the efficiency-safety tradeoff: While CoELA achieves an apparently low caught rate under patrolling sentinels (17.62%), this behavior stems from conservative indecision where agents remain trapped inside initial indoor locations, resulting in timeout failures. In contrast, CoSaR maintains rapid progression (1194.45 steps) while achieving the highest mission success.
  • Foundation model backbone transferability: When replacing GPT-4o with the open-source Qwen3-VL-30B-A3B-Instruct model (evaluated under 5 agents and 5 stationary sentinels), CoSaR retains strong performance with a 60.71% success rate and a 7.69% caught rate (compared to 67.85% and 5.00% for GPT-4o), proving robustness across distinct backbones.

Highlights & Insights

  • Neuro-symbolic synergy for visual detour planning: Synthesizing global topological waypoints through VLM visual reasoning combined with deterministic road-graph snapping (QueryRefinedRoute) effectively sidesteps the failure modes of raw continuous coordinate regression.
  • Event-driven cognitive scheduling: Activating heavy LLM reasoning only upon message receipt, new threat discovery, negotiation conflict, or periodic timeouts minimizes inference latency and provides real-time responsiveness for low-level navigation.

Limitations & Future Work

  • Lack of 3D spatial occlusion modeling: Current 2D bird's-eye schematic maps collapse vertical urban topology, potentially missing multi-level pedestrian overpasses or vertical line-of-sight occlusions.
  • Static assumption of sentinel motion: Danger zones are modeled as static dilated spheres around instantaneous sentinel poses rather than employing predictive trajectory tracking (e.g., Kalman filtering).
  • Idealized communication assumptions: The current benchmark assumes zero packet drop and instantaneous broadcast communication, which can be extended to range-limited ad-hoc networks with transmission latency.
  • vs CoELA [65]: CoELA pioneered LLM-based modular embodied multi-agent systems but is restricted to small-scale indoor rearrangement; CoSaR extends cooperation to 800m × 800m outdoor city domains by incorporating geometric spatial memory and VLM route refinement.
  • vs RoCo [29]: RoCo relies on a pre-task planning phase and cannot adapt to evolving physical hazards; CoSaR supports continuous event-driven replanning throughout navigation.
  • vs MAT [58]: Centralized multi-agent RL struggles with sample complexity in vast continuous urban environments (0.00% success rate); CoSaR highlights the superior sample efficiency of grounded foundation model reasoning.

Rating

  • Novelty: ⭐⭐⭐⭐☆ (Formalizes Cooperative Spatial Intelligence in city-scale environments with visual-linguistic memory integration)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Thorough evaluation across 14 diverse urban maps, multi-agent configurations, ablations, and alternative VLM backbones)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous DEC-POMDP-COM formulation, explicit diagrams, and clean structural progression)
  • Value: ⭐⭐⭐⭐☆ (Establishes a foundational benchmark and reference architecture for urban-scale multi-robot teaming and autonomous fleets)