Agentic Collaborative Cognition for Zero-Shot 3D Understanding¶

Conference: ECCV 2026
arXiv: 2606.24649
Code: Unreleased; Project Page: https://zhangbo135.github.io/agentic-collaborative-cognition/
Area: 3D Vision
Keywords: Zero-Shot 3D Understanding, Multi-Agent Collaboration, View Planning, 3D Visual Grounding, Scene Cognitive Map

TL;DR¶

This paper reformulates zero-shot 3D scene understanding as an iterative collaboration between a Planning Agent and a Perception Agent centered around an explicit holistic cognitive map. By actively planning key views, recording object attributes across viewpoints, and filtering candidates through feedback, the proposed method significantly outperforms existing zero-shot approaches on 6 benchmarks, including ScanRefer, Nr3D, SQA3D, and ScanQA.

Background & Motivation¶

Background: 3D scene understanding requires handling object localization, spatial relations, question answering, dialogue, and task decomposition simultaneously. Traditional 3D grounding/QA methods mostly rely on training labeled 3D-text data, leading to high adaptation costs for novel scenes and tasks. Recent zero-shot paradigms feed keyframes from continuous 3D scanning videos to MLLMs, enabling general vision-language models to indirectly understand 3D scenes from 2D perspectives.

Limitations of Prior Work: Video keyframe paradigms suffer from two key bottlenecks. First, the camera trajectory of a scanning video is fixed and often lacks decisive perspectives required to answer the current query. Second, MLLMs can only implicitly reason about spatial relationships from a set of 2D images, making it difficult to maintain a consistent 3D scene memory under large viewpoint changes, distractors of the same category, or fine-grained attribute constraints.

Key Challenge: What zero-shot 3D understanding truly lacks is not more keyframe sampling, but an explicit 3D cognitive state shared, updated, and verified by agents. Without such a state, the model approaches each round as if observing the scene for the first time, leading to redundant observations of irrelevant objects and difficulty in consistently binding attributes such as "the blue chair," "near the right side of the desk," and "Candidate 7 observed in the previous round."

Goal: To build a general framework that requires no task-specific annotation training, enabling MLLMs to perform 3D visual grounding, 3D QA, situation estimation, 3D-assisted dialogue, and task decomposition under zero-shot settings, and to actively acquire missing viewpoints rather than passively relying on pre-existing video frames.

Key Insight: The problem is decoupled into two complementary roles: a Planning Agent responsible for "where to look" and a Perception Agent responsible for "what is seen and whether it is sufficient." Instead of open-ended long dialogues, the two agents read and write states on a structured holistic cognitive map, forming a closed loop.

Core Idea: An explicit cognitive map serves as a shared workspace for multiple agents: a 3D detector first provides an object list and BEV representation, then the Planning Agent filters candidates and plans key views based on the query and feedback, while the Perception Agent extracts attributes, filters candidates, and determines feasibility from real or rendered views until sufficient evidence is gathered.

Method¶

Overall Architecture¶

The input consists of a 3D scanned scene, a camera trajectory, and a natural language query; the output can be target object localization, QA answers, dialogue responses, or task decomposition results. The framework first constructs a holistic cognitive map \(M={T,I_B}\), where \(T={σ_i}_{i=1}^N\) is an object attribute table, and each object \(σ_i={b_i,l_i,a_i}\) stores a 3D box, category, and progressively completed attributes; \(I_B\) is a BEV map marked with instance IDs. Subsequently, the Planning Agent and Perception Agent run alternately in rounds: the Planning Agent filters candidates from the map and plans views, the system maps the views to real images or renders new images, and the Perception Agent assigns IDs to candidate objects, extracts attributes, verifies spatial relations, and writes feedback back for the next round.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["3D Scene + Query"] --> B["Initialize Holistic Cognitive Map<br/>Object Table + BEV Instance ID"]
    B --> C["Planning Agent<br/>Candidate Filtering + View Planning"]
    C --> D["View Mapping<br/>Real Image Priority / Render If Missing"]
    D --> E["Perception Agent<br/>Attribute Extraction + Spatial Verification"]
    E --> F{"Is Evidence Sufficient?"}
    F -->|No| G["Feedback Mismatched Candidates<br/>and Next Round's Observation Needs"]
    G --> C
    F -->|Yes| H["Output Grounding / QA / Dialogue / Task Decomposition"]

Key Designs¶

1. Holistic Cognitive Map: Turning Implicit Multi-View Understanding into a Readable/Writable Shared State

Previous zero-shot 3D methods directly fed multiple frames to MLLMs, where spatial memory was primarily hidden in the context without stable object-level identities. This work first leverages a 3D detection framework to generate a set of objects, writing the box \(b_i\), category \(l_i\), and attribute slot \(a_i\) into a table, and marking each object in the BEV map with a unique instance ID. The key to this approach is not merely "adding a scene graph", but providing a shared reference system for the two agents: the Planning Agent can instruct to "continue observing the relative positions of Candidates 4 and 7," and the Perception Agent can write newly observed colors, textures, and orientations back into the same object entry.

Attribute slots are initially empty and progressively updated over rounds; if a new view provides attributes with higher confidence, the old attributes are overwritten. This prevents the map from being a static prior, making it a correctable cognitive log instead. This is especially crucial for 3D QA and grounding: the target is rarely a generic category like "chair," but rather "the chair with wheels near the window and to the right of the desk," which requires aggregating constraints across views to be reliable.

2. Planning Agent: Replacing Passive Keyframe Selection with Candidate Filtering and Active View Completion

The Planning Agent first parses key objects \(o_j=(l_j^q,a_j^q)\) from the query, and then filters candidates using the category and known attributes in the cognitive map. The matching rule can be summarized as follows: if scene objects or query attributes are missing, matching is performed by category first; if attributes are known, both category and attributes must match. The resulting initial candidate set \(C^0\) is further narrowed down in subsequent rounds by the feedback of the Perception Agent.

View planning is not random image retrieval, but is structured around two types of information needs: close-up views to examine the attributes of candidate objects themselves, and co-visible views to observe multiple objects simultaneously and judge spatial relations. Each planned view \(V_k^t\) is converted into a 3D camera pose \((R_k^t,T_k^t)\), which is matched against the poses of real video frames. If the distance to the closest real frame is below a threshold \(τ_D\), the real frame is retained to preserve texture and semantic details; otherwise, the view is rendered from the point cloud/reconstruction results, supplementing key observations missing in the video. This real-rendered integration explains why neither image source can be omitted in the ablation study.

3. Perception Agent: Transforming Fragmented Observations into Object Attributes and Feedback Using Consistent Instance IDs

The Perception Agent receives a collection of images \(I^t\) labeled with candidate instance IDs. It does not need to reassign "which object this is," but instead extracts appearance attributes and physical attributes for the candidate IDs and evaluates three conditions: whether the attributes match the query, whether the spatial relationships are correct, and whether the current observation is sufficient to answer the query. Real images primarily provide fine-grained semantic properties like color, texture, and material, while rendered images primarily provide relative positions and spatial layouts; the model is specifically prompted to distinguish the reliability of these two sources.

When a candidate conflicts in terms of attributes or spatial relations, the Perception Agent feeds it back as an incorrect candidate to the Planning Agent; when evidence is insufficient, it specifies which views should be supplemented in the next round. This is the most critical control signal in the closed loop: the Planning Agent no longer blindly expands coverage but targets decisive views based on "what has not yet been resolved," while the Perception Agent goes beyond outputting answers to continuously update the map, filter candidates, and decide when to stop.

A Complete Example¶

Suppose the query is "find the chair with wheels near the right side of the desk." In the initialized map, there might be 8 chairs, and the BEV can only provide coarse positions, with the attribute slots remaining empty. In the first round, the Planning Agent narrows down the candidates to 3 based on the category and proximity to the desk, and plans a view that can observe the desk and the three chairs simultaneously; if the real scanning video does not cover the right side of the desk, a rendered view is generated instead. Upon observing Candidates 2, 5, and 7, the Perception Agent finds that Candidate 2 is on the left side of the desk, Candidate 5 has no wheels, and Candidate 7 matches both color and location but its wheels are occluded, leading to the feedback: "Exclude 2 and 5; supplement the bottom-view of Candidate 7." In the second round, the Planning Agent plans a low-angle or close-up real frame, and the Perception Agent confirms the wheel attribute, halts the process, and returns the 3D box of Candidate 7. This workflow illustrates how the map, active views, and feedback-based filtering tightly integrate.

Loss & Training¶

This framework is a zero-shot system and is not trained end-to-end as a 3D foundation model. In terms of implementation, Qwen2.5-VL-72B is used as the Planning Agent and GPT-4o as the Perception Agent. The resolution of rendered images is \(512×512\), the view matching threshold \(τ_D=0.8\), each round plans \(K=8\) views, and the maximum number of iterations is 6. Ablation experiments also evaluate the impact of using Qwen2.5-VL-7B, Qwen2-VL-72B, Qwen2.5-VL-72B, and GPT-4o as the MLLM, verifying that the performance gains do not solely stem from using stronger models.

Key Experimental Results¶

Main Results¶

The paper evaluates on 6 tasks/benchmarks: ScanRefer, Nr3D, SQA3D situation estimation, SQA3D / ScanQA 3D QA, and 3D-LLM Held-In dialogue / task decomposition. Several highly representative results are shown below:

Dataset	Metric	Ours	Compared Method	Gain
ScanRefer 3DVG	[email protected] / [email protected]	Higher than CSVG	CSVG	+8.5 / +11.1
Nr3D 3DVG	[email protected]	Higher than CSVG	CSVG	+2.0
SQA3D 3D QA	EM	Higher than SpatialPrompting	SpatialPrompting	+2.1
ScanQA	CIDEr	Higher than SpatialPrompting	SpatialPrompting	+3.4
3D-LLM Held-In dialog/task	BLEU-1 / BLEU-4 / ROUGE	Higher than Agent3D-Zero	Agent3D-Zero	+1.8 / +9.4 / +3.7

Ablation Study¶

The authors conduct ablation studies on Nr3D, SQA3D, and ScanQA, which clearly show that the cognitive map and closed-loop feedback are the primary performance contributors.

Config	Nr3D	SQA3D	ScanQA	Note
full model	56.7	53.2	26.1	Full cognitive map + filtering/feedback + real/rendered views
w/o Holistic Cognitive Map	47.8	49.3	18.6	Loses object-level shared state; Nr3D drops by 8.9
w/o Filtering and Feedback	49.3	50.4	20.9	Candidates cannot continuously shrink, easily dragged down by distractors
w/o Rendered Image	52.7	50.6	21.0	Insufficient fixed video viewpoints, spatial localization is compromised
w/o Real Image	53.9	51.7	21.4	Lacks fine-grained texture/material details, attribute recognition is compromised

Key Findings¶

Efficiency gains are observed: compared to SeqVLM, the token cost decreases from 7.5k to 4.3k, single-round latency drops from 42.3s to 30.2s, and Nr3D accuracy increases from 53.2 to 65.6.
View planning strategies matter more than raw coverage: random view strategies score only 13.8/12.6/6.4 on Nr3D/SQA3D/ScanQA, a fixed Co-Visible Template scores 43.1/42.0/13.6, while the proposed method achieves 56.7/53.2/26.1.
Stronger MLLMs yield better performance, but the framework itself remains effective: under the same settings, Qwen2.5-VL-7B achieves 49.3, Qwen2-VL-72B achieves 54.2, Qwen2.5-VL-72B achieves 56.7, and GPT-4o achieves 61.2.

Highlights & Insights¶

Converting the scene graph / map into a shared agent state: This is far more controllable than simply feeding scene descriptions into the prompt, as there are explicit object IDs, attribute slots, and candidate sets in each round.
Clear division of labor between real and rendered views: Real images handle fine-grained details, while rendered images resolve missing spatial relationships; the ablation study proves this is a critical complementary mechanism in zero-shot 3D, rather than a cosmetic module.
Feedback loops impose task constraints on multiple agents: The Planning Agent is not exploring blindly; its planning is guided by the incorrect candidates and evidence gaps identified by the Perception Agent, preventing the tedious yet ineffective collaboration common in multi-agent systems.
Transfer value to embodied AI: Robot navigation, indoor manipulation, and AR scene understanding similarly require "active observation + shared mapping + attribute updates," making this framework easily transferable to an embodied perception loop.

Limitations & Future Work¶

The framework relies on a pretrained 3D detector to provide initial boxes and categories; if the detector misses or misclassifies objects, the subsequent agents struggle to recover.
The current Planning Agent is still driven by textual reasoning of the MLLM; it can still fail on complex spatial relationships, absolute orientations, and fine-grained attributes. The appendix also highlights planning/perception errors as primary failure modes.
Supplementing views with rendered images offers geometric advantages but suffers from unreliable textures. The reliability boundary between real and rendered views currently relies on prompt constraints; future work could model this uncertainty explicitly.
The latency of the iterative agent system remains high (30.2s per round), making it suitable for offline analysis but unsuitable for real-time robot control. Lightweight MLLMs, caching, and local incremental updates are needed.
The candidate filtering function utilizes relatively rigid rule-based attribute matching, which might prematurely discard correct candidates when facing open-vocabulary attributes, synonymous expressions, or ambiguous references.

vs SeqVLM / SPAZER: These methods mainly select keyframes or perform coarse-to-fine reasoning on existing video frames; the proposed method actively plans missing viewpoints and writes back observations to an object-level map.
vs LLM-Grounder / CSVG: Text-based 3D scene graphs represent structure but lose visual detail; the proposed method preserves BEV, real images, and rendered views, allowing MLLMs to iteratively verify on visual evidence.
vs embodied memory / scene documenting: ConceptGraphs, Clio, etc., act more like long-term semantic maps; the proposed cognitive map functions as a query-driven working memory, focusing on rapidly filtering candidates and collecting evidence for the current task.
Insights: Future work can convert the attribute slots of the cognitive map into probability distributions and allow the Planning Agent to directly optimize for "maximizing information gain / minimizing candidate entropy," transitioning from heuristic prompt-based planning to learnable or provably optimal active perception strategies.

Rating¶

Novelty: ★★★★☆ Reformulating zero-shot 3D understanding into an explicit cognitive map + planning/perception loop is not a single-point algorithmic breakthrough, but the combination is highly tailored for zero-shot 3D understanding.
Experimental Thoroughness: ★★★★★ Broad dataset coverage across 6 benchmarks, with thorough ablations on main results, efficiency, core components, view strategies, and choice of MLLMs.
Writing Quality: ★★★★☆ Clear main narrative, complete system flow and algorithm description; some tables are quite long, requiring readers to extract key takeaways.
Value: ★★★★☆ High reference value for 3D VLMs and embodied perception, though actual deployment remains constrained by detector and MLLM latency.