Skip to content

E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://k0uya.github.io/e3vs-proj/
Area: Robotics & Embodied AI
Keywords: active perception, 3D visual search, 3D Gaussian Splatting, embodied question answering, viewpoint planning

TL;DR

E3VS-Bench turns active perception into a quantifiable task: over 99 3D Gaussian Splatting scenes it builds 2,014 question-driven episodes that can only be answered through 5-DoF viewpoint control, enforces that they cannot be solved from prior knowledge or the initial view (unanswerable viewpoints become starts, answerable ones become goals, plus blind and start-view filtering), and evaluates agents with a VLM judge score (1–5), average steps, and collision rate — the best model reaches only 2.79 versus 3.53 for humans.

Background & Motivation

Visual search has been studied for decades in psychology and neuroscience, where the question is how humans locate a target among distractors through goal-directed and saliency-driven attention. The vision community has recently revived it as "2D visual search": V*, SEAL, and DyFo let models crop and zoom inside an image-space loop, while Mini-O3, H*Bench, and O3-Bench benchmark this exploratory reasoning. All of them, however, stay on static 2D imagery — the model's action is to cut a patch out of the same picture, and the 3D structure, occlusions, and depth ambiguity behind that picture never enter the evaluation. On the embodied side, EQA (EQA, MP3D-EQA, OpenEQA) does put an agent inside a 3D scene to navigate, but its action space is essentially that of a 2D camera on a ground plane — planar motion plus limited rotation, testing where to go rather than how to aim the camera. What is far more common in the real world is a different difficulty: the target is right in front of the agent from step one, yet its identifying attributes (the text on a label, a brand logo, whether a lid is open or closed) simply cannot be read from the current angle and demand a step closer, a lateral shift, a raise, or a tilt. This kind of viewpoint-dependent evidence has hardly ever been evaluated head-on.

Two obstacles explain why. The first is representational: mainstream simulators are built on mesh or point-cloud reconstructions (ScanNet, House3D, Matterport3D) where texture degradation and geometric oversmoothing blur out small text or logos, so "evidence visible only from a certain angle" is physically undefined and questions degrade into coarse spatial ones such as "is there an X in the room" or "where is X". The second is the evaluation protocol: scene-centric 3D-QA (ScanQA, SQA3D) assumes direct access to the complete scene representation and never asks how the agent acquires missing evidence, while EQA questions can often be guessed from common-sense priors ("fridges usually contain milk") or from the initial view — so what gets measured is linguistic priors rather than exploration ability.

What makes this feasible now is 3D Gaussian Splatting, whose photorealistic free-viewpoint rendering preserves the fine-grained attributes that meshes lose. In the paper's side-by-side comparison, the white lettering on the red band of a protein jar is an unreadable blob on the ScanNet++ mesh, whereas the SceneSplat++ 3DGS scene renders "WHEY" crisply — and SceneSplat++ happens to supply 3DGS versions of ScanNet++ scenes. With that fidelity in hand, the authors can tie 5-DoF viewpoint control, explicit answerable/unanswerable viewpoint annotation, and a mandatory "the start view must fail" filter into a single design. Core idea: build the benchmark on the photorealistic free-viewpoint rendering of 3DGS so that viewpoint-dependent evidence becomes a first-class object, then define each episode with an unanswerable viewpoint as its start and an answerable one as its goal, and enforce blind-answer plus start-view filtering so that the passing bar is fixed at "you must actually move the camera to score."

Method

Overall Architecture

The task definition is clean: an episode is a triplet \((S, q, v_0)\) where \(S\) is a 3DGS-reconstructed scene, \(q\) a natural-language question, and \(v_0\) the initial camera viewpoint. The agent's state is the viewpoint, \(v_t=(x_t,y_t,z_t,\theta_t,\phi_t)\) — three Cartesian coordinates plus yaw and pitch, five degrees of freedom and no roll; the action space is discrete (translation, rotation, termination), the transition follows the environment dynamics, and an action that causes a collision leaves the viewpoint unchanged. At each step \(t\) the agent renders an egocentric RGB observation \(O_t\) from its current viewpoint, feeds it to a VLM together with the question, and the VLM selects an action; the loop continues until stop, after which the model answers the question from its final observation \(O_T\) in open vocabulary.

Evaluation avoids exact string matching and instead uses VLM-as-a-judge: the judge model (GPT-5.1) receives the agent's final observation \(O_T\), the question \(q\), the predicted answer \(\hat y\), the ground-truth answer \(y\), and the human-annotated goal image \(O_{goal}\), and returns a score from 1 to 5, where 5 is perfectly correct and 1 entirely incorrect. The presence of \(O_{goal}\) in that formula matters: scoring looks not only at whether the answer is right, but also at whether the frame the agent stopped on actually contains the supporting evidence.

The benchmark itself is constructed by a five-stage pipeline — 3D scene curation → multi-view question generation → human filtering and viewpoint annotation → answerability filtering → data cleaning — and is then split at the scene level into train/validation/test. In scale, 105 high-quality reconstructed scenes are hand-picked from SceneSplat++, the VLM stage produces 27,877 candidate QAs over 7,578 object instances, and after human review and filtering this collapses to 1,290 unique question-answer pairs across 99 scenes and 2,014 episodes: 1,406 episodes (900 pairs / 68 scenes) for training, 231 (132 pairs / 10 scenes) for validation, and 377 (258 pairs / 21 scenes) for test.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["3DGS scene curation<br/>105 reconstructed scenes"] --> Q
    subgraph Q["Multi-view QA generation & viewpoint annotation"]
        direction TB
        Q1["Spherical multi-view authoring<br/>VLM drafts 27,877 candidates"] --> Q2["Human filtering & answerable viewpoints<br/>about 1,120 hours"]
    end
    Q --> F["Answerability filtering<br/>blind or start-view correct → drop"]
    F --> B["E3VS-Bench<br/>99 scenes / 2,014 episodes"]
    B --> E["Closed-loop VLM evaluation<br/>judge + steps + collisions"]

Key Designs

1. 3DGS scene curation: making fine-grained viewpoint-dependent evidence actually exist

This design targets rendering fidelity. On mesh or point-cloud reconstructions, texture degradation and oversmoothing erase small text, material textures, and brand logos, so "evidence you can only read from another angle" cannot even be authored. The authors hand-pick high-quality indoor reconstructions from SceneSplat++ (a 3DGS version of ScanNet++), and 99 scenes make it into the benchmark. Choosing 3DGS rather than the usual mesh buys photorealistic free-viewpoint rendering: in the paper's comparison, the red band of a black protein jar is an unreadable blur on the mesh version (the model can only answer "the image is too blurry to make out any text"), while the 3DGS version yields "WHEY". Because the detail survives, the question can be upgraded from "is the jar there" to "what does it say on the red band" — and the answer to the latter is visible only from the few viewpoints aimed at that band, which satisfies "unanswerable from a single view" by construction. A secondary benefit is that 3DGS renders from arbitrary poses in the reconstructed space, supporting the rendering side of continuous 5-DoF viewpoint control, whereas mesh simulators typically confine the agent to planar motion with limited rotation in walkable areas.

2. Multi-view QA generation and viewpoint annotation: from spherical sampling to a set of answerable viewpoints

Authoring questions has to settle both what to ask and how to guarantee the answer depends on the viewpoint. The authors design a three-stage VLM-driven procedure. First, object selection: they prefer categories with very few instances per scene, cutting down the ambiguity (especially for counting questions) that same-class distractors would introduce; for each chosen instance they compute a viewing distance so that the projected bounding box spans roughly three-fifths of the image along either axis, then uniformly sample viewpoints on a sphere centered on the object and render the corresponding multi-view images. The three-fifths constraint makes the evidence large enough to reason about, while spherical sampling guarantees the model sees several sides of the object. Second, view filtering: Gemini 2.5 Flash discards images in which the object is heavily occluded, falls outside the reconstruction bounds, or is itself visually ambiguous. Third, authoring: from the surviving valid multi-view images the VLM synthesizes candidate question-answer pairs covering both intrinsic attributes (color, material, branding) and extrinsic spatial relations. This step yields 27,877 candidate QAs over 7,578 instances.

Then comes the human stage, which is the most expensive part of the benchmark. Annotators check every candidate against predefined criteria, removing ambiguous, multi-interpretable, or objectively unanswerable items and correcting inaccurate answers; the whole annotation process takes roughly 1,120 hours. Expert annotators then label the answerable viewpoints for each retained question — the viewpoints that contain enough information to answer it — discarding physically invalid ones such as viewpoints intersecting 3D geometry, with a second annotator verifying the result for quality and consistency. This single annotation pass is reused three ways: unanswerable viewpoints become episode starts, answerable viewpoints become goals, and the goals render the judge's reference image \(O_{goal}\). The authors also report the category distribution (by unique QA pair): object search about 397, counting about 333, object state about 159, context-guided search about 155, object attribute about 149, and spatial reasoning about 85; scenes are dominated by apartments, bedrooms, and hotel rooms, with the tail covering offices, kitchens, bathrooms, storage rooms, classrooms, conference rooms, gyms, and hallways. (⚠️ Category and scene counts are read off Fig. 4(a)(b); their total of 1,278 differs by a small margin from the 1,290 unique QA pairs stated in the text — refer to the original paper.)

3. Answerability filtering: dropping every question that needs no movement

Even when authoring aims at viewpoint dependence, a batch of generated candidates still contains many shortcut questions: some are guessable from linguistic priors (a binary "is the box open or closed" has a coin-flip chance), and some happen to be legible from the initial view. Keeping them would measure common sense and dataset bias rather than exploration. The remedy is blunt: GPT-5.1 performs VQA under two conditions — blind (no image at all, question only) and start-view (only the render from \(v_0\), with no movement allowed) — both scored with the same 1–5 VLM-judge protocol, and any QA pair scoring ≥3 under either condition is removed entirely. The threshold is the midpoint of the 1–5 scale, a deliberately conservative choice: any indication that the item is answerable without active exploration gets it killed. What survives satisfies a hard condition — not answerable from prior knowledge, not answerable from the initial observation — and therefore requires 5-DoF viewpoint transitions to accumulate information. This step also plants a hook for later analysis: filtering was run with GPT-5.1 only, so other models (Qwen, Gemini) still retain a bias bonus at the start view, which is exactly what explains the anomaly observed on the OST category.

4. Closed-loop VLM evaluation: separating scoring, steps, and collisions

Evaluation plugs a VLM into a closed perception–action loop: each step the current observation \(O_t\) and the question go to the model, it emits an action, the action is executed, and the loop continues until stop, after which the final answer is generated. The interface is a structured prompt with two parts. The system prompt fixes the world coordinate system (Z-axis up) and the action-space constraints (fixed 0.25 m translation, fixed 30° rotation). The user prompt supplies the task input and dynamic state: the question, the current observation image, the step count, the current 3D coordinates, the previous action, and feedback such as collision detection. By default only a single-frame observation is given, with no extra reasoning module, to establish a clean baseline. The metrics are deliberately split into three orthogonal axes: answer correctness via the 1–5 VLM-judge score (Spearman correlation with human evaluation \(\rho=0.54\)), exploration efficiency via average steps, and navigation safety via collision rate (1 if any collision occurs within an episode, 0 otherwise). This separation distinguishes "answered right but crashed around" from "navigated safely but answered wrong". The baselines form a progressive spectrum of perceptual access, from no vision to privileged viewpoints: blind VLM → VQA at start → VQA at birdview → VQA at goal → 2D visual search at start (SEAL, DyFo) → the embodied E3VS agent, letting a reader locate whether a given model fails in perception, exploration, or decision making. Implementation-wise, observations are 512×512 at a 90° field of view, each episode is capped at 25 steps, the model may emit at most 128 output tokens (reasoning disabled) or 256 or more (reasoning enabled), and if no valid action is produced the agent defaults to move_forward.

A Worked Example

Take "What text is visible on the front of the plastic sack?" (answer: "cat litter") through the pipeline. At authoring time the VLM sees "cat litter" printed on the front of the sack in the sampled multi-view images and synthesizes the question; since the text is readable only when the sack faces the camera, it is invisible from most viewpoints and is viewpoint-dependent by nature. During human review an annotator confirms the answer is unambiguous and that the text really is on the front, then labels the answerable viewpoints — a small cluster of poses facing the front — and discards poses intersecting geometry. During filtering, GPT-5.1 cannot answer with no image (it cannot know what is printed) nor from the start view alone (the start is drawn from the unanswerable set), scoring below 3 both times, so the QA enters the benchmark and spawns one or more episodes (the same QA can be paired with different unanswerable starts). At evaluation time the agent begins, say, with the sack edge-on so the printed face is hidden by the fold; it must translate around the occlusion, possibly tilt down onto the printed surface, and only stop once the text is clearly legible. The judge then compares its final frame, predicted answer, and the annotator's goal frame — and if the agent stops before the text becomes readable (the paper's qualitative results show this early stopping is common), it earns a low score even if its answer happens to be right, because the final observation contains no supporting evidence. (⚠️ Per-instance filtering scores for this example are not given in the paper; the account follows the rules in §3.2, and the viewpoint annotation details are illustrative.)

Key Experimental Results

Main Results

Abbreviations for the six question categories: OS object search, OST object state, OA object attribute, CGS context-guided search, SR spatial reasoning, CNT counting. All scores are VLM judge scores (1–5, higher is better); steps and collision rate are dimensionless. (⚠️ The model versions used, such as Gemini 3.0 / GPT 5.1 / Qwen3-VL / Step3-VL, are as stated in the original paper.)

Main results for embodied E3VS agents (Table 2):

Model OS OST OA CGS SR CNT Avg. Avg. Steps Collision
Random Action 2.38 2.18 2.60 2.20 2.50 2.00 2.28 10.15 0.39
Gemini 2.5 Pro 3.14 2.36 2.45 2.60 2.25 2.04 2.54 7.80 0.31
Gemini 3.0 Flash 3.21 2.82 3.18 3.53 2.75 1.88 2.79 11.29 0.43
Gemini 3.0 Pro 3.07 2.55 3.18 3.40 3.00 1.96 2.75 10.17 0.34
GPT 5.1 2.90 2.18 2.53 2.73 2.25 1.88 2.42 15.04 0.49
Qwen3-VL-8B 2.86 2.36 2.60 3.13 1.88 1.96 2.46 18.73 0.30
Qwen3-VL-30B 2.67 2.09 2.60 2.47 2.25 1.84 2.32 14.18 0.50
InternVL3.5-8B 2.66 2.18 2.24 2.60 2.12 1.60 2.21 10.85 0.65
Step3-VL-10B 2.38 2.27 2.38 3.13 2.50 1.64 2.24 11.16 0.42
Human 3.12 3.59 4.06 4.06 3.35 3.59 3.53 11.21 –

Static baselines (Table 3, no viewpoint movement allowed):

Setting Model OS OST OA CGS SR CNT Avg.
Blind VLM Gemini 2.5 Flash 1.72 2.00 2.09 1.67 1.88 1.72 1.82
Blind VLM Qwen3-VL-8B 1.38 2.64 1.58 2.20 1.50 1.52 1.67
2D visual search at start SEAL 1.62 2.45 1.44 1.53 2.50 1.32 1.68
2D visual search at start DyFo 1.90 2.45 1.44 1.67 2.00 1.80 1.86
VQA at start Gemini 2.5 Flash 2.21 2.64 1.73 2.33 2.12 2.00 2.14
VQA at start Qwen3-VL-8B 2.10 2.82 1.87 2.20 2.12 1.56 2.02
VQA at birdview Gemini 2.5 Flash 1.31 1.64 1.44 1.67 1.75 1.48 1.48
VQA at birdview GPT 5.1 1.48 2.09 1.58 2.20 1.62 1.16 1.55
VQA at birdview Qwen3-VL-8B 1.55 2.18 2.09 1.80 1.38 1.12 1.59
VQA at goal Gemini 2.5 Flash 3.79 3.27 3.84 4.20 3.50 3.16 3.58
VQA at goal GPT 5.1 4.17 3.36 3.91 4.20 3.00 2.88 3.60
VQA at goal Qwen3-VL-8B 3.07 3.55 3.40 4.20 3.00 2.20 3.03
VQA at goal Human 4.25 3.82 4.47 4.25 4.14 3.82 4.12

Ablation Study

Effect of the internal reasoning (thinking) switch (Table 4, "Th." marks whether the thinking process is enabled):

Model OS OST OA CGS SR CNT Avg. Steps Collision
Gemini 3.0 Flash 3.21 2.82 3.18 3.53 2.75 1.88 2.79 11.29 0.43
Gemini 3.0 Flash + Th. 3.21 2.82 2.96 3.27 3.00 2.00 2.79 19.80 0.41
GPT 5.1 2.90 2.18 2.53 2.73 2.25 1.88 2.42 15.04 0.49
GPT 5.1 + Th. 2.97 2.64 3.25 2.87 2.38 2.16 2.70 13.78 0.38

Goal-initialized ablation: the initial viewpoint is set to the human-annotated answerable viewpoint, removing the need for exploration and isolating evidence recognition plus stopping behaviour (Table 5):

Model Init. at goal OS OST OA CGS SR CNT Avg. Steps Collision
Qwen3-VL-8B ✗ 2.86 2.36 2.60 3.13 1.88 1.96 2.46 18.73 0.30
Qwen3-VL-8B ✓ 3.90 3.36 3.84 3.93 2.75 2.56 3.38 12.06 0.23
Gemini 3.0 Flash ✗ 3.21 2.82 3.18 3.53 2.75 1.88 2.79 11.29 0.43
Gemini 3.0 Flash ✓ 4.21 3.55 3.33 3.93 3.25 2.38 3.41 6.60 0.29

Key Findings

  • Every model is far from human and close to random. The best model, Gemini 3.0 Flash, reaches 2.79 against 3.53 for humans, while the random-action baseline already sits at 2.28; GPT 5.1 (2.42) is on par with Qwen3-VL-8B (2.46) and Step3-VL-10B (2.24). On a 3D task that requires changing viewpoint, 2D image recognition confers almost no advantage.
  • The start-to-goal gap is the most direct evidence that active exploration is required. Same questions, same models, only the viewpoint changes: VQA at start scores 2.02–2.14 while VQA at goal scores 3.03–3.60. The information is in the scene — just not in the initial field of view, which is precisely the value of 5-DoF viewpoint control.
  • A bird's-eye view is the worst of all. Every "VQA at birdview" result falls in 1.48–1.59, below even VQA at start, showing that fine-grained evidence (text, open/closed state, occluded regions) cannot be summarized by a global overview; seeing everything from above is no substitute for walking up and looking.
  • OS is harder than CGS, contrary to intuition. Models generally do better on context-guided search ("where can one store a large collection of binders and documents in an organized manner?") than on explicit object search ("where is the black office chair with five wheels?"). The former can be answered by recognizing a functional zone or a relevant object, whereas the latter also demands understanding the target's spatial position and placement, and the extra spatial reasoning drags performance down.
  • Counting (CNT) is a systemic weak spot. Gemini 3.0 Flash and GPT 5.1 both score 1.88, below the random-action baseline (2.00), while humans reach 3.82 at the goal viewpoint. Counting requires accumulating evidence along a 5-DoF trajectory (by the third instance you must remember the first two), which single-observation EQA cannot do; models already struggle in the 2D single-view setting and degrade further in full E3VS.
  • Spatial reasoning (SR) shows the first sign of genuine viewpoint selection. SR requires comparing relative position, distance, and size, and the quality of such geometric evidence depends heavily on the viewpoint — comparing two object heights is far easier from the side than from an oblique or top-down view. Gemini 3.0 Pro scores 3.00 on E3VS, matching the VQA-at-goal upper bound on that category (both GPT 5.1 and Qwen3-VL-8B score 3.00 for SR), indicating that it does not merely recognize spatial relations but actively chooses viewpoints that make them more observable.
  • Object state (OST) gains are eaten by bias. Binary judgments such as open versus closed are easy to guess from priors or dataset bias, and because filtering only ran with GPT-5.1, other models keep a bias bonus at the start view — Qwen3-VL-8B already scores 2.82 with VQA at start, close to Gemini 3.0 Flash's 2.82 on full E3VS. Combined with the fact that the viewpoint needed to resolve a state is fairly predictable (face the door to see whether it is open), OST reduces to viewpoint selection plus path planning rather than exploration, and models still fail at even that.
  • Object attribute (OA) is the most viewpoint-sensitive category. The target object is named in the question, but which side of it carries the evidence is unknown a priori, so each instance must be explored. Apart from the two Gemini 3.0 variants (3.18), models hover around the random-action baseline (2.60), showing that acquiring fine-grained evidence requires both exploratory viewpoint discovery and detailed visual recognition.
  • The benefit of thinking is model-dependent. With thinking enabled, GPT 5.1 improves from 2.42 to 2.70 while also cutting steps from 15.04 to 13.78 and collisions from 0.49 to 0.38, so extra internal computation does help viewpoint planning. Gemini 3.0 Flash does not move at all on the average (2.79), but its step count jumps from 11.29 to 19.80. The authors admit the cause is unclear and may relate to architecture or training data.
  • Multi-frame memory improves efficiency, not accuracy. Raising the number of input frames from 1 to 3 and 5 barely moves the judge score but consistently reduces steps and collisions. The step reduction is likely because single-frame agents fall into repeated-action deadlocks; the collision reduction comes from better action–observation understanding, letting the agent anticipate viewpoint changes.
  • The two models' strengths are complementary. In the goal-initialized ablation, Gemini 3.0 Flash averages 3.41 and finishes in only 6.60 steps while Qwen3-VL-8B averages 3.38 but takes 12.06 steps; on OA, however, Qwen3-VL-8B scores higher (3.84 versus 3.33). Back in the standard E3VS setting with unanswerable starts, Gemini overtakes across the board. Together this reads as: Gemini is better at actively exploring to acquire missing information, whereas Qwen3-VL-8B is better at recognizing fine-grained attributes once the evidence is present but does not reliably convert that recognition into correct stopping behaviour.
  • Pure image-space search cannot fix a 3D viewpoint problem. SEAL (1.68) and DyFo (1.86) are essentially indistinguishable from the blind baseline (1.67–1.82) and fall below VQA at start — repeatedly cropping and zooming a 2D image is no substitute for moving the camera to a new pose.
  • Qualitatively, the "answerable" viewpoints agents reach often are not answerable. Human-chosen endpoints typically center the target or deliberately expose a discriminative feature, whereas agent endpoints frequently leave the target partially out of frame with the key feature still unshown. Since the judge penalizes answers unsupported by the final observation, such stops score low.

Highlights & Insights

  • Turning abstract answerability into an executable episode-generation rule. One expert annotation pass — "which viewpoints contain enough information to answer this question" — yields three artifacts at once: the unanswerable set supplies starts, the answerable set supplies goals, and the goals render the judge's reference frame. Annotation cost is reused to the maximum, and "you must move" becomes a property guaranteed by the data rather than by the question author's diligence.
  • Double filtering with a conservative threshold is what makes the benchmark valid. Blind answers block "guessing right from linguistic priors", start-view VQA blocks "the initial view is enough", and taking the midpoint of the 1–5 scale as the cut (drop at ≥3) prefers killing good items to keeping shortcuts. More interesting is the hole the authors themselves point out — filtering used GPT-5.1 only, so other models still enjoy a bias bonus at the start — which becomes the key to explaining the OST anomaly. Making an evaluation flaw into analysis material is refreshingly honest.
  • The goal-initialized ablation decouples exploration from recognition. Same questions, same answers, only the start moves to a human-answerable viewpoint; that isolates "cannot find the evidence" from "found it but cannot read it / does not stop". This cheap diagnostic is what surfaces the "Qwen3-VL-8B can look but cannot search, and cannot stop" profile.
  • Three orthogonal metrics. Correctness (1–5 judge), efficiency (average steps), and safety (collision rate) do not mask each other, so one can read off cases like "answered right while crashing around" (GPT 5.1, 0.49 collision rate) versus "moved little but answered wrong" (Qwen3-VL-30B, 14.18 steps at 2.32). Transferable to any setting that evaluates an embodied agent's judgement.
  • 5-DoF instead of 6-DoF is a deliberate trade. Removing roll keeps the action space discrete and enumerable, matching how human eyes and real pan-tilt units move while avoiding the evaluation noise of continuous control; the fixed 0.25 m translation and 30° rotation also make "steps" a fair cross-model metric.
  • Transferable recipe. Any task whose supervision is only visible from certain angles can reuse the "annotate answerable viewpoints + filter out answerable starts" skeleton — reading code and labels in industrial inspection, judging affordances and grasp poses in robot manipulation, recognizing instrument state in surgical video, or guided AR repair. First label which poses show the evidence, then set the unanswerable ones as starts.
  • The "aha" moment: the qualitative results repeatedly show that failure is usually not "cannot understand" but "stopped too early" — the agent reaches the target's general area and answers without noticing the discriminative evidence is still out of frame. The bottleneck in this direction may therefore lie not in perception models but in the ability to assess whether I have actually seen it.

Limitations & Future Work

  • The limitation the authors admit is about the judge: the correlation between VLM-based judging and human evaluation is relatively weak (\(\rho=0.54\)), and in some cases the agent's final observation differs substantially from the human-annotated goal viewpoint yet receives a similar score, so automated evaluation captures viewpoint adequacy insufficiently. The coarse 1–5 granularity contributes to this.
  • Scale and coverage are limited: 99 scenes and 2,014 episodes, all indoor and static (ScanNet++), dominated by apartments, bedrooms, and hotel rooms. There are no outdoor scenes, no dynamic objects, no interactable objects, and cross-scene generalization cannot be measured separately.
  • Interaction is excluded from the action space entirely. With only translation, rotation, and stop, the agent cannot open a door, move an occluder aside, or pick anything up, so occlusion can only be circumvented by finding another angle. That is the boundary of the task definition, but it is also the gap to real robot manipulation, where much occlusion must be removed physically.
  • Each episode is capped at 25 steps with single-frame observation (multi-frame appears only as an ablation), so long-horizon, multi-target search is not covered — and the memory ablation already shows single-frame agents fall into action deadlocks, meaning part of the 25-step budget is spent on ineffective loops.
  • Cost and bias in construction: roughly 1,120 hours of human annotation does not scale, candidate QAs are all synthesized by a single model (Gemini 2.5 Flash), which imports that model's authoring preferences, and both filtering and judging are tied to GPT-5.1, so version drift in a closed-source model makes results hard to reproduce exactly (the authors already exclude GPT-5.1 from the blind and start-view settings to avoid self-evaluation bias).
  • This work only does zero-shot evaluation; the train and validation splits are released but unused for optimization. The authors leave learning-based agents to future work, so whether training could close the exploration gap remains an open question.
  • Concrete improvements: replace the judge with an open, reproducible model or use multi-judge voting to mitigate the weak correlation; upgrade viewpoint annotation from a binary set to a continuous answerability distribution (the authors' own earlier Answerability Fields points this way) so evaluation no longer hinges on a single final frame; add simple object interaction to the action space so tasks that require moving an occluder are included; and exploit the editability of 3DGS scenes (adding or removing objects, changing lighting) for cheap data augmentation and scene diversity.
  • vs EQA / MP3D-EQA / OpenEQA: they let an agent navigate a 3D environment and answer questions, but the agent is essentially a 2D camera moving on a plane (2-DoF, graph-based or continuous planar motion), so the question is where to go; E3VS raises the freedom to 5-DoF and asks how to aim the camera so the evidence becomes visible. The two are therefore not a matter of scale but of action space and evidence type — at the cost of far fewer scenes than the MP3D-based benchmarks.
  • vs ScanQA / SQA3D: these are scene-centric 3D-QA that take a complete point-cloud or mesh representation as input, implicitly granting the agent access to all scene information, and thus never evaluate how missing visual evidence is acquired. E3VS is less a harder version of them than an explicit model of the evidence-gathering step itself.
  • vs 2D visual search (V* / SEAL / DyFo / Thyme / Mini-O3 / H*Bench / O3-Bench): their actions are image-space crops, zooms, or panoramic pseudo-actions such as turn_left / turn_right, and the pose never changes. This paper makes them static baselines and measures SEAL at 1.68 and DyFo at 1.86, essentially level with blind answering — those numbers are a direct answer to whether 2D search transfers to 3D.
  • vs Aerial VLN / CityNav: they too escape the plane and enter 3D space, but the objective is long-range trajectory following and distant waypoint navigation; what is evaluated is path planning and instruction grounding, not the fine viewpoint adjustment needed to read one label.
  • vs Answerability Fields: that work uses a diffusion model to estimate the spatial answerability distribution of a question — where the evidence needed to answer it can be observed. E3VS pushes the same idea one step further by fixing answerability as an explicitly annotated set of viewpoints and then using it as episode starts and goals. Combining the two (automatic estimation plus annotation verification) is a realistic path to scaling such benchmarks.
  • vs simulation platforms such as Habitat / EXPRESS-BENCH: they provide physics simulation and navigation interfaces while E3VS provides task and data. The 3DGS rendering route and Habitat's mesh route trade off differently — object interaction needs meshes or a physics engine, photorealistic fidelity needs static scenes — and E3VS chooses the latter.

Rating

  • Novelty: ⭐⭐⭐⭐ First to turn viewpoint-dependent evidence and 5-DoF active perception into a comparable, reproducible benchmark, with a clever task definition and filtering protocol; overall, though, it is a compositional innovation of "EQA + stronger rendering + stricter filtering".
  • Experimental Thoroughness: ⭐⭐⭐⭐ Broad model coverage (four proprietary plus four open-source), a well-designed baseline spectrum, and three ablations (thinking, memory, goal-initialized) that each hit a real question; the weaknesses are only 99 scenes, all indoor and static, and a judge whose correlation with humans is just 0.54.
  • Writing Quality: ⭐⭐⭐⭐ The task formalization is clean and the analysis is honest (it flags the filtering blind spot and admits the unexplained thinking results); points off for some figures being overloaded and for pushing much construction detail into the supplementary.
  • Value: ⭐⭐⭐⭐ It converts "active perception" from a slogan into a quantifiable evaluation protocol, and both the unanswerable-start filtering and the answerable-viewpoint annotation are directly reusable; it also lays out clearly where current VLMs fall short in 3D viewpoint planning.