Towards Spatial Supersensing in the Wild¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: spatial supersensing, long-video understanding, world states, cross-view memory, continual counting

TL;DR¶

VSI-Super-Wild uses 442 real long-form videos and 6,980 human-verified question-answer pairs to test continual world-state modeling around the agent, objects, and environment, with even the strongest model, Gemini-3.1-Pro, reaching only 44.36 overall.

Background & Motivation¶

A video model that identifies a street or a walking person has not necessarily remembered its direction of travel, when it visited a place, or whether an object has already been counted. These questions require internal states that change over time, rather than single-frame classification or a plausible verbal explanation. Human spatial memory connects self-motion, external objects, and environmental layout so that past experience can be queried after observation ends. Progress by multimodal large language models (MLLMs) on general video question answering may instead depend on sparse keyframes, local matching, and textual priors, without establishing such memory.

VSI-Super, introduced with Cambrian-S, already advances evaluation toward spatial supersensing: constructing, updating, and using an implicit 3D world model from continual visual experience. However, that benchmark concentrates on object recall and counting in household environments, with long videos assembled from short clips and containing generative in-frame edits. It therefore offers limited coverage of diverse, naturally continuous observations and underrepresents the observer's own state and environmental place states. The present paper does not merely make questions longer; it asks models to preserve queryable spatial relationships under genuine temporal continuity.

The authors select naturally recorded panoramic long videos and use different viewing directions to construct questions about motion, places, and objects. Panoramic sources allow the same place to be shown under a different heading, weakening solutions based on locating an identical image. Distinguishing first from last appearances and counting unique instances requires processing later evidence instead of retaining only an initial impression. Core Idea: jointly constrain question answering with real continuous videos and agent-, object-, and environment-centered states, separating image comprehension from continual world-state maintenance.

Method¶

Overall Architecture¶

The contribution is a dataset, evaluation protocols, and diagnostic analyses, not a new video network or training loss. On the construction side, panoramic videos yield perspective views, object temporal metadata, and camera spatial metadata, from which rules produce questions suitable for human verification. On the evaluation side, the tested model receives video observations and a task query, then returns a multiple-choice answer or an integer count. A crucial boundary is that the detector, segmenter, and pose estimator create annotations; they are not inference modules installed in every evaluated model. Scores reflect each model's own video understanding, so geometric information available to annotation tools must not be treated as an input already supplied to the model. The explanation below covers sources, required states, annotation, and scoring; as a benchmark, it is not depicted as a trainable network architecture.

Key Designs¶

1. Real Continuous Observation: broaden scenes and headings with panoramic sources

The authors collect panoramic YouTube videos covering 8 scene categories: culture and entertainment, industry, medical, office and education, residential, retail, street views, and transportation hubs. Manual filtering primarily retains egocentric, dynamic recordings rather than filling duration with static or visibly poor-quality content. The key distinction from VSI-Super is preserving naturally occurring observation order, without randomly concatenating short clips or relying on generatively inserted objects. Consequently, seeing an object or place again has a real temporal and spatial meaning rather than continuity created by editing.

Panoramas are stored in equirectangular format and projected into four orthogonal perspective directions: front, back, left, and right. These perspective videos become independent inputs to subsequent processing, so the reported 442 videos are view-level videos, not necessarily 442 independently recorded journeys. Front and back views are particularly important: a person walking forward may produce backward motion relative to the viewing direction of a rear-facing camera. This makes a linguistic prior such as "a street walk usually means forward movement" unreliable. The panoramic source also supports realistic, non-generative yaw changes for place queries without changing place identity. Here, "unedited" emphasizes natural content and continuity, not the absence of projection or sampling preprocessing.

2. Three Anchors: separately query motion, place, and object states

VMR, motion orientation recall, supplies a query frame from a specified moment and asks for motion orientation relative to the viewing direction at that moment. It asks how motion relates to the line of sight, not simply where the camera points; a static road image usually cannot determine the answer alone. The model must link the query image to neighboring temporal evidence in the observed video to distinguish forward, backward, and sideways motion. This primarily probes agent state, preventing scene recognition from being equated with self-motion inference.

VPO, place temporal ordering, supplies 4 query frames from different times and places and asks for their visit order. The query frames undergo different yaw rotations, so a queried place may face a direction without a directly matching image in the observation sequence. The model should therefore retain a relatively heading-invariant place representation and map the place back to its temporal position in the experience. This weakens pixel-level or approximate image matching, but does not establish that all visual or semantic shortcuts have been eliminated.

VOO, object temporal ordering, provides object concepts and specifies ordering by their first or last appearance. The first-occurrence task emphasizes retaining the earliest evidence, whereas the last-occurrence task requires updating a record when an object appears again. The dataset contains 1,675 first-occurrence and 1,675 last-occurrence questions, totaling 3,350. Their contrast probes whether a state is updated after its initial creation more directly than merely asking whether something ever appeared.

VOC, continuous object counting, queries an object category and requests the number of unique instances observed in the video. It does not sum per-frame detection boxes or count only the objects simultaneously visible at one instant. When the camera moves around or an object reenters view, the model must preserve identity and avoid treating a repeated observation as a new instance. This task consequently connects all three anchors through object recognition, observer motion, and cross-view association within the environment. Together, the tasks broaden state coverage, but successful answers remain behavioral evidence of world modeling rather than a direct measurement of internal 3D representations.

3. Two Metadata Types and Human Review: ground answers in video evidence

Temporal metadata begins with video-specific candidate objects proposed by human experts and an MLLM, followed by filtering based on YOLO-World occurrence statistics. Retained objects are processed by Segment Anything 3 (SAM3) to produce timestamped instance masks. This intermediate representation links visibility times to instance regions, supporting first appearance, last appearance, and counting questions. Occlusion, blur, and repeated observations in natural video can still cause errors, so automatic annotation alone is not sufficient to establish final ground truth.

Spatial metadata uses Depth Anything 3 (DA3) to estimate time-varying camera poses, supporting the construction of motion and spatial questions. A rule-based question generator then converts temporal and spatial information into multiple-choice or numeric questions; the tested model does not freely author its reference answers. Humans verify question-answer correctness and roll back to metadata or question refinement when problems are identified. The paper also states that answer distributions are controlled to reduce statistical option shortcuts, although the main text does not fully detail filtering thresholds or verification agreement. This is therefore a human-machine data production process, not simply an MLLM automatically generating every reference answer.

4. Fixed Observation Budgets and Permutation Diagnostics: separate content errors from option preferences

The default evaluation uniformly samples 128 frames at 1080P from each video before the model answers its questions. Settings marked with an asterisk in Table 1 use streaming inputs and should not be treated as having identical observation conditions to fixed-frame settings. The paper does not train a new method on this benchmark; the main comparison concerns existing models under specified input protocols. VMR, VPO, and VOO are four-way multiple-choice tasks scored by accuracy, with an expected random-guess accuracy of 25%. VOC directly predicts an integer and uses mean relative accuracy (MRA), so the multiple-choice chance baseline does not apply.

Following the definitions of ground-truth counts, predictions, and proportional error on page 10, MRA is:

\[ \mathrm{MRA}=\frac{1}{N}\sum_{i=1}^{N}\left(1-\frac{|\hat{y}_i-y_i|}{\max(y_i,1)}\right). \]

Here, \(y_i\) is the true count and \(\hat{y}_i\) the prediction; tables report this mean as a percentage. Exact counts receive 1, and error is normalized by the true count; the printed definition does not specify clipping negative values to 0. The text extraction disrupts the equation layout, so this rendering follows its accompanying definition without adding clipping or other modifications. Overall is a question-level aggregate across tasks, not a simple equal-weight average of four task scores or a classification accuracy with one uniform interpretation. Because task sizes differ, the 3,350 VOO questions receive greater weight than the 1,215 VMR questions.

Shuffle24 additionally evaluates all 24 permutations of the four answer options, checking whether equivalent questions produce different predictions solely because of answer position. The paper defines Top-K as the fraction of questions answered correctly at least once in the first K attempts, and Pass@K as the fraction answered correctly at least K times across all 24 attempts. The latter is a paper-specific definition and should not be confused with the usual code-generation pass@k meaning of at least one success in K samples. Option-order bias is the standard deviation of accuracy across permutations, accompanied by the gap between the best and worst permutations. This analysis reveals position sensitivity; it does not justify assuming that every Table 1 score has already been aggregated using Shuffle24.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 1 on page 9; every row is a non-asterisk setting using the default 128-frame input. VMR, VPO, and VOO report accuracy (%), while VOC reports MRA (%); values are means \(\pm\) standard errors of the mean (SEM).

Model	VMR	VPO	VOO	VOC	Overall
Gemini-3.1-Pro	\(34.21\pm2.26\)	\(63.84\pm4.46\)	\(42.54\pm1.35\)	\(38.16\pm5.13\)	\(44.36\pm1.39\)
GPT-5.4	\(37.76\pm3.02\)	\(24.87\pm2.63\)	\(37.62\pm3.13\)	\(32.94\pm1.11\)	\(34.52\pm1.68\)
Cambrian-S-7B	\(26.91\pm1.27\)	\(25.58\pm1.21\)	\(40.36\pm0.85\)	\(33.81\pm0.97\)	\(34.22\pm0.54\)
Cambrian-S-7B-LFP	\(26.58\pm1.27\)	\(26.11\pm1.22\)	\(40.27\pm0.85\)	\(28.36\pm0.94\)	\(33.35\pm0.54\)
InternVL3.5-8B	\(27.90\pm1.29\)	\(24.65\pm1.19\)	\(35.13\pm0.82\)	\(36.74\pm0.96\)	\(32.18\pm0.53\)
Qwen3.5-9B	\(25.68\pm1.25\)	\(25.19\pm1.20\)	\(41.25\pm0.85\)	\(30.76\pm0.95\)	\(33.87\pm0.54\)

Gemini-3.1-Pro's overall lead accompanies particularly strong VPO performance of 63.84%, rather than similarly high scores on every task. Cambrian-S-7B reaches 40.36% on VOO but only 26.91% on VMR and 25.58% on VPO, showing that object-task performance does not represent complete spatial capability. These results support saying that some models are near chance, not that every model is near chance on every spatial task.

Ablation Study¶

There is no module-removal ablation for a new network; the following is the duration-group analysis from Table 2 on page 10, using question-level Overall scores. Different durations also involve different questions and videos, so this is a grouped comparison rather than a controlled experiment that progressively lengthens the same video.

Model / duration (minutes)	0–10	10–30	30–60	60–120	120+
Gemini-3.1-Pro	51.9	42.6	40.1	46.9	41.2
Cambrian-S-7B	38.5	34.9	31.5	31.3	28.9
Cambrian-S-7B-LFP	39.4	32.7	30.0	32.2	29.7
InternVL3.5-8B	37.5	32.8	28.2	30.5	28.2
Qwen3.5-9B	40.1	35.5	29.4	29.3	26.0

Cambrian-S-7B falls from 38.5 to 28.9, a 9.6-point gap between the shortest and 120+ minute groups; not every model declines strictly monotonically with duration. The original table also reports an Average row falling from 35.0 to 26.3, but the prose calls it an open-source average; its included rows need clarification, so no model-group statistic is recomputed here.

Key Findings¶

In Figure 6 on page 13, Cambrian-S-7B achieves 50.7% on first-occurrence ordering and 30.0% on last-occurrence ordering, a 20.7-percentage-point gap consistent with the diagnosis of weak updates after initial registration.
In the same figure's place-rotation comparison, that model falls from 26.2% to 25.6%; the difference is small and near chance, so its significance should not be overstated without corresponding uncertainty estimates.
The rear-facing street example in Figure 5 on page 12 shows a model substituting the semantics of a walking tour for motion analysis, predicting forward movement despite relative-motion evidence.
Figure 7 on page 14 supports greater option-order sensitivity, but reliably readable curve values are unavailable in the cache, so only the qualitative conclusion is retained.

Highlights & Insights¶

The three anchors turn spatial capability into separately inspectable behavioral requirements. Remembering objects does not imply understanding self-motion or place identity across headings.
First- and last-occurrence questions share object concepts but demand different memory operations. Their comparison distinguishes initial registration from later overwriting and updating.
Panoramic data provide controlled contrasts as well as wider views. Changing heading while preserving place identity tests view dependence more directly than general long-video question answering.
Human review covers metadata and final answers rather than merely polishing question wording. This rollback mechanism is a reusable approach to difficult annotations built with automated vision tools.

Limitations & Future Work¶

The authors identify spatial collapse, semantic shortcuts, insufficient updates, and instance confusion, but these behavioral diagnoses do not uniquely determine the internal mechanism of failure.
Reader assessment: fixing input to 128 frames makes temporal sampling sparser in longer videos, so degradation may reflect missing evidence as well as memory difficulty, rather than state-update failure alone.
Reader assessment: future studies should control sampling density, duration, and task composition while comparing streaming and offline settings to separate observation budgets from model memory.
The source contains inconsistent duration statistics: page 9 reports a mean of 38.62 minutes, whereas Figure 4 labels it 41.3 minutes; the prose gives 284.52 total hours, and these conflicting values are not silently harmonized here.
Model-count conventions also differ: the prose says 13 models, while Table 1 lists 14 settings, including streaming and non-streaming versions of the same model.
Figure 6 gives GPT-5.4 first/last scores of 31.3%/22.1%, which do not directly reconcile with its 37.62% VOO score in Table 1; the text does not adequately explain whether subsets or protocols differ.
Annotation-tool and human-verification errors still warrant auditing; stating that answers are human-verified does not replace agreement statistics or review of difficult cases.

Compared with VSI-Super / Cambrian-S: the earlier work established spatial supersensing evaluation through object recall and continual counting; this paper extends it to real continuous scenes and agent and environment states rather than merely increasing question counts.
Compared with general long-video question answering: rotated places and occurrence conditions make scene summaries less useful, but the paper does not prove that every linguistic shortcut has been removed.
Compared with SAM3, YOLO-World, and DA3: these tools form annotation infrastructure, not the paper's model innovation; benchmark performance cannot be attributed to evaluated models using those tools.
Research direction: explicit memory with persistent identities and timestamps could jointly maintain observer poses and place associations; this is a reader-proposed direction, not an improvement validated by the paper.

Rating¶

Novelty: 4/5. Combining three anchors with natural panoramic long videos has clear value, while extending an existing spatial supersensing framework.
Experimental Thoroughness: 3/5. Multiple models, durations, and option-permutation diagnostics are covered, but input-budget confounding and inconsistent statistical conventions remain.
Writing Quality: 3/5. Task motivations are clear, while model counts, mean duration, and some task breakdowns need more consistent explanations.
Value: 4/5. A useful stress test for spatial video models that distinguishes semantic recognition, identity maintenance, and state updates.