CabinSI: Omni-Cabin Spatial Reasoning through Explicit Visual Cognitive Maps

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Area: Vision-Language Reasoning
Keywords: omni-cabin spatial reasoning, multi-view understanding, visual cognitive maps, reference-frame transformation, referring expression grounding

TL;DR

CabinSI evaluates spatial relations and referring localization inside and outside vehicles with 3,758 real-world samples, and converts multi-camera observations into query-relevant top-down cognitive maps that raise Qwen2.5-VL-7B overall RelCabin accuracy from 25.31% to 41.17%, while perception errors and cross-cabin grounding remain substantial bottlenecks.

Background & Motivation

In an intelligent cabin, "left" is not a fixed image direction. When a passenger asks for "the apple behind me on my left," a model must identify the speaker and reference direction, then place objects observed by other cameras into a shared space. For an instruction such as "that white car," an occupant's gesture or orientation may determine which exterior vehicle is intended. Benchmarks such as Talk2Car and DriveLM mainly address the external traffic scene, and merely accepting multiple images does not guarantee an understanding of their geometric correspondence.

The difficulty is not just object recognition. Even if a model detects cars, occupants, and barriers in every image, it can still confuse image-left with the left of a referenced object, or become distracted by repeated views and background details. CabinSI therefore evaluates both relational answers in RelCabin and target localization in RefCabin. Its curation requires relevant samples to genuinely depend on multiple views or interior-exterior information, rather than attaching unrelated pictures to a single-image question.

The authors build on the cognitive-map approach of Ego3D-VLM: establish which objects matter and where they are in a common frame before asking a multimodal model to infer their relationships. Core idea: externalize spatial relations from multi-view pixels into a query-relevant top-down map, so that the model reads a simplified, geometrically aligned scene instead of implicitly converting reference frames within cluttered camera images.

Method

Overall Architecture

The inputs are synchronized multi-view images and a natural-language question; the output is either a directional answer or a localized target. A shared front end performs query pre-parsing, object detection, and depth-based projection to form a labeled two-dimensional cognitive map. RelCabin uses the map to infer relative directions, whereas RefCabin combines attributes and spatial constraints to select a target node. Orientation estimation is optional in RefCabin and should not be interpreted as accurate human orientation being available in every experiment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Synchronized multi-view images<br/>Natural-language question"] --> Parse["Query Pre-Parsing"]
    Parse --> Map["Geometric Cognitive Map"]
    Orient["Optional orientation estimation"] -.-> Map
    Map --> Relation["Map-Based Relational Reasoning"]
    Map --> Selection["Map-Based Node Selection"]
    Relation --> Answer["RelCabin directional answer"]
    Selection --> Box["RefCabin target localization"]

Key Designs

1. Query Pre-Parsing: determine which objects and views are relevant

Passing an entire instruction to a detector, or putting every object from every camera into a map, carries linguistic ambiguity and visual redundancy into reasoning. The authors first use an LLM to parse the question, extract the involved objects, and select semantically relevant views. A referring expression comprehension model then obtains bounding boxes in the corresponding images. Both the queried entity and the reference entity must be retained: detecting only the black car is insufficient to answer where the ego vehicle is from that car's perspective.

This filtering does not replace geometry; it determines what the geometric front end processes. The paper compares complete sentences, rule-extracted objects, and GPT-extracted objects. With map-only input in both cases, rule extraction achieves 37.48%, while GPT extraction achieves 47.46%. Reliable semantic grounding is therefore a prerequisite for useful maps, and the final improvement cannot be attributed entirely to depth estimation or top-down rendering. The cache does not provide the complete parsing prompt, so it does not establish a directly reproducible prompt template.

2. Geometric Cognitive Map: put positions from different cameras in one frame

The geometric front end follows the basic Ego3D-VLM pipeline. It takes the center of each detected box, reads metric depth at that pixel, back-projects the point into camera coordinates using the camera intrinsics, and transforms it into a common frame using known or estimated extrinsics. The paper also describes relational scale normalization using canonical sizes of categories such as pedestrians and sedans to mitigate implausible distances caused by depth errors. This is an approximate object-center representation, not reconstruction of each object's full three-dimensional shape; if center depth falls on an occluder, the reasoning stage may still receive an incorrect layout.

CabinSI's key adaptation is to compress those positions into object nodes on a shared top-down plane and render a visual map with semantic labels and front/back/left/right directions. The paper represents this as \(M=\{(o_i,x_i,y_i)\}_{i=1}^{N}\), where \(o_i\) is a semantic object instance and \((x_i,y_i)\) is its position in the shared plane. The goal is an approximate layout that preserves useful relations, not high-precision reconstruction. The map's graph structure lies mainly in its explicit nodes and spatial relationships; the paper does not introduce a new graph neural network.

Boxes, object identities, source views, and map nodes need to remain associated so that selecting a node can identify an image target. The paper specifies these input-output relationships but does not fully detail cross-view duplicate-instance merging, calibration-error handling, or coordinate-normalization parameters. The cached projection equation is corrupted by text extraction. This note therefore retains the clearly supported verbal pipeline instead of presenting a reconstructed standard projection formula as the paper's original equation.

3. Map-Based Relational Reasoning: let RelCabin read spatial structure alone

For RelCabin, the final MLLM receives the question and rendered cognitive map without a textual serialization of the map. The best ablation configuration also removes the original camera images. Object names and direction labels can still appear in the visual map: "no textual map" does not mean that the image contains no text, but that a separate coordinate listing is not supplied for textual reasoning. The map fixes the output of appearance-based recognition, making the remaining problem closer to locating a reference entity, changing perspective, and assigning a relative direction within a common frame.

An object appearing on the right of a camera image is not necessarily on the right of the entity named as the reference. The shared layout separates these meanings, and the model should interpret directions around the relevant reference object. The paper delegates this ability to an MLLM reading the map; it does not provide enough detail to describe the system as an implemented, fully deterministic angular-threshold classifier. Removing the original images is also consequential: with GPT object extraction and an image map held fixed, accuracy is 35.06% when the target view remains available and 47.46% without raw images. More visual evidence is not automatically better in this setting.

4. Map-Based Node Selection: resolve RefCabin through constrained candidate disambiguation

RefCabin must identify the actual object rather than just answer a direction. The framework first generates proposals, then interprets the expression through semantic attributes such as category, color, and identity, together with constraints such as "at the barrier" or "next to the ego car." These are considered jointly on the cognitive map. Optional orientation estimation can supply additional information for expressions involving an occupant's direction or pointing. Compared with direct coordinate generation, the output is first constrained by existing object proposals rather than freely emitted from the language model's impression of the scene.

The paper expresses target selection in the following form. Only the fragmented typography in the cache is normalized here; \(\mathcal O\) denotes the candidate object set:

\[ o^*=\arg\max_{o_i\in\mathcal O} f_{\mathrm{sel}}(o_i,q_{\mathrm{ref}}). \]

\(f_{\mathrm{sel}}\) jointly measures semantic compatibility and spatial consistency. Spatial phrases can correspond to geometric predicates such as proximity thresholds or relative ordering, but the paper does not specify scoring weights, numerical thresholds, or a new trainable scoring model. The equation should be read as a formulation of node selection, not a fully disclosed optimization algorithm. The selected node identifies an image proposal, and the output is still evaluated through bounding-box localization accuracy. A map cannot recover a correct target that the proposal stage never detected.

A Worked Example

Consider the question in Figure 5: from the perspective of the black BMW hatchback stopped at the barrier in the front view, where is the ego car? Pre-parsing identifies the black BMW, the ego vehicle, and relevant forward-facing views. Grounding and depth information place both objects in a common top-down plane instead of directly using their left-right image positions. The MLLM takes the BMW as the reference when reading the map, and the illustrated answer is "Back."

Changing the question to the Figure 6 instruction, "Locate that black BMW hatchback at the barrier," changes the use of the same layout to candidate selection. Color and vehicle type narrow the semantic candidates, and the spatial relationship to the barrier resolves the reference before the corresponding target is returned. This is a walkthrough of the paper's illustrations, not an invented account of candidate counts, coordinates, confidence values, or success rates.

Loss & Training

The paper treats cognitive-map construction as a post-training procedure on top of existing models. Its main experiments change the input representation and inference procedure rather than introduce an end-to-end training objective. The cache does not report a new loss function, learning rate, training duration, or complete training split, so these details should not be replaced with an assumed supervised map-generator training recipe. Directly generating maps with image models is proposed as future work, not an already trained component of this method.

Key Experimental Results

CabinSI contains 3,758 samples: 1,121 for RelCabin and 2,637 for RefCabin. RelCabin includes 250 in-cabin single-view, 101 in-cabin multi-view, and 770 out-of-cabin multi-view samples. The RefCabin statistics paragraph lists 250 in-cabin single-view, 194 in-cabin multi-view, 514 out-of-cabin single-view, and 1,679 cross-cabin multi-view samples.

For open-ended RelCabin QA, an LLM judges semantic agreement between the predicted and ground-truth direction. Multiple-choice questions use exact option agreement. RefCabin counts a prediction as correct when its box has IoU strictly greater than 0.5 with any ground-truth box. All accuracies below are percentages; gains are percentage points, not relative percentage improvements.

Main Results

RelCabin backbone	Original-input Overall	With proposed framework Overall	Gain
Qwen2.5-VL-7B	25.31	41.17	+15.86
Qwen3-VL-4B	27.17	44.46	+17.29
Qwen3-VL-8B	33.59	45.92	+12.33

Source: Table 2. All three scales benefit, but the highest overall accuracy remains 45.92%; these results do not establish that cabin spatial understanding is close to solved.

Ablation Study

Out-of-cabin multi-view RelCabin configuration	Raw image input	Map representation	Accuracy
Original baseline	Multi-view	None	30.09
Rule-extracted entities	All views	Image	32.99
Rule-extracted entities	Target view only	Image	33.15
Rule-extracted entities	None	Image	37.48
GPT-extracted entities	Target view only	Text and image	33.20
GPT-extracted entities	Target view only	Image	35.06
GPT-extracted entities	None	Image	47.46

Source: Table 3. Changing view selection alone raises accuracy from 32.99 to 33.15, a gain of only 0.16 points. Improving entity extraction under map-only input raises it from 37.48 to 47.46, a gain of 9.98 points. With GPT extraction and an image map held fixed, removing the original target view adds 12.40 points. These factors interact; their differences cannot be summed as independent contributions.

RefCabin configuration, Qwen2.5-VL-7B	In-cabin single-view	In-cabin multi-view	Cross-cabin multi-view	Overall
Direct box prediction	49.41	14.95	Not reported	Not reported
Proposal generation and selection, no map	48.26	22.16	29.32	39.48
Proposal generation and selection, with map	52.90	24.23	30.33	40.40

Source: selected columns from Table 4; Overall is reproduced, not recomputed. Relative to proposal-based selection, the map adds 1.01 points in cross-cabin grounding and 0.92 points overall. For in-cabin multi-view grounding, proposal-based selection first adds 7.21 points, followed by another 2.07 points from the map. Table 4 additionally reports out-of-cabin multi-view results, but the dataset statistics paragraph gives no corresponding RefCabin subset count. This note does not guess its sample source or the overall aggregation denominator.

Key Findings

• Table 5's caption specifies 100 examples per setting, 300 in total, for the human-map study. GPT-5.2 improves from 17, 12, and 38 to 96, 95, and 74 on in-cabin single-view, in-cabin multi-view, and out-of-cabin multi-view respectively, supporting the interpretation that layout quality constrains reasoning.
• Human maps are not better for every model and subset. Qwen2.5-VL-7B scores 50 with predicted maps but 38 with human maps on out-of-cabin multi-view questions. Human-map performance should not be described as a strict upper bound based on the broader wording in the paper's prose.
• Table 5's Overall is not the simple mean of the three equally sized, 100-example subsets, and the cache does not explicitly state aggregation weights. The comparisons above use the reported subset values without combining them with the full-benchmark main results.

Highlights & Insights

• The work moves from understanding multiple pictures to understanding a shared reference frame. The map does not merely add a description; it actively removes appearance information irrelevant to spatial judgments, and the gains from dropping raw images support that design choice.
• Relational reasoning and referring localization share an object layout while retaining different output protocols. This helps distinguish incorrect direction inference from failed proposal generation or target disambiguation, rather than hiding both behind a mixed score.
• The transferable idea is a task-relevant spatial intermediate representation, not an unconditional commitment to precise reconstruction. In settings such as indoor interaction, one can first test whether positions and reference directions are sufficient before paying for greater geometric precision.

Limitations & Future Work

• The authors identify occlusions, cluttered layouts, and cross-cabin relationships as persistent challenges, with map perception quality as a primary bottleneck. Depth, detection, calibration, and entity-parsing errors propagate into the representation; discarding raw images also removes an opportunity for the reasoning model to inspect appearance and correct those errors.
• From a reader's perspective, an approximate two-dimensional layout has limited capacity for height, reaching interactions, and changing poses. Uncertainty-aware nodes, local height information, and selective revisiting of camera views when evidence conflicts are possible extensions, not modules validated in this paper.
• Reporting needs clarification: RefCabin subset statistics do not fully match Table 4, and aggregation in the human-map experiment is unclear. The cache also leaves gaps in cross-view instance merging, complete prompts, latency, and component costs, limiting reproducibility and deployment assessment.
• Cross-cabin localization is only 30.33%, and no end-to-end safety validation of interactive behavior is reported. Offline grounding and QA accuracy should not be extrapolated into a system that can reliably execute vehicle operations.

Related Work & Insights

• Compared with Ego3D-VLM: CabinSI inherits the detection, depth, and shared-coordinate cognitive-map pipeline, emphasizing query-filtered visual top-down representations and cabin-spanning tasks. Its contribution is task definition and validation of representation choices, not the first invention of cognitive maps.
• Compared with Talk2Car and DriveLM: These works primarily address exterior language grounding or driving QA, while CabinSI emphasizes a common reference frame between occupants and external targets. Their benchmark scopes differ, so CabinSI accuracies cannot directly establish overall superiority over them.
• Compared with direct multi-view MLLM inference: The framework makes spatial structure explicit and reduces image distractions, but becomes more dependent on correct front-end geometry. A useful follow-up experiment would systematically perturb map positions and orientations to separate sensitivity to layout noise from sensitivity to linguistic ambiguity.

Rating

• Novelty: 4/5. Omni-cabin tasks and visual-map-only inputs have clear value, while the geometric front end largely builds on earlier work.
• Experimental Thoroughness: 3/5. Multiple backbones, input ablations, and human-map analysis are included, but aggregation and reproduction details remain incomplete.
• Writing Quality: 3/5. The main pipeline is intuitive, but some interpretations are too broad and the tables need closer alignment with the statistics.
• Value: 4/5. The benchmark offers a decomposable entry point for cabin spatial intelligence and exposes the bottleneck between perception and reasoning.