OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Chenfei-Liao/OmniCoT
Area: Vision-Language Reasoning
Keywords: panoramic spatial reasoning, global evidence, multi-hop reasoning, chain-of-thought, GRPO
Dataset: OmniCoT
Model: OmniCoT-R1

Identity check: ECCV ID 3376; the cached title page lists Haocong He, Chenfei Liao, and colleagues. The manifest and full-text titles match, with no subtitle change. This note uses the conference PDF main text; the supplementary material mentioned there is absent from the current cache.

TL;DR

OmniCoT turns panoramic question answering into multi-step See-Locate-Move reasoning that requires cross-view evidence, evaluates both answers and reasoning traces, and raises Qwen2.5-VL-7B accuracy on OmniCoT-B from 23.53% to 59.54% through structured chain-of-thought supervision and GRPO.

Background & Motivation

Panoramas cover the complete horizontal field of view and vertical directions, making them well suited to representing relationships between surrounding objects. Wider coverage does not guarantee broader evidence use, however: a question about an exit or a small region can still be answered using local cues. Meanwhile, equirectangular distortion, continuity across image boundaries, and dense objects make ordinary-image vision-language capabilities difficult to transfer directly to panoramic space.

Existing benchmarks such as OSR-Bench and ODI-Bench provide panoramic understanding tasks. This work pushes models to update orientation repeatedly, locate targets through multiple reference objects, and predict visibility after movement. The central difficulty is grounding every linguistic step in one spatial reference frame: a plausible account of turning, passing a sofa, and finding a target does not establish that the target occupies the predicted image location. Final-answer accuracy alone cannot distinguish lucky guesses, missing evidence, and internally inconsistent reasoning.

Core Idea: use geometry-constrained global multi-hop questions and structured reasoning traces to diagnose answer correctness, evidence coverage, and spatial consistency, then use the same task structure for panoramic model post-training.

Method

Overall Architecture

OmniCoT combines datasets, an evaluation protocol, and a training baseline; it is not a new visual encoder. Inputs are an equirectangular projection (ERP) panorama, a question, and coordinate-and-orientation hints for three objects by default. The model returns a reasoning trace and a final answer. The image supplies visual evidence, while sparse spatial anchors establish a shared reference frame; neither replaces the other.

On the data side, 3D scenes become spatial descriptions, which support candidate generation under See-Locate-Move. Dual-model filtering, expert checks, and reasoning-trace generation produce approximately 14.3K OmniCoT-T training examples and approximately 6.7K OmniCoT-B evaluation examples. Separately captured and manually annotated panoramas form OmniCoT-Real. Training uses supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO), while evaluation reports final answers and six reasoning-quality metrics.

As this is primarily a benchmark contribution, the explanation follows task definitions, data filtering, process evaluation, and post-training without depicting every dataset as a model component.

Key Designs

1. See-Locate-Move taxonomy: require persistent spatial state across steps

See covers viewpoint transformation through Multi-Step Orientation Tracking (MOT) and Relative Angular Calculation (RAC). MOT starts from an initial orientation, applies successive relative rotations, and identifies the resulting target. RAC uses different landmarks as references to compute cumulative rotation or relative bearing. The challenge is preserving directional relationships across projection distortion and image boundaries, rather than merely recognizing objects or treating opposite panorama edges as disconnected.

Locate covers inter-object relationships through Multi-Hop Object Identification (MOI) and Multi-Hop Direction Identification (MDI). Questions chain reference objects using qualifiers such as nearest, second closest, and cardinal directions. The model must resolve an intermediate object before identifying the final target or its relative direction. Skipping a reference can yield a semantically plausible answer that points to the wrong object.

Move covers embodied action simulation through Pure Translational Movement (PTM) and Rotation-Translational Movement (RTM). PTM predicts which object is first encountered along a specified direction. RTM additionally updates position, orientation, and visibility after translation and rotation. Embodiment here means virtual state updates in question answering, not demonstrated closed-loop control of a physical robot.

These three dimensions contain six question types emphasizing viewpoints, relational chains, and action consequences. They share an underlying layout but cannot all be solved with one local object-detection shortcut.

2. Geometry-constrained data filtering: establish answerability before producing checkable traces

The pipeline in Figure 2 starts from reliable 3D scene data, rather than asking a language model to invent a scene from an image. Geometry becomes structured natural-language descriptions, from which candidate questions are generated across the six types. These descriptions provide object identity, position, and relational context, constraining opportunities to invent unsupported layouts.

DeepSeekv3.2 and Qwen3-Max judge candidates for format compliance, object uniqueness, logical consistency, reasoning complexity, answerability, and type-specific penalties. Accepted questions also undergo a rapid expert check for ambiguous references or answers disconnected from the scene. The main text does not disclose every scoring threshold, so it does not by itself specify a fully reproducible filter.

After validating the questions, type-specific prompts generate chains with 2 to 4 clear steps. A summarization stage removes redundancy while preserving essential spatial information. Further scoring checks reasoning format, structure, scene-information use, type-specific penalties, answer format, and correctness. Experts inspect nearly 400 samples in detail, with a 95% accuracy acceptance criterion. This is a sampled quality-control condition, not an individual correctness guarantee for every example.

The real-world subset has a separate acquisition process: 200 Insta-X5 panoramas across 13 indoor scene categories, annotated by three domain experts over approximately 320 hours, yield 1,073 question-answer pairs. Real distortion, lighting variation, and occlusion test transfer beyond simulated scenes. This subset should not be treated as an equal-difficulty copy of the synthetic benchmark.

3. Joint answer and process evaluation: separate plausible language from spatial evidence

By default, the prompt supplies coordinates for three randomly selected objects and orientation conventions, such as positive X pointing east and positive Y pointing north. A spatial anchor is an object-level reference with known position and orientation, not a disclosure of the full scene layout. Anchor density changes the information available to the model, so accuracy comparisons must report the anchor setting rather than treating different densities as identical conditions.

General reasoning metrics follow MME-CoT: Precision measures the fraction of generated steps judged correct, Recall measures coverage of reference solution steps, and F1 combines them. Their harmonic relationship is:

\[ F_1=\frac{2PR}{P+R}. \]

Here \(P\) and \(R\) denote Precision and Recall. This notation expresses the metric definition, not a new training loss. Reported aggregate F1 values should be read from the original tables rather than replaced by recomputing F1 from already averaged P and R.

Panorama-specific metrics inspect spatial content. Viewpoint Consistency (VC) extracts viewpoint-related statements, scores them against ground-truth geometry, and averages the scores. Spatial Evidence Sufficiency (SES) measures whether the trace cites the spatial relationships required to solve the question, using their citation coverage. Reasoning Feasibility (RF) scores movement, turning, and interaction steps against scene constraints and averages their feasibility.

DeepSeekv3.2 judges CoT quality. Although the protocol calls for grounding in spatial facts, the judge is still a language model. High VC, SES, or RF alone does not prove successful pixel-to-geometry correspondence. Qwen3-VL-8B's strong process scores but weaker final accuracy illustrate this distinction.

4. Two-stage post-training: learn evidence organization before optimizing task correctness

OmniCoT-R1 starts from Qwen2.5-VL-7B-Instruct. SFT on OmniCoT-T teaches an explicit <think>...<answer> organization and steps grounded in panoramic evidence. Only the language model is trainable during SFT; the visual encoder and projector remain frozen. The emphasis is therefore on connecting existing visual representations to spatial reasoning, not learning a new panoramic visual frontend.

GRPO initializes from the SFT checkpoint, switches to full-model fine-tuning, and samples 8 responses per question. Rewards cover output format, task accuracy computed by a geometry-grounded executor, and regularization against repetitive or verbose degeneration, with respective weights of 0.1, 1.0, and 0.2. VC, SES, and RF must not be recast as three separate training rewards: Table 6 lists format, accuracy, and repetition.

The main text identifies a geometry-grounded accuracy executor but does not fully specify its tolerances, failure handling, or reward implementations. It is therefore insufficient to reconstruct the entire training program. The central mechanism is to make structured output a stable prerequisite, put most optimization pressure on verifiable task outcomes, and discourage degenerate traces.

Loss & Training

Table 6 specifies OmniCoT-T for both stages, BF16 precision, a maximum sequence length of 4096, and a maximum completion length of 2048. SFT uses a cosine learning-rate schedule and a 0.03 warmup ratio. The table does not supply the initial SFT learning rate, so none is inferred here.

GRPO uses a learning rate of \(10^{-6}\) and a generation group size of 8. Figure 7 tracks the first 6,000 optimization steps; this is not an explicit statement of total training duration. The format reward is already saturated at the start of GRPO, consistent with establishing structure through SFT first.

The authors also report limited KL changes, near-zero clipping ratios, and an improving repetition signal. These observations describe optimization dynamics; they do not independently establish the absence of forgetting on general capabilities, which would require separate task evaluations.

Key Experimental Results

Main Results

OmniCoT-B evaluates 12 open-source and 9 closed-source models using an ERP panorama, a question, and three spatial anchors by default. The following subset comes from Table 2. Accuracy is reported in percent; the last column contains the reported CoT F1 score.

Model	See	Locate	Move	Overall	CoT F1
Qwen2.5-VL-7B-Instruct	33.77	15.53	24.33	23.53	25.76
Qwen2.5-VL-72B-Instruct	52.03	23.69	35.05	35.26	35.39
Qwen3-VL-8B-Instruct	41.52	18.73	26.59	27.56	39.80
GPT-5	57.29	24.40	42.71	39.72	41.51
Gemini3-Flash	47.52	14.17	29.29	28.43	35.01

None of the models evaluated without dataset-specific post-training in Table 2 exceeds 25% on Locate, highlighting relational-chain localization as a bottleneck. GPT-5 achieves 39.72% overall accuracy, but it does not lead every reasoning-quality metric.

Ablation Study

Table 7 compares successive post-training stages. It demonstrates the gain from GRPO after SFT, but includes no GRPO-only control without SFT, so it cannot isolate all interactions between the stages.

Config	See	Locate	Move	Overall	CoT F1
Qwen2.5-VL-7B baseline	33.77	15.53	24.33	23.53	25.76
OmniCoT-R1, SFT	59.41	39.44	52.06	49.31	38.07
OmniCoT-R1, SFT+GRPO	67.97	51.82	61.34	59.54	51.14

SFT increases Overall by 25.78 percentage points; subsequent GRPO adds 10.23 points, producing a total gain of 36.01 points. Locate rises from 15.53% to 51.82%, showing that improvements are not confined to easier viewpoint tasks.

The following comparison uses Overall scores from Tables 2 and 5. Differences are calculated here as real minus synthetic, in percentage points; they are not causal effects measured on strictly paired examples.

Model	OmniCoT-B	OmniCoT-Real	Difference
Qwen2.5-VL-7B-Instruct	23.53	15.28	-8.25
GPT-5	39.72	29.45	-10.27
Gemini3-Flash	28.43	30.10	+1.67

Key Findings

• Process quality is not equivalent to answer correctness: Qwen3-VL-8B scores 85.09 VC and 79.05 SES but only 27.56% Overall in Table 2. This motivates checking visual grounding instead of rewarding fluent reasoning alone.
• CoT does not universally help: GPT-5 falls from 42.66% with direct answers to 39.72% with CoT, a 2.94-point decline in Table 3. This response-mode comparison does not show that an entire model family never needs reasoning.
• Real-world degradation has exceptions: Gemini3-Flash improves from 28.43% to 30.10% Overall in Tables 2 and 5. Likewise, Qwen2.5-VL-7B scores below 3B, so the prose claim of steadily improving performance with scale should not be repeated without qualification.

Highlights & Insights

• Question difficulty is tied to the distinctive value of panoramic input. Updating viewpoints, chaining references, and predicting action consequences test global layout use more directly than simply increasing the number of questions.
• Evidence coverage is separated from step correctness. Short, conservative responses can have high Precision but low Recall, a useful diagnostic distinction for other visual multi-hop benchmarks.
• Training and diagnosis share a task structure rather than optimizing a leaderboard alone. SFT for evidence organization followed by GRPO for outcome constraints provides a clear baseline for targeted small-model post-training.

Limitations & Future Work

• The authors emphasize synthetic-to-real generalization, but the real set contains only 200 indoor panoramas. Outdoor settings, dynamic scenes, and extended interaction remain insufficiently covered, motivating larger real-world datasets.
• Strong language models support both generation and process scoring. Nearly 400 inspected examples and a 95% acceptance criterion do not eliminate systematic question-type bias; judge agreement, failure distributions, and human verification deserve further reporting.
• The current cache excludes supplementary material. Paraphrase robustness, projection robustness, tolerance sensitivity, and data-scaling experiments are mentioned but their numerical results are not reproduced here. Image resolution, complete sampling settings, and executor details also cannot all be recovered from Table 6.
• Table 7 reports neither OmniCoT-R1 performance on the real set nor a GRPO-only control. Its 59.54% accuracy cannot be interpreted as an equivalent improvement in real environments.
• Some prose claims are stronger than the tables support: parameter scaling, anchor increases, and real-world degradation are not uniformly monotonic across models. Further analysis should use model-specific results, confidence intervals, and independent generalization tests.

Related Work & Insights

• Compared with OSR-Bench and ODI-Bench: OmniCoT emphasizes global multi-hop chains and explicit CoT annotations. This extends question structure and diagnostic dimensions; it does not imply that previous benchmarks lacked spatial reasoning entirely.
• Compared with MME-CoT: it inherits general Precision, Recall, and F1 evaluation, then adds geometry-oriented VC, SES, and RF. A transferable principle is defining which evidence a particular visual task requires.
• Compared with 360-R1: both use reinforcement learning for panoramic adaptation, while OmniCoT also supplies data and evaluation centered on global relationships and multi-step state updates. The main text does not provide a direct comparison under unified conditions.
• Research direction: explicitly link intermediate targets and bearings to checkable image regions or geometric states, then compare text-only CoT with evidence-linked reasoning. This is a future direction, not a module already implemented in this paper.

Rating

• Novelty: 4/5. The main contributions are global multi-hop tasks, process diagnostics, and data resources rather than a new architecture.
• Experimental Thoroughness: 4/5. It covers 21 models, post-training stages, and real-world evaluation, but lacks several important controls and transfer results.
• Writing Quality: 3/5. The narrative is clear, but some generalizations exceed the tabulated evidence and reproduction details depend on supplementary material.
• Value: 4/5. A useful resource for panoramic spatial reasoning and evidence-constrained vision-language post-training.