VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning¶
Conference: ECCV 2026
Paper: ECCV paper page
Project: VisReason
Area: Vision-Language Reasoning
Keywords: visual chain-of-thought, multi-round region localization, process supervision, ordinal depth, adaptive zooming
TL;DR¶
VisReason extends visual question answering into multi-round supervision that describes a scene, localizes evidence, crops for inspection, and then answers, providing 489K examples including a 165K VisReason-Pro subset and improving Qwen2.5-VL-7B's sample-size-weighted judge score on the Visual-CoT evaluation suite from 0.770 to 0.802, with gains concentrated in fine-grained recognition and spatial relations.
Background & Motivation¶
Vision-language models (VLMs) can recognize objects, read text, and emit bounding boxes, but these skills do not automatically form a reliable visual reasoning process. Identifying an object to the right of a truck and in front of a bus requires both reasoning about several objects and inspecting the target's details. A single resized global view may support approximate localization while leaving recognition unresolved. Conventional image-question-answer supervision provides only the final answer, without teaching when to inspect the image again or where to look next.
Resources such as Visual CoT, DualFocus, and Chain-of-Spot introduce region-related supervision, while CogCoM and VoCoT explore multi-step visual operations. However, scale, task coverage, and spatial grounding remain difficult to combine. Another gap is that a relation such as left of differs from in front of: two-dimensional bounding boxes describe layout but cannot independently establish depth ordering. VisReason therefore builds on existing question-answering and localization data, using stronger multimodal annotators, segmentation, and monocular depth to turn visible evidence and its inspection process into training targets.
The paper does not introduce a new visual encoder. Its emphasis is process data: the standard corpus covers four task domains, the Pro subset strengthens spatial questions and rationale quality, and zooming is reserved for cases requiring local evidence. Core Idea: teach a global-to-local trajectory with region coordinates, fresh visual observations, and explanations for evidence selection, rather than merely teaching a longer answer.
Method¶
Overall Architecture¶
VisReason connects offline dataset construction with online inference. Offline, images, questions, answers, and target boxes are expanded into per-round scene descriptions, regions of interest (RoIs), and rationales; the Pro branch additionally uses segmentation and ordinal depth to generate more demanding spatial questions. Existing multimodal large language models (MLLMs) are fine-tuned on these trajectories. At inference time, the model predicts regions itself, crops and re-encodes the image, and continues answering with the accumulated history.
The reported 489K examples include the 165K VisReason-Pro subset; these counts must not be added to obtain 654K. The four domains are text/document understanding, fine-grained recognition, general visual question answering, and spatial relation reasoning, spanning 11 source datasets. Pro is constructed primarily from the GQA portion of Visual-CoT.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Images, QA pairs,<br/>and target boxes"] --> B["Region trajectory construction"]
B -->|Pro branch| C["Depth-augmented annotation"]
B -->|Standard trajectories| D["Trajectory supervision"]
C -->|Pro trajectories| D
E["New image and question"] --> F["Adaptive zooming inference"]
D -->|Fine-tuned model| F
F -->|Predict region, crop, and re-encode| F
F --> G["Final answer and<br/>regional evidence"]
Target boxes, segmentation, and depth signals in this pipeline construct supervision; they are not correct answers available at test time. The inference loop succeeds only if the model selects useful regions itself, which motivates evaluating localization separately from answer quality.
Key Designs¶
1. Region trajectory construction: turn a QA pair into inspectable evidence-gathering steps
The standard corpus starts from Visual-CoT image-question-answer triples and uses GPT-4.1-Nano to generate a short scene description, normalized RoI coordinates, and a rationale for selecting that region. Each round updates the local visual observation rather than merely extending an explanation. The annotation procedure adjusts the region to contain the existing target box and progressively narrows the search area. An RoI is represented as \([x_1,y_1,x_2,y_2]\in[0,1]^4\), making where to look a structured component of the model output.
Annotation stops when the RoI area is no more than twice the target-box area or when the budget of at most 3 rounds is reached. If the target is already sufficiently large, for example above 30% of the image area, iterative cropping is skipped in favor of one detailed reasoning step and the answer. These area-based criteria use target boxes known offline; they are rules for generating training trajectories, not a deployment-time controller with access to ground-truth boxes.
This design limits unnecessary long trajectories while discouraging crops that shrink past the answer object. The authors report that answer-critical regions occupy approximately 13.2% of the image on average, supporting the need for local inspection. Nevertheless, target containment constrains visible evidence; it does not establish that a generated rationale is the causally faithful process behind the answer.
2. Depth-augmented annotation: make spatial questions depend on both layout and depth ordering
VisReason-Pro adds object-level spatial information to the annotation paradigm. Figure 3 uses SegmentAnythingV2 and DepthAnythingV2 to obtain segmentation and monocular-depth cues. These become structured descriptions containing object IDs, categories, bounding boxes, and ordinal depths, which are passed to the stronger GPT-4.1-Mini annotator. The generator creates questions combining two-dimensional relations with relative depth and supplies a target box consistent with those relations.
For example, a right-of constraint may leave several candidates, while an additional in-front-of relation can constrain the target further. Verification, correction, and iterative zooming still follow, with at most 4 rounds for Pro. A single-round variant also distills the multi-step content into one rationale and a final RoI. This distilled annotation is distinct from the direct-QA prompt used in the experiments: the former retains explicit process supervision, whereas the latter matches short-answer evaluation formats.
Depth denotes relative ordering supplied by a monocular model, not measured physical distance. Pro simultaneously changes the annotator, rationale quality, and spatial supervision, so its gains cannot be attributed exclusively to depth. This also limits the causal conclusions supported by the ablations.
3. Trajectory supervision: jointly learn verbal reasoning and next-region coordinates
Rather than adding a separate detection head, training places each round's text and serialized bounding box in the autoregressive output sequence. The first round observes the full image; subsequent rounds observe regions selected in the preceding step. The current action can condition on the original question, previous textual actions, and visual features from preceding rounds. Why to inspect a region and where to look next thus become adjacent decisions in the same trainable trajectory.
The objective remains standard next-token prediction. The important change is the target sequence: it contains intermediate rationales and coordinates, not just a final answer. Fresh visual features from a crop allow later steps to obtain evidence that was previously difficult to resolve. The distinction from generating a long chain-of-thought (CoT) over a fixed global image comes from new visual observations, not text length alone.
4. Adaptive zooming inference: turn a predicted region into the next visual input
Inference begins by encoding the full image and producing a rationale plus an RoI to inspect. The execution procedure crops the predicted region, extracts new visual features, and appends them to the context. The model then continues inspecting or returns an answer. Easy training examples that answer directly demonstrate that zooming need not occur in every case. The paper calls this selective local-inspection mechanism Adaptive Zoom-In (AZ).
The mechanism adds no new image sensor; it reallocates observation effort within the available image. Small text, local colors, and targets specified through spatial relations can receive more focused processing. However, retaining historical context does not provide an active recovery policy. The authors explicitly identify the absence of zoom-out or backtracking: an early localization error can keep subsequent observations focused on the wrong region.
Loss & Training¶
Supervised fine-tuning (SFT) maximizes the autoregressive likelihood of the correct trajectory, equivalently minimizing target-token negative log-likelihood. Both text and bounding-box coordinates participate in sequence prediction. Equation (2) is corrupted in the supplied full-text extraction, so this note reports the objective described explicitly in the prose without reconstructing the authors' exact equation.
The main backbone is Qwen2.5-VL-7B, adapted with LoRA while freezing the ViT encoder. The learning rate for the language backbone and projector is \(2\times10^{-5}\). VisReason-7B is trained for 2 epochs on data excluding Pro; VisReason-Pro-7B then receives 1 additional epoch on the full corpus including Pro, not on the Pro subset alone.
For backbone transfer, the authors also fine-tune InternVL-2.5-8B on VisReason for 1 epoch. The main pipeline does not use reinforcement learning (RL), nor does the paper demonstrate that final-answer rewards alone learn the same cropping policy. Using these trajectories for subsequent RL is a proposed application, not an experiment completed here.
Key Experimental Results¶
Main Results¶
The Visual-CoT suite includes 11 source test sets. GPT-4o-mini assigns each example a score in \([0,1]\), and scores are averaged over 5 judge runs. Domain and overall results are weighted by test-set sample counts; they are neither unweighted column averages nor metrics that should simply be called accuracy.
The following results come from Table 3 and compare inference formats for the same Pro checkpoint.
| Model and inference format | Text/Doc | Fine-grained | General VQA | Spatial relations | Overall |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B, direct QA | 0.920 | 0.681 | 0.739 | 0.598 | 0.770 |
| VisReason-Pro-7B, multi-round CoT | 0.892 | 0.831 | 0.745 | 0.722 | 0.802 |
| VisReason-Pro-7B, direct QA | 0.916 | 0.709 | 0.735 | 0.599 | 0.769 |
Relative to the original backbone, multi-round inference gains 0.032 overall, 0.150 on fine-grained tasks, and 0.124 on spatial relations, while losing 0.028 on Text/Doc. Direct QA restores Text/Doc to 0.916 but returns spatial performance almost to the original backbone's level, showing that the inference protocol is part of the benefit.
Table 2 further reports an Open Images increase from 0.498 to 0.805, but a VSR decrease from 0.705 to 0.654. Even within spatial relation reasoning, improvement is not universal across datasets.
Ablation Study¶
The following selection reproduces clearly identified configurations from Table 9. Its numbers retain that ablation's separate settings and should not be conflated with the main trained-model results in Table 2. Pro denotes inclusion of the subset, and AZ denotes Adaptive Zoom-In.
| Config | Text/Doc | General VQA | Relation reasoning | Fine-grained | Overall |
|---|---|---|---|---|---|
| Baseline without VisReason | 0.920 | 0.739 | 0.598 | 0.681 | 0.770 |
| VisReason, without Pro or AZ | 0.864 | 0.744 | 0.678 | 0.798 | 0.777 |
| VisReason + Pro, without AZ | 0.856 | 0.750 | 0.693 | 0.809 | 0.780 |
| VisReason + Pro + AZ | 0.892 | 0.745 | 0.722 | 0.831 | 0.802 |
Compared with VisReason + Pro without AZ, the full configuration gains 0.022 overall, 0.036 on Text/Doc, 0.029 on relation reasoning, and 0.022 on fine-grained tasks, while General VQA decreases by 0.005. AZ therefore does not improve every column monotonically; it benefits the task mix that relies more heavily on local inspection.
Key Findings¶
- Output format affects external evaluation. In Table 7, Pro with multi-round inference scores 0.777/0.603 on MME/V*, increasing to 0.856/0.791 with direct QA, compared with 0.861/0.791 for the original Qwen. This supports a format-mismatch explanation, not the claim that multi-round inference is a better default for general benchmarks.
- Localization has separate supporting evidence. Table 4 uses Pro's annotated final-round RoIs as reference boxes. The full model scores 0.34/0.23 on [email protected]/[email protected], compared with 0.29/0.19 for LLaVA-NeXT. These are localization metrics at the stated IoU thresholds, not general object-detection mAP.
- Backbone transfer still shows specialization. In Table 6, fine-tuning InternVL-2.5-8B raises fine-grained performance from 0.747 to 0.823 and spatial relations from 0.621 to 0.669, but overall performance moves only from 0.738 to 0.740.
- Human approval does not establish fully reliable processes. Table 8 reports Pro scores of 4.19 for answer accuracy, 4.46 for grounded faithfulness, and 4.37 for stepwise clarity and sufficiency on a 1–5 scale. The study involved 30 raters and 20 sampled items per method.
Highlights & Insights¶
- Encoding regional actions matters more than merely lengthening reasoning text. Rationales, coordinates, and subsequent images form an inspectable relationship. Errors can consequently be examined as localization, local recognition, or answer-generation failures.
- Selective zooming is a transferable data-design principle. Direct answers for large targets and iterative inspection for small targets demonstrate different observation budgets. Similar structures could serve high-resolution interfaces and chart reading, although the paper does not evaluate these applications.
- Changing the evaluation format for the same checkpoint is an important control. Recovery on documents and external benchmarks shows that output protocols can obscure retained capabilities. Visual-CoT evaluation should examine both reasoning benefits and compatibility with answer extraction.
Limitations & Future Work¶
- Cropping errors can cascade. The current system has no active zoom-out, return-to-global-view, or candidate-region backtracking policy. Evidence-triggered recovery is a possible extension, not an existing feature.
- Pseudo-depth introduces supervision noise. The depth absolute difference in Table 5 is derived from monocular-depth ordinal cues, not a metric 3D error measured in meters. The supplied main text does not fully specify its scale or the complete Grounded Ratio decision rule, so these results cannot establish physical 3D localization accuracy.
- The quality audit still exposes process errors. The authors conduct a stratified blind audit of 2,200 examples, sampling 200 per source. Answer consistency is 99.1%, target containment 98.5%, and RoI tightness 95.0%, but rationale necessity and faithfulness are both 86.5%. They identify missing global layout cues after cropping as a weakness on Text/Doc tasks.
- Cost and causal attribution remain incomplete. Repeated visual encoding increases latency; the supplied main text lacks a comprehensive latency-budget table and does not independently isolate depth, stronger annotation models, and rationale quality. The main Pro model also receives 1 additional training epoch, so the main-model comparison alone cannot identify the effect of an individual data factor.
Related Work & Insights¶
- vs Visual CoT / VisCoT: VisReason inherits source data and the evaluation framework, extending them with multi-round evidence gathering and stronger ordinal-depth supervision. It expands process annotations over existing resources rather than collecting all images anew.
- vs CogCoM / VoCoT: These approaches also emphasize visual operations or multi-step grounding. VisReason emphasizes four-domain coverage, compact regional trajectories, and Pro spatial annotations. The paper does not supply complete, matched-budget numerical comparisons against all these approaches, so broad superiority claims would be unsupported.
- vs RL approaches such as Chain-of-Focus: VisReason primarily uses SFT to provide explicit demonstrations of regional actions rather than learning a policy from sparse answer rewards. Initializing from these trajectories and then optimizing stopping, recovery, and observation cost is a direct research direction, but the combination requires further experiments.
Rating¶
- Novelty: 4/5. The main contribution combines multi-domain regional trajectories with ordinal-depth annotations rather than introducing a new architecture.
- Experimental Thoroughness: 4/5. Format controls, ablations, a second backbone, and human evaluation are included, but latency and isolated-factor analyses remain limited.
- Writing Quality: 4/5. The pipeline and specialization boundaries are explained clearly, although some metric details depend on supplementary material and the cached equation is corrupted.
- Value: 4/5. A useful resource for supervising visual evidence gathering, provided multi-round and direct-QA modes are selected according to the task.