Skip to content

V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/tianyi-lab/VREX
Area: Multimodal VLM
Keywords: exploratory visual reasoning / visual reasoning benchmark / chain-of-questions / intermediate-step evaluation / planning-following disentanglement

TL;DR

V-REX formalizes open-ended multi-step visual exploration as a Chain-of-Questions, turning an unreproducible free-form process score into discrete, checkable judgments by giving each step a finite set of options; on one shared ground-truth chain it separately measures a VLM's Planning (what to ask next) and Following (can it answer along the chain); across 702 samples and 2,504 questions in 4 categories and 15 scenarios, the strongest model (o3) reaches only 86.1 / 84.2 on the two averages, and small models are consistently better at answering than at asking.

Background & Motivation

The dominant way to evaluate VLM visual reasoning today is passive question answering: hand the model a well-defined question with a fully specified target and let it emit an answer. MathVista puts math problems into visual contexts, VisuLogic collects genuinely vision-centric reasoning items, ZeroBench deliberately targets near-impossible tasks, VisualPuzzles tries to strip away domain knowledge and leave only reasoning, and VGRP-Bench probes multi-step reasoning with visual grid puzzles. What they share is that the question is completely specified at input time, so the model can plan the whole solution in language space before answering and never has to decide "where to look and what to ask" inside the image. Scoring, likewise, only ever looks at the final answer.

Real applications, however, are full of open-ended tasks: inferring where a street-view photo was taken, judging from a set of posted images whether someone is cheating, or reading a system's state off a dashboard screenshot. The goal of such a task cannot be written down up front; the model has to ask a small question first ("what is the black car doing?"), use the clue, then decide the next question, and only then converge on an answer. If the exploration goes off track, getting every later step right does not help. The obstacle to evaluating this ability is that the space of intermediate steps is open: any given task admits countless reasonable walks, so scoring the process directly can only lean on LLM-as-a-Judge or Monte-Carlo-estimated process reward models (the PRMs of Chen et al., the step-wise trace comparison of LlamaV-o1). Such judgments are themselves irreproducible, and they cannot tell "answered wrongly" apart from "never asked the right thing." Decomposition has been tried before—IdealGPT splits questions through an LLM-VLM-LLM pipeline—but its purpose is to raise final answer accuracy, not to diagnose the reasoning process.

This paper's angle is that once each step's candidate set is shrunk to a finite number of discrete options, "exploration" stops being an unscoreable free-form text and becomes a sequence of right-or-wrong choices. The authors therefore write the exploratory process as a Chain-of-Questions (CoQ)—a sequence of interdependent sub-questions and answers \(\{(Q_1, A_{Q_1}), \dots, (Q_T, A_{Q_T})\}\) where each answer is a precondition for later questions—and split it into Planning (picking, among candidate sub-questions, the one that helps the final answer) and Following (answering step by step along the ground-truth question chain), separating the two by handing the model the ground truth at every step. Core idea: model exploratory visual reasoning as a question-answer chain and use a finite-option protocol that supplies the ground truth each step and lets the model choose only that step, so that "what to ask" (Planning) and "what can be answered" (Following) are measured in isolation.

Method

Overall Architecture

V-REX is a benchmark plus an evaluation protocol; it trains nothing. Construction and use follow one fixed chain: human experts annotate a ground-truth QA chain per image (each intermediate answer is a precondition for later questions), that chain is turned into two probes—one turns the question space into a multiple-choice task, the other turns the answer space into one—and intermediate-step accuracy then scores Planning and Following separately. Three parts of the pipeline are the actual contributions: the CoQ formalization that folds exploration into finite options, the construction of human chains under two validity constraints, and a distractor builder that creates both locally and globally misleading alternatives; the disentangled protocol is what turns these designs into two comparable, reproducible numbers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Chain-of-Questions formalization<br/>infinite exploration → finite options"] --> B["Ground-truth QA chain construction<br/>human annotation + two cross-check rounds"]
    B --> C["Distractor construction<br/>step-level + chain-level two-stage pipeline"]
    subgraph P["Planning / Following disentangled evaluation"]
        direction TB
        D["Planning probe<br/>given GT answers, pick what to ask"] --> F["Intermediate-step accuracy<br/>first T−1 steps only"]
        E["Following probe<br/>given GT questions, answer each step"] --> F
    end
    C --> D
    C --> E

Key Designs

1. Chain-of-Questions: folding the infinite exploration space into finite options

The root difficulty of evaluating exploratory reasoning is that the "process" is open: a model may ask anything in any order, and no rater has an objective standard for judging the quality of a free-form exploration. V-REX fixes that path into a definite shape—a \(T\)-step QA chain in which step \(t\)'s question \(Q_t\) and answer \(A_{Q_t}\) become the precondition for the next question, and the final answer is the conclusion the chain arrives at. CoQ is not a prompting trick (the paper explicitly clarifies that it is unrelated to CoT-style prompting and is not meant to push scores up); it is an evaluation substrate. Once each step's question and answer are confined to finite candidates, "was this step the right move" collapses into a discrete choice with a definite truth value, and the model can no longer hide behind hedged wording.

The immediate payoff is reproducible, fine-grained analysis. Every step leaves behind a clear decision point, so one can compute intermediate-step accuracy and also ask the reverse question—if step 3 asked the wrong thing, can the final answer still be rescued? None of this is visible in end-to-end evaluation that only reads the final answer. The paper frames the resulting capability as a diagnosis rather than a score, which is also what separates it from prior visual reasoning benchmarks: VisuLogic, ZeroBench and VisualPuzzles all place exploration in the answer space, whereas V-REX adds the question space.

2. Ground-truth QA chain: human construction under two validity constraints

Finite-option scoring only means something if the ground-truth chain is itself reliable, so the chains are built by human experts rather than generated by a model. Images come from public websites and existing benchmarks (the source list is in Supplementary Section A); five PhD-level annotators each build chains for three scenarios, and every chain must satisfy two properties: Helpfulness—each of the first \(T-1\) answers must be helpful for some later question, ruling out steps that are asked but never used; and Correctly Ordered—the question at step \(t\) must not depend on the answer to any later step, guaranteeing that the chain is a genuinely forward-moving reasoning path rather than a conclusion decomposed backwards (⚠️ the original paper's symbolic rendering of these two properties is garbled in the cached text; this is a semantic restatement—refer to the original).

Construction is followed by two rounds of cross-verification: in each round annotators are randomly shuffled onto three scenarios they did not author, checking the validity of the chain, the usefulness of intermediate steps, and the ordering, with feedback going both to the original annotator and to the previous round's verifier. This process targets the characteristic failure mode of automatically generated chains—models are very good at packaging a known answer into a plausible-looking sequence of follow-up questions, disguising backwards derivation as forward reasoning. The two constraints plus cross-person review block that at the source; without them every downstream metric would be inflated.

3. Distractor construction: step-level and chain-level as complementary stages

A ground-truth chain is an answer key, not an exam; options still have to be built. Planning candidates come from a two-stage automatic pipeline (using GPT-5). The first stage, Distractor Construction, runs two complementary strategies: step-level generation produces, for each reasoning step, a distractor that sounds locally plausible but points in a misleading direction (for example "how many cars are visible?" or "is the ground wet or dry?"), testing whether the model can separate "relevant" from "useful"; chain-level generation instead composes a whole distracting chain that is self-consistent yet subtly deviates from the truth, so the entire exploration path sounds reasonable but cannot reach the correct conclusion. The second stage, Question Integration, automatically filters, refines and merges distractors with the ground-truth chain into one dataset, keeping the number and style of options controlled at each position. Following-side answer options are created manually instead: a few plausible wrong answers are written for each answer step and combined with the ground-truth answer into a multiple-choice item.

The two stages are complementary because each blocks a different shortcut. With step-level distractors alone, a model can score by asking "is this question about an object in the same image?", a purely local cue; the chain-level distractor chain forces a judgment about whether the whole exploration direction reaches the final answer. With chain-level distractors alone, a model could get away with coarse directional judgments and never inspect individual steps. These distractors are also the precondition for the failure-recovery analysis later on: only when wrong options look real does error propagation become worth studying.

4. Planning / Following disentangled evaluation: separating asking from answering by supplying the ground truth

With chains and options in place, the crucial remaining step is to keep the two dimensions from contaminating each other. The Planning probe branches at every question node, mixing the ground-truth question with distractors, and asks the model to select the next question given the original problem and the full preceding history (including each step's ground-truth answer):

\[Q_t \sim p(\cdot \mid Q_{\text{origin}}, Q_1, A^{*}_{Q_1}, \dots, Q_{t-1}, A^{*}_{Q_{t-1}})\]

The design point is that the ground-truth answer is handed to the model every step, so the model never has to solve the sub-question itself and is graded purely on exploration direction. The Following probe reverses this: the question chain is given entirely by ground truth, and the model answers each step from a set of candidate answers, conditioned on all preceding ground-truth QA pairs and the image:

\[A_{Q^{*}_t} \sim p(\cdot \mid Q^{*}_1, A_{Q^{*}_1}, \dots, Q^{*}_{t-1}, A_{Q^{*}_{t-1}}, Q^{*}_t)\]

Both metrics are intermediate-step accuracy, and both count only the first \(T-1\) steps, leaving the final-answer step to the separate end-to-end account: Planning accuracy is the fraction of steps whose question was selected correctly, \(\frac{1}{T-1}\sum_{t=1}^{T-1}\mathbb{I}[Q^{*}_t = Q_t]\), and Following accuracy is the fraction of intermediate questions answered correctly, \(\frac{1}{T-1}\sum_{t=1}^{T-1}\mathbb{I}[A^{*}_{Q^{*}_t} = A_{Q^{*}_t}]\). The two metrics are strictly isomorphic on the same chain and can therefore be compared directly—which is what makes the later finding that small models are markedly better at Following than at Planning hold up.

A Worked Example

Walk one sample through: the accident-liability case from Figure 2. The original question is "who is mainly responsible for the accident?" The ground-truth chain first asks what the black car is doing (the ground-truth answer is backing up), then what the silver car is doing, and only the last step is the conclusion question about responsibility, with candidates "the black car / the silver car". In the Planning task, the candidates the model sees at step one include the ground-truth question plus distractors such as "how many cars are visible?", "is the ground wet or dry?" and "what does the sign say"—all locally true of the scene, none of them useful for assigning blame. Whichever one the model picks, the system feeds back that question's correct answer (distractor questions have their own correct answers too) and moves on. The chain therefore continues, but the exploration direction is already off: if both steps are spent on the weather and the car count, the final liability step receives nothing but irrelevant clues, and the wrong final answer was effectively planted at step one. In the Following task, the chain is replaced by the two ground-truth follow-up questions, and the model simply answers each candidate set in turn before facing the final question. The same sample and the same image expose completely different weaknesses under the two protocols.

Key Experimental Results

Main Results

V-REX comprises 702 samples and 2,504 questions across 4 reasoning categories (Deduction, Guessing, Navigation, Retrieval) and 15 application scenarios (flowchart, pattern, property, relationship, responsibility, intention, location, time, topic, map, GUI, traffic, trend, counting, word puzzle); each sample has 2 to 6 reasoning steps, 3.57 on average. Evaluation spans open-source and proprietary families from 1B to 38B. The table below excerpts Planning / Following average accuracy by size group.

Model Size group Planning avg Following avg
LLaVA-OV-1B <7B 35.1 50.8
InternVL3-1B <7B 42.3 53.9
Qwen3-VL-2B-Think <7B 48.8 53.0
InternVL3-2B <7B 51.9 63.5
InternVL3.5-2B <7B 57.5 60.5
Qwen2.5-VL-3B <7B 48.9 67.2
Qwen3-VL-4B-Think <7B 62.0 64.2
InternVL3.5-4B <7B 63.4 69.1
LLaVA-OV-7B 7B–10B 48.0 66.6
Qwen2.5-VL-7B 7B–10B 57.1 71.1
Qwen3-VL-8B-IT 7B–10B 71.6 67.3
InternVL3-9B 7B–10B 73.1 68.6
InternVL3-14B >10B 74.8 72.4
InternVL3.5-14B >10B 70.7 71.2
InternVL2.5-38B >10B 78.8 76.5
InternVL3-38B >10B 81.6 73.5
GPT-4o Proprietary 72.7 76.4
Gemini 2.5 Flash Proprietary 77.9 77.3
o1 Proprietary 85.3 79.5
GPT-5 Proprietary 84.4 84.6
o3 Proprietary 86.1 84.2

Broken down by category, the abilities are far from uniform. The next table takes three representative models (each cell is Planning / Following accuracy):

Model Deduction Guessing Navigation Retrieval Average
InternVL2.5-38B 85.6 / 71.3 85.0 / 69.0 85.0 / 90.8 59.8 / 74.8 78.8 / 76.5
GPT-5 93.9 / 76.6 91.8 / 86.2 93.9 / 90.9 58.1 / 84.6 84.4 / 84.6
o3 94.8 / 80.2 94.4 / 83.0 93.3 / 89.9 62.1 / 83.7 86.1 / 84.2

Step-Level & Failure-Recovery Analysis

The main text does not report per-step-index accuracy curves—the effect of chain length shows up through the step distribution (2–6 steps, 3.57 on average) and through the stepwise recovery analysis in Supplementary Section H, so no numbers are invented here. What the main text does support is failure recovery: the share of responses that contain at least one wrong intermediate step yet still reach the correct final answer, which reveals how easily errors propagate along the chain (the paper's Table 2, restricted to models above 10B).

Model Recovery from failed Planning Recovery from failed Following
InternVL3-14B 65.9 55.9
InternVL3.5-14B 65.5 52.6
InternVL2.5-26B 66.9 62.1
InternVL2.5-38B 69.2 63.2
InternVL3-38B 65.1 52.9
InternVL3.5-38B 65.4 45.6
GPT-4o 78.9 56.5
GPT-5 84.3 53.6
o1 79.0 59.5
o3 80.6 57.3
Gemini 2.0 Flash 69.5 59.4
Gemini 2.5 Flash 72.7 51.8

Key Findings

  • CoQ hints genuinely help, but only for the reasoning-heavy categories. Comparing final-question accuracy with and without the CoQ intermediate clues, and reporting the change ratio \((\mathrm{Acc}_{\text{CoQ}}-\mathrm{Acc})/\mathrm{Acc}\), shows that the great majority of models improve—evidence that the human-built ground-truth chains are valid. Retrieval (counting, word puzzles) gains the least: those tasks depend more on direct factual visual matching than on hierarchical reasoning, and the paper offers a second explanation, namely that human-designed retrieval paths may diverge from a model's intrinsic retrieval strategy. A few models lose accuracy in every category when intermediate steps are introduced, indicating that extra exploration can inject "cognitive noise."
  • Scaling still holds, but Following is far more stable than Planning. Plotting ability against parameter count shows a clear upward trend; more interestingly, the variance across model families of the same size is markedly smaller for Following than for Planning. In other words, once a question is clearly given, same-sized models exploit visual cues and context to reach similar outcomes—Planning is what actually separates them.
  • Both dimensions correlate positively with end-to-end performance, Following more strongly. The Pearson correlation between Planning ability and final-answer accuracy is 0.858, and between Following ability and accuracy 0.948 (both \(p<1\text{e-}4\)). Following remains the primary determinant of end-to-end competence, but Planning spreads much more widely across models, so today's models differ far more in strategic reasoning.
  • Small models can answer but cannot ask. The ratio of Following to Planning approaches 1 as size grows: small models sit well above 1 (much stronger Following than Planning) while large models become balanced. Scaling capacity therefore does more than raise the total—it lets planning-oriented ability catch up with answering skill.
  • Failed Planning is easier to recover from than failed Following. Every model recovers to some degree, and recovery from failed Planning generally exceeds recovery from failed Following (o3: 80.6 versus 57.3). The natural reading is that distracting questions merely withhold clues and need not derail the conclusion, whereas errors in the answer space propagate down the chain into the final answer. A second observation is that Following recovery is similar between small open-source and large proprietary models, while Planning recovery is clearly stronger for large proprietary ones—they are less susceptible to suboptimal plans and irrelevant cues. (Supporting experiments: a blindfold baseline confirms most categories collapse toward chance without the image; an expanded-candidate-space stress test confirms the benchmark stays discriminative; a human study in Section J validates the benchmark against human performance and the quality of the AI-generated distractors.)

Highlights & Insights

  • Turning "process quality" into "a judgment over finite options" is the cleverest move here. Open-ended reasoning processes are hard to evaluate not because they look messy but because there is no objective criterion; discretizing every step with finite candidates removes the need for an LLM judge or Monte-Carlo estimation, improving cost and reproducibility at once. The same move transfers directly to agent trajectory evaluation and to diagnosing intermediate steps of multi-turn tool use.
  • "Supplying the ground truth" is a cheap and clean disentangling device. The most economical way to make a model display only its questioning ability is to remove the answering burden—feed it the ground truth each step and only decision-making remains. Compared with training two dedicated models or designing elaborate control experiments, the trick costs almost nothing and is worth reusing wherever a composite ability must be split for evaluation.
  • The two-level distractor design exposes a commonly overlooked evaluation hole. With only locally-plausible distractors, "local relevance" becomes a shortcut; adding chain-level distractor chains forces the model to judge whether an entire exploration direction is valid. Any work that scores processes via multiple-choice items should check whether its distractors only create difficulty at a single granularity.

Limitations & Future Work

  • Evaluation covers only multiple-choice exploration, never free-form questioning. Finite candidates buy scoreability at the price of excluding exactly what a model would ask on its own, which in real settings is the hardest part. The paper itself notes CoQ is not a prompting method, so it cannot measure the ceiling of open-ended exploration.
  • Ground-truth chain construction is expensive and the scenario coverage is narrow. Five PhD-level annotators covered only 15 scenarios and 702 samples (roughly 47 per scenario); two rounds of cross-verification raise reliability, but extending to new domains means rerunning the human pipeline, so the marginal cost of scaling is high.
  • Planning distractors are generated by GPT-5, which is also one of the evaluated models. A human study (Supplementary Section J) validates the quality of the AI-generated distractors, but generator and subject coming from the same source may still create a style bias: a model sharing the generator's pragmatic habits may find ground-truth questions easier to spot by their phrasing. Cross-family distractor generation with mutual validation would be safer.
  • The recovery metric deserves scrutiny. It counts "at least one wrong intermediate step but a correct final answer," yet a correct final answer can come from luck or a shallow guess; reporting the joint distribution of error-step type against recovery success (Section H does the stepwise version) would give a sharper diagnosis.
  • vs VisuLogic / ZeroBench / VisualPuzzles: These benchmarks explore the answer space—the question is given and the test is whether the model can compute or guess correctly; V-REX adds exploration of the question space and tests whether the model asks well. The former answers "how strong is this model," the latter "which step is it strong or weak at."
  • vs process reward models such as Chen et al.'s PRM: Both want to score intermediate steps, but PRMs rely on Monte-Carlo estimation and LLM-as-a-Judge, so the scoring itself is irreproducible; V-REX makes each intermediate step a discrete right-or-wrong choice whose results can be recomputed bit for bit.
  • vs LlamaV-o1's fine-grained evaluation: It aligns generated traces to a ground-truth trace step by step, but alignment and correctness judgments are still made by an LLM; in V-REX every step has a unique correct answer and no model judge is needed.
  • vs IdealGPT: IdealGPT also decomposes questions into sub-questions through an LLM-VLM-LLM pipeline, but the decomposition serves accuracy; V-REX's decomposition is an evaluation instrument serving diagnosis rather than score-chasing.

Rating

  • Novelty: ⭐⭐⭐⭐ Formalizing exploratory reasoning as a question chain with finite options, and isolating Planning from Following by supplying the ground truth, brings question-space exploration into evaluation for the first time—a new angle implemented cleanly.
  • Experimental Thoroughness: ⭐⭐⭐⭐ More than 30 models from 1B to 38B plus several proprietary families, with a dedicated analysis behind each of five findings; however, the main text lacks per-step accuracy curves and key evidence such as the human baseline is relegated to the supplement.
  • Writing Quality: ⭐⭐⭐⭐ The protocol is explained clearly and the figures do their job; some equations and symbols are already misaligned in the original, so readers must reconstruct them from context.
  • Value: ⭐⭐⭐⭐ A ready-to-use diagnostic tool for researchers working on VLM reasoning evaluation and training, and the Planning-Following gap points at a clear direction for improvement: strengthen exploration and planning, not just answering.