TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Vision-Language Reasoning (vlm_reasoning)
Keywords: thinking-with-images, tool use, multimodal evaluation, function calling, trajectory fine-tuning
TL;DR¶
TIR-Bench expands visual reasoning evaluation beyond crop-based search to 13 image-operation tasks with 1215 examples, showing substantial benefits from tool execution while the strongest o3-TU configuration still scores only 46.0 overall and struggles with complex spatial planning and tool selection.
Background & Motivation¶
A vision-language model (VLM) can produce a long chain-of-thought without revisiting the visual evidence needed for its reasoning. For inverted text, a longer verbal explanation does not itself restore character orientation; for a jigsaw puzzle, describing each piece does not verify whether neighboring borders match. Traditional multimodal reasoning commonly treats the image as a fixed input and performs subsequent computation mainly in language. Thinking-with-images changes this interface: a model can execute image operations, return the resulting images to its context, and continue making decisions.
Existing evaluations such as V* Bench and HR-Bench advance high-resolution visual search, but primarily emphasize locating targets and cropping them for inspection. They can test whether a model notices small objects, yet reveal less about rotating text, enhancing dark images, calculating pixel properties, or rearranging puzzle pieces. As models become capable of generating and executing Python, tools need not be restricted to registered cropping functions, and evaluation must cover this broader operation space. The paper therefore starts from task requirements, selecting problems that researchers judge to require active changes to visual inputs, rather than fixing a tool first and finding compatible data afterward.
The goal is not to improve the score of a new network, but to expose visual perception, operation selection, code generation, and subsequent judgment together. The paper also compares tool-enabled and tool-disabled versions of the same model, and direct versus tool-trajectory fine-tuning, to investigate possible sources of capability differences. Importantly, a human judgment that a task requires tools is a construction criterion, not a mathematical guarantee that every example is impossible for a static model. Core Idea: require evaluation problems to elicit new visual evidence, then use tool availability, function interfaces, and training-trajectory comparisons to distinguish static recognition from effective reasoning through image operations.
Method¶
Overall Architecture¶
The basic TIR-Bench input consists of images and a question; the output is a verifiable choice, short text, or list answer. It is not a unified solver and does not require all evaluated models to follow a fixed reasoning pipeline. Models without tools answer from the original image, whereas tool-enabled models can generate operation instructions or code, execute them, inspect the results, and continue answering. The important intermediate representation is not another verbal explanation, but new visual evidence such as a rotated image, local crop, enhanced image, or rearranged puzzle. Different tasks require different intermediate representations, so making a tool call does not by itself demonstrate effective visual reasoning.
The study comprises four designs: task coverage, sample construction, scoring and comparison protocols, and function-calling diagnostics. These determine what is tested, where problems and answers originate, how scores should be interpreted, and where tool utilization fails. The main leaderboard uses zero-shot evaluation rather than training evaluated models on TIR-Bench test examples. The rotated OCR fine-tuning study is separate: it constructs training sets from external OCR data and compares training strategies on that task.
Key Designs¶
1. Task coverage: extending visual operations to computation, transformation, and spatial rearrangement
The 13 tasks are Color VQA, Proportion VQA, Symbolic Reasoning, Math, Word Search, Rotated OCR, Maze, Low-Light VQA, Instrument Reading, Spot the Difference, Jigsaw, Visual Search, and Rotation. Color VQA concerns color-related information, proportion questions ask for the image area occupied by a specified object, and symbolic reasoning probes fine-grained geometric structure. These tasks show how tools can support pixel analysis or segmentation-assisted measurement rather than merely provide larger local images. Low-light questions require recovering useful evidence from dark images, instrument reading requires locating pointers and graduations, and rotated OCR benefits from correcting text orientation. They turn an input that is unsuitable for direct reading into an executable visual preprocessing problem. An operation is useful only if its result supports the correct answer, not because the generated code is long.
Another group requires changing spatial organization: mazes involve path planning, jigsaws require recovering piece positions, and geometry questions may benefit from auxiliary lines. Word Search asks for targets in dense character or digit layouts, while Spot the Difference compares changes in corresponding regions of two images. These tasks connect local observation to global structure and cannot be reduced to a single enlargement followed by question answering. Visual Search remains included to connect established high-resolution search evaluation with a broader set of operations. Rotation and Rotated OCR are also distinct: the former asks for the angle needed to restore orientation, whereas the latter asks for the text itself. A model may select an angle without reading accurately, or bypass rotation through static recognition, which is precisely why task-level reporting matters.
2. Sample construction: producing image transformations alongside verifiable answers
According to Table 1, the 13 tasks contain 1215 examples, with individual task sizes ranging from 50 to 120. For mathematics, symbolic reasoning, low-light questions, and instrument reading, researchers collect or create images and annotate questions and answers, with reciprocal checking by two PhD students. The color task selects 100 examples from ColorBench, emphasizing cases judged to require programmatic analysis. Jigsaw uses 120 RefCOCO images, divides them into grids from \(3\times3\) to \(6\times6\), and shuffles them, with the original arrangement providing the answer reference. Rotation transforms 75 CVBench images; Word Search combines 85 programmatically generated examples with 15 examples from the internet. Synthetic transformations provide known targets, but solvers are not thereby given the original arrangement or correct rotation parameter.
Proportion VQA uses RefCOCO ground-truth segmentation masks to compute object area fractions and constructs distractors differing by 8 percentage points. Rotated OCR selects 60 OCRBench images and applies rotations of \(90^\circ\), \(270^\circ\), and \(180^\circ\) with probabilities of 25%, 25%, and 50%, respectively. Spot the Difference partitions image pairs into corresponding grids, with two annotators checking which positions contain changes to produce list labels. Visual Search contains 32 difficult HR-Bench questions and 88 new examples, comprising 25 high-resolution art images and 63 real-world images. These choices provide explicit transformation origins or annotation evidence for several tasks and reduce the share of completely reused question-answer pairs. However, Section 3.2 states that 100 mazes were generated, while Table 1 and Figure 2 report 120; this note preserves the discrepancy and uses Table 1 for dataset sizes.
3. Scoring and comparison protocols: separating correct answers, partial matches, and tool gains
Evaluation first lets a model generate a response, then uses GPT-4o to extract its final answer. For multiple-choice and short answers, the extracted value is compared directly with the reference to calculate accuracy. For list answers such as Jigsaw and Spot the Difference, Section 4.1 specifies intersection over union (IoU), comparing prediction-reference overlap with their combined extent. Thus, although Table 2 is labeled Accuracy (%), it combines different task metrics; not every number represents the proportion of completely correct problems. Jigsaw answers in particular encode an arrangement, but the readable main text does not explain how list IoU preserves positional relationships; this note does not substitute an assumed set formula. This implementation detail affects whether an incorrect arrangement might receive an unjustifiably high score and must be checked in the scoring code during reproduction.
The main evaluations are zero-shot, with open models run on NVIDIA A100 GPUs and proprietary models accessed through APIs.
Tool-disabled GPT models are tested through Azure API, while o3-TU and o4-mini-TU use OpenAI API with code interpreter enabled and the container parameter set to auto.
Consequently, o3 and o3-TU should be read as the paper's two execution configurations, without assuming identical service-side environments.
The tool group also includes Qwen tool configurations, DeepEyes, and GPT-4.1-based PyVision to examine different tool frameworks.
Paired comparisons address the central question more directly than cross-model rankings: does access to an execution environment let a model convert new visual evidence into better scores?
Tool budgets, backend differences, and image-processing costs are not all controlled away, however, so the entire score gap cannot be strictly attributed to one internal reasoning mechanism.
4. Function-calling diagnostics: separating parameter selection from operation implementation
On Rotation, the authors compare two interfaces: supplying a predefined rotation function and requesting its angle parameter, or requiring the model to write the complete rotation code. The former reduces the programming burden but still requires selecting angles and judging whether the resulting image is correctly oriented. The latter allows more flexible operations while introducing implementation failure as an additional risk. Both interfaces allow repeated calls until the model stops requesting function parameters and returns a final answer. The test therefore concerns not only valid function-call syntax, but whether tool outputs help revise subsequent choices.
Three prompt conditions are tested: P1 provides only the question, P2 suggests using the rotation function, and P3 explicitly asks the model to test each candidate angle. P3 places an otherwise implicit search strategy in the prompt, helping distinguish inability to execute tools from failure to organize their use. Figure 3 qualitatively indicates that P3 is usually more effective, stronger models benefit more from code writing, and weaker models are better suited to predefined functions. Gemini-2.5-Flash does not write code under any of the three prompts, so the paper omits its code-writing results. This does not establish failure in every function-calling scenario; it describes behavior under these specific prompts and interfaces. The cache does not reliably expose individual bar values from Figure 3, so this note retains only the experimental design and the authors' qualitative findings.
Loss & Training¶
Section 4.6 uses Qwen-2.5-VL-7B and constructs rotated-text training data from the external OCR-Dataset. Training sets contain 1k, 5k, 10k, or 15k examples, with all parameters fine-tuned for 5 epochs. Direct SFT learns to predict reference text directly from the rotated image, supervising only the final reading result. Tool-Use SFT first supervises the correct rotation angle, executes the restoration, concatenates the upright image with the original context, and then supervises text reading. The trajectory therefore includes action selection, the new image produced by execution, and the answer grounded in that image, rather than merely adding an explanation to direct question answering. The main text does not introduce a new loss formula to reproduce, and this note does not invent an optimization objective for the procedure.
Figure 7 shows that tool-trajectory fine-tuning outperforms direct fine-tuning overall and improves more consistently as data increases; direct fine-tuning lacks the same scaling trend. The authors also observe faster loss reduction for tool-trajectory fine-tuning despite a higher initial loss. Their explanation is that the original model is more familiar with upright OCR, so orientation restoration reuses existing capabilities, whereas direct adaptation to rotated text may interfere with prior knowledge. This is a mechanistic hypothesis, not something established through an independent forgetting test or internal representation experiment. The result comes from a pilot study on rotated OCR and does not establish that the same tool supervision is preferable for every visual task. Exact curve values from Figure 7 are not tabulated in the readable text, so no training configuration's accuracy is estimated here.
Key Experimental Results¶
Main Results¶
The following selection comes from Table 2, page 9, under zero-shot evaluation; scores retain the paper's percentage scale, All is its overall score, and TU denotes its tool-enabled configuration. Overall performance, text reading, dense search, rotation, and object proportions are selected to show both benefits and regressions without presenting different models as one proposed method.
| Model / configuration | All | Rotated OCR | Word Search | Rotation | Proportion VQA |
|---|---|---|---|---|---|
| InternVL3-78B | 21.4 | 3.3 | 8.0 | 26.7 | 21.7 |
| Gemini-2.5-Pro | 28.9 | 25.0 | 12.0 | 30.7 | 21.7 |
| GPT-4.1 | 18.8 | 11.7 | 4.0 | 22.7 | 7.5 |
| PyVision | 31.8 | 63.3 | 10.0 | 46.7 | 26.7 |
| o4-mini | 21.2 | 8.3 | 5.0 | 26.7 | 17.5 |
| o4-mini-TU | 37.5 | 53.3 | 55.0 | 52.0 | 21.7 |
| o3 | 26.9 | 8.3 | 4.0 | 33.3 | 34.2 |
| o3-TU | 46.0 | 53.3 | 64.0 | 77.3 | 31.7 |
The o3 tool configuration raises All from 26.9 to 46.0, a gain of 19.1 percentage points; Rotation rises from 33.3 to 77.3, a gain of 44.0 percentage points. However, Proportion VQA drops from 34.2 to 31.7, a change of -2.5 percentage points, showing that executing Python does not imply accurate object segmentation. The authors connect this regression to o3-TU's inability to call a dedicated segmentation model; this is an explanation based on tool coverage, not a separately validated ablation result.
Ablation Study¶
Not all additional experiments are module ablations; the following uses PyVision interaction-efficiency analysis from Section 4.4, page 12, and Figure 6, page 13, with scores consistent with Table 2. Average interaction count is a proxy for solution cost, not elapsed seconds or a controlled ablation of turn budgets.
| PyVision task | Average interaction turns | Task score |
|---|---|---|
| Rotated OCR | 3.12 | 63.3 |
| Color VQA | 3.23 | 53.0 |
| Maze | 8.58 | 15.8 |
| Word Search | 7.34 | 10.0 |
PyVision averages 4.72 turns across tasks; its error attribution assigns 37% to perception, 61% to tool generation and reasoning, and 2% to other issues. These proportions describe PyVision, not o3's internal error distribution; the authors cannot access o3 and o4-mini internal reasoning to conduct the same analysis. High turn counts with low Maze and Word Search scores support the observation that long interactions can reflect repeated failure, but do not prove that an additional tool call causes errors.
Key Findings¶
- Tool gains are task-dependent. Rotation, OCR, and Word Search improve substantially, while o3-TU still scores only 16.4 on Jigsaw in Table 2.
- Specialized cropping ability does not establish general operation competence. DeepEyes scores 50.8 on Visual Search but 17.3 overall, showing why success with one visual tool cannot replace broad evaluation.
- Effective trajectories need not be longer trajectories. PyVision performs better on OCR and color tasks completed in fewer turns, suggesting that the value of evidence acquired per turn matters.
Highlights & Insights¶
- Changing images during reasoning becomes the evaluation focus rather than treating images as attachments to language questions. This exposes the interaction between perceptual interfaces and planning strategies.
- Tool comparisons retain both successes and failures. The proportion regression highlights that an execution environment enables operations without automatically supplying specialist capabilities such as semantic segmentation.
- Rotated OCR training reframes the problem as restoring a familiar input distribution before solving it. A transferable idea is to supervise meaningful visual transformations rather than merely increase final-answer examples.
Limitations & Future Work¶
- The source contains inconsistent counts: the abstract reports 22 evaluated models, while Table 2 lists 30 non-random model / configuration rows; Maze also differs between 100 in the text and 120 in the table. These discrepancies are not silently normalized here.
- Individual tasks are small, and the requirement to use tools largely rests on human selection judgments. Stronger static models may bypass some operations, motivating stricter paired interventions and difficulty stratification.
- GPT-4o extracts answers, so deterministic final matching does not make the entire evaluation chain model-independent. The main text also leaves positional encoding for list IoU and the weighting details of All insufficiently specified.
- Proprietary-model qualitative trajectories come from reruns in the web interface, not the original API evaluation trajectories; PyVision turn analysis likewise cannot represent every tool model's cost and behavior.
- The readable cache does not contain the referenced appendix, and Figures 3 and 7 lack reliable numeric tables. Reproduction needs full prompts, scoring implementations, budgets, and fine-tuning curves rather than this note's qualitative summaries alone.
Related Work & Insights¶
- Versus V* Bench / HR-Bench: These emphasize high-resolution search, whereas TIR-Bench adds rotation, enhancement, and spatial rearrangement. It tests capabilities beyond search without replacing specialized search evaluation.
- Versus Visual Sketchpad / CoGCoM / DeepEyes: These explore visual tools or operation trajectories, while this paper provides a cross-task evaluation platform. Fixed toolsets and dynamically generated code should be reported separately on shared tasks rather than collapsed into one agent label.
- Versus ViperGPT / PyVision: Program execution can construct visual operations dynamically, and TIR-Bench probes that flexibility across tasks. A useful research direction is to record operation selection, execution correctness, and whether the new image changes the final judgment together.
Rating¶
- Novelty: 4/5. Systematically expands evaluation of tool-assisted visual reasoning beyond crop-based search.
- Experimental Thoroughness: 4/5. Covers tool availability, function interfaces, prompts, training-data scales, and efficiency, though protocol and reporting details need clarification.
- Writing Quality: 3/5. The main argument is clear, but conflicting sample and model counts and underexplained list scoring hinder reproducibility.
- Value: 4/5. Useful for diagnosing visual agent limitations, particularly the gap between having tools and using them effectively.