DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues¶

Conference: ECCV 2026
arXiv: 2606.26602
Code: None
Area: Multimodal VLMs
Keywords: Multi-Image Fine-Grained Perception, Visual Cue Guidance, Benchmark, High Resolution, MLLM Evaluation

TL;DR¶

DiCoBench proposes the first high-resolution multi-image fine-grained perception benchmark driven by implicit visual cues (difference and commonality), comprising 765 high-quality samples and 8 types of tasks. It utilizes a multiple-choice question (MCQ) paradigm to eliminate n-gram evaluation bias. Among 18 evaluated MLLMs, the best-performing model, Gemini-3-Pro, achieves only 58.1% (compared to 98.3% for humans), revealing a severe deficiency in current models' ability to autonomously discover visual cues.

Background & Motivation¶

Existing MLLMs have made significant progress in single-image fine-grained perception—benchmarks such as V, HR-Bench, and TreeBench demonstrate that advanced models can precisely localize and identify minute details in high-resolution images. However, these evaluations fundamentally rely on explicit textual prompts in the questions* (e.g., "Where is the red cup?"). They essentially measure passive text-to-image grounding capabilities rather than the active ability to discover visual cues from the images themselves.

Why has no one previously developed a multi-image benchmark driven by implicit visual cues? Three reasons account for this: (1) Evaluation Paradigm Inertia—existing Image Difference Captioning (IDC) tasks typically employ n-gram matching metrics such as ROUGE-L / CIDEr. However, these metrics are extremely sensitive to variations in MLLM output formats, severely misjudging the true perceptual capabilities of models (as validated by G-VEval). (2) Blind Spots in Task Design—prior works focus solely on visual differences, completely ignoring "commonality" (co-occurring entities/logical relations across completely different scenes), which is an equally vital cognitive dimension. (3) Insufficient Resolution—current multi-image datasets generally utilize low-resolution images, failing to evaluate the perceptual limits of models on micro-scale visual cues.

The key enabling factors that make this task feasible now are: powerful MLLMs like GPT-5.1 can automatically generate diverse editing instructions, image editing models like FLUX.2 Klein allow micro-scale local editing while keeping the background intact, and the MCQ paradigm has matured to directly eliminate evaluation bias.

Design Choices: This work systematically categorizes implicit visual cues into two parallel tracks: Differential Cues and Commonality Cues, with each track containing four perceptual tasks, totaling eight categories. High-resolution images (averaging 1895 pixels, close to 2K) are used to construct MCQ samples, and an option stating "no perceivable difference/commonality" is appended to every question to ensure logical completeness.

Core Idea: Without relying on textual prompts, models must rely purely on cross-image comparison to discover micro-scale visual cues, serving as a true litmus test for fine-grained perceptual capability.

Method¶

Overall Architecture¶

Essentially, DiCoBench is a visual-cue-driven evaluation framework for cross-image fine-grained perception. The input consists of a pair of high-resolution images and a question devoid of any explicit textual prompts, and the output is the model's selection of the correct answer from options A-E. The entire process unfolds along two dimensions: task definition (what to evaluate) and dataset construction (how to build it). The task system categorizes visual cues that humans can capture without textual guidance into two tracks: differential (what is different between the two images) and commonality (what is identical between the two images). Each track is divided into four progressive tasks: attribute, entity, spatial relationship, and reasoning, forming a $2 \times 4 = 8$ task coverage matrix. The construction pipeline adopts a semi-automatic paradigm: "MLLM instruction generation + image editing model execution + rigorous human verification." This ensures four gold standards across 765 samples: text fidelity, visual naturalness, option mutual exclusivity, and micro-scale constraints.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["V* High-Resolution Base Images"] --> B["GPT-5.1 Generates Editing Instructions<br/>with historical de-duplication to prevent collapse"]
    B --> C["FLUX.2 Klein Micro-Scale Local Editing<br/>Modifies target region only, background intact"]
    C --> D["MCQ Construction<br/>4 Options + E: No Perceivable Cue"]
    D --> E["Human Verification<br/>Text loyalty/Visual naturalness/Exclusivity/Micro-scale"]
    E -->|Pass| F["DiCoBench Samples<br/>765 samples / 8 task categories"]
    E -->|Fail| G["Manual fallback supplementation or discard"]
    G --> B

Key Designs¶

1. Dual-Track Eight-Task System: Disentangling Implicit Visual Cues from the Dimension of "With or Without Textual Prompts"

The root problem of existing benchmarks is the misalignment between evaluation targets and human perceptual processes. When humans view two images, their brains spontaneously capture "what is different" and "what is the same" without requiring any textual instructions. Based on this, DiCoBench splits visual cues into two tracks.

The Differential Visual Cues Track evaluates the model's ability to detect micro-scale changes between two highly aligned, high-resolution images. It consists of four progressive levels: (1) Attribute—millimeter-level changes in the spectral reflectance, surface material, or morphological shape of micro-targets, while retaining their category identity and spatial coordinates (e.g., a DIP switch changing from red to green). (2) Entity—categorical substitution, anomalous disappearance, or unexpected appearance of microscopic objects (e.g., replacing a distant "speed limit 60" sign with a "speed limit 80" sign). (3) Spatial Relationship—entities maintain their identity and attributes but undergo micro-displacements or topological reorganization (e.g., moving keys from inside a drawer to a wall hook). (4) Reasoning—micro-scale inconsistencies are not simple pixel-level variations but are "visual traces" of potential physical interactions (e.g., newly added footprints on a beach or micro-cracks on a glass edge), requiring second-order causal reasoning.

The Commonality Visual Cues Track presents a brand-new challenge: the global semantics, illumination, and viewpoints of the two images are completely different, and the model must identify the sole intersection amidst massive visual clutter. (1) Instance—cross-scene instance re-identification, such as locating the same Swiss Army knife with unique wear-and-tear marks in both a messy dormitory and on an outdoor picnic mat. (2) Category—cross-domain category alignment, such as a short red ethernet cable on a server room floor vs. a long blue ethernet cable in a recycling bin; despite differences in color, length, and state, both belong to the RJ45 cable category. (3) Spatial Grounding—determining pairs of objects located in the same relative positions across two images without relying on appearance matching. (4) Reasoning—with zero visual or structural overlap, the model is required to first meticulously perceive almost all details of both images and then deduce whether a functional relationship (combination, supply, nesting, collaboration) exists between the objects (e.g., an old door lock and a rusty key constituting a "combination" relationship).

2. MCQ Evaluation Paradigm: Eradicating n-gram Evaluation Bias with an "Option E + Strict Alphabet Match" Approach

The IDC field has long used ROUGE-L / CIDEr to evaluate the quality of generated textual difference descriptions. However, G-VEval has demonstrated that these metrics are highly sensitive to format variations in MLLM outputs. A model might accurately perceive differences but receive severe penalties simply because its phrasing is not "standard," systematically underestimating its true perceptual capabilities.

DiCoBench designs all tasks as multiple-choice questions (MCQs). For each sample, four successful image-instruction pairings are randomly selected as options A/B/C/D, ensuring the correct answer is uniformly distributed among the alphabetical choices to eliminate linguistic bias guessing. A crucial step is appending Option E: "There is no visible difference between the two images" or "There is no visible commonality between the two images." This logically completes the option space (preventing models from achieving non-trivial scores via random guessing in a 4-choice setup) and quantifies "hallucinations out of thin air" (where differences are "seen" between identical images). The evaluation employs direct alphabetical output with temperature = 0 and exact character matching, completely eliminating formatting bias in text-generation evaluation.

3. Semi-Automatic Construction Pipeline and Quadruple Human Quality Control: Exhaustive Quality Verification Across 765 Samples

Completely manual annotation of high-resolution micro-scale samples is prohibitively expensive with poor consistency, while fully automatic generation suffers from uncontrollable quality. DiCoBench adopts a semi-automatic pipeline of "MLLM instruction generation + diffusion model editing + multi-turn human verification."

Instruction Generation Phase: Base on the base images and annotation masks of the V dataset, the micro-target masks are expanded with a $2\times$ context margin to ensure smooth blending. GPT-5.1 is utilized to generate six diverse, task-aware editing instructions for each mask, with generated instructions iteratively fed back into the context window to prevent mode collapse. Editing Phase: The instructions and expanded masks are fed into FLUX.2 Klein for micro-scale local editing, ensuring that only the intrinsic attributes of the target are modified while the complex high-resolution background remains completely intact. Final Human Verification Phase: All synthesized samples* must pass a checklist of four strict criteria: (1) Text fidelity: modifications must strictly adhere to the instructions; (2) Visual naturalness: the edited areas must show no blurriness, artifacts, or boundary degradation; (3) Option mutual exclusivity: different options (A-D) for the same base image must be mutually exclusive and distinguishable; (4) Scale constraints: the modified areas must strictly occupy $<5\%$ of the total image area. If any criterion is violated, the sample is either routed to a manual fallback mechanism for complementary generation or permanently discarded.

Each of the eight task classes features a customized generation and filtering workflow. For instance, in the commonality-reasoning task: four relationships (combination, supply, nesting, collaboration) are pre-defined and converted into visually recognizable object pairs. After locating suitable masks in two completely different images, corresponding objects are synthesized, and human experts finally verify the synthesis quality and the logical deducibility of the answers.

A Complete Example: Sample Construction and Evaluation for the Attribute Task¶

Taking a specific sample from the differential-attribute task as an example: The base image is a high-resolution photo of an electronics workbench containing a millimeter-scale DIP switch. GPT-5.1 is prompted to generate editing instructions to "only change the color of the DIP switch, without altering its category identity or spatial location," yielding multiple instructions like "change from red to green" and "turn from ON to OFF." FLUX.2 Klein performs editing within the expanded mask region while keeping the rest of the workbench background completely unchanged. Four edited versions that pass automatic checks are randomly chosen to serve as options A-D, and option E, "There is no visible difference between the two images," is appended.

During evaluation, the model receives two nearly identical photos of the electronics workbench along with the question: "What is the difference between these two images?" The options are A: The DIP switch changes from red to green / B: The DIP switch turns from ON to OFF / C: The screw changes from Phillips to flathead / D: The capacitor changes from blue to black / E: No visible difference. While humans can instantly identify the DIP switch as the only changing area and select the correct answer, existing models often overlook the visual cue due to the minute scale of the target, resulting in selecting option E or guessing randomly.

Loss & Training¶

As DiCoBench is an evaluation benchmark, it does not involve model training. However, the quality control standards in its construction pipeline can be considered an "optimization strategy": the four human validation criteria (text fidelity, visual naturalness, option mutual exclusivity, and scale constraints of $<5\%$ area) serve as compulsory gating thresholds for all samples entering the final dataset. Failed samples are iteratively routed to the fallback mechanism or discarded, establishing a closed-loop quality guarantee. Furthermore, to prevent models from exploiting linguistic bias to guess, correct answers are strictly and uniformly distributed across all alphabetical options.

Key Experimental Results¶

Main Results¶

Performance of 18 MLLMs on DiCoBench, with human performance as the upper-bound baseline (refer to Table 1 in the original paper for the complete 8-task breakdown; representative models are shown here):

Model	Attr.	Ent.	Spa.	Rea. (Diff)	Ins.	Cat.	Spa. (Comm)	Rea. (Comm)	Avg
Human	97.6	98.9	98.0	97.3	99.1	99.1	98.4	97.4	98.3
Gemini-3-Pro	49.4	89.5	82.0	28.0	72.7	63.6	58.4	36.4	58.1
Gemini-3-Flash	25.0	55.6	40.0	14.3	45.5	63.6	41.7	36.4	41.9
GPT-4o	32.9	46.3	26.0	25.3	28.2	24.6	26.0	20.0	29.0
GPT-5	35.3	54.7	48.0	22.7	66.4	53.6	36.0	22.6	42.6
TreeVGR-7B (Open-Source)	35.3	57.9	34.0	29.3	76.4	62.7	58.4	20.0	48.8
DeepEyes-7B (Open-Source)	31.8	55.8	36.0	26.7	75.5	60.9	29.6	20.0	42.9
Qwen3-VL-8B (Open-Source)	29.4	51.6	40.0	21.3	71.8	63.6	20.8	20.0	40.3

Ablation Study¶

Focusing on Qwen3-VL-8B, 1/10 of the samples from each task are sampled and manually annotated with the locations of visual cues, leading to two sets of control experiments:

Input Setting	Effect	Analysis
Original High-Resolution Input (Vanilla)	Baseline	Abundant irrelevant visual information overwhelms micro-scale cues
Cropped to Cue Region (Cropped)	Significant improvement	Filtering irrelevant regions reduces interference
Resized post-cropping (Cropped+Resized)	Further improvement	Concurrently increases the relative proportion of the cue, achieving dual-factor improvement

Conclusion: High resolution challenges models from two directions—(1) introducing excessive irrelevant visual information; (2) reducing the relative scale proportion of visual cues. The superposition of both effects makes it extremely difficult for existing models to localize micro-scale cues in global high-resolution scenes.

Human Perception Duration Experiment¶

Eight PhD students who had not been exposed to the evaluation were invited. They were grouped in pairs to complete tasks under four time constraints: 30s / 60s / 120s / unlimited:

Time Constraint	Human Accuracy	Gap with Gemini-3-Pro (58.1%)
30s	~70%	Gap narrows
60s	~80%	Gap widens
120s	~90%	Gap widens significantly
Unlimited	98.3%	Mass gap of 40.2%

Key Finding: Human perception is not instantaneous. Under restricted time budgets, the performance gap between humans and models narrows. However, as the time investment increases, human accuracy steadily climbs to near-perfection, while models remain stuck in bottleneck states. This implies that existing MLLMs' perceptual capabilities, similar to reasoning, could benefit from increased computational investment (e.g., test-time compute scaling).

Key Findings¶

Reasoning Tasks are a Universal Weakness: Regardless of being closed-source or open-source, models see their scores on the reasoning sub-tasks of both the differential and commonality tracks plummet to the random-guess level of 20-30%. In commonality reasoning, the situation is even more severe: almost all open-source models consistently select option E ("no commonality"), locking their performance at exactly 20%.
"Missing Visual Cues" is the Primary Error Type: Error analysis (on Gemini-3-Pro and Qwen3-VL) indicates that missing visual cues is the dominant error mode, exposing a systemic deficiency in models. This is directly related to the historical lack of such evaluations and corresponding signals in training data. Aside from this, Gemini-3-Pro is more prone to "hallucinations out of thin air" (detecting differences between identical images), whereas Qwen3-VL leans toward "factual description hallucinations" (misjudging object orientation/position).
The Closed-Source vs. Open-Source Gap is Not Insurmountable: TreeVGR-7B's score of 48.8% surpasses GPT-4o's 29.0% and GPT-4.1's 25.0%, showing that lightweight task-specific fine-tuned models are capable of competing with large-scale closed-source counterparts.
Commonality-Category Task is Relatively the Strongest: Most models score over 50% on this task, suggesting that basic object recognition and classification capabilities are relatively mature; the bottleneck remains in integrating visual cues that require deep logical analysis.

Highlights & Insights¶

Ingenious "Option E" Design Kills Two Birds with One Stone: It not only completes the logical space of the MCQ (preventing models from gaining points via random guessing in a 4-choice setting) but also directly quantifies "hallucinations out of thin air"—which was completely unmeasurable in previous IDC evaluations. This is an evaluation design trick with extremely low implementation cost but remarkable effectiveness, easily transferable to any VQA benchmark that needs to distinguish between "genuinely not knowing" and "blind guessing."
Insightful Analysis of the "Dual-Challenge" Mechanism of High Resolution: The experiment decouples high resolution into two independent factors: "large volume of irrelevant information" and "small relative scale of cues," utilizing crop vs. crop+resize experiments to unpack them—cropping eliminates the former, while crop+resize eliminates the latter. This ablation strategy can be transferred to any visual task requiring the analysis of resolution effects.
Human Perception Duration Experiment as a Rare "Cognitive Control" Design in Evaluation Papers: Rather than a simple human upper bound vs. model comparison, it reveals the time-varying nature of human perception—climbing from ~70% to 98.3% with increased time investment, while models remain locked at 58%. This directly points to an actionable research direction: introducing test-time scaling to the visual perception of MLLMs.
Commonality Cue Track Fills a Genuine Gap: Identifying the sole intersection across completely different scenes is a capability that has never been independently and systematically evaluated in existing benchmarks, yet it corresponds to the real-world cognitive demand of "human object recognition in complex environments."

Limitations & Future Work¶

Authors' Admission: The construction of DiCoBench heavily relies on strict human verification, limiting its expansion to larger scales due to annotation costs. Although the current sample size of 765 is the largest among similar works, it remains insufficient to cover all real-world scenarios.
Dependency on V* for Image Sources: All base images are sourced from the V* dataset. The scene diversity and distribution bias have not been systematically analyzed, which might affect the generalizability of the evaluation findings.
Operational Definition of Reasoning Tasks May Be Narrow: Commonality reasoning currently covers only four functional relationships (combination, supply, nesting, collaboration), while real-world logical relationships are far richer (e.g., causal chains, temporal dependencies, intent inferences, etc.). Future versions could extend to more relation types.
MCQ Paradigm Eliminates n-gram Bias but Introduces New Limitations: Fixed choices prevent evaluating models' free-form description capabilities. For partially correct but not fully matched perceptions (e.g., perceiving a difference but at a different granularity level of description), MCQ binary-classifies it as correct/incorrect, losing fine-grained ability profiling.
Concrete Improvement Directions: (1) Introduce more diverse base image sources to enhance scene diversity; (2) Expand the repository of relation types in commonality reasoning, such as adding causal, temporal, and counterfactual relationships; (3) Design hybrid evaluation paradigms that append open-ended description tasks alongside MCQs to profile a more fine-grained spectrum of perceptual abilities; (4) Explore a training data synthesis pipeline based on this benchmark, using high-quality synthesized data to feed back into and rectify the models' deficiencies in discovering visual cues.

vs. V* / HR-Bench / TreeBench (Single-Image Fine-Grained Benchmarks): These benchmarks rely on explicit textual prompting in the questions to guide models to focus on specific objects, which evaluates text-to-image grounding. In contrast, DiCoBench completely removes textual prompts to assess a model's capacity to autonomously discover visual cues from images, which is much closer to how humans perceive the real world.
vs. MuirBench / BLINK / MIR Benchmark (Multi-Image Comprehensive Benchmarks): These cover broad logico-semantic relationships like STEM knowledge and scene understanding but employ low-resolution images, making them incapable of evaluating the discovery of micro-scale visual cues. DiCoBench focuses on implicit visual cues at high resolution, acting as a complement rather than a substitute.
vs. DiffTell / OmniDiff / OneDiff (Image Difference Captioning): They evaluate using generated textual descriptions of differences paired with n-gram metrics, which suffers from two fundamental flaws: formatting bias and narrow task scope (focusing only on differences and not commonalities). DiCoBench resolves both issues simultaneously using MCQ and a dual-track system, pushing the resolution to the 2K level.
vs. G-VEval: G-VEval pointed out the failure of existing n-gram evaluation metrics. DiCoBench is the first work to implement this insight into a complete benchmark design, fundamentally bypassing the formatting bias issue with the MCQ paradigm.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Systemizing "implicit visual cues" as an evaluation dimension is a first, with the commonality cue track filling an explicit gap. However, the MCQ paradigm and the semi-automatic construction pipeline themselves are not entirely brand-new; the core novelty lies in the problem formulation rather than the technical execution.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation of 18 models across three categories (closed-source, open-source, and fine-tuning specialized). The high-resolution ablation, human time-variant perception experiment, and error classification analysis all provide insights that extend far beyond the main performance table.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivational logic (three paint-points $\rightarrow$ three designs); explaining task definitions via concrete examples makes them extremely intuitive; experimental analysis is structured and layered rather than just listing numbers.
Value: ⭐⭐⭐⭐⭐ Uncovers a capability gap systematically ignored by the community—the model avg score of ~40% vs. the human score of 98%; this massive gap is sufficient to drive MLLM research in autonomous visual cue discovery for years to come. The benchmark design itself offers a highly reusable evaluation framework (including Option E, MCQ, and dual tracks).