Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting¶

Conference: ECCV 2026
arXiv: 2607.00159
Code: GitHub
Area: Multimodal VLMs
Keywords: KB-VQA, Benchmark Auditing, Answer-Evidence Alignment, Grounding Disambiguation, Multi-Entity Augmentation

TL;DR¶

A systematic audit of the knowledge-based visual question answering (KB-VQA) benchmarks InfoSeek and E-VQA reveals three widespread deficiencies: answer-evidence misalignment, question under-specification, and single-entity visual shortcuts. To address this, a four-stage repair protocol and a controlled multi-entity augmentation protocol are proposed. Experimental results demonstrate model ranking reversals after repairing, along with substantial drops in retrieval recall and QA accuracy after augmentation, indicating that current KB-VQA evaluations significantly distort the measurement of knowledge grounding and reasoning capabilities.

Background & Motivation¶

The original intention of KB-VQA tasks is to evaluate whether VLMs can retrieve, ground, and reason beyond visual evidence using external structured knowledge bases. Answer accuracy is widely accepted as a reliable proxy metric for knowledge grounding and reasoning capability. However, this proxy relationship relies on three key assumptions: (A) annotated answers must be derivable from the associated knowledge base; (B) questions must provide sufficient constraints to uniquely determine the answer; and (C) the visual scene must truly require grounding and disambiguation for correct answering.

Existing representative KB-VQA benchmarks, such as InfoSeek and E-VQA, systematically violate these three assumptions. First, InfoSeek's QA pairs are derived from Wikidata knowledge graph triples, while the evaluation knowledge base utilizes Wikipedia text. This cross-source construction results in approximately 22% of instances having annotated answers that are either missing from the given knowledge base or directly contradicted by the evidence (e.g., labeling the mass of the McLaren 12C as 1302 kg, while the Wikipedia evidence states 1301 kg). Second, templated question generation discards key qualifiers in bulk, leading to about 47%\(\rightarrow\)59% of questions being under-specified: the same question can point to multiple reasonable answers supported by the evidence (e.g., asking "when is the mating season" can be answered with "spring," "March to April," or "around 1 year of age"), yet the evaluation only accepts a single annotated answer, causing correct knowledge reasoning to be systematically misclassified as incorrect. Third, existing benchmark images are dominated by a single salient entity, allowing global image retrieval to hit the target, collapsing the three-step process of localization-grounding-disambiguation into a shortcut; thus, high accuracy does not reflect genuine grounding and reasoning capabilities.

These deficiencies cause benchmark scores to decouple from target capabilities: high accuracy may stem from annotation artifacts, data biases, or shortcut strategies rather than faithful knowledge grounding and reasoning. The goal of this paper is to uncover these issues through auditing and restore the diagnostic power of the benchmarks via repairing and augmenting protocols, re-aligning evaluation representation with actual knowledge reasoning capabilities.

Core Idea: Perform a "physical exam and surgery" on the KB-VQA benchmarks—first, audit annotation validity and question clarity using dual verifiers and repair them; second, force grounding and disambiguation via controlled injection of visual distractor entities, transforming flawed benchmarks into diagnostic testing platforms that truly reflect retrieval-reasoning capabilities.

Method¶

Overall Architecture¶

The proposed method consists of two independent but complementary protocols applied to existing KB-VQA benchmarks (InfoSeek and E-VQA) without requiring new model training. The first is a four-stage auditing and repair protocol addressing Assumptions A (answer derivability) and B (well-defined questions) that outputs repaired instances or filters out invalid ones. The second is a controlled augmentation protocol addressing Assumption C (necessity of grounding disambiguation). By injecting intra- or inter-category distractor entities into the original image, it upgrades single-entity scenes to multi-entity scenes while keeping the original answers unchanged, forcing the model to rely on textual constraints to localize the target entity before conducting retrieval and reasoning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Instance<br/>Image + Question + Answer + Evidence"] --> B["Stage 1: Evidence Verification<br/>Dual Verifiers Judge Support/Contradiction/Missing"]
    B -->|Supported & Matched| C["Stage 2: Answer Derivability Audit<br/>Confirm Answer Can Be Derived from Evidence"]
    B -->|Supported but Mismatched| D["Answer Correction<br/>Replace with Evidence-Supported Value"]
    B -->|Unsupported| E["Filter Out"]
    C --> F["Stage 3: Question Constraint Repair<br/>Attribute/Temporal/Spatial Label Completion"]
    D --> F
    F --> G["Stage 4: Leakage & Consistency Verification<br/>Prevent Answer Leakage + Confirm Uniqueness"]
    G --> H["Repaired Instance"]

The augmentation protocol operates independently of the repair protocol: on the repaired benchmarks, a semantically similar or dissimilar distractor image is concatenated with each anchor image. For intra-category augmentation, a minimal spatial prompt (e.g., "the fish on the left") is appended to the question, whereas for inter-category augmentation, the question remains unchanged, ensuring that performance changes reflect only the increased difficulty of grounding and disambiguation.

Key Designs¶

1. Four-Stage Cascaded Auditing and Repair Protocol: From Evidence Verification to Global Consistency Confirmation

To address the violation of Assumptions A and B, the protocol processes instances sequentially through four cascaded stages, where the output of each stage serves as the input to the next, ensuring the audit trail's traceability. Stage 1 (evidence verification) categorizes instances into three types: supported and matched, supported but mismatched, and unsupported. The key design is the use of two independent verifiers (Qwen3-30B-A3B and DeepSeek-v3.2) to scan the target entity page section by section, removing instances only when both verifiers consistently judge them as "unsupported." This effectively mitigates over-filtering caused by single-model misjudgments. The evidence context is localized at the section level (averaging 868 tokens for E-VQA and 877 tokens for InfoSeek) to avoid full-page noise interfering with verification accuracy. For E-VQA instances, since evidence paragraphs are provided during dataset creation, the protocol directly inspects the cited paragraphs and revisits mismatched cases.

Stage 2 performs evidence-anchored answer correction on "supported but mismatched" instances: DeepSeek-v3.2 rewrites mismatched annotations strictly to values explicitly supported by verified evidence, and does not act as an independent source of answers. If no value can be derived from the evidence, the instance is removed. This step is particularly critical for InfoSeek, as it frequently corrects answer-side mismatches introduced by cross-source construction, whereas it has a minor impact on E-VQA because its answers were originally selected from the evidence articles.

Stage 4 executes a final check after all repairs are completed: it checks whether the question leaks the answer, whether the repaired answer is still supported by the evidence, and whether the repaired question constrains a unique answer. Any editing that leads to leakage is rolled back and conservatively rewritten. This final gating prevents new errors from being introduced during the repair process.

2. Ambiguity Label-Driven Question Constraint Repair: Precise Completion of Three Under-Specification Patterns

To address Assumption B, the paper defines three recurring ambiguity patterns and repairs them via label-driven completion: (1) missing attribute constraints—e.g., asking "how big can this plant grow" without specifying height or diameter; (2) missing temporal scope—e.g., asking "when is the mating season" without restricting to seasons, months, or life stages; and (3) missing spatial reference or granularity—e.g., asking "which country or region does this animal live in" without defining national or regional levels. Audits show that E-VQA has 21.5% missing attribute, 27.5% missing spatial, and 10% missing temporal constraints; while InfoSeek has 17.3%, 30.3%, and 2% respectively.

The repair strategy follows the principle of minimal editing: for InfoSeek, it restores the qualifiers (facets) discarded during the mapping from the KG to questions; for E-VQA, it completes disambiguating qualifiers for fine-grained entities in the originally templated questions. After repair, the original intent and evidence dependencies remain unchanged, ensuring the evaluation still tests the same knowledge point, but with sufficient constraints. For example, "when is the mating season" in E-VQA is revised to "which season of the year is the mating season," while the answer "spring" remains unchanged.

3. Controlled Multi-Entity Augmentation: Intra- and Inter-Category Distractor Injection with Anchored Answer Semantics

To address Assumption C (grounding and disambiguation requirements), the augmentation protocol injects exactly one distractor entity into each anchor instance, creating two scenarios: intra-category augmentation (where the distractor belongs to the same semantic category as the anchor, e.g., stitching two images of lakes side-by-side) with a minimal spatial prompt appended to the question (e.g., "the lake on the right") to specify the anchor; and inter-category augmentation (where the distractor comes from a different category, e.g., placing an animal next to a landmark) keeping the question unchanged. Under both settings, the annotated answers and knowledge bases remain untouched. Performance changes purely reflect the difficulty of grounding and disambiguation after visual ambiguity increases, rather than knowledge or annotation drift.

Intra-category augmentation isolates fine-grained intra-class ambiguity: even if the distractor is semantically homogeneous to the anchor, the model must ground the textual constraint (e.g., "on the left") to local visual evidence instead of relying on coarse-grained global embeddings. Inter-category augmentation tests whether retrieval is overly sensitive to irrelevant visual content: the distractor is semantically unrelated but sufficient to disrupt the global image representation; if the model does not first resolve the textual constraints of the target entity before retrieving, it will be misled by the distractor.

For quality control of the augmented dataset, 100 intra-category and 100 inter-category instances are sampled from each dataset for human evaluation, where two annotators answer the questions based solely on the anchor evidence. Human accuracy is 87.5% for E-VQA intra-category, 96.0% for E-VQA inter-category, 86.0% for InfoSeek intra-category, and 89.5% for InfoSeek inter-category, validating the answerability of the augmented instances.

A Concrete Example: Repair Process of the McLaren 12C Instance in InfoSeek¶

Taking InfoSeek QID 5441 (McLaren 12C) as an example. The original question is "What is the mass of this car?" and the annotated answer is "1302" kg. In Stage 1, after scanning the Wikipedia page of McLaren 12C, the dual verifiers find that the evidence explicitly states "It weighs 1,301 kg." The annotated "1302" differs by a single-value deviation (a typical mismatch introduced during cross-source Wikidata-to-Wikipedia conversion) and is classified as "supported but mismatched." Stage 2 corrects the answer to "1301." Stage 3 identifies that the original question lacks an attribute constraint—"mass" can refer to curb weight, dry weight, or other definitions—and the evidence context points to "dry weight." Thus, the constraint "dry weights" is appended to the question. Stage 4 confirms that the repaired question does not leak the answer and that "1301" uniquely corresponds to the dry weight in the evidence. The final repaired instance consists of the question "What is the dry weight mass of this car?" and the answer "1301." Experiments show that after repair, EchoSight's re-ranker can correctly rank to the target "Design" section and output the correct answer.

Key Experimental Results¶

Main Results: Performance Changes and Ranking Reversals Before and After Repair¶

Keeping the images, retrieval setup, model checkpoints, and evaluation metrics completely identical, only the questions and answers were repaired on the InfoSeek entity-deduplicated subset (1,924 questions: 1,604 String / 223 Numerical / 97 Time) and the E-VQA fixed evaluation set (4,750 questions).

Method	Backbone	InfoSeek Unrepaired	InfoSeek Repaired	E-VQA Unrepaired	E-VQA Repaired
Wiki-PRF	Qwen	44.9	43.6	31.9	33.1
ReflectiVA	Llama	37.3	38.1	36.8	36.6
IBA	Llama	34.5	42.4	41.9	42.7
EchoSight	Llama	30.9	37.2	40.4	40.9
CoMeM	Qwen	24.3	23.5	14.1	13.4
LLaVA-v1.5	Llama	5.8	6.6	12.9	12.7
Qwen2.5-VL	Qwen	21.4	28.3	21.9	21.9

Key Findings: 1. Ranking Reversals: on unrepaired InfoSeek, ReflectiVA (37.3) outperforms IBA (34.5), but after repair, IBA (42.4) reverses the lead over ReflectiVA (38.1), shifting the gap from \(+2.8\) to \(-4.3\). This directly challenges the unrepaired conclusion that "aggregation-filtering is superior to explicit evidence selection." 2. Narrowing of Relative Gaps: unrepaired Wiki-PRF leads IBA by 10.4 percentage points, whereas after repair, it leads by only 1.2 points, weakening the implication that "iterative multi-tool retrieval is inherently superior to modular workflows." 3. Strict Subsets (Time/Num) See More Significant Gains: for instance, Wiki-PRF rises from 35.1 to 43.3 on the Time subset and from 45.7 to 52.9 on the Num subset, indicating that answer mismatches are particularly fatal in these exact-match subsets. E-VQA exhibits smaller variations, consistent with the repair statistics (E-VQA primarily undergoes question modifications, with 48.5% of questions modified vs. only 2.9% of answers modified; whereas InfoSeek has 76.1% of questions and 38.8% of answers modified).

Fine-grained attribution analysis: on a 664-sample subset where EchoSight's re-rank top-1 successfully hits the correct entity, pure question-repair brings a lift of \(60.1 \rightarrow 75.1\), joint question-and-answer repair brings a lift of \(34.0 \rightarrow 45.2\), and pure answer-repair remains almost unchanged (\(44.2 \rightarrow 44.3\)). This indicates that question clarification is the primary driver of accuracy improvements.

Augmentation Experiment: Quantitative Exposure of Grounding Disambiguation Bottlenecks¶

Evaluation on all methods using the anchor subset and its intra-category/inter-category augmented variants (1,604 instances for InfoSeek, 3,871 instances for E-VQA).

Method	Anchor IS	Anchor EV	Intra-category IS	Intra-category EV	Inter-category IS	Inter-category EV
IBA	40.1	38.4	21.4	19.8	21.6	16.6
Wiki-PRF	43.9	32.8	23.6	23.1	25.9	22.5
EchoSight	38.7	42.0	15.9	19.3	17.8	18.5

After augmentation, the QA accuracy of all methods drops significantly, and retrieval recall (R@1) deteriorates rapidly: InfoSeek drops from 43.5% to 14.7% (intra-class) / 20.4% (inter-class), and E-VQA drops from 13.4% to 3.5% (intra-class) / 2.9% (inter-class). Post-retrieval evidence selection cannot offset the loss—IBA's post-retrieval recall on InfoSeek drops from 45.1% to 21.7% / 21.6%, and EchoSight on E-VQA drops from 46.6% to 11.8% / 11.7%. This confirms that once initial grounding fails, subsequent evidence aggregation cannot compensate; thus, grounding disambiguation is the core bottleneck rather than downstream reasoning.

Control ablation: replacing semantic distractor entities with blank panels (Blank) or anchor duplicates (Double) results in a far smaller retrieval degradation than true augmentation (e.g., E-VQA Blank R@1 6.3, Double 9.9 vs. intra-category 3.5, inter-category 2.9), confirming that the complexity introduced by semantic distractors exceeds that of layout shifts.

Key Findings¶

After repair, IBA achieves the largest gain (+7.9) on InfoSeek, reflecting that its explicit entity identification-re-ranking pipeline benefits most from clarified questions. Aggregation approaches like ReflectiVA and CoMeM show limited gains because they rely on the model's internal assessment of evidence relevance, a mechanism which benefits less directly from question clarification than explicit evidence selection does.
Economic implications of ranking reversals: if the community continues to rely on unrepaired benchmarks to assert that "aggregation-filtering is superior to explicit selection," substantial research resources might be misallocated to expensive model training, yielding actual returns disproportionate to the investment.
Wiki-PRF does not actively trigger its grounding tools under the augmented scenarios; the frequency of tool invocation and hit rates both decline significantly from the anchor scenarios (e.g., in InfoSeek intra-class, Caption invocations drop from \(439 \rightarrow 340\), and Hit@1 drops from \(20.3\% \rightarrow 7.9\%\)). This indicates that sparse reward training (optimized solely for final answer correctness) deprives models of the motivation to explore during the retrieval stage.
The cross-dataset ranking difference between E-VQA and InfoSeek is primarily driven by retrieval difficulty: InfoSeek's initial retrieval R@1 is as high as 43.5%, keeping Wiki-PRF competitive (48.6%); whereas E-VQA's R@1 is only 13.4%, causing Wiki-PRF to plunge to 18.2%. This further reinforces the argument that the retrieval phase needs to be challenged.

Highlights & Insights¶

"Physical Exam + Surgery" Double-Layer Audit Framework: Rather than introducing a new model, this work diagnoses and repairs existing benchmarks. This "meta-evaluation" perspective is systematically executed for the first time in the KB-VQA domain, and the auditing framework based on the three assumptions can serve as a general methodology transferable to other knowledge-intensive VQA or reasoning benchmarks.
Cross-Verification with Dual Verifiers to Eliminate Single-Model Biases: Leveraging open-source models (Qwen3-30B-A3B + DeepSeek-v3.2) for evidence verification, and requiring consistency between both models to filter out instances, balances the scalability of automated auditing with the reliability of decisions. This "weak-model cross-validation" concept is highly reusable for any automated annotation quality auditing scenario.
Experimental Design of Controlled Augmentation: The augmentation protocol meticulously isolates variables—holding the answer and knowledge base constant while injecting exactly one distractor entity and appending minimal text prompts, ensuring performance changes are uniquely attributable to the difficulty of grounding and disambiguation. The Blank/Double layout baselines further isolate pure stitching effects, showcasing a highly rigorous experimental design.
Cautionary Tale of Post-Repair Ranking Reversals: It is experimentally demonstrated for the first time that benchmark quality issues do not merely affect absolute scores but can distort comparisons between methods, thereby misdirecting research resource allocations across the community. This finding has profound implications for any domain reliant on leaderboards.

Limitations & Future Work¶

The principle of minimal editing in the repair protocol struggles with deep semantic ambiguities: Human evaluation reveals a residue of cases that are "textually answered but semantically ambiguous" (e.g., the Raglan Castle instance, which asks "which country's residents believe this castle compares to any other," whereas the evidence reads "the castle compares to any other castle in England or Wales"; the subject ambiguity cannot be resolved via minimal editing). The authors acknowledge that such cases may require bolder question rewriting, which alters question semantics and is left for future work.
Distractor selection in the augmentation protocol may introduce unexpected visual confusion: Human evaluation reveals that some inter-category distractor images contain secondary content of the same category as the anchor (e.g., an image of Mont Aiguille that also contains trees, confusing a plant-related question for the anchor), indicating a need for more robust distractor filtering or object detection-based cropping.
Evaluations are limited to single-distractor, single-hop reasoning scenarios: Real-world multimodal queries often involve multi-entity, multi-hop reasoning and complex spatial relationships. The two-entity scenario in this paper represents a backward-compatible first step, and future work should explore more complex multi-entity environments.
Repair and augmentation heavily rely on LLM/VLM automation: Repair quality is constrained by the judgment capabilities of the models used. Although human evaluation verified high consistency (92.9% / 91.5%), model auditing inherently yields edge-case misjudgments. Future work could explore stronger closed-source models for arbitration or introduce more structured formal checks for evidence derivability.

vs InfoSeek / E-VQA: These two benchmarks are the "patients" in this study. InfoSeek's cross-source Wikidata \(\rightarrow\) Wikipedia construction leads to severe answer-evidence mismatches (22% unsupported), and E-VQA's templated generation results in highly under-specified questions (59% ambiguous). Rather than proposing new datasets to replace them, this paper repairs and augments existing benchmarks. This "reuse instead of replace" philosophy offers valuable lessons for teams with limited computing and annotation resources.
vs IBA / EchoSight / Wiki-PRF / ReflectiVA / CoMeM: These methods serve as the baselines evaluated in this study, covering two mainstream KB-VQA paradigms: explicit evidence re-ranking (IBA, EchoSight) and implicit evidence aggregation (ReflectiVA, CoMeM). The paper finds that the gap between these paradigms narrows, and rankings reverse after repair, indicating that paradigm-selection conclusions are highly sensitive to benchmark quality—suggesting that auditing benchmark reliability before designing new methods may be more critical than intuitive development.
vs SK-VQA: SK-VQA also focuses on context understanding in KB-VQA, but its approach is to generate synthetic data to train models. This is highly complementary to the benchmark-repairing approach in this work, as repaired benchmarks naturally provide superior environments for training and evaluation.

Rating¶

Novelty: ⭐⭐⭐⭐☆ First to systematically audit, repair, and conduct controlled augmentation on KB-VQA benchmarks. The three-assumption framework is clear-cut and highly generalizable. However, the repair and augmentation protocols themselves are technically straightforward, with the primary innovation lying in the problem definition and experimental design.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 2 datasets, 5 SOTA baselines, 2 language model backbones, multiple data variants (unrepaired/repaired/anchor/intra-class/inter-class/Blank/Double), human evaluation of 10% of the repaired sets and 200 augmented samples, and a fine-grained attribution analysis distinguishing the contributions of question-repair vs. answer-repair. The experimental design is rigorous and self-consistent.
Writing Quality: ⭐⭐⭐⭐☆ Well-structured, with the three assumptions consistently anchoring the text. The steps of the repair and augmentation protocols are clearly formulated, and the appendix provides rich qualitative cases. However, some tables are highly data-dense with limited layout readability, and there is a missing high-level transitional overview paragraph between the Method and Experiment sections.
Value: ⭐⭐⭐⭐⭐ Highly critical alert for the KB-VQA community—proving that benchmark quality directly distorts method comparisons and resource allocation decisions. The proposed repair and augmentation protocols are directly applicable to other knowledge-intensive reasoning benchmarks, and the "audit-before-evaluate" paradigm is poised to become a standard precursor for benchmark-driven research.