Adaptive Knowledge Routing for Complex Reasoning in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Knowledge-Based VQA, Multimodal LLM, Explicit-Implicit Synergy, Cross-Modal Evidence Selection, Knowledge Routing
TL;DR¶
To tackle retrieval noise accumulation and isolated evidence mismatch in knowledge-based visual question answering, this paper presents MaS-VQA, a selection-driven framework that performs joint cross-modal "Mask-and-Select" filtering over visual regions and text passages to construct compact explicit evidence, which subsequently guides a frozen MLLM to synthesize constrained implicit reasoning paragraphs without any reasoning-stage retraining.
Background & Motivation¶
Knowledge-based Visual Question Answering (KB-VQA) tasks models with answering complex questions by bridging visual observations with external factual knowledge sources such as encyclopedic articles or open-domain web data. While modern Multimodal Large Language Models (MLLMs) display remarkable perception and conversational capabilities on self-contained benchmarks, they struggle significantly when posed with fine-grained species identification, historical landmarks, or multi-hop entity reasoning where visual signals alone lack the necessary factual grounding. Current paradigms address this limitation via explicit retrieval-based approaches, implicit parametric elicitation, or hybrid pipelines that naively concatenate external documents into the reasoning prompt.
However, existing hybrid systems suffer from severe cross-modal evidence noise and isolated filtering deficiencies. External knowledge retrievers inherently return lengthy passages saturated with off-topic facts and superficial correlations; simultaneously, raw images often contain extensive uninformative background clutter. More critically, the utility of visual regions and retrieved text is deeply coupled: visual patches only become semantically meaningful under the guidance of relevant textual facts, whereas the relevance of retrieved knowledge passages strictly hinges on faithful visual entity grounding. Prior approaches invariably perform isolated single-modality filteringโsuch as standalone passage reranking or text-free visual saliency detectionโwhich neglects this bidirectional cross-modal dependency and induces severe hallucinations, attention drift, and error propagation in downstream MLLM reasoning.
To resolve this bottleneck, the authors advocate for a tightly coupled cross-modal evidence selection paradigm that refines raw multimodal inputs within a shared relevance space prior to answer generation. The core idea is to formalize cross-modal evidence refinement as a unified Mask-and-Select mechanism where backpropagated cross-attention gradients simultaneously produce knowledge-guided visual masks and question-conditioned textual keyphrases, which subsequently constrain a frozen MLLM to distill high-signal implicit reasoning paragraphs for robust answer routing.
Method¶
Overall Architecture¶
The MaS-VQA framework comprises three sequential phases: First, given an input image \(I\) and question \(Q\), a multimodal retriever (EchoSight) fetches top-\(k\) candidate passages \(T\) from an external encyclopedic knowledge base. Second, in the explicit knowledge processing phase, an image-text matching (ITM) encoder operates over a unified multimodal relevance space to execute the Mask-and-Select mechanism, yielding a knowledge-guided visual attention mask \(M\) and a question-conditioned keyphrase set \(k\). Third, in the implicit knowledge synthesis and answering phase, the refined explicit knowledge package \(E = \{T, k, M\}\) serves as an inductive scaffold prompting a frozen MLLM to synthesize a grounded 2โ5 sentence implicit context paragraph \(U\), before a unified prompt containing \(\{I, Q, E, U\}\) is fed into the same frozen MLLM to predict the final concise answer.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Inputs: Image I + Question Q + Retrieved Passages T"] --> D1["1. Knowledge-Guided Visual Attention Mask Generation<br/>Gradient-weighted cross-attention + adaptive reweighting + token-wise percentile mask"]
In --> D2["2. Question-Conditioned Textual Keyphrase Selection<br/>Self-attention sensitivity matrix + cross-token aggregation + character span merging"]
D1 --> Merge["Refined Explicit Knowledge Package E = {T, k, M}"]
D2 --> Merge
Merge --> D3["3. Explicit-Constrained Implicit Parametric Knowledge Distillation<br/>Frozen MLLM synthesizes 2-5 sentence grounded prior U under refined context"]
D3 --> Ans["Final Answer Prediction: MLLM g_ฮธ(a | I, Q, E, U)"]
Key Designs¶
1. Knowledge-Guided Visual Attention Mask Generation: Suppressing Background Clutter and Cross-Modal Disconnect To overcome the limitations of isolated visual attention that lacks external factual guidance, this design extracts cross-attention weights and their backpropagated sensitivity gradients with respect to the positive ITM logit from block \(b=7\) of a pretrained BLIP encoder. To address unequal relevance across knowledge tokens and question tokens, an adaptive token reweighting scheme computes per-token spatial relevance, normalizes weights via temperature-controlled softmax (\(\tau=1.0\)) within the knowledge span \(I_K\) and question span \(I_Q\) independently, and balances the two groups with a group-level softmax factor. Instead of applying a brittle single global threshold across heterogeneous image regions, the module computes a token-wise \(\rho\)-percentile threshold (\(\rho=90\)) to generate binary patch masks \(B_{i,p}\), which are aggregated across all tokens via a logical OR operator: $\(B_{i,p} = \mathbb{I}\left[\hat{R}_{i,p} > \text{Quantile}(\hat{R}_{i,:}, \rho)\right], \quad M_p = \bigvee_{i=1}^L B_{i,p}\)$ The resulting patch mask \(M\) is upsampled to the original image resolution and visualized by setting non-salient background pixels to pure white, focusing the downstream visual backbone strictly on question-salient and knowledge-grounded entity regions.
2. Question-Conditioned Textual Keyphrase Selection: Pruning Low-Signal Distractors in Retrieved Passages To prevent noisy, paragraph-length retrieved documents from overwhelming the MLLM's working context with spurious associations, this design constructs a question-to-knowledge interaction matrix \(S\) from the ITM model's text self-attention probabilities modulated by positive gradient sensitivity. Each knowledge token \(j \in I_K\) is assigned an importance score \(s_j\) by averaging its interaction over all question tokens \(i \in I_Q\): $\(s_j = \frac{1}{|I_Q|} \sum_{i \in I_Q} S_{i,j}, \quad S = \frac{1}{H} \sum_{h=1}^H \left(A_h \odot \text{ReLU}(\nabla A_h)\right)\)$ The top-\(m\) tokens (\(m=30\)) with the highest scores are mapped back to character offsets in the raw knowledge text. To bridge fragmented sub-tokens and punctuation gaps, adjacent or overlapping spans separated by fewer than 3 characters are automatically merged into fluent textual phrases, retaining the top 10 merged spans as the high-precision keyphrase set \(k\) to serve as explicit structural anchors.
3. Explicit-Constrained Implicit Parametric Knowledge Distillation: Grounding Parametric Priors in Reliable Semantics To resolve the dilemma where zero-shot MLLM knowledge querying induces severe hallucinations while raw retrieved passages mislead attention, this design constrains implicit knowledge elicitation using the refined explicit package \(E = \{T, k, M\}\). A structured instruction explicitly prompts the frozen MLLM \(f_{\text{imp}}\) not to output the final answer directly, but rather to integrate the visual attention map, raw snippets, and extracted keywords to generate a concise 2โ5 sentence implicit knowledge paragraph \(U\). By restricting parametric memory activation within the grounded semantic envelope defined by \(E\), the model synthesizes missing commonsense priors without hallucinating unverifiable claims, which are subsequently routed into the final answer prediction step.
Benchmark & Evaluation Protocol¶
The method is comprehensively evaluated on two landmark knowledge-intensive multimodal benchmarks: Encyclopedic-VQA (E-VQA) test set (5,750 questions spanning 2 million Wikipedia articles and 16.7k fine-grained entities across single-hop, two-hop, and multi-answer queries, measured via BERT Matching / BEM) and the InfoSeek validation set (71,335 questions testing zero-shot generalization on non-overlapping entities Unseen-E and non-overlapping questions Unseen-Q, measured by VQA accuracy and relaxed numeric accuracy). Experiments are benchmarked across two contemporary open-source MLLM backbones, Qwen3-VL-8B and InternVL3-8B, alongside EchoSight for multimodal passage retrieval (\(k=5\)), maintaining all MLLM parameters strictly frozen during inference.
Key Experimental Results¶
Main Results¶
On both Encyclopedic-VQA and InfoSeek, MaS-VQA establishes new state-of-the-art accuracy among retrieval-augmented MLLM frameworks without requiring any training-time fine-tuning:
| Dataset | Split / Metric | MaS-VQA (Qwen3-VL-8B) | Prev. SOTA Baseline | Gain / Margin |
|---|---|---|---|---|
| E-VQA (Test) | Single-Hop (BEM) | 42.2 | 40.1 (VLM-PRF) | +2.1 |
| E-VQA (Test) | All (BEM) | 41.3 | 39.2 (VLM-PRF) | +2.1 |
| InfoSeek (Val) | Unseen-Q (Acc) | 43.7 | 43.5 (VLM-PRF) | +0.2 |
| InfoSeek (Val) | Unseen-E (Acc) | 43.9 | 42.1 (VLM-PRF) | +1.8 |
| InfoSeek (Val) | All (Acc) | 43.8 | 42.5 (VLM-PRF) | +1.3 |
| E-VQA (Test) | All (InternVL3-8B) | 40.1 | 39.2 (VLM-PRF) | +0.9 |
Ablation Study¶
Systematic ablations on Encyclopedic-VQA (test set, Qwen3-VL-8B backbone) confirm the individual and synergistic contributions of visual masking, phrase selection, and implicit knowledge distillation:
| Config | Attention Mask (\(M\)) | Phrase Selection (\(k\)) | Implicit Knowledge (\(U\)) | E-VQA Single-Hop | E-VQA All | Note |
|---|---|---|---|---|---|---|
| Visual Mask Only | โ | - | - | 39.8 | 36.8 | Localizes visual regions but suffers textual noise |
| Phrase Select Only | - | โ | - | 38.5 | 36.6 | Filters text distractors but lacks spatial focus |
| Explicit Joint Filtering | โ | โ | - | 40.9 | 38.4 | Strong cross-modal synergy (+1.6 over visual alone) |
| Implicit Only | - | - | โ | 40.0 | 35.9 | Parametric distillation distracted by raw noisy text |
| Full MaS-VQA | โ | โ | โ | 42.2 | 41.3 | Best performance; tight explicit-implicit co-modeling |
Key Findings¶
- Synergistic cross-modal explicit filtering: Deploying visual masking alone achieves 36.8 on E-VQA All, while phrase selection alone reaches 36.6. Combining them yields 38.4, confirming that visual spatial localization and textual semantic relevance provide deeply complementary signals that reinforce each other.
- Optimal retrieval depth trade-off: Scaling retrieved passages from \(k=1\) to \(k=5\) drives E-VQA All performance from 31.6 to 41.3. However, further expanding \(k\) to 7 slightly degrades performance to 41.1, demonstrating that excessive candidate volume introduces compounding textual noise that eventually overtakes coverage gains.
- Hyperparameter stability: Sensitivity analyses reveal that the visual threshold \(\rho=90\) yields peak accuracy (41.3), outperforming \(\rho=70\) (39.8, insufficient noise suppression) and \(\rho=95\) (38.1, over-pruning salient entities). Similarly, token count \(m=30\) strikes the optimal balance between information loss (\(m=10\), 37.9) and noise reintroduction (\(m=70\), 40.8).
Highlights & Insights¶
- Unified cross-modal evidence refinement: Rather than treating visual saliency and passage reranking as disconnected post-processing steps, MaS-VQA grounds both in a shared ITM cross-attention and gradient sensitivity space, ensuring fine-grained alignment between visible image patches and textual tokens.
- Explicit gating for implicit parametric memory: By constructing an explicit evidence scaffold before querying the MLLM's internal memory, the method prevents the model from hallucinating or drifting into irrelevant parametric associations, transforming unconstrained generation into bounded, fact-grounded reasoning.
- Inference-only plug-and-play architecture: In contrast to approaches requiring complex reinforcement learning or supervised parameter tuning (e.g., VLM-PRF, ReflectiVA, Prophet++), MaS-VQA achieves superior accuracy on frozen backbones with zero additional training parameters.
Limitations & Future Work¶
- Upper-bound ceiling from initial retrieval quality: The system fundamentally depends on the retriever capturing the target entity in top-\(k\) passages; when the underlying knowledge base lacks coverage or retrieval completely misses, downstream masking cannot recover the absent facts.
- Sensitivity to heuristic hyperparameter tuning: The percentile threshold \(\rho\), number of extracted tokens \(m\), and character merge distance require manual calibration, suggesting that dynamic, instance-adaptive thresholding could be a promising future direction.
- Monolingual evaluation scope: The current evaluation is conducted exclusively on English benchmarks and Wikipedia corpora; verifying cross-lingual entity transfer and multi-lingual knowledge retrieval remains an open research avenue.
Related Work & Insights¶
- vs. Explicit KB-VQA (KAT, RKVQA, EchoSight): Traditional explicit methods feed entire retrieved passages or raw patch embeddings directly to the reasoner, inducing attention distraction; MaS-VQA drastically compresses context into high-salience phrases and white-masked image regions.
- vs. Implicit MLLM Methods (PICa, Prophet, ASB): Implicit baselines rely on heuristic prompts or lightweight pre-answer generators that frequently hallucinate under long-tail queries; MaS-VQA strictly anchors implicit synthesis to the filtered explicit evidence package.
- vs. Trainable Reasoning Pipelines (ReflectiVA, VLM-PRF): Existing top performers rely on fine-tuning MLLM weights or training self-reflective tokens; MaS-VQA demonstrates that inference-time cross-modal evidence distillation alone can outperform fine-tuned baselines across both seen and unseen distributions.
Rating¶
- Novelty: โญโญโญโญ [Unifying visual patch masking and textual phrase extraction within an ITM gradient sensitivity space and using it to scaffold implicit generation is elegant and effective]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across E-VQA and InfoSeek with diverse backbones, fine-grained ablations, retrieval-depth sweeps, and qualitative case studies]
- Writing Quality: โญโญโญโญโญ [Clear motivation, mathematically rigorous formulation, highly cohesive prose, and well-structured empirical analysis]
- Value: โญโญโญโญโญ [Provides a training-free, highly practical blueprint for mitigating retrieval noise in knowledge-intensive multimodal reasoning]