Clue Matters: Empower Video Reasoning with Brain-Inspired Latent Clue Learning

Conference: ECCV2026
Paper: Official page · PDF
Area: Multimodal Reasoning / Video Question Answering
Keywords: Temporal quintuplets, clue cognition, adaptive clue filtering, two-stage supervision, visual compression

TL;DR

ClueNet converts video into temporally grounded entity-interaction clues, learns which clues are both relevant to the question and supported by the frames, and uses two-stage supervision to reach 77.6%, 84.3%, and 69.8% on STAR, NExT-QA, and MVBench; optional visual compression accelerates inference but is a different operating point from the highest-accuracy configuration.

Background & Motivation

Video question answering requires more than recognizing what appears in a frame: the model must connect events across time into evidence. Answering what someone closed after taking a bottle requires distinguishing the timing of taking, holding, and closing. Models such as VideoLLaMA3 and Qwen2.5-VL connect visual tokens to a language model and answer directly. They can describe a scene fluently while attending to the wrong person or mistaking a portrait poster for food, then build a coherent explanation around that initial error. The paper separates these failures into visual perception bias, clue cognition bias, and inductive reasoning bias.

CCoT and Video-of-Thought already introduce scene graphs or intermediate reasoning, but exposing intermediate text does not make it trustworthy. A clue may sound highly relevant yet have no support in the video; another may be visually correct but irrelevant to the question. A subtler problem arises when training supplies perfect annotated clues while inference relies on noisy self-generated ones. The answer model then learns to depend on an evidence quality that deployment cannot provide.

The paper draws an analogy with hierarchical visual cognition, from perception through structured interpretation to evidence integration, rather than implementing biological neural circuitry. Core idea: make temporally localized clues an explicit interface between video and answers, first teach clue extraction and clue use separately, then train filtering and reasoning on self-generated clues under answer supervision so that reasoning adapts to realistic evidence noise.

Method

Overall Architecture

The input is a video and a question; the output is a multiple-choice answer or an open-ended response. Inference proceeds through Keyframe Selection, the Clue Cognizer, the Adaptive Clue Filter, and optional Visual Compression, before interleaving refined clues with visual information for answer generation. The answer model still receives video information: the method does not replace the entire visual input with a textual scene graph.

Training follows two stages. Stage 1 alternates between clue generation and answering with ground-truth clues. Stage 2 switches to model-generated clues and optimizes filtering and answer generation using the answer loss. This training path explains how the Clue Cognizer and filter cooperate, rather than merely forming a pipeline of independently attached tools.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Video + question"] --> KS["Keyframe Selection"]
    KS --> CLUE["Clue Cognizer"]
    CLUE --> ACF["Adaptive Clue Filter"]
    ACF --> VC["Visual Compression<br/>optional"]
    KS --> VC
    VC --> OUTPUT["Interleaved visuals and clues<br/>Answer generation"]
    ACF --> OUTPUT

Key Designs

1. Keyframe Selection: remove repetition before preserving question-critical details

Uniform sampling can miss brief actions, whereas processing every frame with the full language model is expensive. Keyframe Selection, or KS, first extracts frame features using a pretrained ResNet-18 and applies LVNet's Temporal Scene Clustering, or TSC. Its dynamic thresholds group visually similar frames into a reduced candidate set. This step primarily removes repeated scenes; it does not yet claim to identify the evidence needed for an answer.

The second step introduces the question. spaCy extracts nouns and verbs, while the system also retains a representation of the full question; candidate frames are represented by patch features. Each frame is scored using both the maximum cosine similarity between the global question and its patches and the strongest keyword-to-patch alignment. Two weights combine these signals, and the top 16 frames are selected. Max pooling helps a small but crucial object avoid being diluted by a whole-frame average, while the global-question branch supplies context missing from isolated keywords. The cached equation is partially corrupted, so this note reports the recoverable mechanism without reconstructing its full expression or inventing coefficient values.

2. Clue Cognizer: express actions as trackable temporal quintuplets

The Clue Cognizer is not an additional object detector. It uses instruction prompting and supervised fine-tuning to make the multimodal model produce structured clues. Each clue records an interaction's temporal boundaries, subject, relation, and object:

\[ C_i=\langle t_{\mathrm{start}},t_{\mathrm{end}},s_i,r_i,o_i\rangle. \]

For example, <2-4> person take bottle states that a person takes a bottle during frames 2 through 4, rather than merely asserting that a bottle exists somewhere in the video. The complete collection is updated incrementally across video segments to track the evolution of entity states and relations. Relative to a static scene graph, the essential addition is a validity interval for each interaction. This enables temporal ordering and lets the filter revisit the corresponding video segment for visual support.

Stage 1 supervises clue generation with timestamp-aligned scene-annotation quintuplets. A separate task supervises answers conditioned on the full ground-truth clues, question, and video. Alternating the two tasks by batch first establishes the abilities to extract structured evidence and to derive answers from it. Despite the word latent in the title, the generated clues are inspectable textual quintuplets, not exclusively unreadable hidden reasoning states.

3. Adaptive Clue Filter: test relevance and visual support separately

The two Stage 1 tasks do not solve inference-time omissions, recognition errors, or irrelevant relations. Stage 2 therefore starts with candidate clues produced by the Clue Cognizer and assigns each a gate through the Adaptive Clue Filter, or ACF. The semantic-relevance branch compares averaged final-layer language-model token representations of the question and clue. The visual-faithfulness branch compares the clue with averaged visual-token representations from frames within its temporal interval. Each branch uses a linear layer, LayerNorm, and ReLU; the branch outputs are added and passed through a sigmoid to obtain the gate.

Gating does not simply delete a sentence. It scales the input embeddings of every token within a clue, producing a differentiable soft filter. The answer model receives these weighted clues, the video, and the question, with answer cross-entropy providing an external task signal. An L1 sparsity penalty on the gates discourages the degenerate solution of assigning high weight to every clue. This creates a mechanism to downweight both plausible but visually unsupported explanations and real but irrelevant observations.

This is not a guarantee of factual verification. Answer supervision constrains task performance, not independently labeled truth for each clue; the visual and language representations also come from the model system. The paper argues that answer labels prevent a purely self-scoring feedback loop, but this does not establish that incorrect evidence can never survive the filter. Likewise, differentiable gating does not by itself make the discrete generation of clue text differentiable end to end.

4. Visual Compression: let filtered clues determine where complete visual detail survives

Once the filter has estimated evidence utility, Visual Compression, or VC, reuses its visual-faithfulness scores to determine which frames retain all their visual tokens. When several clues cover a frame, the method averages the products of their gate weights and frame-specific support scores. The frame-retention score in Equation 7 is:

\[ S_{\mathrm{ret}}(V_t)=\frac{1}{|\mathcal M_t|}\sum_{i\in\mathcal M_t}g_i\,s_v^{(i,t)}. \]

Here, \(\mathcal M_t\) contains candidate clues whose temporal intervals cover frame \(t\), \(g_i\) is a clue gate, and \(s_v^{(i,t)}\) is the visual-faithfulness score between that clue and frame. Frames reaching the threshold are retained in full; lower-scoring frames undergo adaptive average pooling instead of outright deletion, preserving some background context. If every frame falls below the threshold, the highest-scoring frame remains uncompressed. The main text does not specify behavior when \(|\mathcal M_t|=0\), so no implementation default is assumed here.

The order matters: assess evidence first, then compress visual redundancy outside it. This reduces the risk of removing details before understanding their relevance to the question. Compression is nevertheless not lossless, as the ablation shows a small accuracy decrease. It is best understood as an evidence-guided efficiency option rather than a free performance improvement.

A Worked Example

Figure 1 asks which object the person closed after taking the bottle. Its clues include <2-4> person take bottle, <5-10> person holding bottle, and <6-8> person closing refrigerator. Closing the refrigerator occurs after the taking interval, so the answer should identify the refrigerator rather than the frequently mentioned bottle.

The Clue Cognizer provides interactions and temporal intervals, ACF evaluates relevance and visual support, and VC decides which corresponding frames need full detail. The paper does not publish gate values for this example, so no numerical filtering thresholds or clue-removal counts are invented. This is an illustration of evidence flow, not a measured execution trace.

Loss & Training

The backbone is VideoLLaMA3, with a frozen SigLIP-SO400M visual encoder and a language model initialized from Qwen2.5-7B-Instruct. Stage 1 alternates autoregressive clue-generation cross-entropy with answer cross-entropy conditioned on ground-truth clues. Stage 2 uses self-generated, softly gated clues and optimizes answer cross-entropy plus an L1 gate penalty averaged over the number of clues, weighted by \(\lambda\).

The paper also interprets the method through the information bottleneck principle: preserve information useful for the answer while reducing redundant visual information. It does not actually estimate mutual information and use it as the training loss, so the conceptual interpretation should not be mistaken for an established sufficient-statistic property. Full hyperparameters, further theoretical arguments, and additional experiments are deferred to appendices. The available cache contains the main paper and references but not those appendices, so learning rates, epoch counts, \(\lambda\), and the VC threshold are not supplied here.

Key Experimental Results

Main Results

The following values come from Tables 1 through 3. The first three columns report accuracy percentages; the last two report WUPS scores on open-ended NExT-OE and are not ordinary accuracy. WUPS evaluates semantic similarity using a lexical taxonomy. [email protected] scales matches below the 0.9 similarity threshold by 0.1, making it stricter than WUPS@0.

Model	STAR	NExT-QA	MVBench	NExT-OE WUPS@0	NExT-OE [email protected]
InternVideo2.5	69.1	80.6	68.7	28.4	22.0
Qwen2.5-VL	68.4	81.5	65.2	30.6	24.6
VideoLLaMA3	72.1	81.3	68.3	32.3	25.7
ClueNet	77.6	84.3	69.8	35.5	29.2

Against VideoLLaMA3, the gains are 5.5 percentage points on STAR and 3.0 on NExT-QA; on MVBench, the gain over InternVideo2.5 is 1.1 points. The paper gives VideoLLaMA3 one round of ordinary SFT under identical training settings. Except for Flipped-VQA, ViLA, and VideoChat2, whose numbers are taken from their papers, the other open-source baselines are reimplemented with at most 16 input frames. Pretraining data and model sizes still differ, so this is not a strictly controlled single-module replacement experiment.

Ablation Study

Table 4 most clearly separates the highest-accuracy configuration from the compressed one. Time and GFLOPs below are transcribed as reported. The baseline has no comparable inference-cost entry, so these values cannot establish a speedup over bare VideoLLaMA3.

Configuration	STAR	NExT-QA	Inference time / s	GFLOPs
VideoLLaMA3	72.1	81.3	Not reported	Not reported
Stage 1 only	70.8	80.6	6.5060	54,197.96
Stage 1 + Stage 2	75.1	83.3	5.9963	54,026.85
Add KS	75.4	83.4	6.0443	54,075.02
Add ACF, without VC	77.6	84.3	4.9963	51,355.89
Add VC	77.3	84.1	3.9298	40,622.84

Stage 1 alone underperforms the baseline, showing that learning to answer with perfect clues does not imply robustness to self-generated evidence. Adding Stage 2 raises STAR from 70.8 to 75.1, a 4.3-point improvement; 3.0 points is the net gain relative to the baseline. ACF adds 2.2 points over the preceding KS configuration. The main text's claim of 2.5 points does not match that adjacent-row comparison, so this note follows the table.

Key Findings

• Relative to ACF without compression, VC reduces GFLOPs by approximately 20.9% and inference time by approximately 21.3%, at a cost of 0.3 points on STAR and 0.2 on NExT-QA. This trade-off is internal to the same clue-based pipeline.
• On STAR Feasibility, ClueNet scores 66.7 versus InternVideo2.5's 68.0. Prioritizing observable evidence is not necessarily optimal for questions requiring unobserved commonsense or possibility judgments.
• Descriptive NExT-OE [email protected] rises from VideoLLaMA3's 48.7 to 60.5, but semantic answer scores are not direct measurements of hallucination frequency.
• Figure 7's oracle results, baseline values, and gain annotations are not consistent with the main comparison table. They are not used here as precise improvement estimates. The text mentions VideoMME, VideoHallucer, and an additional experiment skipping Stage 1, but full results are in the unavailable appendices and are not reconstructed into another empirical table.

Highlights & Insights

• The evidence format supports both reasoning and visual compression. Temporal intervals make clues readable while also providing a localization interface for visual-support checks and frame-retention scoring.
• Two-stage training explicitly exposes a distribution mismatch in intermediate representations. The accuracy drop from Stage 1 alone is more explanatory than merely reporting that the full system improves.
• ACF separates relevance to the question from support in the video. This suggests an analogous separation between retrieval relevance and evidence validity in retrieval-augmented QA, although that transfer would require its own experiments.

Limitations & Future Work

• The authors identify a potential cost of evidence-first filtering for feasibility questions requiring unobservable commonsense. A possible extension is to distinguish visual evidence from external commonsense assumptions instead of blending both into one clue weight.
• The method depends on temporally aligned scene-graph annotations. The main text says NExT-QA and STAR provide annotations converted into quintuplets, but the available material does not fully detail their provenance, conversion quality, or cost.
• Incorrect clues may still find support in the model's own representations, and answer loss does not guarantee clue-level faithfulness. Independent clue-accuracy, temporal-localization, and counterfactual evidence-intervention tests would help establish whether the explanations are trustworthy.
• KS ultimately retains only 16 frames, so brief actions can still disappear upstream. VC's treatment of frames covered by no clues and implementation details for visual-textual feature alignment also need fuller specification.
• The cache lacks appendices, some extracted equations are corrupted, and the oracle figure is inconsistent with the main comparison. The available evidence does not suffice to reproduce all training details, confirm hardware-specific timing conditions, or comprehensively verify hallucination-reduction claims.

Related Work & Insights

• vs CCoT / Video-of-Thought: All introduce explicit intermediate structures. ClueNet emphasizes temporal localization, clue filtering, and supervised adaptation to self-generated evidence rather than simply adding a textual reasoning chain.
• vs LVNet: KS reuses its TSC redundancy-reduction strategy before refining selection with the question and keywords. The borrowed sampling foundation should be distinguished from the paper's clue-learning contributions.
• vs visual token pruning: Many approaches prune visual features directly using attention or similarity. ClueNet first extracts clues and then uses evidence support to determine compression, at the cost of additional clue generation and filtering.

Rating

• Novelty: 4/5. Temporal scene clues are not entirely new, but the combination of two-stage supervision, dual-branch filtering, and evidence-guided compression addresses a specific failure mode.
• Experimental Thoroughness: 3/5. Multiple-choice, open-ended, and transfer evaluations are supported by informative negative ablations; missing appendices, limited independent faithfulness evidence, and reporting inconsistencies constrain verification.
• Writing Quality: 3/5. The motivation and main pipeline are clear, but ablation gains, oracle comparisons, and implementation edge cases need clarification.
• Value: 4/5. Useful for video QA requiring traceable evidence, particularly the principle of training downstream reasoning on self-generated intermediate representations.