ReQuest: Rethinking-based Question-Aware Frame Selection for Long-Form Video QA¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://geppa.github.io/ReQuest
Area: Video Understanding
Keywords: keyframe selection / long-form video QA / multimodal large language model / uncertainty-driven routing / adaptive NMS

TL;DR¶

ReQuest distills a lightweight selector from an MLLM's own answers to estimate how much visual evidence each frame contributes, then reuses the prediction entropy of the first inference pass to decide both whether a second thinking round is warranted and how densely frames should be sampled, improving long-video QA accuracy without fine-tuning the underlying MLLM and spending the extra inference only on questions the model is genuinely uncertain about.

Background & Motivation¶

The first hard constraint in long-form video QA is the visual token budget: feeding an entire video into a multimodal large language model (MLLM) is impractical, and as the frame count grows the resolution available to each frame shrinks, so fine detail is lost even faster. Practice therefore falls back to uniform sampling — picking K frames at a fixed interval. But allocating that budget is delicate: sample too few frames and the decisive moment may fall entirely between sampling points; sample too many and redundant frames crowd out resolution and dilute the useful evidence. Running Qwen3-VL with input frames swept from 256 up to 2048, accuracy peaks at 512 frames (70.0%) and drops back to 68.7%/67.7% at 1024/2048 — more frames do not mean more evidence, the most direct empirical statement of the needle-in-a-haystack problem this paper targets.

The established remedy is question-conditioned frame selection: compute frame–question similarity with a vision-language encoder such as CLIP or SigLIP and retrieve the apparently relevant frames, as in BOLT, MDP3, Q-Frame, and AKS. These methods are cheap but share a structural blind spot — they rely on the literal similarity between visual content and question text. When the object needed to answer never appears in the question, the similarity ranking degrades: on the 400 Video-MME questions whose key objects are absent from the question, SigLIP-feature similarity selection reaches only 48.0%, below uniform sampling at 51.3%. The other route hands selection to an MLLM itself (Frame-Voyager, MLLM-based frame selection). It understands questions better but costs too much, forcing a multi-stage sparse-sampling compromise (e.g. 3600 → 128 → 32 frames) — once the first sparse stage misses the evidence, no amount of later refinement recovers it. More fundamentally, the words in a long-video question carry uneven relevance to the answer, so a plain frame–text cosine similarity is not the true contribution of a frame to the correct reasoning trajectory.

The tension is therefore sharp: understanding question intent requires MLLM-level capability, while scanning the whole video globally requires something cheap enough to run per frame, and existing designs cannot have both. This paper's angle is not to use an MLLM as the selector but to distill the MLLM's judgement of "is this footage useful" into a lightweight BLIP-level selector, turning question understanding from one expensive inference into one cheap forward pass; and it accepts a premise — not every question needs careful selection, since re-selecting frames for a question the model already answers confidently on uniform frames is pure waste. Core idea: use the difference between an MLLM's probability on the correct option when shown only a given video segment and when given no visual input at all as a pseudo-label for that segment's visual contribution, train a lightweight selector on it, and drive routing with the prediction entropy of the first pass — triggering the selector and a second inference only when entropy exceeds a length-corrected threshold, while the same entropy decides the suppression interval of frame sampling.

Method¶

Overall Architecture¶

ReQuest is a plug-and-play, three-stage pipeline in which the answering MLLM stays frozen throughout. The input is a long video plus a multiple-choice question: (1) a first-thinking pass runs the MLLM on frames sampled uniformly at 1 fps and computes the prediction entropy from the option probability distribution; (2) Re-thinking Routing compares that entropy against a length-corrected threshold and decides whether the question is answered directly from the uniform frames or enters the re-thinking stage; (3) in the re-thinking stage the question-aware selector scores every frame in the video for its visual contribution, and adaptive NMS sampling rescales the suppression interval with the same entropy, picks K frames, and sends them back to the MLLM for a second inference. Only the selector — a four-layer transformer plus a scoring head — is trained, and even its supervision is generated offline by the MLLM itself; the router and the sampler contain no learned parameters.

It is worth being explicit about what "re-thinking" actually re-thinks: it does not iteratively correct a relevance estimate, nor loop for several rounds. It revises the evidence once: the first pass reasoned over a uniformly sampled set that may have missed the decisive moment, and re-thinking performs a global scan to recover the frames that genuinely contribute to answering this question, substituting a new evidence set for the MLLM to answer from. Whether that happens at all is decided by the model's own uncertainty.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["long video + question<br/>1 fps uniform sampling"] --> B["first thinking<br/>MLLM option probs and entropy"]
    B --> C["Re-thinking Routing<br/>entropy vs length-corrected threshold"]
    C -->|confident| D["answer directly"]
    C -->|uncertain| E["question-aware selector<br/>per-frame visual contribution"]
    E --> F["adaptive NMS sampling<br/>entropy-scaled suppression"]
    F --> G["second thinking<br/>MLLM answers from K frames"]

Key Designs¶

1. Re-thinking Routing: pay for a second round only when the model is genuinely uncertain

Not every question deserves careful selection: for a question the model already answers decisively on uniform frames, re-selecting frames only burns computation, whereas questions with sparse cues that require aggregating evidence across distant moments leave the option probabilities visibly flattened. ReQuest simply reads the entropy of the first pass's option distribution as its uncertainty:

\[u = -\sum_{k=1}^{M} p_k \log p_k\]

where M is the number of options. Comparing this entropy against a threshold is enough to route, but a fixed threshold fails across video lengths: the longer the video, the smaller the fraction a 1 fps uniform sample covers, so entropy inflated by "evidence never seen" mixes into entropy caused by "this question is hard". The paper therefore adds a length-aware correction: the video's frame count N (capped at \(N_{\max}\)) is normalized into \([0,1]\) as \(r_{\text{len}}(N)\), and used to modulate the base threshold \(\tau_0\) into an effective threshold \(\tau_{\mathrm{eff}}\) — stricter for short videos, where the model is unlikely to be uncertain, and more permissive for long ones. ⚠️ Equations (3)(4) are incomplete in the cached text due to OCR; \(\tau_{\mathrm{eff}}=\tau_0\,(1-\gamma_{\text{len}} r_{\text{len}}(N))\) is reconstructed from the paper's description, while \(\gamma_{\text{len}}=0.3\) and \(\tau_0 \in \{0.55, 0.45, 0.3\}\) are as given — refer to the original paper for the exact form.

The routing decision itself carries no learned parameters: \(u > \tau_{\mathrm{eff}}\) takes the selection branch, otherwise the uniform-frame answer is returned. This rule-based design has a practical payoff — the router needs no training and only one threshold must be calibrated when moving to a new benchmark. The price is that the threshold must be tuned per benchmark: on Video-MME the sweep is clean, with all questions routed to selection (2700/0) giving 65.3%, balanced routing (1365 routed, 1335 not) the best 65.6%, and no routing at all (0/2700) falling back to 62.6%; on Qwen3-VL-8B, routing lifts the all-selection setting from 69.9% to 71.1% while cutting average latency from 52.1s to 39.3s. This single module thus guards both the accuracy ceiling and the compute floor.

2. Question-aware selector: distilling "which frame actually matters" from the MLLM's own answers

The core difficulty of this design is where the supervision comes from — using frame–question similarity as the label would only inherit the blind spot of similarity methods. ReQuest instead has the MLLM set its own exam: the video is clustered into segments with FINCH, each segment is fed to the MLLM as visual input to read off the probability of the correct option \(p_{i,g}\), and the same question is asked once more with the visual input replaced by an all-zero dummy tensor to obtain a baseline distribution \(p_{\text{base}}\). Subtracting the two gives the segment's contribution score:

\[\Delta c_i = p_{i,g} - p_{\text{base},g}\]

This subtraction carries the meaning of the whole supervision signal: \(p_{\text{base},g}\) encodes the textual prior — how likely the correct answer is from question text alone, without looking at any pixels — so what remains after removing it is the amount of visual evidence the footage genuinely supplies. The consequence of skipping the baseline is concrete: questions answerable from common sense or language priors would score highly on every segment, lifting the dynamic range of the targets and washing out their discriminative power, and such questions are far from rare in the 178K corpus.

The selector is deliberately lightweight and deliberately non-dual-encoder: frames and the question go through BLIP together, where question tokens attend to visual tokens via cross-attention inside the text-side fusion layers, yielding frame–question fused tokens. During training the fused tokens of all frames in a segment are temporally pooled into a segment-level sequence, a learnable score token is prepended, and the sequence goes through a four-layer transformer encoder whose score token's final hidden state is passed to a light scoring head that regresses \(\Delta c_i\). A key train/inference asymmetry hides here: training operates at segment level (cheap labels, efficient coverage of long videos) while inference operates at frame level, scoring each frame against the question separately. The ablation shows the asymmetry pays off — with the same trained weights, merely switching inference from segment level to frame level lifts Video-MME overall accuracy from 62.9% to 65.6% and the long-video subset from 53.3% to 55.8%, because pooling a segment into a single representation averages away fine-grained cues inside it, and the decisive moment usually lasts only a few seconds. Question information enters the score explicitly (question tokens interact with visual tokens inside one fusion stream), which is why the selector attains 53.0% on the 400 questions whose key objects are missing, surpassing uniform sampling (51.3%) and both similarity selectors (48.0% / 51.0%).

3. Adaptive NMS sampling: letting uncertainty control both where and how densely to select

Once per-frame contribution scores exist, the question becomes how to pick frames along the timeline. Plain top-K keeps selecting neighbouring frames and crams all K frames into one moment; fixed-width greedy NMS (take the current highest score, suppress the window \([i-\delta, i+\delta]\), repeat until K frames are chosen) removes redundancy but its suppression width is hard-coded and cannot adapt to question difficulty. ReQuest lets the first pass's entropy set that width: a base interval \(b_s = N/K\) is defined (N is the total frame count after 1 fps sampling and K the number of frames finally fed to the MLLM), the entropy is clipped into \([u_{\min}, u_{\max}]\) and normalized into \(w\), yielding a scaling coefficient \(s(u)\) and an effective suppression interval \(\delta\):

\[w = \frac{\mathrm{clip}(u;\,u_{\min},u_{\max}) - u_{\min}}{u_{\max}-u_{\min}}, \qquad s(u) = s_{\min} + (1-w)(1-s_{\min}), \qquad \delta = \mathrm{round}\!\left(s(u)\cdot b_s\right)\]

The higher the entropy (the less certain the model), the closer \(w\) is to 1 and \(s(u)\) to its floor \(s_{\min}\), so the suppression window narrows, the selected frames cluster more tightly in time, and the budget concentrates on the suspicious neighbourhood; below \(u_{\min}\) we simply set \(s(u)=1\) and the scheme degrades to globally spaced, near-uniform sampling. Since the entropy was already computed in design 1, this step adds no computation and no learned parameters. The ablation shows "adaptive" is not decoration: with the same selector scores, top-K reaches 64.0%, fixed-width greedy NMS 64.7%, and adaptive NMS 65.6%, with nearly all of the gap coming from the long-video subset (53.0 / 53.3 → 55.8%) — the longer the video, the less a fixed suppression width can satisfy the conflicting demands of "locally dense where the event is" and "globally spread out". Note that this module controls where the budget is spent, not how much of it there is: the frame cap K is fixed in advance and the entropy only reshapes how those K frames are distributed over time.

A Worked Example¶

Take one Video-MME long video (the subset averages 17 minutes): 1 fps sampling yields N = 1020 frames and the budget is K = 32. The first thinking pass runs on uniform frames, producing option probabilities and an entropy of, say, u = 1.02. With \(\tau_{\mathrm{eff}} \approx 0.55\) (lower still for long videos after length correction), the router declares the model uncertain and enters re-thinking; for another question whose entropy is only 0.31, the first-pass answer is returned directly and no extra compute is spent at all.

In the re-thinking stage the selector scores all 1020 frames and the base interval is \(b_s = 1020/32 \approx 32\) frames. Suppose this question's entropy is u = 1.0; with \(u_{\min}=0.45\), \(u_{\max}=1.0\), \(s_{\min}=0.2\) we get w = 1 and \(s(u)=0.2\), so the suppression interval is \(\delta = \mathrm{round}(0.2 \times 32) = 6\) frames — greedy NMS may now take frames no closer than 6 frames apart and packs all 32 inside the few-second window that actually contains the answer. Had the entropy been u = 0.5 instead, we would get w ≈ 0.09, \(s(u) \approx 0.93\) and δ ≈ 30, making the sampling almost a uniform take-one-every-30-frames sweep over the whole video. The 32 selected frames plus the question are then fed to the MLLM for the second and final answer.

Loss & Training¶

The selector is trained so that its predicted score \(\hat c_i\) approaches the pseudo target \(\Delta c_i\) using Smooth-\(\ell_1\) (\(\beta = 0.5\)) plus a pairwise ranking loss that also supervises the ordering between two randomly sampled segments (the paper reports weights \(\alpha = 0.3\), \(\tau = 0.25\), \(\gamma_{\text{loss}} = 2.0\); ⚠️ the exact form of the ranking term is incomplete in the cached text — refer to the original paper). The ranking term matters because the selector is ultimately only ever used to decide which frames are relatively more worth looking at, so the relative ordering matters far more than the calibration of absolute scores.

Supervision data is built in three steps: 808K question-answer pairs are taken from the multiple-choice split of LLaVA-Video-178K and filtered down to the 169K "visually grounded" samples; each video is segmented at 1 fps with FINCH (second hierarchy level) and \(\Delta c_i\) is generated per segment; finally only questions whose \(\Delta c\) distribution has a clear peak (threshold 0.20) are kept, discarding noisy questions where all segments contribute alike. This yields 73,468 training questions and 803,956 segment-level samples. Optimization uses AdamW with batch size 256 and 2,048 randomly sampled segment pairs per batch.

Inference hyper-parameters are K = 32, \(u_{\min} = 0.45\), \(u_{\max} = 1.0\), \(s_{\min} = 0.2\), \(N_{\max} = 3600\), \(\gamma_{\text{len}} = 0.3\), with the base threshold \(\tau_0\) set to 0.55 (Video-MME), 0.45 (MLVU), and 0.3 (LongVideoBench) per benchmark. The underlying MLLM is never fine-tuned; the selector is trained once and then plugged in as-is, and the results on LLaVA-OneVision and Qwen3-VL are zero-shot transfers of a selector trained with LLaVA-Video supervision.

Key Experimental Results¶

Main Results¶

Evaluation covers three long-video benchmarks: MLVU (2,174 questions, 12-minute average), LongVideoBench validation (1,337 QAs, 8-minute average), and Video-MME without subtitles (2,700 QAs, 17-minute average). LLaVA-Video is used both for pseudo-label generation and for the main evaluation.

Model (#Frames)	LVB	MLVU	Video-MME Overall	Short	Medium	Long
LLaVA-Video 7B (32)	58.0	64.7	62.6	76.2	59.3	52.2
+ ReQuest	60.1	71.7	65.6	77.0	64.1	55.8
LLaVA-OneVision 7B (32)	56.6	63.1	58.7	70.3	56.6	49.2
+ ReQuest*	60.2	68.8	60.9	71.7	58.8	52.3
Qwen3-VL 8B (512)	62.7	74.0	70.0	78.6	70.1	61.2
*+ ReQuest (≤512)**	66.3	76.2	71.1	80.0	70.8	62.4

* denotes zero-shot transfer of a selector trained with LLaVA-Video supervision. For reference, the frame-selection/compression methods already in the table report: Frame-Voyager (7B, 8 frames) MLVU 65.6; LongVU (7B) MLVU 65.4, Video-MME 60.9; NVILA (8B, 1024 frames) MLVU 70.1, Video-MME 64.0; VideoTree MLVU 60.4; LLoVi MLVU 55.1.

The long-video subset is the most telling: with uniform sampling Qwen3-VL's accuracy rises then falls as the frame budget grows, whereas ReQuest surpasses every uniform configuration using far fewer than 512 frames.

Qwen3-VL input frames	Overall	Short	Medium	Long	Avg latency (s)
2048	67.7	78.6	68.9	55.6	53.8
1024	68.7	78.6	69.0	58.6	41.6
768	69.1	78.6	69.0	59.9	36.4
512	70.0	78.6	70.1	61.2	33.2
256	69.0	78.6	67.8	60.6	24.7
ReQuest (40/208/369 frames for short/medium/long)	71.1	80.0	70.8	62.4	39.3

ReQuest keeps the frames scoring in the top 50%/40%/15% (short/medium/long) of each video's score distribution; because of this it beats even the 2048-frame uniform setting by 3.4 points while running at lower latency.

Ablation Study¶

Sampling strategy and inference granularity (all on LLaVA-Video + ReQuest, Video-MME):

Config	Overall	Short	Medium	Long	Note
Top-k selection	64.0	76.7	62.3	53.0	no temporal de-duplication
Fixed-width greedy NMS	64.7	77.0	63.7	53.3	suppression width does not adapt
Adaptive NMS	65.6	77.0	64.1	55.8	full model; gains almost entirely on long videos
Segment-level inference	62.9	74.8	60.4	53.3	training and inference both segment level
Frame-level inference	65.6	77.0	64.1	55.8	segment-level training + frame-level inference

Supervision source (same pipeline, only the pseudo-label source varies):

Method	Selection backbone	Training supervision	Overall	Short	Medium	Long
Feature-based Sim (training-free)	SigLIP	–	61.6	74.8	57.8	52.1
Model-based Sim (training-free)	BLIP ITM	–	65.0	76.2	64.0	54.9
Selector	SigLIP	Feature-based Sim	61.7	75.6	57.2	52.3
Selector	BLIP	Model-based Sim	63.6	76.0	61.8	53.0
Selector	SigLIP	MLLM Response (Ours)	64.7	76.2	61.7	56.1
Selector	BLIP	MLLM Response (Ours)	65.6	77.0	64.1	55.8

Router threshold analysis (Video-MME, LLaVA-Video + ReQuest; #Selection/#Uniform is how many questions are routed to each branch):

#Selection	#Uniform	Accuracy (%)
2700	0	65.3
1782	918	65.4
1365	1335	65.6
1051	1649	65.0
567	2133	64.0
0	2700	62.6

Key Findings¶

Gains grow with video length, matching the motivation: on LLaVA-Video the short/medium/long subsets improve by 0.8 / 4.8 / 3.6 points (77.0 / 64.1 / 55.8). Uniform sampling already covers most of a short video, so headroom there is inherently limited; long videos are where evidence localization genuinely fails. ⚠️ Section 4.2 of the text states "gains on MLVU (66.7 → 73.9)" while Table 1 reports MLVU 64.7 → 71.7; the table numbers are recorded here — refer to the original table.
On long videos the sampling strategy contributes as much as the selector: fixed-width greedy NMS and adaptive NMS share the same selector scores and differ by only 0.9 points overall, but by 2.5 points on the long subset (53.3 → 55.8). Scoring well is only half the job; how the budget is distributed over the timeline is equally decisive.
Supervision quality caps the selector: swapping pseudo-labels from feature similarity to MLLM responses lifts BLIP from 63.6% to 65.6% and SigLIP from 61.7% to 64.7%; conversely, training a learned selector on similarity supervision (61.7% / 63.6%) does not even beat the corresponding training-free similarity baseline (61.6% / 65.0%) — changing the architecture without changing the supervision is wasted work. BLIP is slightly ahead of SigLIP (65.6 vs 64.7), which the authors attribute to BLIP performing cross-modal interaction inside the text-side fusion layers.
Routing is a directly tunable cost–accuracy knob: all-selection (65.3%) and no-selection (62.6%) both trail balanced routing (65.6%), and all-selection costs more; at τ_eff = 0 all 2,700 questions enter re-thinking, at τ_eff = 1 all use uniform sampling. On Qwen3-VL-8B, routing lifts 69.9% to 71.1% while reducing average latency from 52.1s to 39.3s.
The gain comes from keyframes, not from re-inference randomness: re-inferring only on the subset uniform sampling got wrong and merging the statistics, uniform sampling re-run with a new random seed gives 62.8%/62.7% (Router-Ours / Router-Oracle), whereas router-guided re-inference gives 65.6%/71.4%. The oracle-routed ceiling of 71.4% also shows the current router leaves room for improvement.
Decoupling training and inference granularity is worthwhile: segment-level inference 62.9% → frame-level inference 65.6%, since pooling a segment averages away fine-grained cues within those few seconds.
The cost structure deserves attention: on 900 long Video-MME questions, ReQuest (LLaVA-Video) takes 102.3s and 428.7 TFLOPs in total against 7.5s / 98.8 TFLOPs for the uniform baseline; the cost is dominated by full-video decoding at 1 fps (89.6s, 256.7 TFLOPs), while the selector itself accounts for only 0.3s / 4.2 TFLOPs. Against plain similarity selection (137.0s / 448.5 TFLOPs, long-video accuracy 54.3%), ReQuest reaches 55.8% at lower latency and compute, confirming that routing avoids unnecessary dense observation (only 640 of the 900 questions enter re-thinking; 390/900 for Qwen3-VL-8B).
Open-ended QA benefits too: replacing uncertainty with the maximum entropy over generated token distributions drives the same routing and sampling, lifting factual accuracy on open-ended Video-MME from 2.77 to 3.00 (overall +8%, long videos +11%), so the method is not confined to multiple choice.
It scales: on Qwen3-VL-32B accuracy rises from 73.9% to 75.7% (short/medium/long 81.6→82.8, 73.7→76.6, 66.5→67.8), so the benefit persists as the answering MLLM grows stronger.

Highlights & Insights¶

Letting the MLLM grade itself: \(\Delta c_i = p_{i,g} - p_{\text{base},g}\) turns "how much is this footage worth" into a computable quantity requiring no human annotation and no extra model. Its clever part is the zero-visual baseline — explicitly subtracting the textual prior leaves only the visual contribution; without it, common-sense questions would score every segment highly and the targets would lose discriminative power. The recipe transfers verbatim to any setting with multiple-choice supervision and splittable evidence.
One uncertainty signal used twice: the same prediction entropy decides both whether to re-think (routing) and how densely to select (NMS suppression interval). These look like two different problems, yet both are resolved from a scalar already computed, at essentially zero overhead, leaving the whole pipeline with a single interpretable knob.
Decoupled training and inference granularity: segment-level training solves the cost of asking the MLLM for per-frame labels, while frame-level inference avoids the cue-averaging of pooling. This "supervise coarsely, infer finely" pattern transfers to any dense prediction task where annotation cost scales with resolution, such as evidence-sentence selection in long documents or event localization in long audio.
Improving what is fed to the MLLM rather than the MLLM itself: the answering model stays frozen and only a four-layer transformer selector is trained, and it transfers zero-shot to other MLLMs. This "controllable periphery, untouched core" approach is easier to deploy in practice — swapping the base model does not require retraining the pipeline, only regenerating the selector's supervision.
A counter-intuitive conclusion about frame budgets: Qwen3-VL can swallow 2048 frames, yet accuracy peaks at 512; ReQuest exceeds the 2048-frame setting on long videos with roughly 369 frames. The intuition that a high-capacity MLLM should simply be fed more frames is wrong: the bottleneck is effective evidence density, not the frame cap.

Limitations & Future Work¶

The second-round savings are eaten by decoding cost: the cost table shows 89.6s of the 102.3s total (about 88%) is 1 fps full-video decoding, while the selector accounts for only 0.3s. The method does save MLLM forward passes on samples that do not need dense observation, but it still decodes the complete 1 fps frame sequence for every sample, and the authors offer no plan to reduce that (for example coarse low-resolution pre-screening, decoding only samples routed to re-thinking, or partial decoding around I-frames).
The router needs per-benchmark threshold tuning: \(\tau_0\) is 0.55 / 0.45 / 0.3 on the three benchmarks and no automatic calibration is given, which is a real usage cost when transferring to a new benchmark with a different video-length distribution; \(\gamma_{\text{len}}\) is likewise fixed at 0.3.
The selector still needs one supervised training round: although the answering MLLM stays frozen, the selector's supervision depends on the 808K→169K→73K filtering pipeline and on running the MLLM over the whole corpus offline, and it is matched to the target MLLM's distribution (cross-model results are marked * as zero-shot transfers, and the gain on LLaVA-OneVision's MLVU is smaller than on LLaVA-Video itself). The \(\Delta c\) peak filter discards a substantial fraction of questions (169K → 73K), hinting that very long questions requiring aggregation across segments rarely yield clean labels.
A single re-thinking round with no fallback: routing is a binary decision, and once triggered the evidence is re-selected only once — there is no "still uncertain, re-think again" mechanism and no fallback when re-thinking fails. Router-Oracle reaches 71.4% against the actual router's 65.6% in Table 6, indicating that a meaningful share of questions routed as "confident" are in fact wrong, so router quality is the clearest headroom.
The main evaluation remains multiple choice: open-ended QA is covered only by a maximum-entropy heuristic and only on Video-MME (2.77 → 3.00), with no validation on benchmarks dominated by open-ended questions.
Improvement directions: turn the binary re-think decision into entropy-banded sampling budgets (the more uncertain, the more frames) so that K itself adapts per question; use agreement between first-pass and post-re-think answers as self-verification, triggering another round or abstaining on disagreement; and replace regression on \(\Delta c\) with ranking learning directly on "do the selected frames let the MLLM answer correctly".

vs similarity-driven frame selection (BOLT / MDP3 / Q-Frame / AKS): they select frames by CLIP / SigLIP cosine similarity or policy-based retrieval, training-free and extremely cheap. The difference is that similarity reflects only the literal proximity between visual content and question text, so the ranking is unreliable when the key object is absent — on the 400-question control, SigLIP similarity reaches only 48.0%, below uniform sampling's 51.3%. ReQuest supervises the selector with MLLM-response contribution scores and reaches 53.0% on those questions; the price is one offline training round, while inference cost stays comparable (0.3s).
vs Frame-Voyager / MLLM-based frame selection: they build relative rankings of frame combinations from a Video-LLM's prediction loss and train a model to pick frames — semantically stronger, but the compute forces a multi-stage sparse pipeline (3600 → 128 → 32) that hard-trades global exploration against efficiency, and evidence missed by the first stage is unrecoverable. ReQuest keeps dense 1 fps observation (no sparse pre-filtering) and compresses "question understanding" into a cheap BLIP + four-layer transformer selector to make global scanning feasible.
vs hierarchical / agent-style selection (VideoTree / VideoAgent / LLoVi / SeViLA): these let an LLM or MLLM agent ask questions layer by layer and localize recursively, semantically strong but requiring many model calls; ReQuest distills question understanding into the selector once, so inference adds only one selector forward pass plus one MLLM pass, with a much flatter cost curve.
vs LVNet: LVNet is a training-free hierarchical keyframe selector targeting the same redundancy problem in long-form video QA. ReQuest keeps the training-free parts (routing and adaptive NMS learn nothing) and assigns only the hardest judgement — "which frame is useful" — to a supervised lightweight selector, trading one training round for question-conditioned selection.
vs simply enlarging the frame budget (LongVU / NVILA and other high-frame routes): that route relies on compression or larger frame capacity, whereas ReQuest's result (roughly 369 frames beating 2048 frames on long videos) shows that under a fixed visual token budget evidence density matters more than frame capacity — a direct counter-argument to any "just feed more frames" method.

Rating¶

Novelty: ⭐⭐⭐⭐ Using the MLLM's own answer-probability difference as frame-level visual-contribution supervision, and one entropy signal driving both routing and sampling density, is a clean and reusable combination; each component (distilled selector, entropy routing, NMS sampling) has prior art on its own.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Three long-video benchmarks, three backbones including 32B, cost–accuracy comparison, randomness control, and ablations over granularity and supervision source, plus the key frame-budget sweep.
Writing Quality: ⭐⭐⭐⭐ The motivation chain matches the ablation design and formulas are restrained, but Equations (3)(4) are not clearly presented and the MLVU figures in Section 4.2 disagree with Table 1 — both need revision.
Value: ⭐⭐⭐⭐ Plug-and-play improvement without touching the base model, with a clear use case for high-frame-capacity MLLMs; the main discount is the unsolved 1 fps full-video decoding overhead.