Skip to content

QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding

Conference: ECCV 2026
Paper: ECCV Official
Area: Video Understanding
Keywords: keyframe selection, long video understanding, training-free, query-aware, frame budget allocation

TL;DR

QCA splits a fixed frame budget across temporal segments using two signals — query semantic matching and content deviation — and then, within each segment, anchors on the most query-relevant frame and greedily adds the frames farthest from the selected set; it is fully training-free and plugs into existing Video-LLMs, consistently beating uniform sampling and prior frame-selection methods on four long-video benchmarks across three MLLM backbones (with Qwen3-VL-8B at a 64-frame budget: 66.9 / 69.5 / 75.7 / 51.8 on LongVideoBench / Video-MME / MLVU / LVBench).

Background & Motivation

Feeding long videos to MLLMs hits a token wall first: frames must be turned into visual tokens before they reach the language model, and even at 1 fps an hour-long video yields several thousand frames, which inevitably blows past the context window — the input is either truncated or reasoning quality collapses. The standard workaround is to compress the video into a fixed frame budget (say 64 frames) before feeding it in, most simply by uniform sampling. But uniform sampling is query-agnostic: it picks the same frames no matter what the user asks, so the budget is easily spent on long static shots and repeated footage while the one or two seconds that actually answer the question are dropped. Fig. 1 in the paper makes this vivid: asked "what are these two shirtless men in white shorts doing?", uniform sampling sees running and answers "running race", while query-driven selection answers "wrestling match". A second family of methods scores each frame against the query and keeps the Top-k, or filters by attention. This does introduce query awareness, but its cost grows with video length, and more importantly, chasing "looks like the question" concentrates the selected frames on a few locally salient moments and discards the rest of the video — which is worse for questions that need global scene understanding.

The real tension is that "is this piece of video worth a frame?" has two distinct sources of signal that do not coincide. One is semantic relevance: whether the segment matches the question. The other is content salience: whether the segment is unusual within the whole video — densely packed events, frequent cuts, large visual change — which is often where the critical context lives (for instance, what happened before the queried object appeared). Use only the former and the selected frames clump together and lose coverage; use only the latter (clustering- and diversity-based methods) and selection becomes query-independent, possibly picking "representative" frames that are irrelevant to the question. Uniform sampling uses neither.

This paper therefore formulates the task as joint relevance–diversity modeling and frame allocation under a limited frame budget: each segment first estimates its own information contribution, the budget is split according to those contributions, and each segment then fills its quota while balancing relevance and diversity. The entire pipeline uses only an off-the-shelf image-text matching model (BLIP-2) as a scorer plus plain vector distances — no training, no extra parameters, no change to the MLLM. Core idea: decompose "which frames to select" into a three-stage pipeline — segment-level contribution scoring (query matching \(M_s\) plus content deviation \(D_s\)), contribution-weighted budget allocation, and intra-segment "semantic anchor + distance-greedy diversity expansion" — replacing query-agnostic uniform sampling with a training-free scoring preprocessor.

Method

Overall Architecture

QCA takes a query \(q\) and a frame sequence sampled at 1 fps, and outputs a keyframe subset of fixed size \(N'\) that is handed to a downstream Video-LLM to produce the answer; the whole pipeline is pure inference-time preprocessing with no learnable parameters. It proceeds in three steps. First, the video is split into \(S\) uniformly spaced temporal segments. Second, each segment receives an information contribution score — a weighted combination of the average image-text matching degree between its frames and the query, and its content deviation relative to the whole video — and these scores are normalized into weights that split the total budget \(N'\) into a per-segment quota \(q_s\) (with the shortfall from the floor operation redistributed to the highest-scoring segments). Third, inside each segment the most query-relevant frame serves as the semantic anchor, all sufficiently relevant frames form the candidate set, and the frame with the largest total distance to the already-selected set is repeatedly added until the segment's quota is filled. The selections from all segments are merged into the final set \(\mathcal{K}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Long video, frames at 1 fps"] --> B["Uniform temporal segmentation"]
    Q["Query q"] --> C["Segment contribution scoring"]
    B --> C
    C --> D["Budget allocation"]
    D --> E["Semantic anchor and relevance candidate set"]
    Q --> E
    E --> F["Distance-greedy diversity expansion"]
    F --> G["N' keyframes to Video-LLM"]

Key Designs

1. Segment contribution scoring: fusing "relevant to the question" and "unusual in content" into one score

Ranking purely by ITM score and taking the top-k spends the entire budget on a few locally salient frames, which hurts tasks such as Video-MME that need global scene understanding — in the paper's tables, QCA beats Top-k by 1.4 points on Video-MME with LLaVA-Video-7B. QCA instead computes two quantities per segment and fuses them. The first is the semantic matching degree \(M_s\), the mean ITM score between the segment's frames and the query (ITM is implemented with BLIP-2's image-text matching capability). The second is the content deviation \(D_s\), which sums two terms: the L2 distance between the segment's mean feature and the whole video's mean feature (how different this segment is from the video at large — the more different, the more likely it holds an information-dense event), and the trace of the segment's feature covariance matrix (how much visual variation occurs inside the segment):

\[M_s=\frac{1}{|\mathcal{X}_s|}\sum_{x_i\in\mathcal{X}_s}\mathrm{ITM}(x_i,q),\qquad D_s=\|\mu_{\mathcal{X}_s}-\mu_{\mathcal{X}}\|_2+\mathrm{Tr}(\Sigma_{\mathcal{X}_s})\]

Because the two terms have different scales, each is normalized first (softmax across segments), after which they are fused as \(c_s=\alpha M_s+\beta D_s\) with \(\alpha=\beta=0.5\) by default. The weighted sum makes the score complementary rather than winner-take-all: a segment that both matches the question and stands out from the rest of the video scores highest; a segment entirely unrelated to the question is not heavily rewarded even if its content is unique, while a relevant but visually plain segment still keeps a basic quota through \(M_s\). Ablations show that removing either term hurts, which is exactly the evidence for this complementarity.

2. Budget allocation: splitting frames by contribution weight, with the floor remainder going to top segments

The segment scores cannot be used directly as frame counts, because scores from different segments are not on a comparable scale and the total must equal the budget \(N'\) exactly. The paper converts contribution scores into weights that sum to one via a power normalization, \(w_s=c_s^{\tau}/\sum_{j=1}^{S}c_j^{\tau}\), where \(\tau\) controls how smooth the allocation is: a larger \(\tau\) approaches winner-take-all, a smaller one approaches an even split, and the default is \(\tau=0.5\). Each segment's quota is \(q_s=\lfloor N'\cdot w_s\rfloor\); since flooring leaves the total below \(N'\), the remaining frames are handed out one by one to the highest-scoring segments so that the final count matches the target budget exactly. The point is that the budget is not spread evenly: a long static stretch inside an hour-long video automatically loses its quota, which is transferred to event-dense or strongly query-relevant segments — precisely what uniform sampling cannot do. (⚠️ If a segment's weight is low enough that \(q_s=0\), it contributes no keyframes; the paper only specifies the rule for the floor-induced deficit and does not discuss a segment being skipped entirely, so refer to the original paper.)

3. Semantic anchor and relevance candidate set: fix the reference frame first, then narrow selection to highly relevant frames

The hazard of intra-segment selection is that pursuing diversity across the whole segment easily admits irrelevant frames (background boards from a cut, unrelated passers-by) and wastes the quota. QCA first forces the frame with the highest ITM score in the segment to be the semantic anchor \(\mathcal{K}_s=\{x^*\}\) — this sets a semantic reference for the whole segment, so that every subsequent comparison happens at the scale of "as relevant as this reference" — and then collects all frames whose ITM score is at least a fraction \(\gamma\) of the anchor score into the candidate set:

\[\mathcal{C}_s=\{x_j\in\mathcal{X}_s \mid \mathrm{ITM}(x_j,q)\geq\gamma\cdot R^*\},\qquad R^*=\max_{x_i\in\mathcal{X}_s}\mathrm{ITM}(x_i,q)\]

Diversity expansion afterwards happens only inside this candidate set. The threshold \(\gamma\) (0.7 by default) is an explicit relevance–diversity knob: raising it keeps the pool closer to the question but leaves fewer frames and limits diversity, while lowering it enlarges the pool at the cost of noisy frames. In the ablations, replacing the anchor with a random in-segment frame or removing the candidate set entirely degrades all four benchmarks (without the candidate set, MLVU drops from 75.8 to 72.4 and LVBench from 52.4 to 48.7), showing that "anchor first, then talk about diversity among relevant frames" is a mutually supporting pair rather than two independently removable pieces.

4. Distance-greedy diversity expansion: each round picks the candidate farthest from the selected set

The rest of the quota is filled with a plain greedy rule: repeatedly add the unselected candidate frame with the largest total distance to the current keyframe set, using Euclidean distance, until \(|\mathcal{K}_s|=q_s\):

\[\mathcal{K}_s\leftarrow\mathcal{K}_s\cup\Big\{\arg\max_{x_i\in\mathcal{C}_s\setminus\mathcal{K}_s}\sum_{x_j\in\mathcal{K}_s}\phi(x_i,x_j)\Big\}\]

This greedily maximizes marginal information gain: if a new frame closely resembles the already-selected ones, its added information is close to zero, so maximizing the total distance to the selected set amounts to covering visual content the current set has not yet captured. This is exactly where it differs from Top-k — Top-k takes the highest-ITM frames, which typically come from one continuous shot with near-identical content, collapsing diversity, whereas QCA deliberately spreads out inside the "relevant enough" pool. Replacing this step with plain top-matching in the ablations degrades every benchmark (MLVU 75.8→75.3, LVBench 52.4→49.2), confirming that explicit diversity modeling is not optional. The paper also adds a fallback: if a segment's candidate set initially cannot fill the quota (\(|\mathcal{K}_s|+|\mathcal{C}_s|<q_s\)), \(\gamma\) is reduced to widen the pool so the segment is not left short.

A Worked Example

Take a roughly one-hour video (about 3,600 frames at 1 fps; this is a schematic walk-through and the actual numbers vary with content): with \(S=12\) and budget \(N'=64\), each segment averages about 300 frames. The query is "what are these two shirtless men in white shorts doing?". After computing \(M_s\) and \(D_s\) per segment, suppose segment 4 (a ring, two people grappling) scores high on both and segment 9 (a long static corridor shot) scores low on both; after cross-segment normalization the former might weigh 0.15 and the latter 0.02, so on a 64-frame budget segment 4 receives about 9 frames and segment 9 only 1, with the floor remainder redistributed to the top-scoring segments so the total stays 64. Inside segment 4, the anchor frame with ITM 0.71 is selected first (the two just locking up), then with \(\gamma=0.7\) roughly 180 frames scoring ITM \(\geq\) 0.497 form the candidate set; each subsequent round picks the candidate farthest from the selected set, which likely lands on shots from the same scene but with very different viewpoint or pose (close-up, wide shot, referee entering frame) rather than nine near-identical consecutive frames. The result goes to the Video-LLM and yields "wrestling match", where uniform sampling on the same video answers "running race".

Loss & Training

QCA is entirely training-free: there is no loss function and no parameter is updated. The only "model" is the image-text matching model used as a scorer (BLIP-2 by default), which is called but never fine-tuned, so QCA can be attached directly in front of off-the-shelf Video-LLMs such as LLaVA-Video, InternVL-3.5, and Qwen3-VL as a preprocessor. All hyper-parameters are fixed and need no per-backbone retuning: \(\alpha=\beta=0.5\) (equal weight for semantic relevance and content deviation), softmax temperature \(\tau=0.5\), candidate threshold \(\gamma=0.7\), segment count \(S=12\), frame budget \(N'=64\) (the paper also varies \(N'\) for a budget–accuracy analysis). To keep scoring cheap, frames are first extracted at 1 fps before ITM is computed; experiments run on 8×A800 80G GPUs.

Key Experimental Results

Main Results

Comparison with uniform sampling and existing frame-selection methods at a 64-frame budget on four long-video benchmarks (paper Table 1; Top-k selects the frames with the highest ITM score; results for BOLT / FRAG / E-VRAG are in the original paper):

Backbone / Method LongVideoBench Video-MME MLVU LVBench
LLaVA-Video-7B + Uniform 58.9 64.4 70.8 41.9
LLaVA-Video-7B + Top-k 61.6 63.7 72.8 47.2
LLaVA-Video-7B + AKS 62.7 65.3 71.8 47.6
LLaVA-Video-7B + OneClip-RAG (trained) 62.5 65.2 71.2 -
LLaVA-Video-7B + QCA 62.9 66.1 74.1 48.9
InternVL-3.5-8B + Uniform 61.3 61.9 69.9 42.8
InternVL-3.5-8B + AKS 62.9 62.8 70.5 47.9
InternVL-3.5-8B + QCA 63.5 63.9 71.3 50.0
Qwen3-VL-8B + Uniform 63.1 67.6 71.0 43.8
Qwen3-VL-8B + Top-k 64.4 67.4 74.2 50.7
Qwen3-VL-8B + AKS 64.7 68.6 74.2 50.8
Qwen3-VL-8B + Q-Frame 65.8 67.9 74.7 50.7
Qwen3-VL-8B + QCA 66.9 69.5 75.7 51.8

The gains also hold on a stronger backbone: with Qwen3-VL-30B-A3B-Instruct, QCA lifts the 64-frame uniform baseline from 67.2/69.9/72.8/44.0 to 69.9/71.4/77.1/52.6. The abstract's headline comparison is QCA reaching 67.8% on LongVideoBench with 128 frames, versus GPT-4o at 66.7% with 256 frames.

Frame budget versus accuracy (paper Table 5, Qwen3-VL-8B; LongVideoBench is split into Med / Long / Avg subsets):

Method Frames LVB Med LVB Long LVB Avg MLVU LVBench
Uniform 64 65.0 52.8 63.1 71.0 43.8
QCA 16 63.1 56.6 62.7 71.3 46.3
QCA 32 66.0 57.4 64.5 73.3 48.9
QCA 64 67.2 59.2 66.9 75.9 52.4
QCA 128 69.7 59.4 67.8 76.8 53.2

Ablation Study

Component and setting ablations (paper Table 3, Qwen3-VL, 64 frames):

Config LongVideoBench Video-MME MLVU LVBench
full model 66.9 70.1 75.8 52.4
w/o \(D_s\) (semantic matching only) 66.3 70.4 75.4 51.0
w/o \(M_s\) (content deviation only) 65.8 69.5 75.1 51.1
w/o \(D_s,M_s\) (both removed) 66.0 68.3 75.2 51.2
w/o Anchor (random frame) 63.7 68.2 74.1 47.2
w/o Candidate Set (select over the whole segment) 64.5 69.0 72.4 48.7
w/o Diversity (top matching) 66.2 68.6 75.3 49.2
Uniform Selection 62.7 67.6 71.8 43.9

Key Findings

  • Semantic matching carries the most weight, and all three selection mechanisms are needed: removing both terms drops Video-MME from 70.1 to 68.3 and LongVideoBench from 66.9 to 65.8; yet keeping only semantic matching (w/o \(D_s\)) actually gives 70.4 on Video-MME, indicating that query awareness is the main driver and content deviation is more of a coverage-boosting addition. Among the three intra-segment pieces (anchor, candidate set, diversity), the costliest is the semantic anchor: replacing it with a random frame loses 3.2 points on LongVideoBench and 5.2 on LVBench — a strongly relevant reference frame matters more than spreading frames out.
  • Hyper-parameters show a clear inverted-U behavior: balanced \(\alpha,\beta\) (\(\beta=1-\alpha\)) works best; too small a \(\gamma\) admits noise into the candidate set while too large a one limits diversity; the segment count \(S\) first improves then degrades, with LongVideoBench rising from 66.0 at \(S=8\) to a peak of 66.9 at \(S=16\) and falling beyond that (too few segments means one segment spans several events and the statistics are over-averaged; too many means too few frames per segment and unstable statistics). The main text states a default of \(S=12\) while the ablation peak is at \(S=16\) — ⚠️ the two are not consistent, refer to the original paper. The authors also tried a naive dynamic strategy scaling \(S\) linearly with video length, which did not consistently beat a fixed \(S\).
  • Selection itself costs almost nothing: as input frames grow from 128 to 1024, ITM matching time rises from 0.595 s to 1.165 s, while the selection process itself takes about 0.005 s — negligible against the Video-LLM's cost of encoding dense visual tokens.
  • Frame efficiency is high, and the gains concentrate on questions needing long-range evidence: QCA with 32 frames already beats 64-frame uniform sampling (LongVideoBench 64.5 vs 63.1, MLVU 73.3 vs 71.0, LVBench 48.9 vs 43.8); at 16 frames it still beats 64-frame uniform sampling on MLVU/LVBench, and on the Long subset of LongVideoBench 16 frames (56.6) even exceeds 64-frame uniform sampling (52.8), while the Med subset at 63.1 is slightly below 65.0. Accuracy rises steadily as the budget grows from 16 to 128, indicating that extra frames do not bring severe redundancy.
  • The gains come from the selection strategy, not the encoder: swapping the scorer for CLIP, BLIP, or LongCLIP improves all of them, with BLIP best (66.9/70.1/75.8/51.8, i.e. +4.2/+2.5/+4.8/+8.0 over the respective baselines).
  • An advantage over token pruning: under the same 64-frame effective budget, QCA (Video-MME 66.1 / MLVU 74.1) beats ForestPrune pruning 75% of 256 frames (64.2 / 72.5). The authors attribute this to QCA choosing semantically information-dense frames before encoding, whereas token pruning operates after encoding and may discard tokens belonging to entire low-relevance frames.
  • ⚠️ Note on number consistency: in Table 1, "Qwen3-VL-8B + QCA" reports 69.5 on Video-MME, whereas Table 3 and Table 4 report 70.1 for the identical setting (64 frames, BLIP); the main text cites 70.1, matching the latter two. The tables above reproduce each source table as-is without harmonizing them — refer to the original paper.

Highlights & Insights

  • Reframing frame selection from ranking into allocation: prior work mostly scores each frame and sorts. QCA allocates the budget at the segment level first and selects within segments afterwards. As a result, a frame that would never crack the global top-64 can still be chosen as long as its segment holds a quota — this is the key to preserving both relevance and coverage at once.
  • Content deviation is captured by an almost free statistic: \(D_s\) uses only the distance between the segment mean and the global mean plus the trace of the in-segment covariance, requiring no extra saliency model or event detector to approximate "is this segment unusual within the video and does it change rapidly?". Replacing a learned saliency signal with a second-order statistic is cheap and reusable.
  • Two explicit knobs for the relevance–diversity tension: \(\gamma\) sets the relevance floor of the candidate set and \(\tau\) sets how skewed the budget allocation is, both mapping directly onto interpretable trade-offs rather than one implicit mixing weight. They are easy to tune in practice and easy to calibrate per benchmark.
  • Training-free and model-agnostic make it naturally transferable: no MLLM modification, no retraining, and no dependence on a specific encoder (CLIP, BLIP, and LongCLIP all work) mean it can serve as an inference-time plugin that swaps data without swapping models, stackable onto any frame-based Video-LLM. The same "estimate segment contributions, then allocate budget by contribution" recipe transfers to paragraph selection for long documents or image selection in multi-image inputs — any setting with a limited budget over structured candidates.

Limitations & Future Work

  • The authors acknowledge that temporal segmentation is the simplest possible uniform split. When event boundaries do not align with the uniform grid (a critical event straddling two segments, for instance), the \(M_s\) and \(D_s\) statistics get diluted. The authors list content-aware segmentation (e.g. shot boundary detection) as future work — which is arguably the most natural next step.
  • The method assumes image-text matching scores are reliable, and every signal comes from single-frame matching against the query with no temporal relation involved. If the decisive evidence is the ordering of an action ("what happened after he jumped over the steps?"), a single-frame ITM score may not reflect it; questions like the one in Fig. 4 are carried by content deviation and coverage rather than by a direct mechanism.
  • Keyframes are compared for dissimilarity by Euclidean distance, a weak proxy in feature space: frames that are semantically close but visually very different are misjudged as "different information", and vice versa. Learned similarity or an explicit temporal-adjacency constraint could help.
  • Evaluation covers four benchmarks, a fixed 1 fps pre-sampling, and a 64-frame budget for the most part; the 1 fps pre-sampling is itself lossy for very long videos (an hour compressed to 3,600 frames and then to 64), and the paper does not analyze what 1 fps misses.
  • One detail gap: the floor operation in budget allocation can leave some low-scoring segment with \(q_s=0\), and the paper only gives the deficit rule for topping up high-scoring segments without saying whether such entirely skipped segments hurt coverage.
  • vs Uniform Sampling: uniform sampling is query-agnostic with fixed temporal intervals and misses semantically critical moments (the running/wrestling distinction in Fig. 1). QCA lets the query signal decide where frames land, at the cost of one ITM pass (about 1.165 s for 1024 frames), which buys consistent gains on all four benchmarks and beats 64-frame uniform sampling with only 32 frames.
  • vs Top-k / relevance filtering (Q-Frame and similar): these rank frames by query similarity and keep the top ones — query-aware but relevance-only, with no notion of content structure, so selection concentrates on locally salient moments. QCA keeps a relevance floor through the candidate set but forces dispersion via distance greedy; the gaps on LongVideoBench and LVBench largely come from this step.
  • vs AKS / BOLT: AKS performs adaptive keyframe sampling and BOLT does training-free frame selection for long video, but neither explicitly separates "segment contribution → budget allocation → intra-segment selection" into three levels as QCA does; with Qwen3-VL-8B, QCA is 2.2 points above AKS on LongVideoBench and 2.0 points above Q-Frame on Video-MME, though it only edges AKS on MLVU (75.7 vs 74.2).
  • vs OneClip-RAG / FRAG / E-VRAG: these either rely on additional training (OneClip-RAG) or call in larger models (an LLM or a 7B MLLM) to pick frames. QCA trains nothing and introduces no large model, yet with LLaVA-Video-7B it gains +2.9 on MLVU over OneClip-RAG, +2.4 on Video-MME over FRAG, and +3.9 on MLVU over E-VRAG. This suggests the payoff in frame selection comes more from whether the structure of the selection is right than from how large the scoring model is.
  • vs token pruning such as ForestPrune: pruning drops tokens after encoding and may remove every token of a low-relevance frame; QCA decides before encoding which frames deserve budget, giving higher information density at the same effective budget (Video-MME 66.1 / MLVU 74.1 vs 64.2 / 72.5).

Rating

  • Novelty: ⭐⭐⭐ The problem framing is clean and the combination is sensible, but each component (ITM relevance, greedy intra-segment diversity, weight-proportional budget allocation) is an existing tool; the contribution lies mainly in organizing them into a training-free three-stage pipeline rather than proposing a new scoring primitive.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Four long-video benchmarks, three MLLM backbones, three VL embeddings, frame budgets from 16 to 128, component ablations plus an overhead analysis, and a same-effective-budget comparison against token pruning; it loses a star because the ablation tables report two different Video-MME numbers (69.5 and 70.1) for the same setting, and the default \(S\) disagrees with the optimal \(S\).
  • Writing Quality: ⭐⭐⭐⭐ The method and formulas are clearly presented and the figures are intuitive, but a few formulas (how the two terms of \(D_s\) are normalized, and where exactly the softmax normalization is applied) require the reader to reconstruct them, and the manuscript contains some garbled equations.
  • Value: ⭐⭐⭐⭐ Plug-and-play, training-free, and model-agnostic, beating 64-frame uniform sampling with 32 frames — very practical for deploying long-video MLLMs; the gains are 2–8 points and depend on an external scoring model, making this a solid, pragmatic engineering improvement.