Reinforcing Video Reasoning with Focused Thinking

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/longmalongma/TW-GRPO
Area: Vision-Language Reasoning (Video Reasoning)
Keywords: focused thinking, group relative policy optimization, token importance, multi-answer soft reward, question-answer inversion

TL;DR

TW-GRPO reallocates learning strength using within-group token distributional differences at aligned positions and grants partial credit to multi-answer predictions under a strict false-positive constraint, improving Qwen2.5-VL-7B's CLEVRER accuracy from 41.1% with matched GRPO to 50.4% using only 1K CLEVRER training samples, although gains do not extend to every video benchmark.

Background & Motivation

Video reasoning requires more than a plausible explanation: a model must extract the entities, actions, and temporal relationships that determine an answer from dynamic visual content. Methods such as Video-R1 already apply Group Relative Policy Optimization (GRPO) to video question answering, but final-answer rewards typically provide a shared learning signal across an entire chain-of-thought (CoT). Generic openings and concrete descriptions of objects colliding and separating consequently share the same sequence-level advantage despite their different informational value. The paper asks whether learning effort is being spent on boilerplate and repeated verification, rather than assuming that longer reasoning is necessarily better reasoning.

A second bottleneck comes from the question format. Single-answer questions usually provide only correct-or-incorrect rewards. When a task requires identifying multiple valid events, omitting one correct event should not necessarily be treated like getting every judgment wrong. However, simply switching to multiple correct answers while retaining exact-match rewards makes positive feedback harder to obtain. The authors therefore change the task format and reward design together, making partial correctness useful for learning while preventing a model from collecting credit by selecting every option.

These problems operate between responses and within responses, respectively: multi-answer soft rewards distinguish candidate responses to the same question, while token weighting determines which positions within a response receive stronger updates. CLEVRER counterfactual questions provide native multi-answer data; question-answer inversion extends this format to some single-answer datasets. Core Idea: model how much of an answer is correct separately from which tokens deserve stronger learning, using partially scored answer sets to rank responses and positional distributional differences to emphasize visual and logical evidence.

Method

Overall Architecture

The input consists of a video, a question, and candidate options; the output contains a reasoning trace and one or more selected answers. Qwen2.5-VL-7B is the backbone. Training first uses native multi-answer questions or constructs them through question-answer inversion, then samples a group of responses for each input. Multi-answer soft reward supplies response-level feedback, while token importance weighting uses the same group's predictive distributions to produce position-level weights. Both signals feed the policy update.

Here, focusing takes place in the training loss: it does not mean cropping the video, learning a visual attention mask, or invoking an inference-time verifier. Question-answer inversion is offline preprocessing; soft rewards require training labels; token weights depend on grouped sampling. At inference time, the trained model simply generates a response without access to correct answers or an additional scoring model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Training videos and single-answer questions"] --> QAI["Question-Answer Inversion"]
    QAI --> Sample["Sample a group of responses"]
    Native["Native multi-answer questions"] --> Sample
    Sample --> Reward["Multi-Answer Soft Reward"]
    Label["Training answer sets"] -.-> Reward
    Sample --> Weight["Token Importance Weighting"]
    Reward -->|Response-level advantages| Update["Policy update"]
    Weight -->|Position-level weights| Update
    Update -.-> Model["Trained model"]
    Test["Inference video and question"] --> Model
    Model --> Output["Reasoning trace and answer"]

Key Designs

1. Question-Answer Inversion: turn single-answer data into partially scorable answer sets

Question-Answer Inversion (QAI) negates a question and inverts its correct answer set. A question originally containing one correct option among five can then have several correct options. The authors also randomly remove some correct options to discourage a shortcut in which negation automatically triggers selecting many answers. This supplies multi-answer examples for the NExT-GQA and STAR data-construction experiments, but is not required in every TW-GRPO setting: CLEVRER already provides multi-answer counterfactual questions, and the main 4K Video-R1 setting does not activate multi-level soft rewards.

The critical assumption is that originally incorrect options become valid under the negated question. This is stronger than swapping labels, particularly because asking why something happened and why it did not happen does not always create logical complements. Random option removal can reduce format shortcuts but cannot guarantee factual validity. The authors use a model judge on 528 inverted samples drawn from 1,000 NExT-QA instances, reporting 86.0% validity and a 4.0% linguistic-artifact rate. A separate 50-example human review reports 96.0% validity. These findings support QAI as useful but noisy augmentation, not as a strictly lossless transformation.

2. Multi-Answer Soft Reward: reward omissions without rewarding false positives

The model predicts an answer set, which a rule-based scorer compares with the correct set. If every selected option is correct, the reward equals the fraction of correct options covered. Including any incorrect option instead makes the response's task reward zero. A model can therefore progress from conservatively selecting some correct answers toward recovering the full set, but cannot exploit the reward by selecting everything. This is neither ordinary intersection-over-union nor F1: it is stricter about false positives while permitting graded feedback for omissions.

The following rule follows the explicit prose accompanying Equation (7); \(P\) is the predicted set and \(G\) is the nonempty correct set.

\[ R_{\mathrm{soft}}(P,G)= \begin{cases} |P|/|G|, & P\subseteq G,\\ 0, & P\nsubseteq G. \end{cases} \]

When there is only one correct option, this reward naturally reduces to binary feedback. Its granularity comes from multiple correct options, not from interpreting the confidence of a single-answer prediction as a soft label. After scoring the group, GRPO still constructs relative advantages using the group's reward mean and standard deviation. This improves discrimination between responses; it does not certify every textual reasoning step. A response containing faulty reasoning but coincidentally selecting the correct set can still receive a high outcome reward.

3. Token Importance Weighting: use within-group disagreement to identify positions worth updating

For responses sampled from the same question, the authors align generation positions. At each position, they compute the KL divergence between each response's token predictive distribution and the group-average distribution, then sum these divergences into a positional disagreement score. This measures distributional differences rather than counting how often a word appears or using the predictive entropy of only one response. Generic openings tend to be similar across the group, whereas critical entities, actions, and logical choices are more likely to branch. The method treats high-disagreement positions as more informative and assigns them stronger training weights.

Responses differ in length. The paper fills missing positions with a uniform distribution over the vocabulary before computing the scores, then applies min-max normalization and a scaling hyperparameter. Padding avoids simply discarding shorter responses, but positional alignment is still not semantic alignment: the tenth token in two responses need not represent the same reasoning step. High disagreement is therefore an empirical proxy, not verified causal importance. Word clouds and trajectory examples show high weights concentrating on entity, collision, and temporal terms, supporting this interpretation without establishing that every highly weighted token is necessary for reasoning.

Implementation details require caution. In the local PDF text, Equation (5)'s layout is ambiguous relative to the accompanying claim that low-disagreement positions retain nonzero weights, and extraction also damages the clipping terms in Equation (6). Appendix E's normalization and padding details are absent from this cache. This note therefore preserves the verifiable computation flow without reconstructing the exact weighting formula, clipping-branch placement, or zero-denominator handling. The mechanism changes token-level contributions to policy optimization; it does not add a module to the visual encoder.

A Worked Example

Consider a training update using the paper's illustrative correct set B and D. Suppose four sampled responses select B; B and D; B and C; and C alone. Their rewards are 0.5, 1, 0, and 0. The four-response group is illustrative, not a reported sampling-group hyperparameter. Its reward mean is 0.375, so the first two responses receive positive relative advantages and the remaining two receive negative relative advantages. Strict exact-match rewards would instead give the first response zero as well.

Next, disagreement is computed across predictive distributions at corresponding generation positions. If a position concerning which two objects collide has high disagreement, its policy update receives greater weight, while shared boilerplate contributes less. Weighting does not turn an incorrect trajectory into a positive example: the response-level advantage still determines the update direction. This illustrates how the signals cooperate and why the method is not simply deleting low-weight words and answering again.

Loss & Training

Training retains GRPO's grouped sampling, relative advantages, and clipped policy-ratio formulation, incorporates positional importance into the token-level objective, and normalizes by the total number of output tokens. The preliminary standard-GRPO equation includes reference-policy KL regularization, but the cache does not unambiguously establish the exact combination of terms in the final objective. Reproduction should consult the authors' implementation and supplementary material rather than treating this workflow description as a line-by-line specification.

The main low-data setting uses 1,000 CLEVRER counterfactual training examples, 500 RL steps, and 2 NVIDIA H800 GPUs. The authors report about 4 hours for TW-GRPO versus about 6 hours for matched standard GRPO. Additional token-weight computation takes less than 0.01 seconds per step. This overhead describes weight computation alone, not the complexity of full training; the paper's claim of linear scaling with output length should likewise be restricted to this module.

A separate setting trains for 1,000 steps on 4,000 Video-R1 examples to compare against Video-R1-Zero. The authors explicitly state that these data contain only open-ended and single-answer questions, making multi-level soft rewards inactive. These results therefore primarily test token weighting rather than the full combination. Evaluation uses top_p = 0.001 and temperature = 0.01; the paper reports a batch size of 16 and follows the relevant Video-R1 prompting settings. Appendix D is absent from the cache, so exact benchmark splits, video-sampling parameters, and all training hyperparameters cannot be filled in from it.

Key Experimental Results

Main Results

Table 1 below selects results from the paper's Table 1. Values are accuracy (%); MMVU uses its multiple-choice portion, and VideoMME is evaluated without subtitles. CLEVRER overall accuracy is distinct from Soft Acc. below. All TW-GRPO rows use Qwen2.5-VL-7B. The 1K and 4K rows differ in data source and reward activation as well as sample count, so they are not a pure data-scaling ablation. The main text does not list the exact test-split names individually.

Model and training setting	CLEVRER	NExT-GQA	MMVU_mc	MVBench	TempCompass	VideoMME without subtitles
Qwen2.5-VL-7B, zero-shot	30.5	75.9	65.4	63.3	72.5	56.5
GRPO, 1K CLEVRER RL	41.1	75.2	65.1	62.8	71.9	55.9
TW-GRPO, 1K CLEVRER RL	50.4	76.1	65.8	63.3	73.3	55.1
Video-R1-Zero, 4K RL	Not reported	Not reported	63.8	60.4	70.9	53.8
Video-R1, 165K SFT + 4K RL	31.6	74.3	64.2	62.7	72.6	57.4
TW-GRPO, 4K Video-R1 RL	32.0	76.3	65.6	64.2	73.3	56.1

Under the matched 1K setting, CLEVRER improves by 9.3 percentage points, whereas VideoMME falls by 0.8 points. Against Video-R1-Zero with the matched 4K data setting, improvements on MMVU, MVBench, TempCompass, and VideoMME are 1.8, 3.8, 2.4, and 2.3 points, respectively, averaging approximately 2.6 percentage points. This 2.6-point figure must not be described as the average gain over full Video-R1, which additionally uses 165K SFT examples.

Ablation Study

Table 2 below selects matched configurations from the paper's Table 3, evaluated on CLEVRER and MMVU. CLEVRER overall and multi-answer Acc. measure exact correctness; multi-answer Soft Acc. averages the Equation (7) scores and expresses the result as a percentage. It is not another measure of output confidence.

Config	CLEVRER overall Acc.	Multi-answer Acc.	Multi-answer Soft Acc.	MMVU Acc.
GRPO, multi-answer training, fixed reward	41.1	32.3	55.4	65.1
TW-GRPO, multi-answer training, fixed reward	42.6	26.0	54.2	65.0
GRPO, multi-answer training, soft reward	41.2	28.1	57.6	64.6
TW-GRPO, single-answer training only	38.9	22.8	55.9	63.7
TW-GRPO, multi-answer training, soft reward	50.4	42.5	64.4	65.8

Key Findings

• Holding token weighting and multi-answer training fixed, replacing fixed rewards with soft rewards raises overall accuracy from 42.6 to 50.4, a 7.8-point gain. Nearby source prose calls this a soft-accuracy improvement, but these values belong to overall Acc.; this note follows the table header.
• Holding multi-answer training and soft rewards fixed, adding token weighting raises overall accuracy from 41.2 to 50.4. Soft rewards alone improve only from 41.1 to 41.2 over fixed rewards. This supports complementary effects rather than the claim that changing rewards alone is sufficient.
• Token weighting does not improve every subset monotonically: with fixed rewards, multi-answer accuracy falls from 32.3 to 26.0 even as overall accuracy rises from 41.1 to 42.6. Evidence for the complete combination is stronger than for either component alone.

Highlights & Insights

• Separate two forms of credit assignment. Answer sets determine a trajectory's quality relative to other trajectories, while positional disagreement determines where learning concentrates within it. This addresses video-reasoning bottlenecks more directly than merely tuning coefficients around a binary outcome reward.
• Keep an explicit anti-exploitation boundary in soft rewards. Allowing omissions but rejecting false positives provides graded feedback without an additional judge model. It also suggests that multi-label reward design should consider both learnability and opportunities for reward hacking.
• Offer a practical low-data starting point. The 1K-example, dual-GPU setup lowers the barrier to experimentation. The four-hour result remains an author-reported measurement and should not be extrapolated directly to other video lengths, hardware, or output budgets.

Limitations & Future Work

• Author-reported scope: The 4K Video-R1 setting does not activate multi-level rewards, and the main paper has no standalone limitations section. Its cited appendices are missing from the cache, so their hyperparameter sensitivity results, F1-reward comparisons, and complete implementation details have not been verified here.
• Assessment in this note: Distributional disagreement at the same position may reflect phrasing, length padding, or uncertainty rather than critical temporal reasoning. Semantic-step alignment and failure-trajectory analysis would test the proxy more strongly than word clouds.
• Assessment in this note: QAI negation introduces semantic noise, and model-judged and human-judged validity rates differ. The small human sample cannot establish reliability across all question types. Future audits could separate causal, descriptive, and counterfactual questions when measuring validity and shortcuts.
• Assessment in this note: Some benchmark results regress, and the main table provides no multi-seed variability. Neither a single-run gain nor shorter reasoning alone establishes stronger general reasoning. A fuller reproduction should jointly report performance, generation length, compute budget, and uncertainty.

Related Work & Insights

• vs Video-R1 / Video-R1-Zero: These methods focus on training video temporal reasoning, while this paper emphasizes token-level update allocation and answer-set rewards. The direct matched-4K comparison should use Video-R1-Zero; full Video-R1 includes an additional SFT stage.
• vs VideoChat-R1: Its video-grounding IoU reward exploits overlap between temporal intervals. TW-GRPO extends partial-correctness feedback to answer sets but applies a stricter rule that zeros the reward upon any false positive. The two soft rewards are not the same objective.
• vs TokUR and critical-token analysis: This work turns within-group positional distributional differences from an analysis signal into training weights. Reusing existing grouped samples is transferable; whether disagreement identifies useful information must still be validated for each new task.

Rating

• Novelty: 4/5. Combines positional weighting, multi-answer rewards, and data inversion into an interpretable training scheme, while building on established ideas.
• Experimental Thoroughness: 4/5. Six benchmarks and component ablations provide substantial evidence, but main-table variability and locally verifiable supplementary details are missing.
• Writing Quality: 3/5. The problem and workflow are clear, but the abstract's comparator, some metric descriptions, and extraction ambiguities require care.
• Value: 4/5. Useful for low-cost video VLM post-training, particularly for studying how response-level and token-level supervision interact.