Skip to content

VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting

Conference: ECCV 2026
Paper: ECCV 2026 official page Β· Project page
Area: Multimodal VLM / Video Understanding
Keywords: video reasoning, visual prompting, reinforcement learning, self-distillation, spatio-temporal grounding

TL;DR

VisionCoach turns visual prompting from an inference-time tool into a training-time coach: a VP-Selector picks one prompt type (red circle / darkening / frame numbering, etc.) for hard samples only, overlays it on key frames and re-samples trajectories, and self-distillation internalizes those high-reward trajectories into the policy, while an object-aware spatial grounding reward keeps object identity consistent β€” so at inference the model takes a single forward pass over the raw video, improving +15.0 mAM / +25.1 mLGM over Qwen2.5-VL-7B on V-STAR and surpassing GPT-4o.

Background & Motivation

Video reasoning requires a model to localize and track question-relevant evidence across frames. The recent line of reinforcement learning with verifiable rewards (Video-R1, VideoRFT, and others) has indeed pushed answer accuracy up, but spatio-temporal grounding during the reasoning process remains unreliable. Each of the three existing families fails in its own way. Text-centric reasoning models fabricate explanations from language priors β€” the evidence is simply not in the video. Tool-calling approaches (VideoZoomer, Conan, Video-o3) repeatedly crop, zoom, and retrieve at inference time; they can recover the evidence, but the reasoning model itself never learns to look, so it degrades as soon as the tools are removed, and every extra call adds latency. The recent grounded reasoning methods that interleave grounding with reasoning inside one model (Open-o3-video being the strongest) go one step further, yet they still emit wrong object references and hallucinated boxes, and these errors propagate along the reasoning chain all the way to the final answer.

The root cause is the absence of a mechanism that enforces alignment between intermediate reasoning steps and spatio-temporal evidence. In practice, improving grounding means one of two things: scaling up training data with dense spatio-temporal annotations, which is extremely expensive to label, or attaching extra perception modules at inference time. Both are ultimately heavier external intervention: the model's own perception behavior does not change β€” it is merely fed more data or propped up by more tools. This paper moves the intervention from the data side and the inference side to the training side: give visual guidance only during training, bake the resulting improvement in spatial-temporal grounding into the parameters, and keep the lightest possible path at inference.

The idea rests on two small analyses. First, on a PerceptionTest subset, correctly answered samples consistently show higher temporal match rate, object name match rate, and box IoU than incorrectly answered ones β€” grounding quality is not a by-product of answering but a precondition for it. Second, different visual prompts produce wildly different results for the same model: Qwen2.5-VL scores 52.2 on the raw key frame, drops to 43.3 with darkening, is essentially unchanged by a red circle (51.5), and reaches 70.8 when every sample gets its own best prompt (the oracle setting; Gemini-2.5-Flash shows the same pattern at 47.4 / 50.4 / 49.3 β†’ 75.2). Core idea: since choosing the right prompt matters far more than giving a prompt at all, first distill a VP-Selector from the behavior of several proxy reasoners so it can pick prompts for hard samples, then let the ST-Reasoner re-sample on prompt-augmented inputs and use self-distillation to internalize the grounding gains β€” ending up with an inference pipeline that uses neither prompts nor tools.

Method

Overall Architecture

VisionCoach takes a raw video \(x\) and a question \(q\) as input and produces a reasoning trajectory with explicit spatio-temporal annotations (<obj> / <box> / <t>) plus the final answer. Training has two stages. First the VP-Selector is trained offline: several proxy reasoners are run over candidate prompts to turn "which prompt is best" into supervision labels, and a small VLM is LoRA-tuned as an |V|-way classifier; once trained it is frozen. Then GSPO reinforcement learning trains the ST-Reasoner. During RL, each sample first gets G initial rollouts whose average reward decides whether it is a hard sample; only hard samples invoke the VP-Selector, which picks one prompt type that is overlaid on the key frames, together with a textual hint appended to the question, and G new trajectories are sampled. Comparing the two rounds, the top-2 trajectories whose reward exceeds the initial average and whose answering reward is highest are selected for token-level NLL self-distillation, which is added to the GSPO loss. At inference, the VP-Selector, the visual prompt, and the textual hint are all removed: the model performs a single forward pass directly over the raw video.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["video + question"] --> B["VP-Selector<br/>trained offline then frozen"]
    B --> C["hard-sample detection + prompt-guided rollout<br/>prompt applied only below the reward threshold"]
    C --> D["self-distillation<br/>NLL internalizes high-reward trajectories"]
    D --> E["object-aware spatial grounding reward<br/>identity consistency + multi-region IoU"]
    E --> F["GSPO policy update"]
    F --> G["inference: single forward pass<br/>raw video, no prompt"]

Key Designs

1. VP-Selector: turning "which visual prompt to give" into a supervised classification problem

Prompt effectiveness depends heavily on the input β€” on the same key frame, darkening destroys Qwen2.5-VL's evidence (52.2 β†’ 43.3) while a red circle barely changes anything (51.5), and the ordering is different again for Gemini-2.5-Flash (darkening 50.4 beats the red circle 49.3). So "which prompt to give" is itself something to predict per input, and it has no gold standard: which prompt is better depends on the downstream model's reaction and cannot be labeled a priori. The solution is to define the label from the behavior of several proxy reasoners. The candidate pool \(\mathcal{V}\) contains four visual prompt types β€” red circles, attention-based prompts, frame numbering, and darkening. For each candidate \(v_k\), a prompted input \(x'_k\) is built and run through multiple proxy reasoners (Qwen3-VL, Gemini 2.5, GPT-4o and the like); their binary answer accuracy \(A_k\) and grounding score \(G_k\) are averaged across reasoners, and the candidate with the largest product is taken as the optimal prompt for that \((x, q)\):

\[v_i = \arg\max_{v_k \in \mathcal{V}} (A_k \cdot G_k)\]

(⚠️ Equation (3) is garbled in the cached text; whether "Β·" denotes a product should be checked against the original.) With \((x, q, v_i)\) triples in hand, training reduces to ordinary |V|-way classification: \((x,q)\) is formatted as an instruction to choose one method from \(\mathcal{V}\), and the lightweight VLM is trained with LoRA to emit the corresponding prompt label as a short response under token-level cross-entropy. Using proxy reasoners rather than training a dedicated scorer is what makes the labels reflect a cross-model notion of prompt effectiveness instead of overfitting to one backbone β€” and the plug-and-play experiment in Tab. 7 shows this worked: the trained selector improves Qwen2.5-VL, GPT-4o, and Gemini-2.5-Flash alike.

2. Hard-sample detection and prompt-guided rollout: help only when help is due

Applying prompts indiscriminately is wrong. In the ablation, fixing the prompt to darkening (the VP-F row) drags V-STAR mAM from 30.4 down to 29.7, because it destroys evidence that was already intact on easy questions; on genuinely hard samples, in contrast, prompting yields a relative reward gain of +56% to +66%. The question is therefore not whether to prompt but whom to prompt. Mechanically, each sample \((x_i, q_i)\) first gets G rollouts under the current policy, whose rewards average to \(\bar R_i\); if \(\bar R_i < k\) (with \(k\) a predefined hard-sample threshold) the sample is marked hard and the indicator \(I_i = 1\), otherwise it follows the plain GSPO branch. The decision uses the full four-term reward, so "hard" covers both wrong answers and poor grounding rather than wrong answers alone. Only hard samples query the frozen VP-Selector for \(v_i\); the prompt is applied to the key frames of \(x_i\) to form \(x'_i\), and a sentence describing the prompt is appended to the question to form \(q'_i\). That textual hint is easy to overlook but necessary β€” otherwise the model only sees a red circle or a globally darkened frame and has no way to know it is an intentional guidance signal. G new trajectories are then sampled from \((x'_i, q'_i)\). The whole procedure is input-level perception control: no architecture change and no extra parameters, just amplifying question-relevant evidence and suppressing distractors in pixel space β€” a different route from implicit feature-space attention modulation.

3. Self-distillation: internalizing the gains that prompting produced

Prompts exist only during training; if the model merely answers better on prompted inputs, nothing changes at inference. Self-distillation addresses exactly that, by filtering across the two rollout rounds. Writing the prompted-trajectory rewards as \(\{R'^{(g)}_i\}\), the candidate set is \(C_i = \{g : R'^{(g)}_i > \bar R_i\}\) β€” the rollouts that beat the initial average. If \(C_i\) is empty (prompting did not help) the sample is simply skipped; otherwise the candidates are further ranked by answering reward, the top-\(N\) (the paper uses top-2) form \(\mathcal{S}_i\), and a token-level negative log-likelihood is applied to them:

\[\mathcal{L}_{\mathrm{SD}}(\theta) = -\frac{1}{|\mathcal{S}_i|}\sum_{j\in\mathcal{S}_i}\frac{1}{|y'_{i,j}|}\sum_{t=1}^{|y'_{i,j}|}\log \pi_\theta\big(y'_{i,j,t} \mid y'_{i,j,<t}, x'_i, q'_i\big)\]

The "teacher" here is the model's own sampling on the prompted input β€” no external model is involved. Note that the conditioning input in this loss is the prompted \(x'_i\), so "internalize" is best read as a behavioral description: on one hand the good trajectories written while looking at the prompt are reinforced as positive samples, and on the other hand the GSPO term for the same samples still updates on the raw input, and the two together migrate that grounded reasoning pattern onto raw, prompt-free video β€” the paper calls this "self-evolving". ⚠️ The mechanism is supported only by the behavioral evidence that removing the prompt at inference does not hurt, without an ablation that varies whether the distillation loss is conditioned on the prompted or the raw input.

4. Object-aware spatial grounding reward: claim an object, and you must box that object

The remaining rewards (accuracy, format, temporal) follow Open-o3-video; the spatial grounding reward is this paper's addition. The earlier formulation scored only the single predicted box with the maximum IoU and never checked whether the predicted object name corresponds to that box, which taught the model to emit several boxes so that one of them would land well, together with object-agnostic hallucinated boxes. The new reward requires that every claimed object be boxed in its own place. Concretely, spatio-temporal triples (object name \(o_m\), timestamp \(t_m\), box \(b_m\)) are parsed from the reasoning text, and soft identity matching is applied: the predicted name must exactly match a ground-truth name or be a substring of it (or vice versa, as in car vs black car), while mismatched pairings such as the black-car location assigned to "white van" in Fig. 2 score 0. For each matched object, the predicted box is scored against every ground-truth box in the temporally aligned key frame \(\tilde t_m\), taking the maximum IoU; the average is then computed only over the predictions that are both temporally aligned (\(\Delta t_m \le \tau\)) and identity-consistent, i.e. the set \(\mathcal{I}\):

\[r_{\text{spa}}(x,y)=\frac{1}{|\mathcal{I}|}\sum_{m\in\mathcal{I}}\frac{1}{|\mathcal{O}^{\text{match}}_m|}\sum_{o\in\mathcal{O}^{\text{match}}_m}\max_{b^{\text{gt}}\in\mathcal{B}^{\text{gt}}(\tilde t_m)}\mathrm{IoU}(b_m,b^{\text{gt}})\]

(⚠️ Equation (8) is garbled in the cached text; this is rewritten from the surrounding prose, and the exact symbol meanings should be checked against the original.) The temporal gate \(\Delta t_m \le \tau\) is not decoration: if the timestamp itself is wrong, it is moot which frame the box lands on, and without the gate a prediction that is "boxed correctly but at the wrong time" would count as a spatial gain, handing the model a shortcut around temporal localization. Averaging over all matched regions rather than taking the maximum removes the payoff of single-box guessing and turns every claimed object into a promise that must be kept.

Loss & Training

The total reward sums four terms:

\[r(x,y) = r_{\text{acc}}(x,y) + r_{\text{fmt}}(x,y) + r_{\text{tmp}}(x,y) + r_{\text{spa}}(x,y)\]

The accuracy reward \(r_{\text{acc}}\) adapts to the task format: binary correctness for multiple-choice questions, ROUGE textual similarity for open-ended questions, visual IoU between predicted and ground-truth boxes for spatial grounding tasks, and temporal IoU between predicted and ground-truth intervals for temporal grounding tasks β€” one unified signal covering heterogeneous supervision. The format reward \(r_{\text{fmt}}\) is binary and checks that the <think> / <answer> tags and the <obj> / <box> / <t> annotations are well formed, giving 1 when they are. The temporal reward \(r_{\text{tmp}}\) follows Open-o3-video: each of the M generated timestamps is matched to its nearest annotated temporal position, scoring a full 1 if it falls inside a ground-truth interval and otherwise decaying as \(\exp(-\Delta t^2 / 2\sigma^2)\) with the deviation \(\Delta t\) from the nearest ground-truth position; the M values are then averaged. All four rewards are group-normalized across rollouts to compute the advantages used for the GSPO update.

As for the training recipe, the ST-Reasoner keeps the standard two-stage pipeline of SFT cold-start initialization followed by RL with GSPO. The VP-Selector is trained separately on the training splits of TVQA+ and VideoEspresso as a LoRA classifier and stays frozen throughout the RL stage. Self-distillation uses top-2 candidates with weight \(\alpha = 0.1\), and the hard-sample threshold \(k\) is predefined (its ablation lives in the appendix). For a fair comparison the SFT and RL data of the ST-Reasoner match those of Open-o3-video. Inference keeps a single forward pass with no prompt and no tool; the latency comparison is measured on one NVIDIA RTX 6000 (40GB).

Key Experimental Results

Main Results

The main arena for spatio-temporal reasoning is V-STAR (What accuracy plus When/Where IoU on two chains; mAM and mLGM are the aggregate metrics):

Model What Acc When Chain1 When Chain2 Where Chain1 Where Chain2 mAM mLGM
GPT-4o 60.8 16.7 12.8 6.5 3.0 26.8 38.2
Gemini-2-Flash 53.0 24.5 23.8 4.6 2.2 26.9 35.6
Conan (tool-calling) 54.5 14.4 14.9 4.4 0.6 23.9 32.4
Video-o3 (tool-calling) 29.7 5.5 23.1 0.5 0.0 13.3 15.4
Qwen2.5-VL-7B 33.5 15.4 13.8 17.0 2.5 19.3 22.4
Video-LLaMA3 41.9 23.0 23.1 0.9 0.2 21.7 27.0
LLaVA-Video 49.5 10.5 12.2 1.9 1.3 20.8 27.3
VideoChat-R1.5 49.7 14.4 5.0 14.8 1.5 22.5 29.3
Open-o3-video (best prior tool-free) 60.2 25.0 24.5 24.8 5.9 33.4 46.0
VisionCoach (Ours) 61.1 25.7 25.4 27.2 5.3 34.3 47.5
Ξ” vs. Qwen2.5-VL-7B +27.6 +10.3 +11.6 +10.2 +2.8 +15.0 +25.1

General video understanding (VideoMME / WorldSense / VideoMMMU / PerceptionTest; * marks the authors' own reimplementation):

Model VideoMME Overall VideoMME Long WorldSense Overall WorldSense Recognition VideoMMMU Overall VideoMMMU Perception PerceptionTest
GPT-4o 71.9 – 42.6 – 61.2 66.0 –
EgoR1 (w/o RAG) (tool-calling) 58.7 50.8 42.0 40.3 34.4 37.6 65.8
LongVT-RL-7B (tool-calling) 66.1 – 22.1 22.5 42.6 50.0 55.4
Qwen2.5-VL-7B 62.4 50.8 36.1 33.7 51.2 64.7 66.2
VideoRFT-7B 59.8 50.7 38.2 36.6 51.1 66.0 –
VideoR1-7B 61.4 50.6 35.5 32.8 52.4 65.3 –
Open-o3-video-7B 63.1* 53.1* 37.5 36.8 52.3 68.0 67.5
VisionCoach-7B (Ours) 63.3 53.2 43.8 41.8 54.4 70.3 68.7

Ablation Study

Switching on the four components one at a time (\(r_{\text{spa}}\): object-aware spatial reward; LSD: self-distillation; VP-F: fixed darkening prompt; VP-S: VP-Selector):

\(r_{\text{spa}}\) LSD VP-F VP-S Acc mAM mLGM
βœ“ – – – 59.4 31.1 42.7
βœ“ βœ“ – – 59.6 30.4 41.6
βœ“ βœ“ darken – 58.3 29.7 40.6
βœ“ βœ“ – βœ“ 60.7 31.3 43.1

VP-Selector as a plug-and-play module on different backbones (TVQA+ in-domain, a PerceptionTest subset out-of-domain):

Model TVQA+ (in-domain) PerceptionTest (out-of-domain)
Qwen2.5-VL-7B 54.5 52.2
+ VP-Selector 56.2 56.7
GPT-4o 71.8 61.5
+ VP-Selector 75.3 62.4
Gemini-2.5-Flash 72.4 47.4
+ VP-Selector 76.3 50.4

Key Findings

  • Prompting itself is not the source of the gain; selecting the prompt per input is. In the ablation, fixing the prompt to darkening (third row) is consistently worse than using no prompt at all (second row) β€” Acc 59.6 β†’ 58.3, mAM 30.4 β†’ 29.7 β€” whereas swapping in the VP-Selector (fourth row) gives the best result on every metric (60.7 / 31.3 / 43.1). Having both directions of the comparison is far more convincing than the gain alone.
  • Self-distillation on its own actually hurts the grounding metrics. Adding LSD alone nudges Acc from 59.4 to 59.6, but mAM drops from 31.1 to 30.4 and mLGM from 42.7 to 41.6. The paper's reading is that self-distillation slightly improves VQA accuracy without consistently improving grounding β€” which suggests the distilled trajectories are not necessarily better grounded, and that "prompting + distillation" is the effective unit. It should also be said plainly that the net grounding gain from the VP-Selector is small (against \(r_{\text{spa}}\) alone at 31.1 / 42.7, it adds +0.2 mAM / +0.4 mLGM); most of the benefit lands on answer accuracy (+1.3 Acc).
  • Distribution and payoff of adaptive prompting: 58% of samples are classified as hard. On these, the VP-Selector mainly chooses attention-based prompts (25.4%) and frame-numbering prompts (20.1%), with red circles and darkening making up the rest; every prompt type yields a positive gain, with relative reward improvements \(\left(R'_i - \bar R_i\right)/\bar R_i\) between +56% and +66%.
  • VP-Selector transfers across models: attached as a standalone module it improves Qwen2.5-VL (out-of-domain 52.2 β†’ 56.7, +4.5), GPT-4o (61.5 β†’ 62.4), and Gemini-2.5-Flash (47.4 β†’ 50.4), indicating it learns an input-to-prompt matching rule rather than the preference of any single model.
  • Inference latency: on a single RTX 6000, VisionCoach beats both text-centric reasoning models (Qwen2.5-VL, Video-R1) and tool-calling ones (EgoR1, LongVT-RL) while running at substantially lower latency than anything that depends on external tools β€” better results without paying extra overhead.
  • Finer-grained grounding benchmarks: on HCSTVG, VisionCoach reaches 19.3 m_vIoU, 25.4 [email protected], and 6.1 [email protected], i.e. +12.9% / +12.4% / +17.3% over Open-o3-video (17.1 / 22.6 / 5.2). Against GPT-5.4 (20.2 / 25.9 / 5.1) it is slightly lower on m_vIoU and @0.3 but ahead at the stricter @0.5 threshold (6.1 vs 5.1), with a far smaller backbone. On Charades-STA the paper reports gains over all compared VideoLLMs (Tab. 4 of the original); ⚠️ that table is interleaved across two columns in the cached text and its per-row values cannot be reliably reconstructed, so they are not reproduced here β€” refer to the original paper for the numbers.

Highlights & Insights

  • Turning "prompting" from an inference-time tool into a training-time supervision signal is the "aha" moment of this paper: once the grounding information carried by the prompt is baked into the parameters, the inference cost is unchanged, and the VP-Selector can double as a plug-and-play enhancement module (Tab. 7 is exactly that second use). The general recipe β€” heavy input during training, light input at inference β€” transfers to any "input transformation" choice, such as multi-frame sampling strategy, resolution selection, or retrieval-augmented context expansion.
  • Using the votes of several proxy models as labels sidesteps the fact that "which prompt is better" has no gold standard. The ranking function \(A_k \cdot G_k\) multiplies two off-the-shelf metrics and is remarkably cheap, yet it converts an unlabelable question into a supervised classification problem; the same trick applies to selecting data augmentations, prompt templates, or query rewrites.
  • Object identity consistency + multi-box IoU + temporal gating targets reward hacking directly: replacing "the single best box" with "every claimed object must be delivered", and excluding "right box, wrong time". Any RL task with structured output (citation grounding, table extraction, chart annotation) faces the same "emit several candidates and hope one lands" loophole, and this reward design is directly reusable.
  • Hard-sample gating is nearly free: deciding difficulty needs only the average reward of the first rollout batch, a by-product of sampling, with no separate difficulty predictor or human difficulty labels.

Limitations & Future Work

  • The explanation of "internalization" is largely narrative. The evidence is behavioral (removing the prompt at inference does not hurt), but there is no ablation that varies whether the distillation loss is conditioned on the prompted or the raw input, and no check that the model reproduces the prompt-induced attention redistribution once the prompt is gone (Fig. 5 only compares attention maps during training).
  • The ablation shows only a small net gain on the grounding metrics (+0.2 mAM over using \(r_{\text{spa}}\) alone), while the headline V-STAR gains come from answer accuracy. For localization β€” the pain point the paper keeps emphasizing β€” the evidence is weaker than for answer correctness.
  • The cost of constructing the VP-Selector data is not reported: each \((x,q)\) needs one inference per candidate prompt per proxy reasoner just to produce a single training label, and the paper gives neither the compute overhead nor the dataset size of this step.
  • Ablations on the key hyper-parameters (hard-sample threshold \(k\), temporal tolerance \(\tau\), Gaussian decay \(\sigma\)) live in the appendix; this note is based on the main-text cache and did not read the appendix.
  • The relationship between the in-domain data (V-STAR, TVQA+) and the training data may introduce an optimistic bias, and cross-domain evidence comes mainly from one uniformly sampled PerceptionTest subset.
  • vs Open-o3-video: Both belong to the "interleave grounding and reasoning inside one model" line, and this paper explicitly inherits the backbone (GSPO training, three of the four rewards, the two-stage SFT/RL recipe); the additions are the training-time visual prompt coach, self-distillation, and the object-aware spatial reward. The measured gains run from 60.2 β†’ 61.1 Acc and 33.4 β†’ 34.3 mAM on V-STAR, and 17.1 β†’ 19.3 m_vIoU on HCSTVG β€” consistent in direction but not an order of magnitude, which makes the contribution read as reinforcing the perception side of an existing framework, with gains that depend heavily on the inherited base model.
  • vs tool-calling methods (VideoZoomer / Conan / Video-o3 / EgoR1 / LongVT-RL): They intervene at inference time by repeatedly cropping, zooming, and retrieving, which costs latency and never gives the model itself perceptual ability; this paper moves the intervention to training time and keeps inference to a single forward pass. The price is an extra VP-Selector training pipeline and a proxy-reasoner data-construction stage.
  • vs text-centric RL (Video-R1 / VideoRFT): They align only the final answer and hope grounding emerges; this paper constrains it explicitly with grounding rewards and prompt guidance. The gap is largest on V-STAR's When / Where dimensions β€” Qwen2.5-VL-7B scores 17.0 on Where Chain1 against VisionCoach's 27.2 β€” suggesting explicit grounded supervision is the more effective route.

Rating

  • Novelty: ⭐⭐⭐⭐ Recasting visual prompting from an inference tool into a training-time supervision signal with self-distillation is a clean angle; the backbone (GSPO plus four rewards) clearly inherits Open-o3-video, so this is a significant increment rather than a new framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Six benchmarks (V-STAR, VideoMME, WorldSense, VideoMMMU, PerceptionTest, Charades-STA and HCSTVG) plus ablations, plug-and-play transfer, attention visualization, and a latency comparison; the gaps are key hyper-parameter ablations relegated to the appendix, unreported prompt-data construction cost, and small ablation gains on the grounding metrics.
  • Writing Quality: ⭐⭐⭐⭐ The two small analyses (Fig. 3 / Tab. 1) establish the problem before proposing the method, and the algorithm listing states the hard-sample branch and distillation conditions clearly; marks lost for the equation typesetting (Eqs. 1, 2, 3, and 8 are visibly garbled in the cached text).
  • Value: ⭐⭐⭐⭐ It provides a reusable route to better spatio-temporal grounding at zero extra inference cost, and the VP-Selector can be reused on its own as a plug-and-play module, which makes the engineering value concrete.