VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement¶

Conference: ECCV 2026
arXiv: 2607.00446
Code: https://mlvlab.github.io/VideoSearch-R1/ (Open Sourced)
Area: Multimodal VLM / Video Understanding / Agent
Keywords: Video Retrieval, Temporal Grounding, Soft Query Refinement, GRPO, Agentic AI

TL;DR¶

VideoSearch-R1 proposes an agentic framework that unifies video retrieval and temporal grounding in an iterative interaction loop. It replaces traditional text-level query rewriting with "soft query refinement" in a continuous latent space, and jointly optimizes retrieval and reasoning via GRPO, achieving state-of-the-art performance on three VCMR benchmarks.

Background & Motivation¶

As video corpora continue to expand, users' demands have progressed beyond coarse-grained retrieval like "finding relevant videos." A truly meaningful scenario often involves: given a natural language query, the system first needs to locate the correct video from a corpus containing tens of thousands of videos (video retrieval), and then precisely predict the start and end timestamps of the event described by the query within that video (temporal grounding). This joint challenge of video retrieval and temporal grounding is known as Video Corpus Moment Retrieval (VCMR).

However, almost all existing pipelines treat these two stages in a decoupled manner. Retrieval models (e.g., CLIP4Clip, TS2-Net) handle coarse-grained cross-modal alignment to output a list of candidate videos, while downstream temporal grounding models (e.g., Vid2Seq, VideoLLaMA, FlashVTG) operate under the assumption that "the relevant video is already provided," performing only fine-grained intra-video reasoning. This decoupled architecture introduces a fatal cascading error: if the first-stage retrieval fails to retrieve the correct video, all subsequent temporal grounding becomes impossible, with no mechanism for recovery. Meanwhile, although existing video agent frameworks incorporate external tools (object trackers, OCR, grounders) for long video understanding, they also generally assume the query-relevant video is already known, bypassing the step of corpus-level retrieval.

The core insight of this paper is that retrieval and reasoning should not be a "one-shot deal" but should form a closed loop: the model retrieves first, verifies the result, reformulates the query upon finding a mismatch, and repeats this until the correct video is found before performing temporal grounding. To this end, the authors propose VideoSearch-R1, an agentic framework that unifies retrieval and reasoning in an iterative interaction loop. It introduces an interesting mechanism: using low-cost soft query refinement to replace lengthy and messy text-level rewriting, combined with end-to-end reinforcement learning. Core Idea: modeling video retrieval and temporal grounding as an iterative agent-search engine multi-turn interaction process, where query optimization is conducted in a continuous representation space by directly generating a small number of soft tokens to finely adjust the query semantics (instead of natural language rewriting in the discrete text space, i.e., hard query refinement). It then jointly optimizes multiple objectives for retrieval verification and temporal grounding using GRPO.

Method¶

Overall Architecture¶

The core of VideoSearch-R1 is an iterative retrieve-verify-refining loop. Given a user query \(q_1\), the model first calls a video search engine (a cross-modal dense retriever such as Qwen3-VL-Embedding-2B) to return the top-1 candidate video \(v_t\) from the corpus \(\mathcal{V}\). The model then analyzes the video frame by frame to determine whether it matches the requirements of the query. If it matches, the model immediately switches to temporal grounding mode, predicting precise start and end timestamps. If it does not match, the model generates a set of continuous soft tokens to perform fine-grained adjustments on the query, and then calls the search engine again with the optimized query. This process is repeated for a maximum of \(T\) turns; if no matching video is found within these turns, it is considered a failure.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Query q1"] --> B["Call Search Engine<br/>Retrieve top-1 video"]
    B --> C{"Verification:<br/>Video matches query?"}
    C -->|Match| D["Temporal Grounding<br/>→ Predict start/end"]
    C -->|Mismatch| E["Generate Soft Query Tokens<br/>SQR: Continuous Latent Space Optimization"]
    E --> F["Append soft tokens to original query<br/>qt+1 = [q1 || qt_soft]"]
    F --> B
    D --> G["Output Timestamps"]

The training process is divided into two stages:

Stage 1 SFT Warm-up: Supervised learning is first used to enable the model to follow structured reasoning templates, while contrastive learning (InfoNCE) is employed to train the capability of generating soft tokens—forcing the soft tokens to be similar to the correct video embeddings and distant from negative sample embeddings.
Stage 2 GRPO Reinforcement Learning: Building upon SFT, GRPO is used to explore better reasoning trajectories and query refinement strategies. The four reward signals correspond to format compliance (format), matching verification correctness (verification), soft token retrieval quality (retrieval), and temporal grounding accuracy (temporal grounding).

Key Designs¶

1. Soft Query Refinement (SQR): Query Optimization in Continuous Latent Space

Traditional query refinement approaches direct the model to rewrite a more precise query in natural language—termed Hard Query Refinement (HQR). However, this approach has several drawbacks: first, the model needs to generate a large number of tokens (averaging 26.8 tokens in experiments) to describe visual details, and these texts may introduce semantic noise into the cross-modal retriever, thereby degrading retrieval; second, text-level rewriting is restricted to a discrete vocabulary, making fine-grained continuous adjustments impossible.

The mechanism of SQR is entirely different: during autoregressive decoding, the model maps the hidden state of the previous token to the input embedding of the next token through a linear projection layer, directly generating \(N\) continuous vectors \(\mathbf{q}^{\text{soft}}_t \in \mathbb{R}^{N \times D}\). These soft tokens do not correspond to any word in the natural language vocabulary but are purely latent space representations. They are concatenated to the end of the original query \(q_1\) to form \(q_{t+1} = [q_1 \parallel q^{\text{soft}}_t]\), which is then fed into the search engine. The soft tokens are trained via the InfoNCE loss, aiming to minimize the distance between the optimized query and the correct video embedding while maximizing the distance to negative samples. This contrastive supervision provides richer discriminative signals than simple next-token prediction in HQR. Experiments demonstrate that using only 8 soft tokens significantly improves retrieval performance, whereas HQR requires 26.8 tokens and yields smaller retrieval gains.

2. Iterative Retrieval-Verification Loop: Instilling Self-Correction Capability into Retrieval

This is the top-level architecture of the entire framework. Each turn of interaction consists of three pieces of information: the current query \(q_t\), the top-1 video \(v_t\) returned by the search engine, and the reasoning trace \(r_t\) of the model. Instead of simply outputting "match/mismatch," the model first generates an explicit <think> reasoning chain, in which it compares the visual evidence required by the query with actual observations in the video item by item (e.g., "the query requires a dark gray T-shirt, but the person in the video is wearing a light gray T-shirt") before delivering the verification decision \(y^{\text{ret}}_t\). If a mismatch is identified, the model additionally outputs a special <REFINE> tag at the end, triggering soft token generation.

This "reasoning before verification" design is crucial—it forces the model to truly comprehend the semantic gap between the query and the video content before deciding whether to search again. Remarkably, even when the match is successful, the <REFINE> token is still generated to ensure the model always utilizes the soft query generation path during inference, preventing it from collapsing into a lazy strategy of "always saying mismatch to keep retrieving" during RL training. When a match is successful, the model proceeds to predict precision temporal boundaries \(y^{\text{time}}\), completing the entire chain of retrieval and grounding.

3. Joint Optimization via GRPO Multi-Rewards: Aligning Four Objectives

Most existing works train retrieval and reasoning separately, or merely use SFT to mimic fixed patterns. VideoSearch-R1 uses GRPO to optimize the entire iterative process as a trajectory under policy optimization, designing four complementary reward signals:

Format Reward \(R^{\text{format}}\): Ensures the output strictly adheres to the <think>/<answer>/<start>/<end>/<REFINE> template—serving as the foundational infrastructure for stable multi-turn interactions.
Verification Reward \(R^{\text{verif}}\): Correctly determines if the video matches the query, acting as a safeguard to prevent performing temporal grounding on an incorrect video.
Retrieval Reward \(R^{\text{ret}}\): Reuses the InfoNCE loss from the SFT stage, encouraging soft tokens to more accurately pull the query embedding towards the correct video via \(R^{\text{ret}} = \exp(-\mathcal{L}_{\text{ret}})\).
Temporal Grounding Reward \(R^{\text{time}}\): Directly computes the IoU between the predicted frame interval and the ground truth, which is only activated on successful matches.

Weighted summation of these four rewards is followed by group-relative normalization to calculate the Advantage. This design allows the GRPO signal to simultaneously backpropagate to both retrieval enhancement (embedding optimization of soft tokens) and reasoning enhancement (temporal boundary prediction), achieving true end-to-end iterative optimization. Ablation studies reveal that adding \(R^{\text{ret}}\) primarily improves retrieval recall, adding \(R^{\text{verif}}\) significantly improves verification accuracy, and \(R^{\text{time}}\) substantially enhances temporal localization IoU—making all three indispensable.

Loss & Training¶

Stage 1 Joint SFT Loss: \(\mathcal{L}_{\text{SFT}} = \mathcal{L}_{\text{verif}} + \mathcal{L}_{\text{ret}} + \mathbb{1}_{y^{\text{ret}}=\text{'match'}} (\mathcal{L}_{\text{time}})\). Here, \(\mathcal{L}_{\text{ret}}\) is the InfoNCE contrastive loss for soft tokens, while \(\mathcal{L}_{\text{verif}}\) and \(\mathcal{L}_{\text{time}}\) are cross-entropy losses.
Stage 2 RL: GRPO, with KL coefficient \(\beta=0.01\), rollout size \(G=8\), and decoding temperature 1.0. Learning rate is \(5\times10^{-7}\) and weight decay is 0.01.
Backbone Model: Qwen3-VL-2B-Instruct, with Qwen3-VL-Embedding-2B as the search engine. The number of visual tokens is 4096 (64 frames @1FPS), the number of soft tokens is \(N=8\), and the maximum iteration turn is \(T=2\).

Key Experimental Results¶

Main Results¶

On three VCMR benchmarks (ActivityNet-FIG, Charades-FIG, DiDeMo-FIG), using Qwen3-VL-2B as the backbone, compared against zero-shot and fine-tuning baselines as well as conventional methods (CONQUER, SQuiDNet):

Dataset	Method	VCMR 0.5/R@1	VR R@1	VER Acc
Charades-FIG	Qwen3-VL-2B (FT)	10.4	-	74.7
	VideoSearch-R1	13.4	24.6	75.7
DiDeMo-FIG	Qwen3-VL-2B (FT)	22.1	-	73.1
	VideoSearch-R1	30.2	59.0	74.6
ActivityNet-FIG	Qwen3-VL-2B (FT)	19.2	-	83.1
	VideoSearch-R1	22.3	61.1	83.3

VideoSearch-R1 achieves comprehensive dominance across all datasets and indicators. Notably, even though the fine-tuned baseline sequentially checks the top-2/top-3 candidate videos returned by the search engine during multi-turn reasoning, VideoSearch-R1 boosts R@1 by 6-7 percentage points by iteratively optimizing query embeddings (since the same search engine returns a more accurate top-1 video for the optimized query).

Ablation Study¶

Configuration	VCMR 0.5/R@1	VR R@1	VER Acc	Description
ZS baseline	10.6	54.8	62.8	Without any fine-tuning
Stage 1 (SFT only)	18.7	57.4	66.0	Learn template + soft tokens, but limited temporal grounding gain
Stage 1 + Stage 2 (Full)	30.2	59.0	74.6	GRPO significantly improves grounding accuracy
Full w/o \(R^{\text{ret}}\)	17.3	58.0	65.0	Retrieval reward lost, soft tokens degenerate
Full w/o \(R^{\text{verif}}\)	17.3	59.7	75.0	Verification is more accurate but grounding drops
Full (all rewards)	30.2	59.0	74.6	Full version, balance between the three

Key Findings¶

Huge Token Efficiency Gap between SQR and HQR: SQR achieves a retrieval R@1 gain of 7.2 with only 8 soft tokens; HQR achieves an R@1 gain of only 3.7 with an average of 26.8 tokens. The longer sequences in HQR actually result in smaller retrieval gains, which the authors attribute to semantic noise introduced in the cross-modal embedding space by verbose text rewriting.
Rapid Saturation of Iteration Turns: VCMR performance nearly saturates after the 2nd turn, with almost no improvement in the 3rd turn—indicating that setting \(T=2\) efficiently balances computation and accuracy.
Gradual Refinement of Soft Tokens is Observable: As the number of soft tokens is progressively increased (from 0 to 8), the retrieved videos evolve from "a woman brushing her teeth" to "blonde hair, someone combing hair" and further to "light blue wall background"—demonstrating that soft tokens step-by-step approach the target video embedding in the continuous space, forming a gradient path in the embedding space.

Highlights & Insights¶

The Design of "Soft Query" is Highly Clever: Moving query refinement from the discrete vocabulary space to a continuous vector space reduces token overhead (8 vs. 26.8) and avoids the semantic noise introduced by text rewriting. This is a successful migration of the "soft reasoning" concept to video retrieval, akin to explorations like CoCoT/Latent CoT in NLP.
<think> Reasoning Chain + <REFINE> Special Token in Verification: Requiring the model to explicitly state its reasoning process (comparing query requirements with video evidence item by item) before determining match status improves verification reliability and provides a natural, interpretable interface. Users can inspect exactly "why the model claimed a mismatch."
Progressive Integration of the Four GRPO Rewards: Format -> Verification -> Retrieval -> Grounding, where each step resolves a specific sub-problem. The ablation table cleanly displays how each added reward directly impacts its corresponding metric, validating the structured design of rewards in multi-objective RL.
Generalizability of this Framework: Any two-stage task requiring "retrieval followed by fine-grained reasoning"—such as cross-modal fact-checking (retrieving images then verifying) or document retrieval-QA (retrieving documents then extracting answers)—can readily adopt this SQR + iterative verification loop framework.

Limitations & Future Work¶

The search engine is fixed (Qwen3-VL-Embedding-2B), meaning the optimization of soft tokens is constrained by the embedding space representation capacity of this specific retriever. If the search engine is switched, SQR must be retrained to adapt to the new embedding space.
The maximum iteration limit (\(T=2\)) might be inflexible for queries of varying difficulty—easy queries might be matched in one shot, whereas difficult queries could require more retrieval turns. A better design would let the model dynamically decide when to stop iterating (similar to halting in ReAct).
Experiments are restricted to VCMR; whether the model can generalize to more open scenarios (such as real-time web video retrieval without a pre-defined corpus) remains unverified.
Insufficient interpretability of soft tokens: while hard queries allow users to read "how the model modified the query," soft tokens are entirely uninterpretable, making debugging and troubleshooting much harder.

vs. CONQUER / SQuiDNet: These are decoupled two-stage approaches—video retrieval followed by temporal grounding, leaving retrieval errors uncorrectable. VideoSearch-R1 turns both into an iterative closed loop, correcting errors via feedback.
vs. Search-R1: Search-R1 uses GRPO to optimize search queries and reasoning in pure-text RAG. VideoSearch-R1 extends this to the video domain and replaces text query rewriting with SQR—a pivotal advancement from text to cross-modal scenarios.
vs. VideoAgent / VidoRAG: Existing video agent frameworks assume relevant videos are known and only perform intra-video reasoning. VideoSearch-R1 incorporates retrieval into the agent's toolbox, upgrading the agent from "analyzing known videos with tools" to "finding videos itself + analyzing."
vs. CoCoT / Coconut (NLP Soft Reasoning): These works replace text-based CoT reasoning chains with continuous hidden states. VideoSearch-R1 applies a similar philosophy to query optimization rather than the reasoning sequence itself, demonstrating the portability of the "soft representation" concept to retrieval scenarios.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ SQR successfully migrates soft reasoning from NLP inference to video retrieval, and the structure of the iterative retrieval-verification loop is clean and well-designed.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on three benchmarks with comprehensive ablations (training stages, reward designs, SQR vs. HQR, token count scans), establishing a complete validation chain.
Writing Quality: ⭐⭐⭐⭐ Clear motivation, well-coordinated diagrams (the illustrations in Fig.1/2/3 are highly intuitive), but the method section is relatively long and slightly redundant.
Value: ⭐⭐⭐⭐⭐ Offers an elegant and practical unified solution to the disjoint issues of video retrieval and grounding, where the token efficiency advantages of SQR are highly notable.