VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning¶
Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.1373/
Code: https://github.com/Stephen-gzk/VideoSearcher
Area: LLM Reasoning
Keywords: Video Deep Research, Multimodal Agent, Reinforcement Learning, Multi-Tool Reasoning, Policy Optimization
TL;DR¶
Addressing the disconnect between open-world evidence seeking and video perception, VideoSearcher presents a closed-loop agentic framework that unifies temporal localization, spatial zooming, and multimodal web search, optimized by a novel Bi-branch Sequence Policy Optimization (BiSPO) algorithm that decouples answer accuracy from tool invocation behavior.
Background & Motivation¶
Video understanding is undergoing a fundamental paradigm shift from closed-evidence perception to open-world evidence exploration. In real-world visual problem-solving, videos frequently provide only sparse, fleeting visual clues—such as a specific company logo on a jacket, an unknown public landmark in the background, or an unfamiliar robotic entity—while the definitive facts needed to answer complex queries reside across open-web resources, including news reports, encyclopedic entries, and multimodal knowledge repositories. The emerging Video Deep Research (VDR) paradigm formalizes this setting, positioning video as an initial perceptual gateway from which autonomous agents must track dynamic spatio-temporal cues, formulate targeted external inquiries, and synthesize verifiable multi-hop reasoning chains.
Despite rapid advancements in tool-augmented multimodal agents, current architectures remain poorly equipped for genuine VDR challenges. Existing multimodal search agents, such as MMSearch-R1 and DeepEyesV2, are almost exclusively engineered for static images, lacking mechanisms to filter temporal redundancy, localize ephemeral moments, or manage long video timelines. Conversely, prior exploratory benchmarks like VideoDR treat video evidence through a text-centric lens, converting visual anchors into textual descriptions before initiating web search. This artificial serialization discards crucial geometric, spatial, and visual features, precluding direct visual retrieval (e.g., reverse image search on small visual crops) and compromising retrieval fidelity for long-tail entities.
Furthermore, training an agent across long-horizon trajectories that interleave video manipulation with web retrieval exposes fundamental vulnerabilities in reinforcement learning. Standard policy gradient algorithms (e.g., GRPO/GSPO) evaluate trajectories via monolithic reward aggregation, confounding tool usage patterns with final answer correctness. Consequently, models often learn pathological shortcuts—either guessing prematurely based on parametric language priors or succumbing to runaway query loops and redundant tool calls. Core idea: develop a closed-loop Video Deep Research agent, VideoSearcher, that seamlessly coordinates temporal localization, spatial inspection, and dual-modality search in a single trajectory, optimized via Bi-branch Sequence Policy Optimization (BiSPO) to decouple answer accuracy from tool-behavior credit assignment under a correctness-gated, bell-shaped search reward.
Method¶
Overall Architecture¶
VideoSearcher models Video Deep Research as a multi-turn, interactive sequential decision process. The environment receives an input video \(V\), an open-domain query \(q\), and access to the external web evidence space \(\mathcal{W}\). At each interaction turn \(t\), the agent emits a hidden reasoning trace within <think> tags, invokes an action \(a_t\) from its toolset, and receives a compact multimodal observation \(o_t\) updating its interaction history \(H_{t+1} = (H_t, a_t, o_t)\), continuing until an <answer> tag is generated.
The model is equipped with five dedicated tools: choose_frames for coarse-grained temporal interval selection, find_frame for keyframe indexing, zoom_in for high-resolution region cropping, web_search for textual query retrieval, and image_search for visual entity retrieval using localized bounding boxes. Training follows a two-stage paradigm: a cold-start Supervised Fine-Tuning (SFT) phase over verified hierarchical synthetic trajectories, followed by online Reinforcement Learning (RL) using the proposed BiSPO objective.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Long Video & Query<br/>V, q with Initial 64 Sampled Frames"] --> B["Video-Centric Synthesis & Cold-Start SFT<br/>Entity-driven acquisition + hard mining + 3811 trajectories"]
B --> C["Temporal Screening & Keyframe Localization<br/>choose_frames coarse interval + find_frame exact index"]
C --> D["Spatial Focusing & Multimodal Search<br/>zoom_in regional crop + image_search & web_search"]
D --> E["Bi-branch Sequence Policy Optimization BiSPO<br/>Decoupled accuracy branch & correctness-gated tool branch"]
E --> F["Gated Bell-Shaped Search Reward Shaping<br/>Diminishing marginal returns penalizing excessive calls"]
F --> G["Multimodal Evidence Synthesis & Termination<br/>Verifiable final answer grounded in video and web evidence"]
Key Designs¶
1. Video-Centric Synthesis Pipeline & Cold-Start SFT: Bootstrapping multi-turn long-horizon interaction protocols High-quality, multi-tool trajectories grounded in dynamic videos are virtually nonexistent in open datasets. To bridge this data scarcity, the authors design an automated synthesis pipeline. Starting from knowledge-intensive image-text QA datasets (FVQA, DeepEyes), core entities are extracted and cross-verified via Qwen3.5-397B and Kimi-K2.5. Verified entities serve as query targets to scrape real-world videos from public platforms. Collected videos are resampled to 1 FPS with overlaid temporal frame markers, and the QA pairs are rewritten to strictly align with video contexts. An adversarial hard sample mining step using Qwen3-VL-8B-Instruct retains only instances where the model consistently fails (\(\le 1\) successful rollout out of 8). Trajectories are hierarchically synthesized using Gemini-3.1-Flash-Lite (coarse temporal selection) and Gemini-3.1-Pro (fine-grained inspection, image search, and web search), followed by strict heuristic and Kimi-K2.5 hallucination verification. This yields 3,811 high-quality SFT trajectories that establish syntax and tool coordination before RL.
2. Unified Spatio-Temporal Focusing & Multimodal Search: Eliminating information loss from text serialization
Rather than forcing visual anchors through an intermediate text description, VideoSearcher operates directly across visual and web domains within the same reasoning trace. Given an uncurated long video, the agent first invokes choose_frames to narrow down the temporal horizon to a relevant snippet, followed by find_frame to lock onto the sharpest view of the target entity. When the entity is visually small (e.g., an emblem, wristband, or background banner), zoom_in extracts a focused high-resolution crop. Crucially, the model can feed this crop directly into image_search to perform reverse visual retrieval on the open web, uncovering proper entity names that lack explicit textual identifiers in the video. Subsequently, the agent invokes web_search to verify fine-grained factual relationships (e.g., inaugural deployment dates or administrative jurisdictions). This synergistic closed-loop eliminates the bottleneck of premature visual-to-text summarization.
3. Bi-branch Sequence Policy Optimization (BiSPO): Resolving credit assignment confounding between answers and tools In long-horizon VDR tasks, naive policy optimization that sums accuracy and tool rewards encourages false credit assignment: correct guesses based on language priors reinforce arbitrary tool calls, while slight tool deviations on difficult queries penalize valid search strategies. Built upon Group Sequence Policy Optimization (GSPO) with sequence-level importance sampling ratio \(s_i(\theta)\) and clipping (\(\epsilon_{\text{low}}=0.2, \epsilon_{\text{high}}=0.28\)), BiSPO splits the objective into two independent branches. The accuracy branch evaluates all \(G\) sampled rollouts via group-relative advantages: $\(A_i^{\text{acc}} = \frac{R_i^{\text{acc}} - \mu_{\text{acc}}}{\sigma_{\text{acc}} + \epsilon}, \quad R_i^{\text{acc}} = R_i^{\text{judge}} + R_i^{\text{fmt}} + R_i^{\text{dep}}\)$ where \(R_i^{\text{dep}} = -\min(0.1 N_{\text{dep}}^i, 0.5)\) penalizes calling search/zoom tools without locking a keyframe. Conversely, the tool branch applies a correctness gate: tool behaviors are evaluated strictly within the subset of rollouts that yield correct answers (\(Q = \{j \mid R_j^{\text{judge}} > 0\}\)): $\(A_i^{\text{tool}} = \begin{cases} \frac{R_i^{\text{tool}} - \mu_Q^{\text{tool}}}{\sigma_Q^{\text{tool}} + \epsilon}, & i \in Q, |Q| \ge 2 \\ 0, & \text{otherwise} \end{cases}\)$ The composite loss optimizes the actor at the sequence level: \(\mathcal{L}_{\text{BiSPO}} = w_{\text{acc}} \mathcal{L}_{\text{GSPO}}(A^{\text{acc}}) + w_{\text{tool}} \mathcal{L}_{\text{GSPO}}(A^{\text{tool}})\) with \(w_{\text{acc}} = 1.0\) and \(w_{\text{tool}} = 0.15\), isolating answer supervision while steering correct rollouts toward optimal tool efficiency.
4. Gated Bell-Shaped Search Reward Shaping: Preventing over-searching loops while incentivizing evidence exploration To prevent agents from exploiting search tools through repetitive calls or avoiding retrieval out of penalty fear, tool behavior rewards are structured with non-linear diminishing marginal returns. Valid video-grounding actions receive a base observation reward \(b + \gamma \sqrt{T_i^{\text{vid}}}\) (\(b=0.3, \gamma=0.35\)). For multimodal web retrieval, unique successful queries under image and web modalities are tracked independently (\(U_{i,m}\)). The marginal reward for the \(k\)-th unique useful call follows a predefined bell schedule: \([+0.35, +0.20, +0.10, -0.10, -0.25]\) for \(k=1, \dots, 5\), and \(-0.60\) for every subsequent call. Failed executions, blank image search returns, and duplicate queries yield zero reward and do not advance the counter. This shaped schedule actively rewards exploratory initial searches while imposing steep penalties on redundant queries, guiding the model toward efficient evidence convergence.
Loss & Training¶
The framework is realized on Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct. SFT is executed with LLaMA-Factory across 3,811 trajectories using bfloat16, learning rate \(1 \times 10^{-5}\), with frozen vision encoders and multimodal projectors. Online RL is implemented on veRL across 16 H20 GPUs for one epoch. The batch size is 64 prompts with 4 rollouts per prompt (\(G=4\)). The actor learning rate is \(1 \times 10^{-5}\), with maximum response length 16,384 tokens, context length 49,152 tokens, and a maximum turn budget of 12. Vision parameters remain frozen during RL to stabilize long-context gradient propagation.
Key Experimental Results¶
Main Results¶
VideoSearcher was evaluated across two Video Deep Research benchmarks (VideoDR and the newly constructed VideoSearch-QA) alongside six open-domain multimodal search benchmarks (MMSearch, HR-MMSearch, FVQA-test, InfoSeek, SimpleVQA, LiveVQA).
| Model | VideoDR | VSQA | MMSearch | HR-MMSearch | FVQA-test | InfoSeek | SimpleVQA | LiveVQA | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Direct Answer | |||||||||
| Qwen3-VL-4B-Instruct | 8.00 | 12.59 | 13.45 | 3.61 | 25.33 | 25.75 | 46.89 | 23.35 | 19.87 |
| Qwen3-VL-8B-Instruct | 8.00 | 16.91 | 11.70 | 12.13 | 24.22 | 23.15 | 42.94 | 23.18 | 20.28 |
| GPT-4o (Direct) | 35.00 | 36.33 | 23.39 | 13.11 | 48.00 | 52.90 | 51.73 | 28.18 | 36.08 |
| Gemini-3-Flash (Direct) | 57.00 | 56.12 | 57.31 | 21.97 | 56.50 | 54.85 | 63.57 | 38.90 | 50.78 |
| Agentic (Zero-shot) | |||||||||
| Qwen3-VL-4B-Instruct | 25.00 | 29.14 | 49.12 | 24.92 | 31.72 | 28.10 | 41.95 | 26.15 | 32.01 |
| Qwen3-VL-8B-Instruct | 28.00 | 30.94 | 47.37 | 27.87 | 53.61 | 46.15 | 62.29 | 39.37 | 41.95 |
| GPT-4o (Agentic) | 48.00 | 49.64 | 49.12 | 30.16 | 66.34 | 59.55 | 63.67 | 40.09 | 50.82 |
| Gemini-3-Pro (Agentic) | 77.00 | 65.11 | 74.27 | 48.52 | 72.61 | 66.45 | 75.91 | 59.69 | 67.45 |
| Agentic Models (Trained) | |||||||||
| Visual-ARFT | 5.00 | 8.99 | 34.50 | 24.92 | 41.72 | 37.95 | 42.45 | 25.40 | 27.62 |
| MMSearch-R1 | 5.00 | 11.87 | 53.80 | 20.33 | 58.40 | 55.10 | 57.40 | 48.40 | 38.79 |
| DeepEyesV2 | 19.00 | 21.58 | 63.70 | 14.10 | 60.60 | 51.10 | 59.40 | 24.43 | 39.24 |
| Webwatcher-7B | 37.00 | 34.53 | 49.10 | 26.89 | 58.17 | 56.90 | 54.30 | 51.20 | 46.01 |
| VideoSearcher-4B (Ours) | 46.00 | 49.28 | 66.08 | 39.67 | 55.17 | 49.25 | 65.84 | 50.72 | 52.75 |
| Gain vs Qwen3-VL-4B | +21.00 | +20.14 | +16.96 | +14.75 | +23.45 | +21.15 | +23.89 | +24.57 | +20.74 |
| VideoSearcher-8B (Ours) | 53.00 | 51.80 | 67.84 | 43.61 | 64.10 | 58.30 | 67.20 | 55.40 | 57.66 |
| Gain vs Qwen3-VL-8B | +25.00 | +20.86 | +20.47 | +15.74 | +10.49 | +12.15 | +4.91 | +16.03 | +15.71 |
On general video understanding benchmarks, VideoSearcher-8B achieved an average score of 60.39% across MMVU (70.40%), TempCompass (73.14%), VideoMMMU (66.33%), and VideoMathQA (31.67%), outperforming dedicated video reasoning models such as Video-R1 (53.18%), VideoRFT (54.63%), and VideoCoM (53.68%). This demonstrates that open-web search reinforcement training does not compromise closed-domain video reasoning.
Ablation Study¶
Ablations on RL algorithms, reward formulations, and tool compositions were conducted on the 8B backbone:
| Aspect | Config / Algorithm | VideoDR | VSQA | MMSearch | HR-MMSearch |
|---|---|---|---|---|---|
| RL Algorithm & Reward | SFT Baseline | 46.00 | 48.16 | 65.50 | 40.98 |
| GRPO (\(R_{\text{acc}}\) only) | 45.00 | 47.84 | 58.48 | 41.14 | |
| GSPO (\(R_{\text{acc}}\) only) | 47.00 | 50.72 | 63.16 | 40.66 | |
| GSPO (\(R_{\text{acc}} + 0.15 R_{\text{tool-mono}}\)) | 50.00 | 51.08 | 65.50 | 40.33 | |
| BiSPO (\(R_{\text{acc}} + 0.15 R_{\text{tool-mono}}\)) | 47.00 | 51.44 | 65.50 | 40.33 | |
| BiSPO Full (Decoupled + Bell-shaped) | 53.00 | 51.80 | 67.84 | 43.61 | |
| Tool Ablation | Locate Tools Only | 12.00 | 19.78 | 12.28 | 3.93 |
| Search Tools Only | 43.00 | 47.48 | 66.67 | 40.98 | |
| Full Toolset (Locate + Search) | 53.00 | 51.80 | 67.84 | 43.61 |
Key Findings¶
- Sequence-Level Importance Sampling is Critical: Token-level clipping in GRPO causes performance degradation under multi-turn long contexts (VideoDR drops from 46.00% to 45.00%), whereas sequence-level GSPO successfully maintains policy stability.
- Superiority of Decoupled Correctness Gating: Merging tool rewards into a single advantage causes models to learn spurious tool behaviors. Decoupling the policy updates and restricting tool advantage estimation to successful trajectories enables a substantial leap on VideoDR (from 47.00% to 53.00%).
- Interdependence of Temporal Localization and Retrieval: Operating without search tools collapses performance on VSQA (19.78%), confirming that open-world questions cannot be resolved through closed-video perception. Omitting localization tools drops VideoDR accuracy from 53.00% to 43.00%, proving that grounded visual anchors are indispensable for precise web queries.
Highlights & Insights¶
- Correctness-Gated Tool Advantage: By isolating tool behavior comparison exclusively to correct trajectory subsets (\(|Q| \ge 2\)), BiSPO prevents suboptimal or failing trajectories from corrupting the gradient updates of search actions, ensuring every reinforced tool invocation directly supports verifiable factual deduction.
- Entity-to-Video Reverse Synthesis: Overcoming the challenge of annotating complex VDR benchmarks, the pipeline extracts verified entities from static QA, mines corresponding web videos, rewrites video-aligned queries, and filters hard negatives, yielding a cost-effective, scalable recipe for interactive agent data creation.
- Bell-Shaped Marginal Search Utility: The non-monotonic reward schedule penalizes search frequencies beyond 3-5 calls while zeroing out failed or duplicate queries, elegantly balancing the trade-off between under-exploration and infinite loop collapse.
Limitations & Future Work¶
- Vulnerability to External Tool Noise: Because the agent relies on real-time search APIs, erroneous visual retrieval results propagate down the reasoning chain without self-correction (e.g., misclassifying the Grömitz Diving Gondola as Zinnowitz led to an incorrect factual year retrieval). Future iterations require explicit cross-source consistency checking and uncertainty calibration.
- Prohibitive Compute Overhead in Multi-Turn RL: Sampling 12-turn rollouts over 49k context windows creates severe memory and throughput bottlenecks, forcing the vision encoder to remain frozen throughout RL. Exploring parameter-efficient multimodal visual adapter tuning remains an open challenge.
- Downsampling Resolution Bottlenecks: The uniform 1 FPS downsampling strategy can miss fleeting micro-actions or transient visual cues lasting only fractions of a second, limiting application to high-speed dynamic scene analysis.
Related Work & Insights¶
- vs VideoDR / WebWatcher: VideoDR relies on text-serialized retrieval that loses fine-grained visual features, whereas WebWatcher and MMSearch-R1 are tailored for static images. VideoSearcher is the first unified agent capable of end-to-end temporal interval selection, spatial zooming, and image/web retrieval over dynamic video streams.
- vs Video-R1 / FrameThinker: Prior video RL methods operate within closed-context environments where reward signals derive strictly from internal perception. BiSPO pioneers multi-objective sequence policy optimization that bridges video spatio-temporal modeling with external web evidence retrieval.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first closed-loop multi-tool VDR framework and introduces a principled dual-branch decoupled RL algorithm.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 12 comprehensive benchmarks spanning VDR, multimodal search, and general video reasoning with detailed ablations and qualitative error analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous task formulation, clear mathematical exposition, and logically coherent narrative progression.
- Value: ⭐⭐⭐⭐⭐ Sets a foundational open-source benchmark and methodology for transitioning multimodal agents from passive video perception to active web deep research.