MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://marineevt.hkustvgd.com
Area: Multimodal VLM / LLM Reasoning
Keywords: Marine Video Understanding, Event-Centric Reasoning, Visual Tool Reasoning, Reinforcement Learning, GRPO

TL;DR

To tackle sparse, unpredictable events and the scarcity of temporal-causal reasoning in underwater videos, this work constructs MarineEVT, the first 20K multi-dimensional event-centric marine video benchmark, and proposes EVT-R1, a multi-turn visual tool-integrated reasoning framework optimized via dual-reward GRPO.

Background & Motivation

Oceans cover over 70% of Earth's surface, and understanding marine ecosystems is paramount for biodiversity preservation, climate change monitoring, and sustainable ocean management. Vision-language models (VLMs) have recently made substantial strides in terrestrial video comprehension and static marine imagery tasks such as species identification, underwater object detection, and instance segmentation. However, when deployed on underwater video streams, VLMs experience severe performance degradation. Underwater environments impose severe visual bottlenecks: optical distortion, low contrast, suspended sediment, and persistent background redundancy caused by slow-moving cameras or stationary sensors. Conversely, informative ecological occurrences—such as predator-prey dynamics, territorial defense, mating rituals, and cross-species interactions—are sparse, transient, and unpredictably distributed across lengthy recordings.

General-purpose video VLMs fundamentally favor global scene summarization, frequently overlooking nuanced visual dynamics and temporal causality essential to biological discovery. When tested on complex underwater footage, commercial foundation models such as GPT-5 fail to detect subtle but critical behavioral nuances; for instance, interpreting an aggressive territorial charge by a Red Sea dascyllus against a masked pufferfish as simply "a reef fish calmly swimming out from coral across the seabed." This failure stems from a core tension: comprehensive marine video understanding requires domain expertise combined with the capability to identify and magnify localized spatio-temporal intervals buried within redundant frames. Current foundation models lack domain-tailored event benchmarks and possess no autonomous mechanism to discard extraneous visual tokens via external tool feedback. Straightforward supervised fine-tuning (SFT) can supply domain terminology, but fails to endow models with dynamic localization and adaptive evidence gathering.

To resolve these challenges, this paper frames complex underwater video reasoning as a sequential decision-making process involving multi-turn external visual tool invocation. Core idea: curate MarineEVT, the first event-centric marine video benchmark covering 20 fine-grained reasoning dimensions, and develop EVT-R1, a reinforcement learning framework that sequentially invokes spatio-temporal visual tools to prune redundant frames and applies dual-reward GRPO to align tool usage and answer correctness.

Method

Overall Architecture

EVT-R1 conceptualizes event-centric marine video comprehension as a sequential decision process over an interactive trajectory. Given a user query \(q\), initial video sequence \(V_0\), and external visual toolbox \(\mathcal{T}\), the policy model \(\pi_\theta\) conducts multi-turn reasoning over \(K\) steps. At step \(k\), conditioned on the context \((q, V_k, H_k)\), the model outputs intermediate reasoning \(r_k\) and decides whether to output a final answer directly or invoke visual tools \(T_k \subseteq \mathcal{T}\) with parameterized calls \(\{c_{k,j}\}\). When a tool is triggered, the execution engine retrieves localized temporal boundaries or crops key spatial objects, producing updated visual observation \(o_k\) that replaces prior redundant visual tokens in the dialogue history. This dynamic visual replacement purges outdated background clutter and focuses representation learning onto critical evidence until the policy terminates at final answer \(a\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Marine Video V0 and Query q"] --> B["Multi-Stage Event Data Generation & Verification<br/>TransNet segmentation + SAM3 grounding + dual verification"]
    B --> C["Multi-Turn Visual Tool Reasoning Trajectory<br/>Dynamic spatio-temporal tool invocation & visual replacement"]
    C --> D["Dual-Reward Reinforcement Learning Optimization<br/>Decoupled tool reasoning reward and multi-task answer reward"]
    D --> E["Precise Event-Centric Answer a"]

Key Designs

1. Multi-Stage Event Data Generation & Verification: High-Density Domain Benchmark Construction

Directly prompting foundation models on raw underwater footage yields severe hallucinations and temporal misalignment due to extreme visual sparsity. The authors address this by designing a coarse-to-fine data synthesis pipeline. TransNet partitions 7,300 raw marine videos into 97,284 coherent scenes, and GPT-5 produces domain-structured descriptions across species, human activities, environmental contexts, and notable ecological events. Next, grounding toolkits—including SAM3 for mask/bounding box detection, DepthAnythingV2 for metric depth, and OrientAnything for 3D orientation—extract 94,028 spatial-geometric grounding annotations across 240,017 bounding boxes. Finally, QwenVL-Max generates question-answer pairs over 20 distinct reasoning dimensions, followed by a rigorous two-tier verification stage involving automated three-LLM filtering and validation by three professional marine scientists.

2. Multi-Turn Visual Tool Reasoning Trajectory: Active Spatio-Temporal Redundancy Elimination

Standard Chain-of-Thought (CoT) operates exclusively within the language modality, rendering it incapable of resolving underwater token bloat and low signal-to-noise ratios. EVT-R1 equips the VLM policy with an active visual interaction loop. Across consecutive turns, the model issues structured API calls targeting two primary tool categories: temporal grounding tools that pinpoint start and end timestamps of relevant events, and spatial grounding tools that segment regions containing target marine organisms or diver equipment. Crucially, the resulting visual outputs actively replace previous high-overhead visual inputs in the dialogue history, discarding non-informative frames and driving the internal representation toward the localized physical interactions.

3. Dual-Reward Reinforcement Learning Optimization: Decoupling Intermediate Tool Use from Final Answers

Standard Group Relative Policy Optimization (GRPO) relies on a single scalar outcome reward evaluated at the end of the trajectory, which frequently causes reward hacking, repetitive looping, or unjustified tool invocations in multi-turn tool-augmented settings. EVT-R1 formulates a turn-level dual-reward function that decouples tool execution quality from answer correctness:

\[R(y_i, g_i) = \lambda \cdot \mathbb{I}[\text{tool\_turn}] \cdot R_{\text{tool}}(y_i, g_i) + (1-\lambda) \cdot \mathbb{I}[\text{answer\_turn}] \cdot R_{\text{ans}}(y_i, g_i)\]

The tool reward \(R_{\text{tool}}\) assesses Invocation Validity (binary agreement with ground-truth tool-trigger requirements) and Invocation Accuracy (spatial and temporal IoU matching ground-truth boundaries). The answer reward \(R_{\text{ans}}\) checks Format Compliance alongside Semantic Correctness (exact matching for closed-form questions and embedding cosine similarity with group normalization for open-ended queries). Optimizing this objective under GRPO stabilizes policy gradient updates and guides the model on when and how to invoke visual tools.

Loss & Training

EVT-R1 adopts a two-stage training paradigm: a 1-epoch supervised fine-tuning (SFT) warm-up on MarineEVT to establish baseline marine domain knowledge and tool-calling syntax, followed by 4 epochs of GRPO reinforcement learning. Policy updates maximize the clipped surrogate objective with a KL divergence penalty relative to the reference policy:

\[\mathcal{L}_{\text{GRPO}}(\theta) = \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\left[\min\left(\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q,o_{i,<t})}\hat{A}_{i,t}, \text{clip}\left(\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q,o_{i,<t})}, 1-\epsilon, 1+\epsilon\right)\hat{A}_{i,t}\right)\right] - \beta D_{\text{KL}}[\pi_{\theta}||\pi_{\text{ref}}]\]

Training uses LoRA parameter-efficient fine-tuning on Qwen3-VL-8B using AdamW with learning rate \(5 \times 10^{-6}\), weight decay \(1 \times 10^{-2}\), micro-batch size 2 supporting sequences up to 50k tokens, and reward weighting factor \(\lambda = 0.75\).

Key Experimental Results

Main Results

Evaluation is conducted on a dedicated 2,000 QA testing split from MarineEVT across semantic reasoning (SemR), contextual reasoning (ConR), spatial reasoning (SpaR), temporal reasoning (TemR), and causal reasoning (CasR):

Model	Tool Invocation	Training Paradigm	SemR	ConR	SpaR	TemR	CasR	Average (Avg.)
Video-LLaVA-7B	✗	Zero-shot	24.40	29.00	8.40	5.80	42.00	21.92
LLaVA-NeXT-Video-7B	✗	Zero-shot	35.40	34.33	9.00	6.75	42.67	25.63
VideoLLaMA3-7B	✗	Zero-shot	41.00	46.33	5.40	3.50	61.77	31.60
InternVL3-8B	✗	Zero-shot	53.40	50.33	17.20	10.33	71.77	40.61
Qwen3-VL-8B-Instruct	✗	Zero-shot	58.20	52.77	22.40	14.00	71.00	43.67
Grok-4-1-FR	✗	Zero-shot	41.20	37.33	17.40	6.25	50.67	30.57
Gemini-3.0-Flash	✗	Zero-shot	48.20	43.00	27.00	7.75	62.00	37.59
GPT-5-Mini	✗	Zero-shot	58.40	30.67	22.80	10.00	66.67	37.71
GPT-5-Mini	✓	Prompt-based	58.40	43.67	22.40	13.00	64.33	40.35
Qwen3-VL-8B	✓	GRPO (Post-train only)	44.60	37.00	20.00	10.75	62.66	35.58
Qwen3-VL-8B	✓	SFT	61.40	53.33	22.60	15.00	74.00	45.27
EVT-R1 (Ours)	✓	SFT + RL	65.80	53.33	30.60	20.75	74.00	48.89

Ablation Study

The ablation experiments investigate token compression approaches, temporal localization methods, tool-calling behavior, training pipelines, and reward coefficient \(\lambda\):

Configuration / Baseline	Category	SemR	ConR	SpaR	TemR	CasR	Average (Avg.)	Note
VisionZip (Qwen2.5-VL)	Token Compression	26.60	13.00	10.80	2.75	47.33	20.42	Aggressive pruning drops tiny marine organisms
PVC (InternVL2-8B)	Token Compression	45.60	24.67	9.40	2.50	69.00	30.23	Lacks grounding, leading to degraded temporal reasoning
MaxInfo (Key-frame)	Temporal Baseline	48.00	23.67	19.20	11.00	50.00	30.37	Volume maximization fails on sparse dynamic events
VideoITG (Temporal Grounding)	Temporal Baseline	47.40	48.67	20.20	14.25	67.33	39.57	Strong temporal localization competitor
Qwen3-VL (SFT-only)	Training Scheme	61.40	53.33	22.60	15.00	74.00	45.27	Pure SFT cannot master complex policy exploration
EVT-R1 (RL-only, no SFT)	Training Scheme	44.80	39.00	21.40	13.75	63.33	36.52	Domain knowledge gap causes infinite tool loops
EVT-R1 (\(\lambda = 0.00\))	Reward Weight	44.80	38.67	18.80	14.50	65.00	36.35	Zero intermediate tool reward slows convergence
EVT-R1 (\(\lambda = 0.50\))	Reward Weight	54.00	38.33	18.20	12.25	66.67	37.89	Equal weighting of tool and answer rewards
EVT-R1 (\(\lambda = 0.75\))	Full Model	65.80	53.33	30.60	20.75	74.00	48.89	Optimal setup, setting new SOTA across tasks

Key Findings

• SFT warm-up is mandatory prior to multi-turn RL: Applying GRPO directly from a raw foundation checkpoint without SFT leads to degraded accuracy (35.58/36.52), suffering from tool invocation loops and domain vocabulary deficiency. A 1-epoch SFT cold start provides essential grounding, enabling the subsequent RL phase to achieve 48.89.
• Improved tool invocation accuracy: Compared to the SFT-only baseline, EVT-R1 enhances tool selection validity by +4.45% (reaching 70.00%), step correctness by +2.95% (87.35%), and spatial localization IoU \(\ge 0.5\) by +3.75%, directly improving attention concentration on small marine targets.
• Active tool-driven grounding beats heuristic token compression: Passive token compression frameworks (VisionZip, PVC) score only 20~30% on average, as unguided feature pruning deletes critical sparse event cues. Active spatio-temporal tool calling achieves an 18.66-point gain over the strongest compression baseline.

Highlights & Insights

• Visual Replacement Trajectory: Rather than endlessly appending bounding boxes or cropped images to the conversation, EVT-R1 overwrites previous redundant visual representations with refined spatio-temporal crops. This mechanism enforces constant token consumption and completely prevents background distraction.
• Turn-Level Decoupled Reward Modeling: Splitting intermediate tool selection fidelity from final answer accuracy under \(\lambda=0.75\) circumvents sparse reward bottlenecks in multi-step RL, offering stable gradient guidance across multi-turn tool trajectories.
• Generalizable Domain-Specific Agent Recipe: The paradigm of coupling an SFT cold start with tool-integrated reinforcement learning establishes an actionable template for transferring foundation VLMs to low signal-to-noise domains, such as surgical laparoscopy and remote sensing monitoring.

Limitations & Future Work

• Constrained Toolset Scope: The current visual toolbox focuses predominantly on temporal window extraction and spatial segmentation (SAM3). Incorporating underwater image enhancement (e.g., dehazing, color restoration) and 3D multi-object tracking tools represents a natural extension.
• Inference Latency Overhead: Multi-turn sequential tool execution incurs notable computational latency, underscoring the necessity for future distillation of external tool capabilities directly into the core vision backbone for edge-deployed autonomous underwater vehicles (AUVs).

Related Work & Insights

• vs UVLM: While UVLM pioneered video language benchmarks for underwater environments, it remains restricted to static species classification and isolated actions; MarineEVT introduces structured evaluations over fine-grained temporal causality and behavioral interactions.
• vs VideoITG & Chain-of-Frames: Existing temporal grounding techniques employ unidirectional heuristic sampling; EVT-R1 integrates bidirectional tool invocation directly into the policy reasoning loop, allowing the model to iteratively inspect, verify, and resolve complex video inquiries.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First event-centric marine video benchmark; novel integration of spatio-temporal visual tools into dual-reward GRPO reasoning.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking against 11 open/closed VLMs, token compressors, and temporal grounding methods with granular ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with rigorous formal definitions, crisp flowcharts, and exhaustive qualitative analyses.
• Value: ⭐⭐⭐⭐⭐ High scientific and engineering relevance for marine biology, autonomous ocean exploration, and domain-specialized vision agent design.