Skip to content

FeVOS: Foresight Expression Video Object Segmentation

Conference: ECCV 2026
arXiv: 2606.25585
Code: None
Area: Video Understanding
Keywords: Video Object Segmentation, Predictive Reasoning, Referring Expression, Reinforcement Learning, Chain-of-Thought

TL;DR

Proposes the FeVOS (Foresight Expression Video Object Segmentation) task, which requires predicting future events and segmenting related objects based on visual cues in observed frames. It introduces a dataset containing 968 videos and 14,525 predictive expressions, accompanied by 2,904 synthetic Chain-of-Thought (CoT) annotations. A two-stage training framework, FeVOS-R1, is designed (CoT supervised fine-tuning cold start + GRPO reinforcement learning with pure IoU rewards for refined reasoning), achieving 42.3 J&F on FeVOS—a 6.5-point improvement over the fine-tuned Sa2VA baseline—while demonstrating strong zero-shot generalization capabilities on ReVOS (60.3) and MeViS (49.5).

Background & Motivation

Referring Video Object Segmentation (RVOS) requires models to segment corresponding objects in videos based on linguistic expressions, demonstrating potential in video editing, autonomous driving, and robotic planning. However, existing RVOS tasks and datasets share a fundamental limitation: all referring expressions describe events or attributes that occur within the observed frames. Early datasets like Ref-DAVIS and Ref-YouTube-VOS primarily focus on static attributes inferable from a single frame (appearance, category, position). Although MeViS introduces motion expressions requiring spatiotemporal understanding across frames (e.g., "the sponge moved"), its reference scope remains confined to the provided observed frames. In other words, existing RVOS is essentially a "post-hoc description" task, where the model only needs to locate the described object within the existing footage.

This design paradigm conflicts with actual applications: in scenarios such as robotic assistance and autonomous driving, agents often need to predict which objects will participate in the next interaction before the action occurs, rather than identifying them post-hoc. For instance, in a kitchen scenario, the question "What tool will be used?" requires the model to infer that a sponge will be used based on visual cues—such as a dirty pot with dishwashing foam, the left hand holding the pot while the right hand is free—and accurately locate the sponge. This is a "pre-emptive prediction" problem rather than a "post-hoc description." Existing RVOS models generally score below 31.0 J&F under zero-shot settings when facing such predictive expressions, exposing a severe limitation. The reason this direction has not been explored previously is twofold: first, the lack of dedicated annotation data for predictive reasoning, requiring annotators to possess both an understanding of video causal chains and precise pixel-level annotation capabilities; second, the traditional RVOS community has long focused on "how to better align language and vision," rather than "how to infer the future from vision."

The core idea of this paper is to advance RVOS from observational reasoning to predictive reasoning, requiring the model to predict the objects involved in future events and generate pixel-level segmentation masks based solely on implicit visual cues in observed frames. To this end, the authors construct a specialized FeVOS benchmark dataset and design a training framework that empowers the model with predictive spatiotemporal reasoning capabilities via explicit Chain-of-Thought (CoT) reasoning and reinforcement learning.

Method

Overall Architecture

The complete FeVOS system covers two levels: benchmark dataset construction and model training methodology. The dataset is constructed via a five-stage pipeline: gathering candidate clips from multi-source videos like COIN, STAR, and EPIC-KITCHENS-VISOR; automatic filtering via Qwen2.5-VL (retaining only ~30.5% of videos with causal chains); manual determination of the split point between observed and future frames (retaining ~44.0%); two-stage expression annotation (design + independent validation to ensure expressions require genuine predictive reasoning and are reasonably inferable, retaining ~85.9%); and SAM2-based interactive mask annotation. This process yields 968 videos, 14,525 predictive expressions, and corresponding pixel-level masks. Building on this, 2,904 Chain-of-Thought annotations (3 per video) are generated automatically using Qwen2.5-VL with visual prompts, providing diverse step-by-step reasoning paths.

The FeVOS-R1 model uses Sa2VA as its backbone (InternVL2.5-4B as the MLLM and SAM2-L as the segmentation module) and adopts a two-stage training paradigm. During inference, given an input video frame sequence and a predictive expression, the visual encoder extracts frame features and feeds them into the LLM. The LLM generates a text response containing the step-by-step reasoning process and designates the segmentation target using a special token [SEG]. The hidden state of [SEG] is then projected and fed into the SAM2 mask decoder to predict target segmentation masks frame-by-frame on the observed frames.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Video Frame Sequence<br/>+ Predictive Expression Q"] --> B["Visual Encoder<br/>Frame-by-frame Visual Feature Extraction"]
    B --> C["LLM Reasoning<br/>Analyze Spatiotemporal Cues & Causal Relations<br/>Generate CoT → Output [SEG]"]
    C --> D["SAM2 Mask Decoder<br/>Based on [SEG] Hidden State<br/>Frame-by-frame Mask Prediction"]
    D --> E["Output: Target Segmentation<br/>Mask on Observed Frames"]
    F["Stage 1: CoT SFT<br/>Chain-of-Thought Cold Start"] -.->|"Establish Reasoning Format"| C
    G["Stage 2: GRPO RL<br/>End-to-End Optimization via IoU Reward"] -.->|"Refine Reasoning Quality"| C

Key Designs

1. Synthetic Chain-of-Thought Annotation and Supervised Fine-Tuning Cold Start: Establishing the "Observation \(\rightarrow\) Reasoning \(\rightarrow\) Segmentation" Cognitive Link

Existing MLLM segmentation methods (e.g., Sa2VA, VideoLISA) perform well on traditional RVOS but lack explicit reasoning processes when faced with predictive expressions—the model directly attempts to map observations to segmentations without intermediate analysis of spatiotemporal cues and causal relationships. This is particularly fatal in predictive scenarios, where the semantics of predictive expressions (e.g., "What tool will be used?") cannot be directly aligned with the visual content of the observed frames—there is no event of "the tool being used" in the observed frames, only implicit causal precursors.

To address this, the authors first automatically generate Chain-of-Thought annotations using Qwen2.5-VL. Specifically, they overlay ground-truth masks onto video frames as visual prompts (highlighting the target object) and feed both the annotated video and the predictive expression to Qwen2.5-VL, asking it to generate step-by-step reasoning to explain "why this highlighted object is the answer." The resulting Chain-of-Thought typically contains analysis of visual cues, temporal context, and causal relationships. Three chains are generated per video to provide diverse reasoning perspectives.

In Stage 1, these CoT annotations are used to perform supervised fine-tuning (SFT) on Sa2VA. The loss function is a weighted combination of mask quality loss and text generation loss: \(\mathcal{L}_{total} = \alpha_{ce}\mathcal{L}_{ce} + \alpha_{dice}\mathcal{L}_{dice} + \alpha_{text}\mathcal{L}_{text}\). The model learns to generate a "reason-then-segment" response format: first performing step-by-step reasoning in natural language (analyzing visual evidence and causal relations in the observed frames), and then outputting the [SEG] token to trigger mask generation by SAM2. The key to this step lies in the "cold start"—it not only teaches the model the correct output format but, more importantly, establishes the "observation \(\rightarrow\) reasoning \(\rightarrow\) segmentation" cognitive link. Ablation studies show that training with pure RL from scratch (without SFT) yields only 36.0 J&F, while starting with SFT cold-start reaches 37.2 J&F even without RL, verifying the foundational role of explicit reasoning supervision.

2. Pure IoU-Driven GRPO Reinforcement Learning: Removing Format Rewards to Align Reasoning Directly with Segmentation Quality

While the SFT stage endows the model with basic reasoning capabilities, the quality of the reasoning chain remains to be optimized, as the generated reasoning process may not align perfectly with the final segmentation target. Traditional GRPO in vision-language reasoning tasks typical employs both accuracy rewards and format rewards (such as requiring outputs to comply with <think></think><answer></answer> tags and JSON formats) to enforce clean output structures.

The key insight of the authors is that when the reasoning format has already been established via CoT-SFT, format rewards are not only redundant but also distract the model capacity—the model might sacrifice reasoning quality to satisfy formatting constraints. Therefore, Stage 2 uses only a pure accuracy reward based on IoU: \(R_{IoU} = \frac{1}{T}\sum_{t=1}^{T}\text{IoU}(\hat{M}_t, M_t)\), where \(\hat{M}_t\) and \(M_t\) represent the predicted and ground-truth masks of frame \(t\), respectively. For each query, GRPO samples \(|G|=4\) outputs and computes the group-normalized advantage \(A_i = (r_i - \bar{r}_G) / \sigma_G\) to update the policy \(\pi_\theta\):

\[\mathcal{J}(\theta) = \mathbb{E}_G\left[\frac{1}{|G|}\sum_{i\in G}\left(\min\left(s_i A_i, \text{clip}(s_i, 1-\epsilon, 1+\epsilon)A_i\right) - \beta\mathbb{D}_{KL}(\pi_\theta||\pi_{ref})\right)\right]\]

where \(s_i = \pi_\theta(o_i|q) / \pi_{old}(o_i|q)\) is the probability ratio between the new and old policies, and the KL divergence regularization term \(\beta\mathbb{D}_{KL}(\pi_\theta||\pi_{ref})\) prevents the model from shifting too far from the reference model.

Unlike RL methods like Seg-Zero that require the model to output JSON-format bounding box coordinates, FeVOS-R1 directly bridges the LLM and SAM2 through the hidden state of the [SEG] token. This allows the IoU reward signal to propagate backward through the SAM2 decoder into the LLM's reasoning process, making the entire pipeline end-to-end optimizable. This avoids the information loss and format alignment overhead brought by intermediate format representations.

Ablation studies validate the correctness of removing the format reward: the IoU-only setup (42.3 J&F) outperforms IoU+Format (40.9) by 1.4 percentage points, while pure format reward yields only 37.7. This demonstrates that when the reasoning format is already solidified by SFT, allowing RL to directly optimize segmentation quality aligns the reasoning chain more closely with the segmentation targets.

A Complete Example

Take a kitchen cleaning scene as an example. The input consists of an observed video (showing a dirty pot with dish soap foam in the sink, the left hand holding the pot handle, and the right hand naturally resting next to the sponge on the right side) and the predictive expression "What tool will be used?". The visual encoder first encodes the observed frames frame-by-frame into a sequence of features. Upon receiving the visual features and the text prompt, the LLM starts generating a Chain-of-Thought: "Given the visual evidence — a dirty pot with dish soap applied in the sink, the left hand holding the pot handle while the right hand is free — the scene indicates a transition to cleaning action. The right hand is positioned near the sponge and is unoccupied, making it the most likely tool to be used next. Therefore, the sponge on the right side will be used." During this reasoning process, the model not only infers the category "sponge" but also resolves spatial ambiguity through the position of the right hand—locating the sponge on the right instead of other potential cleaning tools. After the reasoning is complete, the LLM outputs the [SEG] token, whose hidden state is projected as a prompt embedding for SAM2. Based on this, the SAM2 mask decoder predicts the pixel-level segmentation mask of the sponge on each observed frame. The final output mask sequence precisely localizes the sponge on the right in the observed frames, discarding the left hand, the pot, the dish soap bottle, and other distractors.

Loss & Training

Stage 1 (CoT-SFT): Only the LLM and the SAM2 mask decoder are updated, while other parts are frozen, leveraging LoRA (rank=128) for efficient fine-tuning. The optimizer uses a learning rate of \(2\times10^{-5}\) paired with a cosine annealing scheduler, a batch size of 4, and 4 gradient accumulation steps, training for 4 epochs on the CoT annotations of the FeVOS dataset.

Stage 2 (GRPO RL): The SAM2 mask decoder is frozen, and only the LLM is fine-tuned (with the same LoRA configuration). For each group, \(|G|=4\) outputs are sampled for GRPO advantage estimation, with a learning rate of \(1\times10^{-5}\) on the same referring data as the SFT stage. All experiments are conducted on 4 NVIDIA RTX 4090 GPUs. During inference, the model generates the CoT response in an autoregressive manner, extracts the position of the [SEG] token, and uses its hidden state to prompt SAM2 to generate the final frame-by-frame masks.

Key Experimental Results

Main Results

Method Backbone J F J&F
ReferFormer ResNet-50 16.4 20.0 18.2
LMPM Swin-T 17.1 20.7 18.9
VISA Chat-UniVi-7B 22.9 28.2 25.6
VideoLISA LLaVA-Phi-3-V-3.8B 22.9 29.4 26.1
VideoGLaMM Phi3-Mini-3.8B 21.7 26.7 24.2
VRS-HQ Chat-UniVi-7B 28.8 33.3 31.0
GLUS Chat-UniVi-7B 27.4 31.7 29.6
Sa2VA InternVL2.5-4B 23.7 27.2 25.4
GLUS* (FeVOS Fine-Tuned) Chat-UniVi-7B 31.0 35.9 33.5
Sa2VA* (FeVOS Fine-Tuned) InternVL2.5-4B 33.1 38.4 35.8
FeVOS-R1 InternVL2.5-4B 39.5 45.1 42.3

All zero-shot methods score below 31.0 J&F, confirming that the predictive reasoning task is significantly more challenging than traditional RVOS. Traditional methods like ReferFormer (18.2) and LMPM (18.9) fail almost completely, as they are solely designed to align explicit descriptions with visual content. Sa2VA improves from 25.4 (zero-shot) to 35.8 with direct SFT fine-tuning (+10.4), indicating that domain adaptation is effective; FeVOS-R1 further pushes this to 42.3 (+6.5), proving that Chain-of-Thought reasoning and RL optimization are indispensable in predictive scenarios. Notably, the best performance on FeVOS (42.3 J&F) is far below the model's performance on traditional RVOS benchmarks (ReVOS 60.3), quantifying how much harder "prediction" is compared to "description."

Generalization Results

Method ReVOS All (J&F) ReVOS Inference Subset MeViS (J&F)
Sa2VA (Zero-Shot) 59.1 55.6 46.4
Sa2VA* (FeVOS Fine-Tuned) 58.1 55.2 46.5
FeVOS-R1 60.3 57.8 49.5

Direct SFT fine-tuning on Sa2VA actually drops by 1.0 point on ReVOS (59.1 \(\rightarrow\) 58.1), suffering from catastrophic forgetting. However, FeVOS-R1 not only restores performance but also outperforms the zero-shot baseline (60.3), displaying a notable gain on the inference subset (57.8 vs 55.2, +2.6). On MeViS, FeVOS-R1 (49.5) exceeds similarly sized models like VideoLISA (44.4) and VideoGLaMM (45.2). This indicates that CoT+RL training does not lead to overfitting; instead, it achieves cross-domain generalization by enhancing reasoning capabilities.

Ablation Study

Ablation of Training Stages:

Configuration CoT-SFT RL J F J&F
SFT only Yes No 34.7 39.8 37.2
RL only No Yes 33.5 38.6 36.0
Two-stage Yes Yes 39.5 45.1 42.3

Pure RL (36.0) is lower than pure SFT (37.2). This is because, without a cold-start, the model tends to produce meaningless, trivial responses, making it difficult for RL to spontaneously yield effective reasoning without structural guidance. Combining the two stages (42.3) yields a +5.1 gain over SFT-only, proving that SFT for establishing reasoning formats and RL for refining reasoning quality are highly complementary.

Ablation of Reward Designs:

Configuration IoU Reward Format Reward J F J&F
Format only No Yes 35.1 40.3 37.7
IoU + Format Yes Yes 38.3 43.5 40.9
IoU only Yes No 39.5 45.1 42.3

Once the model has mastered the output format through CoT-SFT, additional format rewards are not only unhelpful but even detrimental: the IoU-only setup (42.3) achieves a 1.4 point improvement over IoU+Format (40.9). Format rewards force the model to dedicate part of its capacity to satisfying redundant formatting constraints rather than focusing on the segmentation targets.

Key Findings

  • Both Stages Are Indispensable: Direct SFT or direct RL alone is far inferior to their combination. SFT provides the "syntax" of reasoning (output format and fundamental reasoning patterns), while RL provides the "semantics" of reasoning (aligning reasoning directly with segmentation quality). Skipping SFT cold-start drops performance by 6.3 points; bypassing RL refinement drops it by 5.1 points. Both contribute comparably and are highly complementary.
  • Format Rewards Lead to Negative Optimization when Formatting is Solidified: This is a counterintuitive finding. While most GRPO works treat format rewards as a standard component, experiments in this paper clearly demonstrate that when SFT has already established output structures, format rewards are redundant and hinder task objective optimization by dispersing model capacity. This has general reference value for reward design in vision-language reinforcement learning.
  • The Difficulty of Predictive Reasoning is Severely Underestimated: The gap between the best performance on FeVOS (42.3 J&F) and ReVOS (60.3 J&F)—amounting to an 18-point difference—quantifies the incremental difficulty of "predicting the future" compared to "describing the present." This is not merely a model capacity issue, but a fundamental challenge induced by a paradigm shift in the task itself.
  • Direct SFT Fine-Tuning on FeVOS Suffers Overfitting Risks: Direct SFT (Sa2VA*) suffers a performance drop on ReVOS (59.1 \(\rightarrow\) 58.1), whereas the RL-refined FeVOS-R1 surpasses the zero-shot baseline (60.3). This suggests that the RL stage encourages the model to acquire transferable reasoning abilities rather than merely memorizing statistical patterns within FeVOS.

Highlights & Insights

  • Paradigm Shift from "Post-hoc Description" to "Pre-emptive Prediction" in RVOS: This is more than a simple dataset expansion; it fundamentally redefines the cognitive requirements of the task. The authors' analysis of why existing RVOS is insufficient is highly accurate—the core limitation is not model power, but rather the task definition itself excluding predictive reasoning. This problem-driven approach to motivation writing is highly commendable.
  • Generating CoT Annotations with Visual Prompts (Overlaid GT Masks) instead of Text Descriptions: This is an elegant design choice. It allows Qwen2.5-VL to directly "see" the answers and then justify them, avoiding ambiguities and inaccuracies inherent in linguistic spatial descriptions. The generated CoTs ensure high-quality reasoning while naturally supporting diversity (3 different perspectives per video).
  • The "Detrimental Format Reward" Discovery Offers Generalizable Value: In other vision-language reasoning contexts using GRPO, if the output structure is already well-learned during the SFT stage, removing format rewards should be considered to unleash full model capacity. This insight challenges the standard practice of treating format rewards as an essential suite since DeepSeek-R1.
  • End-to-End RL Optimization based on the [SEG] Token Avoids Intermediate Representation Loss: Unlike methods like Seg-Zero that require models to output JSON-formatted bounding box coordinates, the hidden state of the [SEG] token serves as a continuous representation directly interfacing with SAM2. This allows the IoU gradient to be backpropagated to the LLM's reasoning process without information loss, realizing genuine end-to-end optimization.

Limitations & Future Work

  • Massive Headroom for Absolute Performance Improvement: The 42.3 J&F on FeVOS lags significantly behind traditional RVOS benchmarks (60+), suggesting a long path ahead given the inherent difficulty of predictive reasoning. Future directions suggested by the authors include overcoming frame limits in current MLLMs to harness richer global contexts, optimizing sampling strategies to capture transient visual cues, incorporating CoT-SFT and RL training on wider RVOS datasets to raise generalization limits, and mitigating hallucinations during thinking.
  • CoT Annotations Depend on a Simple MLLM (Qwen2.5-VL): All 2,904 CoTs are generated by the exact same model, meaning reasoning styles and coverage are restricted by its latent capacity and preferences. If Qwen2.5-VL exhibits systematic biases in specific types of causal reasoning, this bias could be magnified during SFT. Future updates could consider multi-model ensembling or introducing manually checked CoT annotations to enhance diversity and robustness.
  • RL Stage Training Confined Solely to FeVOS: Although generalization experiments demonstrate zero-shot transfer capabilities, mixing partial traditional RVOS data during RL training might further mitigate overfitting and bolster cross-domain performance.
  • Underspecified Prediction Uncertainty: Real-world futures are non-deterministic and can unfold in various plausible ways (e.g., using a sponge or a brush). The current model only generates a single deterministic prediction; future studies should explore multi-modal outputs or probabilistic mask predictions.
  • Lack of Mechanisms to Detect and Alleviate Reasoning Hallucinations: During CoT reasoning, the model might produce analyses that sound plausible but conflict with visual evidence. While listed as a future direction, the authors do not provide concrete solutions. A potential workaround is injecting reasoning-visual consistency checks as auxiliary signals in the RL rewards.
  • vs MeViS: MeViS introduces motion expressions that require spatiotemporal understanding (e.g., "the sponge moved"), but still describes events that have occurred within the observed frames—fundamentally representing observational reasoning. FeVOS requires predicting future events based on observed frames (e.g., "What tool will be used?"), which constitutes predictive reasoning. The core difference between the two lies in the temporal reference point—MeViS faces the past and present, while FeVOS faces the future. The evolutionary trajectory from MeViS to FeVOS (static attributes \(\rightarrow\) motion attributes \(\rightarrow\) future events) reveals a notable trend.
  • vs Sa2VA / VISA / VideoLISA: These methods excel on traditional RVOS but yield extremely low zero-shot performance on FeVOS (25.4–31.0). This indicates that while MLLM+segmentation architectures possess innate reasoning potential, they lack dedicated training for predictive reasoning. FeVOS-R1 achieves substantial improvements using the identical Sa2VA architecture purely by amending training strategies, proving that "how to train" is currently more critical than "which architecture to use."
  • vs Seg-Zero / Veason-R1: These concurrent works also introduce RL to video segmentation but focus on observational segmentation in traditional RVOS settings. FeVOS-R1's novelty lies in utilizing RL to optimize the predictive reasoning chain itself—realizing end-to-end IoU optimization without middle format conversions like JSON via the [SEG] token's hidden state, avoiding formatting overheads and information loss in intermediate representations.
  • vs DeepSeek-R1 / VLM-R1: DeepSeek-R1 proved that RL can elicit LLM reasoning capabilities, and VLM-R1 extended this to vision-language tasks. While inheriting this paradigm, FeVOS-R1 uncovers an important, counterintuitive phenomenon: given sufficient SFT cold-start, pure task rewards (IoU) outperform the combination of task and format rewards. This finding carries general reference value for reward engineering in visual-language reinforcement learning and deserves validation across a wider spectrum of VL tasks.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Advancing RVOS from observational reasoning to predictive reasoning is a clear paradigm innovation, where both the task definition and dataset construction display distinct differentiation. On the methodology side, combining CoT SFT and GRPO RL is not entirely pioneering, but the insight of removing format rewards and the application of RL to predictive segmentation are highly novel.
  • Experimental Thoroughness: ⭐⭐⭐⭐☆ Main results cover 8 zero-shot baselines and 2 fine-tuned baselines, generalization experiments span the ReVOS and MeViS benchmarks, and ablation studies systematically disentangle two key variables: training stages and reward designs. It lacks an independent assessment of CoT annotation quality (e.g., human-reviewed accuracy) and comparisons between different CoT generation strategies (e.g., different MLLMs, various prompting formulations).
  • Writing Quality: ⭐⭐⭐⭐☆ The problem motivation is clearly articulated; Figure 1 (task comparison) and Figure 2 (CoT examples) visually demonstrate the difference between predictive and observational reasoning. The data construction pipeline and the diagram of the two-stage training are highly illustrative. The layout of some tables could be further optimized.
  • Value: ⭐⭐⭐⭐⭐ Predictive video understanding is a vital step toward proactive intelligence. This work not only introduces a high-quality benchmark but also demonstrates the effectiveness of Chain-of-Thought reasoning coupled with RL in this direction, carving out clear research pathways for subsequent work (such as predictive perception in robot planning, intent prediction in autonomous driving, pre-action judgment in embodied AI, etc.). The 42.3 J&F upper bound indicates substantial headroom for development, which is highly likely to draw follow-up research from the community.