Boosting Text-Driven Video Segmentation via Geometry-Aware Distillation¶

Conference: ECCV2026
Official Paper: 4230
PDF: Full paper
Authors: Tianyu Zhu, Yingping Liang, Hesong Li, Ying Fu
Code: https://github.com/Tony1882880/GeoLaV
Area: Model Compression (Knowledge Distillation; Referring Video Object Segmentation)
Keywords: referring video object segmentation, monocular geometry pretraining, geometry-aware distillation, novel-view synthesis, dual-teacher supervision

TL;DR¶

Built on SAMWISE, GeoLaV first pretrains on continuous novel views synthesized from single images, then distills semantic and 3D geometric teachers on real videos, reaching 70.5 J&F on Ref-YouTube-VOS—1.3 points above the equally sized 202M-parameter SAMWISE—without retaining teachers or projection heads at inference.

Background & Motivation¶

Referring video object segmentation (RVOS) requires a model to produce target masks throughout a video from a natural-language sentence. The challenge extends beyond recognizing the object in the first frame: the model must preserve the referent through camera changes, occlusion, and object motion. Methods such as SAMWISE already combine SAM2 segmentation priors with text fusion and temporal memory. However, their image pretraining typically uses RefCOCO/+/g, which teaches correspondence between descriptions and single-frame regions without directly providing spatial relationships between different views of the same scene.

Translating or scaling a static image into multiple frames increases training inputs, but does not change parallax and occlusion according to scene depth. Meanwhile, real-video fine-tuning often learns temporal relationships through mask losses: segmenting the current frame correctly is not equivalent to understanding spatial structure across frames. The paper therefore examines two parts of training: whether pretraining exposes the model to continuous views with plausible geometric changes, and whether fine-tuning supplies structural supervision beyond 2D masks.

Rather than designing a new large language model, the paper changes training data and auxiliary supervision within an existing lightweight video segmentation architecture. Core idea: extend single-image supervision into geometry-consistent multi-view learning, then compress semantic and 3D geometric teacher knowledge into memory features, allowing the deployed model to retain its segmentation path while becoming more robust to viewpoint changes.

Method¶

Overall Architecture¶

The input is an RGB video and a sentence describing the target; the output is a binary target mask for each frame. GeoLaV retains the main SAMWISE architecture, using a SAM2 image encoder, text encoder, cross-modal adapters, memory mechanisms, and a mask decoder for language-guided segmentation. Its additions are primarily training components, rather than an extra 3D reconstruction process at inference.

Training consists of monocular geometry pretraining (MGP) and geometry-aware distillation (GAD). MGP first turns a single image into a pseudo video through “Geometry-Consistent View Synthesis,” then learns its representations through “Semantic Memory Alignment.” Starting from this initialization, GAD performs “Geometric–Semantic Dual-Teacher Distillation” on real videos. The three designs in the diagram correspond, in order, to data construction, first-stage supervision, and second-stage supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single image, text, and mask"] --> B["Geometry-Consistent View Synthesis"]
    B --> C["Semantic Memory Alignment<br/>MGP pretraining"]
    C --> D["Geometric–Semantic Dual-Teacher Distillation<br/>GAD fine-tuning"]
    V["Real video, text, and masks"] --> D
    D --> E["Deployed segmentation model<br/>Video and text to per-frame masks"]

Key Designs¶

1. Geometry-Consistent View Synthesis: expand single images through genuine projection relationships rather than 2D perturbations

A feed-forward geometry model first predicts depth or a point map from a single image; the paper names π3 and VGGT as possible models. To account for monocular scale ambiguity, the synthesis process applies random scaling and a bias to predicted depth, then lifts pixels into 3D using sampled camera intrinsics. The requirement is relative depth compatible with the original image structure, rather than precise metric ranging: near and distant points should undergo different projected displacements as the camera moves.

The process then moves these 3D points along a continuous virtual camera trajectory and reprojects them. Trajectories can be linear, piecewise linear, or curved, with neighboring poses related by small rigid transformations rather than sampled independently. An occlusion mask identifies holes exposed by the new viewpoint, and an image inpainting network fills them. In the implementation, five novel views are generated from each COCO source image and combined with the original to form a pseudo video. These sequences contain changes in parallax and depth ordering, making them closer to camera motion than a homography that treats the whole image as a plane. They do not, however, simulate genuine nonrigid object motion, and inpainted regions are not equivalent to real observations.

2. Semantic Memory Alignment: supplement synthetic-frame supervision with dense teacher features

Multi-frame inputs must teach the segmentation network something beyond repeated exposure to an image. The pseudo video passes through the frozen SAM2 image encoder, while text passes through a frozen text encoder. Trainable cross-modal adapters at each level allow the modalities to interact in a shared representation. Fused visual features enter memory attention and combine with prompt-encoder outputs for mask decoding. Mask and image information pass through the memory encoder into a memory bank that subsequent frames can read. This path largely comes from SAMWISE/SAM2: the paper uses memory features that already contain historical context, rather than creating a separate tracking branch.

A parallel visual foundation model teacher supplies semantic features for the pseudo video, using DINOv3-ViT-L in the experiments. A two-layer MLP with GELU projects the student's memory-attention features to the teacher feature dimension, followed by cosine-distance alignment. Following Equation (8), the central constraint is:

\[ \mathcal{L}_{\mathrm{sim}}=1-\frac{F_{\mathrm{proj}}\cdot F_{\mathrm{VFM}}}{\|F_{\mathrm{proj}}\|_2\|F_{\mathrm{VFM}}\|_2}. \]

Here, \(F_{\mathrm{proj}}\) denotes projected student features and \(F_{\mathrm{VFM}}\) denotes teacher outputs. Alignment occurs before prompt-conditioned mask decoding. Rather than asking the teacher to specify which object a sentence refers to, it constrains an intermediate representation carrying global frame information; text-driven segmentation supervision still determines target identity. This also explains the use of a visual teacher that does not depend on the particular referring expression: it adds frame-level structure and semantics instead of replacing language grounding.

MGP jointly optimizes this auxiliary loss and the segmentation loss. The paper explicitly states that feature supervision compensates for missing supervision in synthetic frames. However, the available full text does not clearly explain how novel-view masks are generated or which pixels contribute to the segmentation loss. It therefore does not justify claiming that every synthetic view has complete, human-quality mask annotations. This is a reproduction detail to verify, not a missing step that the note should invent.

3. Geometric–Semantic Dual-Teacher Distillation: retain semantics while adding 3D structural priors on real videos

The second stage loads the MGP-trained model and switches to real videos with text and masks. Alongside the visual foundation model teacher, it introduces a frozen 3D-aware teacher, using π3 in the experiments. The former supplies object appearance and semantic boundaries, while the latter supplies cross-frame structural and depth relationships. The student continues to use the same memory and segmentation path; predicted teacher depth or camera parameters do not directly become the final segmentation output.

Because the teachers have different feature spaces, each receives a separate two-layer MLP projection head. Both heads start from the same student memory representation and align it to the corresponding teacher. Each branch uses a cosine-similarity loss, and their sum forms the distillation objective, trained alongside text-conditioned segmentation supervision. Separate projections avoid forcing teachers with different dimensions and semantics into a single output space, while allowing the memory representation to retain both semantic boundaries and 3D structure.

This supervision does not guarantee explicit recovery of correct 3D geometry. More precisely, teacher representations act as structural regularization. At inference, both teachers and projection heads are removed: real video and text pass directly through the trained segmentation model, retaining its per-frame prediction and memory updates. Thus, “no additional inference overhead” refers to these training-only branches; it does not mean data synthesis or teacher-based training is free.

Loss & Training¶

The first-stage objective adds the segmentation loss to the similarity loss above. Following prior methods, the segmentation loss combines Dice and mask losses. The second stage retains segmentation supervision and adds distillation from semantic and geometric branches. Equations (9) and (10) are corrupted in the cached text, but the surrounding prose explicitly supports two independent projection heads and two cosine losses. No unspecified extra weights or spatial reduction details are inferred here.

Pretraining runs for 6 epochs on RefCOCO/+/g, with the learning rate decaying from \(1\times10^{-4}\) to \(2\times10^{-5}\). Fine-tuning then runs for 4 epochs on Ref-YouTube-VOS at \(2\times10^{-6}\), and the resulting model is evaluated on Ref-YouTube-VOS and Ref-DAVIS17; MeViS training runs for 2 epochs under the stated protocol. Optimization uses Adam on 4 NVIDIA A100 GPUs with 80GB memory each. The paper describes the image encoder, text encoder, and memory attention as frozen, while cross-modal adapters, projection heads, and some prompt parameters are trainable. Other fine-grained parameter settings still require checking against the implementation.

Key Experimental Results¶

Main Results¶

Region similarity \(\mathcal{J}\) measures mask-region overlap, contour accuracy \(\mathcal{F}\) measures boundary matching, and J&F is their mean; higher is better for all three. The table below retains the values reported in Table 1. Ref-YouTube-VOS and MeViS use official evaluation servers, whereas Ref-DAVIS17 uses official evaluation code. Baseline results mainly come from their respective papers rather than uniform retraining, so comparisons should account for their training protocols.

Method	Total parameters	Ref-YouTube-VOS J&F ↑	Ref-DAVIS17 J&F ↑	MeViS J&F ↑
SAMWISE	202M	69.2	70.6	49.5
ReferDINO	230M	69.3	68.9	49.3
GeoLaV	202M	70.5	72.5	50.0
GLUS-A (large VLM, separate reference)	7B	67.3	Not reported	51.3

Compared with the equally sized SAMWISE, gains are 1.3, 1.9, and 0.5 points across the three datasets. Relative to the previously best non-large-VLM methods in the table, gains are 1.2, 1.9, and 0.5 points. These are score differences, not relative percentages. GeoLaV leads this non-large-VLM group, but its MeViS result is below the large-VLM method GLUS-A at 51.3; the conclusion should not be broadened to unconditional superiority over all methods.

Table 2 separately measures generalization when models are trained only on images and evaluated directly on videos. Those results must not be conflated with the video-fine-tuned results above.

Direct evaluation after image pretraining	Ref-YouTube-VOS J&F ↑	MeViS J&F ↑
SAMWISE	31.9	26.4
GeoLaV	47.0	31.6
Absolute gain	15.1	5.2

Ablation Study¶

The following reproduces the J&F column of Table 3. All configurations undergo image pretraining followed by video fine-tuning. The caption does not separately identify the dataset, so the results are presented as component ablations, not relabeled as averages over multiple benchmarks.

Config	Stage I	Stage II	J&F ↑
Vanilla Model	Base pretraining	Base fine-tuning	65.1
Planar augmentation	Planar homography	Base fine-tuning	66.5
MGP only	MGP	Base fine-tuning	67.2
GAD only	Base pretraining	GAD	67.5
Full GeoLaV	MGP	GAD	70.5

Relative to the Vanilla Model in this table, MGP alone adds 2.1 points, GAD alone adds 2.4, and their combination adds 5.4. MGP exceeds planar augmentation by 0.7 points, supporting a benefit beyond simply exposing the model to more frames. However, MGP includes both synthesis and semantic alignment, so the table cannot isolate their individual contributions. Also, the Vanilla Model scores 65.1 in Table 3, whereas SAMWISE scores 69.2 in the main table; they should not be treated as identical baselines. Some J and F components in the original table do not fully agree with the listed means. This note retains the reported J&F rather than silently correcting the table.

Key Findings¶

The clearest improvement appears in transfer from image training to video testing: Ref-YouTube-VOS rises from 31.9 to 47.0, highlighting the value of cross-view structure in pretraining inputs. This is not zero-shot learning without any pretraining.
In the component experiments, the combined configuration exceeds MGP alone by 3.3 points and GAD alone by 3.0. This is consistent with complementary stages, although the paper does not report repeated-run variance to establish the stability of these differences.
On motion-centric MeViS, the fully trained model improves over equally sized SAMWISE by 0.5 points, less than on the other two benchmarks. Geometric priors do not by themselves solve complex motion-language understanding.

Highlights & Insights¶

Single-image data becomes a source of controlled viewpoint variation rather than merely more static supervision. The important factor is not frame count alone, but how depth, camera trajectories, and occlusion jointly determine new frames.
Alignment targets shared features after memory attention rather than final target masks. Teachers provide general visual structure while the text-driven segmentation loss remains responsible for the referent, keeping their roles distinct.
The two teachers are used during training but do not accompany deployment. Structural supervision can therefore enhance an existing video segmenter without adding a complete 3D model to its online processing path.

Limitations & Future Work¶

Single-image synthesis depends on estimated depth and inpainting. It cannot recover the true content of unseen regions or replace genuine dynamic video. The authors identify open-vocabulary and larger-scale 3D-aware multimodal segmentation as future directions.
The available full text lacks detailed synthetic-mask supervision, a breakdown of teacher costs, and sufficient branch-level ablations. It does not establish that all MGP gains come from geometric reprojection or that a geometric teacher necessarily outperforms other dense teachers.
Evidence mainly consists of standard segmentation metrics and qualitative feature visualizations, without an independent quantitative geometric-consistency metric. Together with unclear values and baseline protocols in parts of the tables, this limits the conclusion to segmentation improvements under the reported settings.

vs SAMWISE: SAMWISE supplies the SAM2, text-adaptation, and memory-fusion architecture; GeoLaV mainly adds two stages of geometric training signals. Its contribution is better understood as a training-paradigm enhancement rather than an entirely new segmentation backbone.
vs L2M: L2M provides precedent for lifting a single image into 3D to learn dense matching. This paper applies a related view-construction approach to RVOS pretraining and connects it to teacher distillation on real videos.
vs GLUS: GLUS uses global and local reasoning within a large language model. GeoLaV instead combines a smaller segmenter with training-time teachers and remains competitive on most listed metrics, but does not exceed GLUS-A on MeViS.

Rating¶

Novelty: 3/5 — The combination of single-image geometric augmentation and dual-teacher distillation suits RVOS, although its individual components have precedents.
Experimental Thoroughness: 3/5 — Three benchmarks, image-to-video transfer, and stage ablations are included, but supervision details, statistical stability, and some numerical protocols need clarification.
Writing Quality: 3/5 — The two-stage motivation is understandable, while formula extraction and several tables in the available full text have readability or consistency issues.
Value: 4/5 — A practical route to stronger training supervision without retaining extra inference branches, especially relevant to readers studying image-to-video transfer.