Segmenting, Fast and Slow: Real-Time Open-Vocabulary Video Instance Segmentation with Dual-Path Processing¶

Conference: ECCV 2026
arXiv: 2607.00124
Code: To be confirmed
Area: Segmentation / Video Instance Segmentation
Keywords: Open-Vocabulary Video Instance Segmentation, Real-Time Inference, Dual-Path Architecture, Mobile Deployment, Feature-Space Propagation

TL;DR¶

SegFS proposes a dual-path fast-and-slow framework for open-vocabulary video instance segmentation (OV-VIS). It runs a full object-centric model on sparse keyframes to extract instance embeddings, and then projects these embeddings back into the backbone feature space. A lightweight Fast Feature Aggregator is utilized on intermediate frames to complete efficient instance relocation and segmentation. This approach achieves up to a 14x reduction in on-device latency and crosses the 30 FPS real-time threshold on mobile devices for the first time, while limiting the performance drop to within 1 AP.

Background & Motivation¶

Open-vocabulary video instance segmentation (OV-VIS) requires continuous localization, tracking, and segmentation of object instances of arbitrary categories specified by text prompts. The current mainstream paradigm is represented by DETR-like object-centric architectures. A typical system (such as GLEE) consists of three parts: a visual backbone (to extract multi-scale features), a feature enhancer (containing a pixel decoder and early text-visual fusion modules), and an object decoder (which interacts with the enhanced features via learnable queries to output instance embeddings and masks). The feature enhancer, involving dense cross-modal interactions between text embeddings and multi-scale features, represents the most severe computational bottleneck in the pipeline. Even though mobile optimization solutions like MOBIUS (which fuses features by selecting a single scale) and TROY-VIS (which decomposes the enhancer into lightweight operations) have recently emerged, the inference cost of the feature enhancer still dominates on edge devices (as shown in the FLOPs and on-device latency analysis in Figure 1). This bottleneck deteriorates sharply as image resolution and vocabulary size scale up.

To resolve this challenge, existing works have leveraged the concept of keyframes. For instance, MobileInst directly reuses keyframe object embeddings on intermediate frames after running the full model on keyframes, thereby bypassing the object decoder overhead. However, this approach still needs to execute the feature enhancer on every frame—which is indeed the true computational bottleneck. The Key Challenge is that high-precision OV-VIS requires deep multi-modal fusion to understand open semantics, while mobile real-time inference (over 30 FPS) requires per-frame processing time within 33 ms. These two objectives are mutually exclusive under existing architectures.

The Key Insight of this work is that the spatial semantic information used for fine-grained localization in the current frame is already encoded in the multi-scale features of the visual backbone, making it unnecessary to repeatedly run the expensive feature enhancer on every frame. Based on this, the authors propose SegFS—a dual-path framework that runs the full object-centric model (including the feature enhancer and object decoder) only on sparse keyframes ("slow path"). It extracts instance embeddings, projects them back into the backbone feature space, and then employs an extremely lightweight Fast Feature Aggregator on intermediate frames ("fast path") to perform instance segmentation in conjunction with the current frame's backbone features. Core Idea: Decouple the computationally heavy multi-modal semantic understanding from the efficient dense mask prediction in OV-VIS. By feeding instance embeddings from the object decoding space back into the backbone feature space, the fast path can achieve frame-level instance relocation and segmentation using only a lightweight aggregation module, completely bypassing the feature enhancer.

Method¶

Overall Architecture¶

The overall architecture of SegFS is centered around a dual-path spatiotemporal alternating execution strategy. Given a video sequence, the system selects keyframes at a fixed interval (typically every 6 frames as a keyframe), forming a propagation window of "1 keyframe + 5 intermediate frames".

The slow path runs on sparse keyframes. Upon receiving a keyframe image, a frozen object-centric OV-VIS model (which can be GLEE, MOBIUS, or TROY-VIS) sequentially executes the backbone network to extract multi-scale features, the feature enhancer to complete text-visual fusion and multi-scale interactions, and the object decoder to generate instance queries and output instance embeddings and masks. This process provides high-quality open-set detection and segmentation results, but is computationally expensive.

The fast path runs on all intermediate frames. Each frame first passes through the same visual backbone (sharing weights with the slow path, also frozen) to extract multi-scale feature maps {P2, P3, P4, P5}. After projecting these feature maps into a unified channel dimension, they enter the proposed Fast Feature Aggregator. This aggregator injects instance embeddings from keyframes via the Object Guidance module at the coarsest semantic level P5, and then progressively upsamples and fuses them with finer-scale feature maps to ultimately generate high-resolution feature maps for 1x1 convolutional mask prediction.

The instance embeddings output by the slow path are projected into the backbone feature space (features understandable by the fast path) via a 3-layer FFN. The system then selects the Top-K embeddings with the highest similarity to the text categories, appends a learnable background token, and refines them using self-attention and cross-attention before feeding them into the fast path. Temporal association across the video sequence reuses the tracking-by-matching paradigm of MinVIS, which performs optimal bipartite graph matching by computing the cosine similarity of instance embeddings between frames.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video frame sequence"] --> B{"Is it a keyframe?"}

    B -->|Yes| C["Slow Path<br/>Full OV-VIS Model<br/>(Backbone + Enhancer + Decoder)"]
    C --> D["Instance embeddings + Masks"]

    B -->|No| E["Fast Path<br/>Shared Backbone"]
    E --> F["Multi-scale feature maps<br/>{P2, P3, P4, P5}"]

    D --> G["Projection FFN +<br/>Top-K selection +<br/>Background token"]
    G --> H["Self-attention +<br/>Cross-attention refinement"]

    H --> I["Object Guidance<br/>(Injection + Gated Fusion)"]
    F --> I

    I --> J["Progressive upsampling<br/>P5→P4→P3→P2"]
    J --> K["1x1 Convolution mask prediction<br/>(Instance embeddings as kernels)"]

    C --> L["MinVIS<br/>Inter-frame match tracking"]
    K --> L
    L --> M["Final instance mask sequence"]

Key Designs¶

1. Dual-Path Fast-and-Slow Separation: Shifting Feature Enhancers to Sparse Keyframes

The fundamental bottleneck of existing OV-VIS models is that the feature enhancer must perform dense multi-scale cross-modal interactions on every single frame, with its computational load scaling linearly with the vocabulary size. The core strategy of SegFS is to break the convention of "executing all three components on every frame" by restricting the heaviest feature enhancer and object decoder to sparse keyframes (running only once every 6 frames), while intermediate frames bypass them entirely. This seemingly simple "skipping" operation is feasible because intermediate frames do not need to perform open-set detection from scratch; keyframes already provide the category information and initial embeddings of the instances. The task of intermediate frames is simply to relocate and refine existing instances based on current frame changes (e.g., object movement, deformation, occlusion changes), which does not require heavy multi-modal semantic understanding.

In implementation, the slow path adopts a frozen pre-trained object-centric model (GLEE/MOBIUS/TROY-VIS), outputting 300 object queries. The fast path shares the same backbone, but the backbone output directly generates masks via the Fast Feature Aggregator. Since the backbone is also frozen, the fast path effectively has no learnable visual encoder; all learnable parameters reside in the lightweight aggregator. This design ensures that the computational cost of the fast path is virtually independent of the backbone network size and vocabulary size, offering excellent scalability.

2. Object Guidance: Instance Semantic Injection and Gated Fusion

How to efficiently feed the instance embeddings extracted from keyframes into the feature space of the current frame is the key technical challenge of the fast path. The Object Guidance module achieves this goal and operates only on the coarsest-scale feature map P5 (to avoid expensive cross-modal operations at high resolutions).

The module operates in two steps. The first step is Object Injection: for each spatial location in P5, the module computes its multi-head cosine similarity with the K+1 tokens (K selected instance embeddings + 1 background token). These similarities are scaled by a learnable temperature parameter tau and softmax-normalized along the instance dimension to obtain the attention distribution of each spatial location over the instances. By multiplying these probabilities with the instance embeddings and summing them, the "Object-Aware Feature Map" I5 is formed. When the network learns to decrease the tau value, the softmax distribution becomes sharper, effectively replacing each spatial position with its semantically closest instance embedding—acting as a soft instance assignment in the feature space.

The second step is Gated Fusion: a direct issue with using I5 alone is that it only carries coarse semantic information from the keyframes, lacking structure and details of the current frame. Hence, a DSConvGN module (depthwise separable convolution + GroupNorm + SiLU) is first used to spatially smooth I5, and then a gating mechanism is utilized to fuse I5 with the original P5 feature map:

\[ \tilde{P}_5 = P_5 + \sigma(\text{GateConv}(P_5 \parallel I_5)) \odot \text{DSConvGN}(I_5) \]

where sigma is the sigmoid function, and GateConv predicts a spatial mixture mask that controls to what extent each location should ingest instance semantic information. This design ensures that the network preserves precise visual cues from P5 while absorbing instance semantics from I5, with the gating being fully avoidable for end-to-end learning of the optimal mixture ratio.

3. Progressive Upsampling and Lightweight Feature Fusion

After obtaining the semantics-infused P5, it needs to be progressively restored to the high resolution of P2 to output detailed masks. To avoid repeating Object Guidance at high resolutions (which would introduce extra overhead), SegFS adopts an extremely lightweight progressive upsampling fusion scheme. The core operation is: concatenate the upsampled P5 with P4 along the channel dimension, pass them into a DSConvGN block for local fusion and refinement, to obtain the enhanced P4 feature; this operation is iteratively applied down to P3 and P2.

The key design decision is using DSConvGN—a lightweight module consisting of a 3x3 depthwise separable convolution, a 1x1 pointwise convolution, GroupNorm, and SiLU activation. This design keeps the compute of the entire Fast Feature Aggregator basically constant (approx. 10.1 GFLOPs, independent of backbones and vocab structures) and running with extremely low latency (only 8.3 ms on an S25 Ultra for the MobileNetV4-CM variant).

Finally, the high-resolution feature map at P2 resolution passes through a 1x1 convolution to output mask logits, where keyframe instance embeddings act as the convolution kernels—each group of kernels corresponding to a mask activation map of one instance. This naturally translates the semantic information of the instance embeddings into spatial segmentation signals, avoiding additional classification heads.

Loss & Training¶

SegFS is trained entirely on image-level instance segmentation datasets, without requiring video ground truths. During training, the same image passes through the slow and fast paths sequentially. The slow path (frozen) propagates forward normally, outputting category logits, bounding box predictions, and object queries. These queries are projected into the fast feature space and fed into the fast path. Hungarian matching utilizes a hybrid matching cost: category logits and bounding boxes come from the slow path, while mask predictions come from the fast path. Once the optimal assignment is settled, the fast path is supervised with Mask Loss and DICE Loss. Training uses the AdamW optimizer with a learning rate of 1e-4 and a weight decay of 0.05 (increased to 0.1 after 400k iterations). The model is trained on 4 A100 GPUs with a batch size of 128 for 500k iterations using multi-scale training (short side 320 to 640 pixels).

Key Experimental Results¶

Main Results¶

SegFS is evaluated using 5 different frozen slow networks as backbones: GLEE, MOBIUS (MNv4-CM/CL, ResNet50), and TROY-VIS. It is compared against propagation strategies such as Copy, Reuse Objects, LiteFlowNet2, RAFT, and MPVSS on four datasets: YouTubeVIS19, OVIS, BURST, and LV-VIS. The table below shows the configurations using MOBIUS-MNv4-CM as the slow network (AP metrics, and FPS is measured as the amortized value over a 6-frame sliding window on an S25 Ultra):

Dataset	Metric	Full Model (Upper Bound)	Reuse Objects	MPVSS	SegFS (Ours)	SegFS FPS
YouTubeVIS19	AP	48.7	42.1	39.8	41.6	38.2
OVIS	AP	23.6	15.0	13.9	14.1	38.2
BURST	HOTA	40.1	30.6	28.4	30.3	38.2
LV-VIS	AP	16.7	15.8	14.8	15.2	38.2

Under all 5 slow network configurations, SegFS is the only propagation method to cross the 30 FPS real-time threshold (reaching 38.2 FPS on the MOBIUS-MNv4-CM configuration). While its performance is very close to the Reuse Objects baseline (with a gap of only -0.5 to -5.4 AP), it achieves a 2-3x FPS improvement.

Ablation Study¶

Component-wise ablation based on MOBIUS-Mini-M on YouTubeVIS19 (trained for 100k iterations):

Config	Injection	Background token	Attention refinement	Smoothing	YTVIS19 AP	OVIS AP
Baseline	✗	✗	✗	✗	37.9	10.7
+ Injection	✓	✗	✗	✗	39.6	12.5
+ Background token	✓	✓	✗	✗	39.6	12.8
+ Attention refinement	✓	✓	✓	✗	40.2	13.6
+ Smoothing	✓	✓	✓	✓	40.9	13.7

Key Findings¶

Injection module makes the biggest contribution: Simply adding Object Injection boosts AP from 37.9 to 39.6 (+1.7 AP), indicating that injecting instance semantics into the feature space is a prerequisite for the fast path to function. Attention refinement (+0.6 AP) and smoothing (+0.7 AP) also yield consistent gains, whereas the background token makes a minor contribution (+0.3 AP on OVIS but no change on YTVIS19).
Robustness to propagation intervals: As shown in Figure 5, methods propagating semantic embeddings like SegFS and MPVSS exhibit very graceful performance degradation when the propagation interval T increases from 1 to 10, whereas pixel-mask-warping-based methods (Copy, RAFT, LiteFlowNet2) drop sharply. This proves that "propagating semantic embeddings + relocation based on current frame features" is far more robust than "propagating pixel masks."
Significant on-device latency edge: SegFS-MNv4-CM requires only 8.3 ms/frame on the S25 Ultra, which is 14 times faster than MOBIUS (115.7 ms), and its latency is almost independent of vocabulary size (while the feature enhancer's latency spikes significantly as vocabulary goes from 40 to 1196, SegFS's fast path remains constant).
Sensitivity to Top-K: The AP peaks at K=50, and slightly decreases when K is further increased to 100. This indicates that excessive instance embeddings might act as noise disrupting the P5 features, while the latency only marginally increases from 0.324 ms to 0.444 ms, remaining negligible.

Highlights & Insights¶

"Feature-Space Propagation" replacing "Pixel-Space Propagation": Most temporal propagation methods (optical flow warping, mask warping) operate in the pixel space, where errors accumulate over frames. The significance of SegFS lies in elevating the propagation from the pixel level to the feature level—propagating the semantic representation of "what the object looks like" rather than "where the pixels of the object are", the former being naturally more robust to deformation and occlusion.
Clever Kernel Reuse Design: At the 1x1 convolution stage, keyframe instance embeddings are directly employed as convolution kernels. This realizes "instance-specific decoding" at the mask prediction level—different instances use different kernels to decode their masks, precluding the need for separate classification or instance heads.
Adaptive Sharpening of Temperature tau: The learnable temperature parameter allows the softmax attention to automatically transition from a uniform distribution to a hard assignment during training ("each spatial location belongs to a unique instance"). This elegant implicit learning mechanism requires no explicit supervision to guide instance-spatial correspondence.
Inherent Advantage in Large-Vocabulary Recall: Since the fast path does not involve any text-visual cross-attention, its computational cost is decoupled from the number of categories. When handling large-vocabulary scenes like LV-VIS (1196 classes), the speedup ratio of SegFS is further elevated from 14x to 16x.
Pure Image-Based Training: The propagation capability can be learned using only image-level annotations, without video-level ground truths, dramatically lowering training data barriers—during training, each image is treated as a "keyframe + current frame" combination to simulate video scenarios.

Limitations & Future Work¶

Dependency on slow network quality: The performance upper bound of the fast path is heavily constrained by how well the slow network detects and extracts instance embeddings on keyframes. If an object is missed or falsely detected on a keyframe, the error will propagate directly to subsequent frames. Practical deployment might require introducing a re-detection mechanism to discover newly appearing instances.
Fast path is unaware of new objects: By design, the fast path only propagates existing instance embeddings and cannot discover or segment newly appearing objects on intermediate frames (until the next keyframe). In scenarios requiring real-time discovery of new objects (e.g., surveillance, autonomous driving), the keyframe interval must be set carefully to balance compute and recall.
Propagation window length limit: Experiments show that T=5-6 is the sweet spot balancing performance and cost, but AP drops noticeably when T exceeds 10. In practical applications with severe content changes (e.g., fast motion, frequent entering/leaving the field of view), a shorter keyframe interval might be required, which partly offsets the efficiency gains.
Optical flow methods still hold advantages in certain scenarios: For highly non-rigid deformations (e.g., a person bending or jumping in a video), the "coarse localization + refinement" strategy based on feature injection is less precise than pixel-wise optical flow warping. SegFS shows an AP drop (-0.8 to -4.6) compared to the Reuse Objects baseline on highly occluded datasets like OVIS, indicating that propagation itself still suffers from information loss.
Cross-modal generalization boundaries: This paper focuses on segmenting instances guided by textual categories. It remains unverified whether the slow network can produce sufficiently strong instance embeddings to guide the fast path for fine-grained instructions (such as referring expression comprehension like "the person in red on the left").

vs MobileInst / TROY-VIS: These methods only reuse the output of the object decoder (i.e., skipping the object decoder but keeping the feature enhancer). SegFS goes a step further to skip the feature enhancer and elevates the propagation granularity from "instance embeddings" to "feature-space conditioning", yielding much larger acceleration.
vs MPVSS: MPVSS predicts instance-level motion fields based on optical flow conditioning to warp masks, operating in the pixel space. SegFS operates in the feature space without explicit optical flow estimation, making it faster and more robust to aggressive motion or boundary appearances. However, MPVSS might have higher propagation precision in closed-set scenarios as it aligns pixels using motion fields.
vs SAM2 / MobileSAMv2: SAM2 adopts a mask propagation + memory bank paradigm that requires maintaining long-term memory. The design of SegFS is closer to an extension of the traditional DETR paradigm, requiring no memory storage, but it also cannot accept user clicks/boxes to specify arbitrary segmentations like SAM2 does.
vs MinVIS: MinVIS proved that DETR instance embeddings are discriminative across frames and can be matched directly for tracking. SegFS deepens this idea: not only are instance embeddings stable across frames, but they can also effectively condition feature maps once projected back into the feature space, which is a stronger conclusion.

Rating¶

Novelty: ⭐⭐⭐⭐ The concept of dual-path fast-and-slow execution is not entirely new in video understanding, but applying it to OV-VIS and designing Object Guidance to propagate instance semantics in the feature space represents an elegant, system-level innovation. In particular, the decision to "bypass the feature enhancer" strikes at the heart of the issue.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The paper conducts exhaustive comparisons against 5 different slow networks and 6 propagation baselines on 4 datasets. The ablation studies cover every single component, the efficiency analysis spans 4 mobile devices + 2 GPUs, and sensitivity analyses are provided for different K and T values. The experiments are exceptionally thorough.
Writing Quality: ⭐⭐⭐⭐ The motivation is clear, the methodology is introduced step-by-step, and the experimental design is highly logical. The downside is that Tables 1/2 are too dense (spanning many rows, causing average readability), and the citation order of some key figures/tables could be optimized.
Value: ⭐⭐⭐⭐⭐ It achieves a breakthrough of 30+ FPS real-time OV-VIS on mobile devices, providing a highly feasible engineering paradigm for edge deployment. The method is clean (with the fast path using only 10.4M parameters and 10.1 GFLOPs), easy to reproduce, and holds high practical value.