MiNQVIS: Mitigating Noisy Queries for Robust Online Video Instance Segmentation¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://nothing898.github.io/MiNQVIS
Area: Segmentation
Keywords: Video Instance Segmentation, Online Tracking, Noisy Queries, Prototypical Contrastive Learning, Dual-Memory Association
TL;DR¶
Targeting the corruption of temporal association and contrastive supervision caused by noisy queries under heavy occlusion, blur, and clutter, MiNQVIS introduces query-level gating, video-level prototypical contrastive learning, and dual-memory fusion to achieve robust, drift-resistant online video instance segmentation.
Background & Motivation¶
Video Instance Segmentation (VIS) requires models to simultaneously perform pixel-level instance localization, segmentation, and temporal identity association across video sequences. Due to the strict requirements for causal processing and low latency in real-world scenarios, online VIS—which operates strictly on streaming inputs without accessing future frames—serves as a core perceptual foundation for autonomous driving, mixed reality, and embodied intelligence. With the emergence of Transformer-based architectures like Mask2Former, prevailing online frameworks such as IDOL, CTVIS, CAVIS, and VISAGE leverage learnable instance queries to jointly manage per-frame detection and temporal linking, while adopting contrastive objectives or momentum-based tracking memories to enhance identity consistency.
However, existing frameworks fundamentally rely on a critical premise: per-frame predicted instance queries provide sufficiently reliable visual cues for cross-frame association and contrastive supervision. In realistic, unconstrained video scenarios, severe object occlusions, rapid motion blur, background clutter, and dense object interactions frequently violate this assumption. The resulting noisy queries—manifested as false positives, fragmented masks, and poorly localized predictions—trigger a cascading three-fold failure: first, corrupted embeddings distort cosine-similarity matching, inducing frequent identity switches and fragmented tracks; second, unreliable queries contaminate positive and negative pair construction during contrastive learning, injecting contradictory gradient updates; third, in momentum memory banks governed by exponential moving averages, transiently corrupted embeddings are written into the tracking state, accumulating temporal drift that leads to irrecoverable identity loss over long sequences.
While prior research has continually scaled model capacity and devised increasingly complex contextual interaction modules, the root cause—noisy query contamination—has remained unaddressed. The central insight of this work is that robust online tracking cannot be achieved by relying naively on single-frame predictions; instead, temporal information flow must be stabilized by explicitly identifying and suppressing query noise throughout the pipeline. Core idea: propose MiNQVIS, the first noise-aware online video instance segmentation framework, which integrates query-level gating to suppress unreliable predictions, video-level prototypical contrastive learning to provide stable representation supervision, and dual-memory fusion combining momentum updates with history-best retrieval to eliminate tracking drift.
Method¶
Overall Architecture¶
MiNQVIS builds upon the standard Mask2Former online VIS architecture. Given an incoming video frame sequence, the backbone and Transformer decoder extract a set of instance queries per frame along with their classification logits and predicted mask embeddings. On top of this baseline, MiNQVIS integrates three tightly coordinated modules: first, a query-level gating mechanism operating across both training and inference to filter out low-confidence and background queries via classification thresholding and \(\ell_2\)-norm normalization; second, a video-level prototypical contrastive learning (VPCL) objective that aggregates reliable cross-frame embeddings within a training clip into robust identity prototypes, ensuring noise-resilient representation learning; third, a dual-memory fusion association module during inference that balances short-term continuity and long-term historical anchors via bi-directional Softmax calibration, producing highly stable online tracks.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input frame sequence It<br/>Mask2Former extracts per-frame queries and embeddings"] --> B["Query-level Gating<br/>Confidence-guided filtering + L2-norm stabilization"]
B -->|Training stage| C["Video-level Prototypical Contrastive Learning (VPCL)<br/>Aggregate reliable cross-frame embeddings into stable prototypes"]
B -->|Inference stage| D["Dual-memory Fusion Association<br/>Momentum embedding + history-best retrieval with bi-directional Softmax"]
C --> E["Joint optimization: segmentation loss + gating hinge loss + VPCL loss"]
D --> F["Output stable tracks and consistent instance masks"]
Key Designs¶
1. Query-level gating: confidence-guided thresholding and norm decoupling
Online VIS models produce hundreds of per-frame instance queries, many of which capture background regions or noisy artifacts under severe occlusion and clutter. Conventional detectors relegate confidence filtering to post-processing, which leaves contrastive supervision and online memory banks vulnerable to noisy query intrusion. Furthermore, earlier methods relying on unnormalized inner products often suppress background queries implicitly by shrinking their feature magnitudes, coupling matching scores with feature norms and destabilizing similarity calculations.
To address these vulnerabilities, this design establishes a lightweight query-level gating mechanism applied during both training and inference. For each query \(j\) at frame \(t\) with classification logits \(\mathbf{z}_{t,j} \in \mathbb{R}^{C+1}\) (where the \((C+1)\)-th index represents background), foreground and background probabilities are defined as \(p^j_{t,\text{bg}} = \text{softmax}(\mathbf{z}_{t,j})_{C+1}\) and \(p^j_{t,\text{fg}} = 1 - p^j_{t,\text{bg}}\). A query is retained only when its foreground confidence exceeds an empirical threshold \(p^j_{t,\text{fg}} > \gamma\), suppressing low-confidence queries before they can corrupt downstream modules. To reinforce the decision boundary, the authors introduce a log-domain hinge regularization loss \(\mathcal{L}_{\text{gate}}\). For Hungarian-matched foreground queries \(\mathcal{S}^+\), the model encourages \(p^j_{t,\text{fg}}\) to exceed an upper margin \(p^\star_{\text{fg}} = \gamma + \mu\); for unmatched background queries \(\mathcal{S}^-\), it enforces \(p^j_{t,\text{bg}}\) to exceed \(p^\star_{\text{bg}} = 1 - \gamma + \mu\) (with margin \(\mu = 0.1\gamma\)): $$ \mathcal{L}{\text{gate}} = \frac{1}{|\mathcal{S}^+|} \sum + \epsilon) \right]}^+} \left[ p^\star_{\text{fg}} - \log(p^j_{t,\text{fg}+ + \frac{1}{|\mathcal{S}^-|} \sum + \epsilon) \right]_+ $$ Every retained embedding }^-} \left[ p^\star_{\text{bg}} - \log(p^j_{t,\text{bg}\(\mathbf{e}_t^j\) is subsequently \(\ell_2\)-normalized as \(\hat{\mathbf{e}}_t^j = \frac{\mathbf{e}_t^j}{\|\mathbf{e}_t^j\|_2 + \epsilon}\), completely decoupling similarity scores from feature magnitudes and bounding similarities onto a stable hyperspherical manifold.
2. Video-level prototypical contrastive learning (VPCL): temporally aggregated stable supervision
Standard contrastive learning in VIS pulls embeddings of the same instance across adjacent frames closer while pushing different instances apart. Under severe occlusion or motion blur, single-frame embeddings often experience severe degradation or semantic corruption; forcing the network to align with such distorted anchors compromises the discriminability of the learned metric space. VPCL mitigates this issue by exploiting temporal redundancy across video clips to construct noise-robust instance prototypes.
For each ground-truth instance \(k\) in a training clip, the model collects only the gated, reliably matched embeddings \(\mathcal{E}_k = \{\hat{\mathbf{e}}_{t,j(t,k)}\}\). Averaging these normalized vectors yields a centralized, temporally aggregated prototype \(\mathbf{p}_k\), which is then \(\ell_2\)-normalized to produce \(\hat{\mathbf{p}}_k\): $$ \hat{\mathbf{p}}k = \frac{\mathbf{p}_k}{|\mathbf{p}_k|_2}, \qquad \mathbf{p}_k = \frac{1}{|\mathcal{E}_k|} \sum}} \in \mathcal{Ek} \hat{\mathbf{e}} $$ During contrastive pair construction, each single-frame embedding \(\hat{\mathbf{e}}_{t,j(t,k)}\) serves as an anchor, and the corresponding instance prototype \(\hat{\mathbf{p}}_k\) acts as the positive target \(e^+\). The negative set \(\mathcal{N}_{t,k}\) combines high-confidence foreground queries from the previous frame with prototypes of other instances present in the clip. Because all embeddings are normalized, cosine similarities are naturally bounded, allowing contrastive supervision to operate cleanly without requiring a temperature hyperparameter: $$ \mathcal{L}_k) \right) \right] $$ By anchoring supervision to clip-level prototypes rather than fluctuating frame embeddings, VPCL prevents transient visual noise from destabilizing the learned identity space.}} = \log \left[ 1 + \sum_{e^- \in \mathcal{N}_{t,k}} \exp\left( \text{sim}(\hat{\mathbf{e}}, e^-) - \text{sim}(\hat{\mathbf{e}}, \hat{\mathbf{p}
3. Dual-memory fusion association: combining momentum continuity and history-best retrieval
Online temporal association struggles when current-frame appearance undergoes abrupt alterations due to partial occlusion or extreme pose changes. Conventional online trackers rely strictly on a single momentum memory updated via exponential moving averages (EMA). When an object suffers consecutive frames of degradation, the momentum state is overwritten by corrupted features, leading to irreversible tracking drift and loss of track continuity once the object re-emerges.
To resolve this limitation, MiNQVIS establishes a dual-memory architecture that simultaneously tracks a momentum embedding \(\mathbf{m}_k\) and maintains a historical cache \(\mathcal{R}_k = \{\mathbf{r}_{k,1}, \dots, \mathbf{r}_{k,N_k}\}\) of past high-quality observations for each tracklet \(k\). For each candidate query \(\hat{\mathbf{e}}_{t,q}\), two complementary similarities are evaluated: the momentum similarity \(s^{\text{mom}}_t(q,k) = \langle \hat{\mathbf{e}}_{t,q}, \hat{\mathbf{m}}_k \rangle\) (capturing smooth short-term evolution) and the history-best similarity \(s^{\text{best}}_t(q,k) = \max_{\mathbf{r} \in \mathcal{R}_k} \langle \hat{\mathbf{e}}_{t,q}, \hat{\mathbf{r}} \rangle\) (retrieving the most salient historical match). Because raw cosine scores can be skewed across different candidate pools, a bi-directional Softmax normalization is performed across queries (\(k \to q\)) and across tracklets (\(q \to k\)): $$ f^u_t(q,k) = \frac{1}{2} \left( \frac{\exp(s^u_t(q,k))}{\sum_{k'} \exp(s^u_t(q,k'))} + \frac{\exp(s^u_t(q,k))}{\sum_{q'} \exp(s^u_t(q',k))} \right), \quad u \in {\text{mom}, \text{best}} $$ The final association affinity is computed as a balanced fusion \(S_t(q,k) = 0.5 f^{\text{best}}_t(q,k) + 0.5 f^{\text{mom}}_t(q,k)\). Matches satisfying the threshold \(S_t(q,k) > \delta\) (\(\delta = 0.3\)) are assigned via one-to-one Hungarian matching. The history-best branch provides a reliable anchor preventing track loss after long occlusions, while the momentum branch maintains smooth frame-to-frame tracking.
Loss & Training¶
The overall training objective of MiNQVIS unifies the standard segmentation loss \(\mathcal{L}_{\text{seg}}\) with the proposed gating regularization loss \(\mathcal{L}_{\text{gate}}\) and video-level prototypical contrastive loss \(\mathcal{L}_{\text{vpcl}}\): $$ \mathcal{L}{\text{total}} = \mathcal{L}}} + \lambda_{\text{gate}} \mathcal{L{\text{gate}} + \lambda $$ where }} \mathcal{L}_{\text{vpcl}\(\mathcal{L}_{\text{seg}} = \lambda_{\text{cls}} \mathcal{L}_{\text{cls}} + \lambda_{\text{ce}} \mathcal{L}_{\text{ce}} + \lambda_{\text{dice}} \mathcal{L}_{\text{dice}}\). Loss coefficients are set to \(\lambda_{\text{gate}}=5.0\), \(\lambda_{\text{vpcl}}=2.0\), \(\lambda_{\text{cls}}=2.0\), \(\lambda_{\text{ce}}=5.0\), and \(\lambda_{\text{dice}}=5.0\). The gating threshold is configured as \(\gamma=0.1\) (\(\mu=0.01\)) for YouTubeVIS 2019/2021 and \(\gamma=0.05\) (\(\mu=0.005\)) for the more densely occluded OVIS. Models are initialized with COCO-pretrained weights and trained under the COCO joint training (CJT) scheme for 16,000 iterations using mini-batches of 16 clips (clip length of 8 frames on YTVIS and 10 frames on OVIS).
Key Experimental Results¶
Main Results¶
The framework is evaluated against leading online and near-online VIS methods across YouTube-VIS 2019/2021 and the heavily occluded OVIS benchmark under ResNet-50 and Swin-Large backbones:
| Dataset | Backbone | Metric | Ours (MiNQVIS) | Prev. SOTA (CAVIS) | Gain |
|---|---|---|---|---|---|
| YTVIS 2019 | ResNet-50 | AP | 55.6 | 55.7 | -0.1 |
| YTVIS 2019 | ResNet-50 | AP50 | 78.5 | 78.3 | +0.2 |
| YTVIS 2019 | Swin-L | AP | 66.2 | 66.0 | +0.2 |
| YTVIS 2019 | Swin-L | AP75 | 73.8 | 73.3 | +0.5 |
| YTVIS 2021 | ResNet-50 | AP | 51.7 | 50.5 | +1.2 |
| YTVIS 2021 | ResNet-50 | AP75 | 56.6 | 54.9 | +1.7 |
| YTVIS 2021 | Swin-L | AP | 61.6 | 61.1 | +0.5 |
| OVIS | ResNet-50 | AP | 38.2 | 37.6 | +0.6 |
| OVIS | ResNet-50 | AR10 | 45.4 | 43.5 | +1.9 |
| OVIS | Swin-L | AP | 49.1 | 48.6 | +0.5 |
| OVIS | Swin-L | AP75 | 52.6 | 52.5 | +0.1 |
| OVIS | Swin-L | AR10 | 53.9 | 53.3 | +0.6 |
Note: On OVIS, MiNQVIS outperforms earlier seminal methods CTVIS (35.5 AP, ResNet-50) and IDOL (28.2 AP, ResNet-50) by +2.7 AP and +10.0 AP, respectively.
Ablation Study¶
1. Individual and Combined Contributions of Core Components (Baseline: CTVIS without memory noise injection):
| Query-level Gating | VPCL | Dual-memory Fusion | YTVIS21 AP | OVIS AP | Note |
|---|---|---|---|---|---|
| - | - | - | 47.5 | 34.7 | Baseline online model |
| ✓ | - | - | 49.8 | 35.9 | Gating filters noisy queries (highest single-module gain) |
| - | ✓ | - | 48.7 | 36.3 | Video prototypes remove single-frame distortion |
| - | - | ✓ | 48.1 | 35.5 | Dual-memory enhances association stability |
| ✓ | ✓ | - | 51.4 | 37.8 | Gating and VPCL demonstrate strong synergy |
| ✓ | ✓ | ✓ | 51.7 | 38.2 | Full model: comprehensive noise-aware pipeline |
2. Training vs. Inference Gating Application:
| Config | YTVIS21 AP | OVIS AP | Note |
|---|---|---|---|
| Training-only gating | 39.6 | 25.2 | Severe train-test mismatch; unfiltered noise floods inference |
| Inference-only gating | 48.3 | 34.3 | Moderate post-processing gain, but lacks feature regularization |
| Training + Inference gating | 51.7 | 38.2 | Consistent distribution throughout the pipeline yields optimal gain |
3. Tracker Memory Strategy:
| Config | YTVIS21 AP | OVIS AP | Note |
|---|---|---|---|
| Momentum memory only | 51.4 | 37.8 | Provides short-term smoothness, but suffers long-term drift |
| History-best memory only | 51.1 | 36.1 | Strong re-identification, but lacks adaptive smoothing |
| Dual-memory fusion | 51.7 | 38.2 | Balances short-term continuity and long-term re-identification |
Key Findings¶
- Query-level gating is the primary performance catalyst: Incorporating query-level gating alone delivers a remarkable +2.3 AP gain on YTVIS21. More importantly, maintaining gating consistency across both training and inference is mandatory: disabling gating during inference causes performance on OVIS to collapse from 38.2 AP to 25.2 AP (-13.0 AP), proving that downstream association degrades catastrophically when bombarded with unregularized noise queries.
- Scene-dependent sensitivity of gating threshold: On YTVIS21, where instances are relatively isolated and clear, a stricter threshold \(\gamma=0.10\) achieves the highest AP (51.7). On OVIS, where severe occlusions and low-visibility targets dominate, a more permissive \(\gamma=0.05\) proves optimal (38.2 AP), preventing premature filtering of partially occluded foreground instances.
- Dual-memory fusion resolves occlusion-induced drift: In OVIS, dual-memory association surpasses pure momentum tracking by +0.4 AP and pure history-best retrieval by +2.1 AP, driving AR10 from 43.5 up to 45.4. This proves that historical retrieval acts as a critical failsafe when momentum states are degraded during extended occlusions.
Highlights & Insights¶
- Paradigm shift toward noise awareness in online VIS: While previous research concentrated almost exclusively on architectural capacity and feature cross-attention, MiNQVIS identifies the unchecked propagation of noisy queries as the true bottleneck in online tracking, formulating an end-to-end framework to clean the representation and association pipelines.
- Bounded hyper-spherical contrastive formulation: By enforcing \(\ell_2\) normalization on all embeddings, the similarity space is strictly bounded within \([-1, 1]\). This not only eliminates norm-coupling artifacts where background queries shrink feature lengths, but also removes the need for heuristic temperature tuning in contrastive loss.
- Bi-directional Softmax score calibration: Calculating similarity scores via symmetric normalization (\(q \to k\) and \(k \to q\)) effectively counters the "winner-takes-all" bias where highly salient tracklets dominate associations, providing a plug-and-play trick applicable to general multi-object tracking pipelines.
Limitations & Future Work¶
- Ambiguity under extreme motion and severe pose variations: As highlighted in the failure case analysis, when visually similar objects undergo rapid motion or abrupt non-rigid deformation, appearance cues become ambiguous, occasionally inducing temporary identity switches or fragmenting tracks.
- Underutilization of spatial and motion dynamics: The framework relies predominantly on visual embeddings for association without incorporating explicit motion priors such as Kalman filtering, optical flow warping, or temporal trajectory forecasting.
- Future directions: Integrating scene-adaptive dynamic gating thresholds alongside 3D spatial motion constraints could provide an orthogonal layer of robustness when visual features are degraded by extreme deformation.
Related Work & Insights¶
- vs CTVIS [ICCV 2023]: CTVIS injects synthetic Gaussian noise into momentum embeddings during training to mimic inference inaccuracies. In contrast, MiNQVIS directly identifies and suppresses real query noise at the source using confidence gating and replaces vulnerable frame-level contrastive targets with video-level prototypes.
- vs CAVIS [ICCV 2025]: While CAVIS introduces context-aware matching and prototype contrastive supervision, it maintains single-path momentum tracking and assumes individual queries are reliable. MiNQVIS exposes the vulnerability of single-path momentum to multi-frame occlusion drift, rectifying it via dual-memory fusion and bi-directional calibration.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Systematically diagnoses noisy query contamination in online VIS and resolves it via coordinated gating, prototype aggregation, and dual memory.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across YTVIS19/21 and OVIS under dual backbones, paired with rigorous ablation and sensitivity studies.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, technically crisp, with well-motivated design choices and transparent failure analyses.
- Value: ⭐⭐⭐⭐☆ Provides an effective, modular blueprint for noise suppression in video perception, offering practical insights for online tracking and embodied AI systems.