FSD-Net: Foundation-Guided Spatiotemporal Distillation for Video Polyp Segmentation¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Medical Imaging
Keywords: Video Polyp Segmentation, Spatiotemporal Knowledge Distillation, Weakly Supervised Learning, Foundation Models, Cycle Consistency

TL;DR¶

To alleviate the prohibitive cost of dense pixel-level annotations and eliminate mask hallucinations caused by mucosal occlusions, FSD-Net introduces a foundation-guided spatiotemporal distillation framework with occlusion-aware cycle consistency and momentum prototype re-identification, allowing a lightweight student model to surpass fully supervised methods at 46.7 FPS under sparse annotations.

Background & Motivation¶

Colorectal cancer remains a leading cause of cancer-related mortality worldwide, yet early screening and endoscopic resection of polyps during colonoscopy can interrupt malignant progression and substantially improve overall patient survival rates. Although automated deep learning models have achieved impressive strides in polyp segmentation, conventional approaches predominantly process colonoscopy video streams as independent static frames. This static paradigm discards inherent temporal correlations across frames, frequently leading to false negatives, fragmented boundaries, and temporal flickering when polyps undergo rapid non-rigid deformations, specular mucosal reflections, or violent camera motion. Consequently, Video Polyp Segmentation (VPS) has emerged to enforce spatiotemporal coherence. However, high-performing VPS models depend heavily on dense, pixel-level ground truth masks for every single video frame—an annotation requirement that demands prohibitive labor and scarce clinical expertise, severely impending real-world deployment.

Weakly supervised and semi-supervised video segmentation alternatives struggle fundamentally in the endoluminal domain. Conventional frameworks rely on low-level optical flow estimation or local block matching to propagate supervisory signals from sparse keyframes to unlabeled frames. Yet, colonoscopy videos are characterized by non-rigid peristalsis of the colon wall, slippery fluid interference, and severe specular highlights, causing optical flow algorithms to fail catastrophically and accumulate boundary drift across time. More critically, polyps frequently undergo dynamic occlusion as they disappear behind mucosal folds and subsequently reappear. Existing cycle-consistency frameworks lack high-level semantic visibility awareness: oblivious to the polyp's absence, they blindly enforce backward-forward alignment across frames, actively compelling the network to hallucinate false-positive masks onto healthy mucosal folds. The resulting accumulation of corrupted gradients damages feature memory, causing the tracker to completely lose the target upon reappearance.

To overcome the fragility of low-level motion vectors and the blindness of standard cycle constraints, this paper transitions the supervision paradigm from unreliable pixel flow to robust semantic flow. The authors employ a frozen Segment Anything Model (SAM) as a spatiotemporal teacher to supervise a lightweight student network, incorporating uncertainty-aware confidence weighting to filter noisy pseudo-labels, dynamic visibility gating to block erroneous cycle gradients during occlusions, and momentum-updated prototype memory to instantly recapture reappearing lesions. Core idea: leverage a frozen SAM foundation model as a spatiotemporal teacher to guide a compact student network through uncertainty-weighted semantic flow distillation, occlusion-aware cycle consistency, and momentum prototype re-identification, establishing a hallucination-free, label-efficient video polyp segmentation framework capable of real-time clinical deployment.

Method¶

Overall Architecture¶

FSD-Net aims to optimize a compact, high-speed student network under a sparsely annotated video setting where only the first frame of a 5-frame clip possesses an expert ground truth mask. The framework consists of two core components: a frozen SAM-ViT-B teacher model $\Phi_T$ and a lightweight student network $\Phi_S$ parametrized by learnable weights $\theta_S$, featuring a PVTv2-B0 backbone paired with a lightweight FPN decoder (12.7M parameters, 18.4 GFLOPs, 1.62GB inference memory).

During training on an input sequence $V = \{I_t\}_{t=1}^T$, the teacher model receives the first-frame ground truth mask $M_1$ and recursively propagates semantic cues forward across consecutive frames to generate dense pseudo-labels $\hat{M}_t$. Simultaneously, the student model processes each frame independently to produce prediction maps $P_t$. To form a closed temporal loop, the student's prediction $P_t$ serves as a backward prompt fed into the teacher model to deduce the antecedent mask $\tilde{M}_{t-1}$. Discrepancies between forward pseudo-labels and student predictions, alongside cycle consistency errors modulated by dynamic target visibility, formulate the composite training objective. During clinical deployment, the massive foundation teacher is discarded entirely, and the lightweight student executes single-frame forward passes at 46.7 FPS on an RTX 4090.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Endoscopy Video Sequence<br/>(Sparse Annotation: 1st Frame Only)"] --> B["Semantic Flow Distillation<br/>Frozen SAM Forward Tracking & Uncertainty Weighting"]
    B --> C["Occlusion-Aware Cycle Consistency<br/>Backward Propagation & Visibility Gated Truncation"]
    B --> D["Prototype Re-identification<br/>Momentum Memory Bank & Instant Reappearance Activation"]
    C --> E["Joint Spatiotemporal Loss<br/>Distillation + Gated Cycle + Smoothness"]
    D --> E
    E --> F["Deployable Student Network<br/>(PVTv2-B0 + FPN, 46.7 FPS Real-Time Inference)"]

Key Designs¶

1. Semantic Flow Distillation: Confidence-weighted high-level cross-frame supervision

To bypass the catastrophic breakdown of optical flow caused by mucosal glare and non-rigid peristalsis, this module employs the frozen SAM-ViT-B teacher to construct a robust semantic transfer path. With only the initial frame containing an expert ground truth mask $M_1 \in \{0, 1\}^{H \times W}$, the teacher iteratively maps representations forward to synthesize pseudo-labels $\hat{M}_t$. Because foundation model pseudo-labels in complex endoscopic fields inevitably contain noise, the framework introduces a pixel-level deterministic confidence matrix $Q_t$. By combining SAM's predicted IoU score $s_t^{iou}$ and the pixel-level foreground sigmoid probability $p_t = \sigma(z_t)$: $$Q_t = s_t^{iou} \cdot |2p_t - 1|$$ The factor $|2p_t - 1|$ reaches maximum response near definitive foreground or background regions while smoothly approaching zero in ambiguous boundary zones ($p_t \approx 0.5$). The confidence matrix $Q_t(x, y)$ modulates the standard binary cross-entropy loss $\mathcal{L}_{BCE}$ at each spatial location, which is temporally averaged across the sequence into $\mathcal{L}_{distill}$. This confidence weighting strictly prevents the student network from fitting corrupted teacher boundaries and enforces robust cross-frame semantic constancy learning.

2. Occlusion-Aware Cycle Consistency: Dynamic visibility gating to eliminate hallucinated gradients

Standard cycle consistency assumes target presence across all consecutive frames. When a polyp slides into deep intestinal folds and vanishes, backward temporal alignment forces the model to reconstruct absent targets from background tissue, hallucinating false-positive masks onto healthy mucosa. To eliminate this pathology, the backward loop—which deduces $\tilde{M}_{t-1}$ from the student's current prediction $P_t$ via the teacher—integrates a normalized global target visibility score $v_t \in [0, 1]$: $$v_t = \mathrm{mean}_{\Omega_t}(Q_t) \cdot \min\left(1, \frac{|\Omega_t|}{|\Omega_{t-1}| + \epsilon}\right)$$ where $\Omega_t = \{p_t > 0.5\}$ denotes the detected foreground region. Sudden shrinkage of the polyp region under occlusion collapses the area ratio, sharply suppressing $v_t$. Introducing a visibility tolerance threshold $\tau$ (set to 0.80), an adaptive weight coefficient is obtained via $w_t = \max(0, v_t - \tau)$. For frames where the polyp is severely occluded or absent, $w_t$ drops strictly to zero, dynamically cutting off backward mean squared error ($\mathcal{L}_{MSE}$) gradient propagation. Bidirectional temporal constraints are only active when the target is reliably visible, eradicating occlusion-induced false positives.

3. Prototype Re-identification: Momentum-updated global memory for long-term target recapture

Prolonged occlusions combined with rapid scope movement trigger severe feature drift in conventional tracking heads, irreversibly forfeiting target identities. The Prototype Re-identification Module (PRM) leverages a lightweight feature extractor to project the input frame into a compact semantic feature map $F_t \in \mathbb{R}^{D \times H' \times W'}$. The initial prototype $C_1$ is masked-pooled from the first frame's ground truth $M_1$. Across subsequent high-confidence frames, the prototype updates smoothly via a momentum coefficient $\alpha = 0.90$: $$C_t = \alpha C_{t-1} + (1 - \alpha) \frac{\sum_{x,y} F_t(x,y) \cdot \hat{M}_t(x,y)}{\sum_{x,y} \hat{M}_t(x,y)}$$ During occlusion intervals, prototype updating freezes to preserve template integrity. In incoming frames, the model computes the spatial cosine similarity matrix $S_t(x, y)$ against $C_{t-1}$ and tracks the maximum response $s_t^{max} = \max_{x,y} S_t(x, y)$. When $s_t^{max}$ surpasses the re-identification threshold of 0.72, the model signals the polyp's emergence and triggers a prompt to immediately reactivate the spatiotemporal tracking flow, resolving long-term lost-track failures.

Loss & Training¶

The framework is trained end-to-end using a joint objective. In addition to the weighted semantic flow distillation loss $\mathcal{L}_{distill}$ and the occlusion-aware cycle consistency loss $\mathcal{L}_{cycle}$, a temporal smoothness regularization term $\mathcal{L}_{smooth}$ is introduced to penalize absolute differences between consecutive student prediction masks, suppressing high-frequency mask flickering: $$\mathcal{L}_{smooth} = \frac{1}{T-1}\sum_{t=2}^T \sum_{x=1}^H \sum_{y=1}^W |P_t(x,y) - P_{t-1}(x,y)|$$ The overall training loss linearly balances the three components: $$\mathcal{L}_{total} = \lambda_1 \mathcal{L}_{distill} + \lambda_2 \mathcal{L}_{cycle} + \lambda_3 \mathcal{L}_{smooth}$$ with empirically optimized weights $\lambda_1 = 1.0, \lambda_2 = 0.5, \lambda_3 = 0.2$. Optimization is performed on a single NVIDIA RTX 4090 GPU (24GB) using AdamW with an initial learning rate of $1 \times 10^{-4}$ and a cosine annealing schedule over 50 epochs. All input frames are resized to $352 \times 352$.

Key Experimental Results¶

Main Results¶

Quantitative evaluations were performed across the standardized SUN-SEG video benchmark (tested separately on Easy and Hard subsets), the CVC-612 video dataset, and the unseen static CVC-300 dataset. Metrics include mean Dice (mDice), mean Intersection over Union (mIoU), and boundary mean Hausdorff Distance (mHD in mm).

Method	Year	Backbone	Supervision	SUN-SEG-Easy mDice / mIoU / mHD	SUN-SEG-Hard mDice / mIoU / mHD	CVC-612 mDice / mIoU / mHD	CVC-300 mDice / mIoU / mHD
DCNet	2023	PVT-V2	Fully Supervised	78.36 / 68.63 / 29.24	75.04 / 65.55 / 28.48	88.21 / 80.85 / 25.77	86.24 / 78.25 / 18.26
PNS+	2022	Res2Net-50	Fully Supervised	79.26 / 70.24 / 26.38	76.51 / 68.11 / 28.67	90.06 / 83.43 / 21.79	86.59 / 78.24 / 15.98
META-UNet	2023	ResNet-34	Fully Supervised	81.17 / 72.43 / 25.71	80.16 / 70.55 / 26.27	90.64 / 84.39 / 20.49	86.64 / 78.55 / 16.02
Polyp-SAM-FS	2023	ViT-B	Fully Supervised	81.33 / 73.18 / 25.19	80.52 / 71.33 / 25.82	90.47 / 84.58 / 21.29	86.83 / 78.72 / 16.53
MedSAM	2024	ViT-B	Semi Foundation	69.04 / 60.29 / 220.35	68.23 / 58.71 / 207.92	76.08 / 66.82 / 182.58	79.02 / 70.69 / 113.32
SAMUS	2024	ViT-B	Semi Foundation	81.99 / 73.37 / 25.55	80.41 / 71.37 / 26.44	90.23 / 83.68 / 23.98	86.81 / 79.23 / 16.12
SP-SAM	2025	ViT-B	Semi Foundation	84.34 / 77.21 / 22.36	83.25 / 73.34 / 24.86	90.47 / 85.06 / 21.15	87.09 / 79.52 / 14.98
VP-SAM	2024	ViT-B	Semi Foundation	84.83 / 77.48 / 21.72	84.11 / 75.04 / 22.52	91.12 / 85.15 / 20.24	87.17 / 79.41 / 14.33
Ours (prompt-free)	2026	PVTv2-B0	Distilled (Ours)	85.62 / 78.16 / 21.22	85.28 / 77.16 / 21.38	92.33 / 86.79 / 19.34	88.26 / 80.17 / 14.06
Ours (1 pt/frame)	2026	PVTv2-B0	Distilled (Ours)	87.56 / 80.04 / 19.80	87.04 / 79.20 / 20.64	93.54 / 88.83 / 17.86	89.93 / 82.38 / 12.38

Ablation Study¶

1. Comprehensive Orthogonal Module Ablation (SUN-SEG Official Test Set)

Config	SFD	OACC	PRM	SUN-SEG-Easy mDice (%)	SUN-SEG-Easy mIoU (%)	SUN-SEG-Easy mHD (mm)	SUN-SEG-Hard mDice (%)	SUN-SEG-Hard mIoU (%)	SUN-SEG-Hard mHD (mm)
1	✓			85.62	79.50	20.85	79.42	70.15	25.66
2		✓		85.11	79.05	20.55	79.15	71.04	24.89
3			✓	85.26	78.42	20.97	79.56	70.33	24.12
4	✓	✓		86.65	79.80	20.30	85.31	76.54	21.05
5	✓		✓	86.20	79.67	20.40	85.11	76.88	21.45
6		✓	✓	86.90	79.78	20.90	85.79	76.02	21.11
7 (Full model)	✓	✓	✓	87.56	80.04	19.80	87.04	79.20	20.64

2. Component Mechanism Comparisons (SUN-SEG-Hard)

Investigation	Variant	SUN-SEG-Hard mDice (%)	SUN-SEG-Hard mIoU (%)	SUN-SEG-Hard mHD (mm)	Note
Temporal Propagation	Baseline (Optical Flow)	75.24	64.88	31.45	Low-level flow suffers from peristalsis and mucosal glare
	+ Spatial KD	78.37	67.52	28.16	Static spatial distillation yields modest gains (+2.64% mIoU)
	+ SFD (Ours)	79.42	70.15	25.66	Uncertainty-weighted semantic distillation (+5.27% mIoU)
Cycle Consistency	Baseline + SFD	79.42	70.15	25.66	Open-loop forward supervision without backward constraint
	+ Standard Cycle Consistency	78.56	69.87	26.36	Blind alignment hallucinating false positives (-0.86% mDice)
	+ OACC (Ours)	85.31	76.54	21.05	Visibility gating suppresses occlusions (+5.89% mDice, -4.61mm mHD)
Long-term Memory	Baseline + SFD + OACC	85.31	76.54	21.05	Lacks adaptive memory to handle prolonged target absence
	+ Static Prototype	85.35	78.10	20.65	Inelastic template unable to accommodate non-rigid deformation
	+ PRM (Ours)	87.04	79.20	20.64	Momentum updating tracks morphing polyps across occlusions

Key Findings¶

Occlusion-aware gating is the definitive factor for challenging endoscopic video: Standard cycle consistency directly degrades performance on the Hard subset by 0.86% in mDice and 0.70 mm in mHD due to backpropagating hallucinated gradients over mucosal folds. Introducing OACC's visibility gating triggers an immediate surge of +5.89% in mDice and a 4.61 mm reduction in mHD, proving that blocking erroneous alignment during disappearance is critical.
Momentum updating preserves re-identification efficacy: A static prototype improves mDice on the Hard subset by only 0.04%, as it fails under severe endoscopic angle changes. In contrast, momentum updating ($\alpha = 0.90$) elevates mDice to 87.04% and drives cosine activation above 0.72 upon polyp exposure, eliminating post-occlusion tracking failures.
Visibility threshold $\tau$ demonstrates clear convexity: Sweeping $\tau$ between 0.60 and 0.95 reveals that 0.80 is strictly optimal (87.04% mDice on Hard). Values below 0.80 prematurely cut off valid learning gradients, while values above 0.80 admit false-positive noise into the cycle loss (dropping mDice to 86.12%).
Superior label efficiency and deployment speed: Under 20% sparse annotations (1 frame per 5-frame clip), the prompt-free student model scores 85.28% mDice on SUN-SEG-Hard, decisively beating fully supervised baselines (META-UNet's 80.16% and Polyp-SAM-FS's 80.52%) while operating at 46.7 FPS with only 12.7M parameters.

Highlights & Insights¶

Semantic flow supersedes fragile optical flow: The paradigm replaces error-prone pixel motion vectors with high-level foundation representations, naturally conferring invariance against non-rigid peristalsis, specular reflections, and endoscopic fluid artifacts.
Gated consistency resolving the hallucination paradox: By framing temporal alignment as a conditionally gated process rather than an absolute truth, the model prevents cycle consistency from hallucinating phantom targets during disappearance events.
Asymmetric distillation architecture: Confining the heavy SAM-ViT-B teacher strictly to offline distillation while deploying a lightweight PVTv2-B0 student achieves an ideal compromise between deep representation learning and ultra-fast (46.7 FPS), low-footprint (1.62GB) clinical utility.

Limitations & Future Work¶

Vulnerability to flat camouflaged polyps: OACC depends on semantic visibility scores estimated by the frozen SAM teacher. Highly camouflaged sessile or flat polyps sharing identical texture with intestinal folds may yield false-negative visibility, prematurely dampening student gradients.
Memory degradation under extended scope retraction: The prototype bank updates via momentum. If violent endoscope maneuvers cause polyps to disappear for minutes, severe perspective shifts upon re-entry may exceed the prototype's cosine retrieval basin.
Future trajectories: Incorporating monocular depth priors and 3D endoluminal surface reconstruction to inform geometric occlusion reasoning, accompanied by INT8 quantization and structured pruning for direct on-device surgical deployment.

vs PNS+ / META-UNet (Fully Supervised VPS): Fully supervised models rely on complete, expensive pixel-level ground truth. With only 20% keyframe labels, FSD-Net outperforms META-UNet by 6.88% mDice on SUN-SEG-Hard, demonstrating the strong label efficiency of foundation model distillation.
vs VP-SAM / SP-SAM (Semi-Supervised Foundation Adaptation): Prior adaptation strategies retain massive foundation backbones at test time and demand interactive prompts, precluding real-time clinical use. FSD-Net discards the teacher at inference, operating fully prompt-free at 46.7 FPS while exceeding VP-SAM in accuracy (85.28% vs 84.11% mDice).
vs Classic Cycle-Consistent Video Segmentation (e.g., Cen et al., Wu & Wang): Traditional methods blindly enforce temporal alignment across all frames. FSD-Net's occlusion gating provides a plug-and-play principle that generalizes to broader video tasks plagued by dynamic object disappearance.

Rating¶

Novelty: ⭐⭐⭐⭐ [Clever integration of foundation-guided semantic distillation and occlusion-aware gated cycle consistency]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive multi-benchmark validation with rigorous orthogonal ablations and parameter sensitivity sweeps]
Writing Quality: ⭐⭐⭐⭐⭐ [Extremely clear logical motivation, robust mathematical formulations, and well-grounded clinical rationale]
Value: ⭐⭐⭐⭐⭐ [Directly resolves the high-cost annotation bottleneck while delivering genuine real-time inference for clinical endoscopy]