Exclusivity-Guided Mask Learning for Semi-Supervised Crowd Instance Segmentation and Counting

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/JoyceeH0127/ECCV2026XMask
Area: Segmentation
Keywords: semi-supervised learning, crowd counting, crowd instance segmentation, exclusivity-guided mask learning, pseudo-mask constraints

TL;DR

The paper introduces Exclusion-Constrained Dual-Prompt SAM (EDP-SAM) with Nearest Neighbor Exclusion Circle (NNEC) priors to create benchmark mask annotations from point labels, paired with an Exclusivity-Guided Mask Learning (XMask) framework and pseudo-mask constraints that bridge dense crowd instance segmentation and crowd counting in a unified semi-supervised pipeline.

Background & Motivation

Crowd scene analysis serves as a cornerstone in computer vision for critical applications such as public safety, intelligent transportation management, and urban surveillance. However, acquiring dense pixel-level annotations in heavily congested environments is notoriously labor-intensive and cost-prohibitive, forcing existing benchmarks to rely almost exclusively on sparse head-point annotations. Because dimensionless point labels completely discard an individual's spatial extent and shape boundary, conventional crowd analysis methods have historically been limited to density regression or point localization, both of which struggle under severe occlusion, heavy overlap, and wide perspective variations. Conversely, standard object detection models frequently collapse in dense crowds due to severe bounding box ambiguity and overlapping suppression failures.

Although foundation models like SAM and FastSAM exhibit remarkable zero-shot transfer capabilities, directly prompting them with sparse points in crowded scenes causes significant cross-instance mask leakage, merging adjacent heads into indistinct blobs. Furthermore, their heavy computational footprint prevents their direct use during iterative semi-supervised pseudo-labeling. While recent semi-supervised counting works propose pseudo-point matching or local region smoothing, they continue to treat pedestrians as dimensionless entities, leaving the crucial structural semantic cues untapped.

To overcome these barriers, this work bridges crowd instance segmentation and counting through a topological exclusivity constraint. By transforming sparse supervision into structured masks and internalizing local metric learning into a lightweight architecture, the two tasks mutually reinforce one another in a semi-supervised closed loop. Core idea: guide SAM with dual prompts and Nearest Neighbor Exclusion Circle constraints to generate robust crowd masks, design an exclusivity-guided discriminative embedding framework (XMask) with Gaussian smoothing, and regularize crowd counting using high-confidence instance mask priors.

Method

Overall Architecture

The proposed approach establishes a three-stage end-to-end framework uniting crowd instance segmentation and counting under a Mean Teacher architecture. In Stage 1, Exclusion-Constrained Dual-Prompt SAM (EDP-SAM) synthesizes high-quality instance masks from point prompts and SLIC superpixels under strict exclusion geometry, providing ground truth masks for initial training. In Stage 2, Exclusivity-Guided Mask Learning (XMask) optimizes a feature embedding network using local topological partitioning, depthwise Gaussian smoothing, and differentiable center sampling to learn discriminative pixel embeddings with pull-push loss objectives. In Stage 3, the trained mask head infers pseudo-masks on unlabeled data, filtering valid predictions via a dual-threshold scheme and enforcing foreground instance constraints and background suppression losses to regularize the crowd counting model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Sparse Point Annotations"] --> B["Stage 1: EDP-SAM Mask Generation<br/>Dual prompts + NNEC topological exclusion"]
    B --> C["Stage 2: XMask Discriminative Mask Learning<br/>Gaussian smoothing + center sampling + pull-push loss"]
    C --> D["Stage 3: Mask-Constrained Counting Optimization<br/>Dual-threshold filtering + foreground/background regularizers"]
    D --> E["Output: High-quality crowd masks & accurate count predictions"]

Key Designs

1. EDP-SAM Mask Generation: dual-prompt guidance and Nearest-Neighbor Exclusion Circle constraints In congested crowd scenes, standard point prompts supplied to SAM frequently spill across boundaries into neighboring heads. EDP-SAM addresses this by pairing the annotated head point with SLIC-generated superpixels as dual prompts, anchoring SAM's attention to local texture boundaries. To physically restrict mask bleeding, the method introduces the Nearest-Neighbor Exclusion Circle (NNEC) constraint. For any annotated head point \(\mathbf{p}_i\), its exclusion radius \(r_i\) is computed from the Euclidean distance to its closest neighboring annotation:

\[r_i = \min_{j \neq i} \|\mathbf{p}_i - \mathbf{p}_j\|_2\]

The raw segmentation output from SAM is intersected with this circular exclusion zone, strictly guaranteeing first-order spatial separation between adjacent individuals. With minimal manual quality verification, this pipeline upgrades point-annotated crowd datasets into high-fidelity instance mask supervision.

2. XMask Discriminative Mask Learning: separable Gaussian smoothing and adaptive local metric learning To bypass the extreme computational cost of global multi-class mask classification, XMask reformulates instance segmentation as local metric learning over dynamically instantiated regions. A lightweight backbone produces a multi-channel feature embedding map. To suppress high-frequency interpolation artifacts and local noise without destroying discriminative boundaries, a separable depthwise Gaussian filter decomposes 2D isotropic blurring into sequential 1D horizontal and vertical convolutions applied independently per channel. Within each candidate region defined by the adaptive radius \(r_i\), pixels are partitioned into positive set \(\mathcal{R}_i^+\) and negative set \(\mathcal{R}_i^-\) according to ground truth or pseudo-masks. Using bilinear grid sampling, sub-pixel differentiable center sampling extracts prototype feature vector \(\mathbf{c}_i\), optimized via an asymmetric margin contrastive objective:

\[\mathcal{L}_i = \frac{1}{|\mathcal{R}_i|} \sum_{(y,x) \in \mathcal{R}_i} \left[ \alpha_{y,x} \left( \|\tilde{\mathbf{E}}(y,x) - \mathbf{c}_i\|_2 - (\tau + \delta \alpha_{y,x}) \right) \right]_+\]

where \(\alpha_{y,x} \in \{+1, -1\}\) denotes positive and negative pixel identity, \(\tilde{\mathbf{E}}\) represents smoothed embeddings, and \(\tau, \delta\) govern margin boundaries. During inference, a normalized squared geometric distance is fused with feature distance into a joint energy function, completely preventing incomplete mask clipping in early training stages.

3. Mask-Constrained Counting Optimization: dual-threshold filtering and topological regularization Semi-supervised counting models often suffer from location drift and noisy pseudo-point proposals. Rather than relying on simple point matching, this framework uses XMask's dense shape predictions as structured priors. To counter teacher network uncertainty, a dual-threshold scheme is deployed: instances are proposed at a low confidence threshold to guarantee exhaustive coverage and ensure background purity, but only masks surpassing a stringent upper confidence threshold are admitted into the active instance set \(K_v\). The counting prediction map \(\hat{\mathbf{f}}\) is then regularized via dual objectives: a background penalty loss \(\mathcal{L}_{\mathrm{bg}}\) forces true background responses to negative values, while a foreground constraint loss \(\mathcal{L}_{\mathrm{fg}}\) enforces unified response density and exact single-point activation per valid instance mask:

\[\mathcal{L}_{\mathrm{fg}} = \frac{1}{|K_v|} \sum_{k \in K_v} \left( |N_k^+ - 1| + |S_k - 1| \right), \quad S_k = \sum_{p \in \Omega_k} \max(0, \hat{\mathbf{f}}(p)), \quad N_k^+ = \sum_{p \in \Omega_k} \mathbb{I}(\hat{\mathbf{f}}(p) > 0)\]

This allows structural geometric priors from the segmentation branch to directly constrain the likelihood landscape of the counting branch.

Loss & Training

The overall training pipeline operates under the Mean Teacher paradigm. Labeled samples are directly optimized with supervised mask loss and point classification loss, while unlabeled samples receive pseudo-mask constraints generated via teacher model predictions updated through student exponential moving average (EMA). The combined optimization objective integrates supervised classification loss, mask discriminative pull-push loss \(\mathcal{L}_i\), foreground constraint loss \(\mathcal{L}_{\mathrm{fg}}\), and background penalty loss \(\mathcal{L}_{\mathrm{bg}}\). For counting model refinement, freezing the backbone and updating only the decoder yields the optimal performance-efficiency trade-off.

Key Experimental Results

Main Results

The framework is evaluated across three challenging benchmarks (ShanghaiTech A, UCF-QNRF, and JHU++) under 5%, 10%, and 40% labeled supervision protocols.

1. Semi-Supervised Crowd Instance Segmentation ([email protected])

Methods	NNEC Constraint	ShTech A (5%)	ShTech A (10%)	ShTech A (40%)	UCF-QNRF (5%)	UCF-QNRF (10%)	UCF-QNRF (40%)	JHU++ (5%)	JHU++ (10%)	JHU++ (40%)
FastSAM	✗	0.279	0.281	0.291	0.203	0.211	0.213	0.186	0.185	0.189
FastSAM	✓	0.287	0.290	0.300	0.223	0.229	0.233	0.197	0.182	0.201
S⁴M	✗	0.005	0.007	0.011	0.010	0.012	0.015	0.013	0.014	0.021
CrowdSAM	✓	0.201	0.201	0.212	0.149	0.156	0.163	0.187	0.189	0.192
Point only	✓	0.195	0.194	0.202	0.198	0.201	0.207	0.180	0.182	0.183
XMask (Ours)	✗	0.314	0.318	0.331	0.269	0.277	0.278	0.249	0.253	0.257
XMask (Ours)	✓	0.313	0.318	0.333	0.282	0.290	0.297	0.258	0.263	0.270

2. Semi-Supervised Crowd Counting Performance (MAE / MSE)

Labeled Pct.	Methods	ShTech A MAE↓	ShTech A MSE↓	UCF-QNRF MAE↓	UCF-QNRF MSE↓	JHU++ MAE↓	JHU++ MSE↓
5% labeled	MT	104.7	33.2	172.4	284.9	101.5	363.5
	L2R	103.0	27.6	160.1	272.3	101.4	338.8
	DAC	85.4	134.5	120.2	209.3	82.2	294.9
	OT-M	83.7	133.3	118.4	195.4	82.7	304.5
	P2R	69.9	119.5	100.1	182.5	77.8	293.5
	Ours	64.6	106.2	98.4	163.7	77.1	289.5
10% labeled	MT	319.3	94.5	156.1	245.5	250.3	90.2
	L2R	90.3	153.5	148.9	249.8	87.5	315.3
	DAC	74.9	115.5	109.0	187.2	75.9	282.3
	OT-M	80.1	118.5	113.1	186.7	73.0	280.6
	P2R	64.2	114.6	94.9	167.2	68.7	272.3
	Ours	62.4	104.4	93.9	155.7	68.7	258.7
40% labeled	MT	88.2	151.1	147.2	249.6	121.5	388.9
	L2R	86.5	148.2	145.1	256.1	123.6	376.1
	DAC	67.5	110.7	91.1	153.4	65.1	260.0
	OT-M	70.7	114.5	100.6	167.6	72.1	272.0
	P2R	55.6	95.0	86.0	144.3	63.3	271.1
	Ours	56.2	93.5	84.0	143.6	61.9	256.1

Ablation Study

Ablation experiments analyze individual architectural designs and optimization settings:

1. Segmentation Loss Functions and Gaussian Blur Configurations (ShTech A)

Ablation Target	Configuration	Settings / Loss Type	IoU↑	F1↑	Note
Loss Type	Baseline Loss	Dice loss	0.2381	0.0946	Overlap ratio suffers severely from crowd overlaps
	Baseline Loss	Focal loss	0.2893	0.2070	Standard cross-entropy shows limited boundary discrimination
	Proposed	MaskHead Discriminative	0.3119	0.2574	Pull-push contrastive embeddings perform best
Gaussian Blur	No Smoothing	-	0.3050	0.2481	High-frequency noise and boundary artifacts degrade features
	Depthwise Blur	\(k=5, \sigma=1.5\)	0.3111	0.2566	Smooths spatial transitions
	Depthwise Blur	\(k=7, \sigma=1.5\)	0.3119	0.2574	Balances spatial scale and sharp boundaries
	Depthwise Blur	\(k=9, \sigma=1.5\)	0.3120	0.2577	Marginally higher IoU with increased computation
	Optimal Blur	\(k=7, \sigma=3.0\)	0.3130	0.2584	Best overall IoU and F1 balance, adopted as default
	Depthwise Blur	\(k=5, \sigma=3.0\)	0.3119	0.2575	Slight drop compared to \(k=7\)
	Depthwise Blur	\(k=9, \sigma=3.0\)	0.3130	0.2570	Marginally lower F1 score

2. Fine-Tuning Module Selection (ShTech A 5% labeled)

Head (Detection / Count)	Backbone (Feature Extractor)	Trainable Parameters	MAE↓	MSE↓	Note
✓	✓	16.77M	68.1	106.8	Full parameter training leads to mild overfitting under 5% data
✓	✗	15.58M	70.2	111.3	Updating head alone leaves features unaligned
✗ (Decoder-only fine-tuning)	✓ (Frozen backbone)	2.05M	64.6	106.2	Reduces active parameters by 88% while yielding best accuracy

Key Findings

• Instance mask priors substantially outperform point cues: Conditioning crowd counting on XMask-derived structural shapes drops ShTech A MAE from 69.9 to 64.6 and UCF-QNRF MSE from 182.5 to 163.7 under the 5% labeled protocol, demonstrating that structural boundary information effectively overcomes sparse point ambiguity.
• NNEC geometric priors are vital for congested benchmarks: On the denser UCF-QNRF and JHU++ datasets, enforcing NNEC constraints improves XMask's [email protected] by 1.3% to 1.9%, cleanly cutting off cross-instance mask leakage.
• Inference speed is completely invariant to crowd density: On ShTech A, XMask averages 0.17 seconds per image compared to FastSAM's 1.01 seconds (a 5.9× acceleration). Regardless of whether an image contains dozens or thousands of individuals, XMask's latency stays below 0.3 seconds, overcoming the latency explosion typical of foundation models in dense scenarios.

Highlights & Insights

• Topological exclusion deciphers crowd ambiguity: Formulating a simple geometric first-order constraint (NNEC) based on nearest-neighbor Euclidean distance elegantly solves cross-instance mask spilling without requiring complex boundary supervision.
• Local metric learning avoids global classification bottlenecks: Reformulating instance segmentation into local discriminative embedding learning with depthwise Gaussian smoothing enables fast, lightweight mask extraction on par with foundation models at a fraction of the compute cost.
• Synergistic multi-task feedback loop: The architecture unifies instance segmentation and counting into a closed loop where the segmentation head regularizes counting prediction landscapes through dual-threshold filtering and topological consistency.

Limitations & Future Work

• Boundary degradation under severe body occlusions: Because the approach primarily conditions exclusion geometry on head locations, heavily overlapping torsos or extreme perspective foreshortening can still produce jagged mask boundaries.
• Sensitivity to perspective scale variations: NNEC assumes nearest neighbor distance accurately captures local instance scale; in scenes with steep tilt angles where near and distant persons project close to each other in 2D space, incorporating monocular depth priors would offer further scale calibration.

Related Work & Insights

• vs FastSAM / SAM: FastSAM and SAM are category-agnostic foundation models that suffer severe computational lag and frequent mask merging in dense crowds. EDP-SAM introduces superpixel and exclusion guidance to build offline training masks, while the online XMask runs in only 17% of FastSAM's time with superior segmentation accuracy.
• vs P2R / OT-M: P2R and OT-M confine pseudo-supervision to point coordinates or localized Gaussian heatmaps, treating people as dimensionless targets. This work establishes rich instance mask priors, using foreground activation counts and background suppression to achieve state-of-the-art semi-supervised counting accuracy.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegantly adapts topological exclusion circles and dual-prompt foundation models to crowd segmentation and counting.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across three major crowd benchmarks under three label ratios, supported by detailed ablations on losses, smoothing kernels, and tuning modules.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, mathematically concise, and self-consistent throughout.
• Value: ⭐⭐⭐⭐⭐ Provides a practical, highly efficient blueprint for scaling semi-supervised crowd instance segmentation and fine-grained counting.

Related Papers

• [ECCV 2026] Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentation
• [CVPR 2026] From Softmax to Dirichlet: Evidential Learning for Semi-supervised Semantic Segmentation
• [AAAI 2026] S5: Scalable Semi-Supervised Semantic Segmentation in Remote Sensing
• [ECCV 2026] ZMIS-SAM: Segment Anything Model Enhanced With Wavelet Transform For Zooplankton Microscopy Image Instance Segmentation
• [ECCV 2026] Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation