ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentation¶

Conference: ECCV2026
arXiv: 2604.07021
Code: https://github.com/Autumnair007/ModuSeg
Area: Semantic Segmentation
Keywords: Weakly Supervised Semantic Segmentation, Decoupling, Training-Free, Feature Retrieval, Foundation Models

TL;DR¶

ModuSeg explicitly decouples weakly supervised semantic segmentation into two independent phases: "class-agnostic object discovery" and "semantic retrieval." It extracts geometric proposals using a generic Mask Proposer and performs non-parametric retrieval using an offline-constructed feature bank. Without any fine-tuning, the method achieves state-of-the-art performance on the Pascal VOC dataset with 86.3 mIoU using only image-level labels during training.

Background & Motivation¶

Weakly Supervised Semantic Segmentation (WSSS) aims to make pixel-level predictions using only image-level labels, thereby significantly reducing the cost of pixel-wise annotations. Conventional methods typically adopt a standard paradigm: generating initial localization cues via Class Activation Maps (CAMs), and then expanding the segmented regions through multi-stage network retraining or end-to-end joint optimization. The fundamental bottleneck of this paradigm is that semantic recognition and object localization are coupled into the same classification objective. Classification networks naturally focus only on the most discriminative local regions, which causes CAM activation maps to cover only sparse object fragments (e.g., a bird's head instead of its entire body), yielding naturally incomplete foreground regions with blurry boundaries. To refine these incomplete seeds, conventional paradigms must rely either on multi-stage retraining (computationally expensive and complex) or end-to-end joint optimization (unstable training, and hard-to-suppress pseudo-label noise).

The rise of vision foundation models (such as DINOv2/v3, C-RADIOv4, SAM, and EntitySeg) opens up new possibilities for resolving this dilemma. Class-agnostic segmentation models exhibit exceptional boundary awareness, while vision-language and self-supervised models learn highly powerful dense semantic descriptions. However, many existing methods still follow the well-trodden path of coupled optimization by integrating foundation models for fine-tuning or end-to-end distillation. Consequently, they still suffer from the inherent noise in weakly supervised pseudo-labels, failing to truly unleash the potential of foundation models.

The key insight of this work is: since geometric localization and semantic classification are inherently distinct tasks, why should they be optimized together? A generic Mask Proposer (such as EntitySeg) is inherently proficient at extracting object proposals with clean boundaries without requiring any semantic information; simultaneously, a semantic foundation model (such as C-RADIOv4) can naturally provide high-quality dense features. Combining these two—using proposals for localization and feature retrieval for classification—bypasses all the complications of coupled optimization. Core Idea: Explicitly decouple weakly supervised semantic segmentation into two independent stages: class-agnostic object discovery and semantic retrieval. A generic Mask Proposer is employed to extract geometric proposals, and an offline-built feature bank is used to assign semantics via non-parametric KNN retrieval, requiring zero parameter fine-tuning throughout the entire pipeline.

Method¶

Overall Architecture¶

The core of ModuSeg is to split WSSS into two entirely independent phases—offline feature bank construction and online inference retrieval—with only a single non-parametric feature similarity calculation between them, eliminating any cross-stage gradient backpropagation or joint optimization.

Phase 1 (Offline): Building a robust feature bank based on training set image-level labels. A pre-trained CorrCLIP is used to generate initial pseudo-masks for each training image. However, directly extracting features using these masks poses two issues: the pseudo-masks suffer from severe feature mixing at object boundaries due to ViT patch alignment and inherent weakly supervised noise, and hard quantization sampling introduces quantization noise. To address this, the authors propose Semantic Boundary Purification (SBP, discarding boundaries via morphological erosion) and Soft Mask Feature Aggregation (SMFA, area interpolation weighting) to extract high-quality class prototypes, followed by outlier removal based on clustering assumptions to obtain a clean feature bank.

Phase 2 (Online Inference): Extracting object proposals using a class-agnostic Mask Proposer, followed by retrieval-based voting in the feature bank to complete semantic classification. Given a test image, EntitySeg generates a set of binary mask proposals (identifying "where an object is" without knowing its class). For each proposal, the same SMFA is applied to extract a query feature, and its Top-\(K\) nearest neighbors are retrieved from the feature bank to determine its class via majority voting. Finally, intra-class NMS and confidence-prioritized rasterization are performed on overlapping proposals to yield the final segmentation map.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training Images<br/>+ Image-level Labels"] --> B["CorrCLIP<br/>Pseudo-Mask Gen"]
    B --> C["Semantic Boundary Purification<br/>Morphological Erosion (Discard Edge)"]
    C --> D["Soft Mask Feature Aggregation<br/>Weighted Area Interpolation"]
    D --> E["Prototype-level Feature Purification<br/>Outlier Removal"]
    E --> F["Offline Feature Bank"]

    G["Test Image"] --> H["EntitySeg<br/>Mask Proposer"]
    H --> I["SMFA Query<br/>Feature Extraction"]
    I --> J["Top-K Retrieval<br/>+ Majority Voting"]
    F --> J
    J --> K["Intra-class NMS<br/>+ Confidence Rasterization"]
    K --> L["Final Segmentation Map"]

Key Designs¶

1. Semantic Boundary Purification (SBP): Actively Discarding Unreliable Boundaries for Feature Purity

There is an inherent structural misalignment between ViT feature maps and images—a \(16\times16\) patch straddling an object boundary encodes a mixture of foreground and background signals. Meanwhile, weakly supervised pseudo-masks are inherently highly uncertain at object boundaries (since classifiers exhibit low confidence near boundaries). If these "dirty" patches are used to build the feature bank, each class prototype becomes corrupted with noise, degrading the discriminative power of subsequent retrieval.

The solution of SBP is remarkably simple yet effective: shaving off a boundary layer of the pseudo-mask via morphological erosion. Specifically, for the binary mask of each class, a \(3 \times 3\) structuring element is applied with 20 erosion iterations. This is equivalent to actively discarding the "gray areas" near object boundaries and retaining only the core regions of the objects—where patches are almost guaranteed to belong to the target class, yielding highly pure features. Although boundary geometric integrity is sacrificed, the resulting features achieve high semantic purity. Note that this operation is only applied during offline feature bank construction (where precision is the goal); during inference, the raw, uneroded proposals from EntitySeg are directly used (where recall and boundary fidelity are the goals), creating an intentional asymmetry between the training and inference stages.

2. Soft Mask Feature Aggregation (SMFA): Replacing Hard Quantization with Area Interpolation to Suppress Quantization Noise

When mapping a high-resolution mask to a low-resolution ViT feature grid, naive nearest-neighbor downsampling assigns a binary flag ("contained" or "not contained") to each feature grid point. However, grid points along the mask's boundary are likely only partially covered by the object. Hard quantization either includes the whole block (introducing background noise) or completely excludes it (discarding valid information).

SMFA replaces hard quantization with area interpolation: it downsamples the high-resolution purified mask to the feature grid resolution, assigning each grid point a soft weight \(w \in [0, 1]\) representing the actual fraction of the patch covered by the object. The prototype feature is then computed as a weighted average: \(v = \text{normalize}(\sum (w \cdot F) / \sum w)\). Consequently, patches completely inside the object center contribute near \(1\), while partially covered boundary patches decay proportionally, making the entire aggregation process differentiable and clean. This design also decouples the module from any specific ViT backbone—whether using DINOv2 or C-RADIOv4, the SMFA pipeline remains identical, and only the feature extractor needs to be swapped.

3. Prototype-level Feature Purification: Outlier Removal Driven by Clustering Assumptions

Even after SBP + SMFA, noise in pseudo-labels cannot be completely eliminated—some training samples may have severely shifted pseudo-masks, or their feature vectors may deviate from the intra-class distribution center. This work performs a second round of filtering based on the clustering assumption (reliable features of the same class form tight clusters in high-dimensional space, whereas noisy samples are sparsely distributed at the periphery).

For each foreground class, the mean of all candidate feature vectors is first calculated as the global prototype \(\mu_c\), and then the Euclidean distance \(d_v\) of each feature vector \(v\) to \(\mu_c\) is computed. The top \(\alpha\%\) (default 25%) of samples with the largest distances are discarded. A crucial design decision here is: no purification is applied to the background class. The background is naturally multi-modal (sky, roads, vegetation, buildings, etc.), making the single-center assumption completely inapplicable; forcing filtration would lose background diversity and damage the model's discriminative ability in complex environments. Retaining all background embeddings ensures comprehensive coverage of negative samples.

4. Retrieval-Augmented Semantic Assignment: Top-\(K\) Search + Hierarchical Voting + Confidence-Prioritized Rasterization

Proposals generated by entity segmentation models (EntitySeg) often exhibit over-segmentation—an object is chopped into multiple fragments, or the background is split. This requires the retrieval stage to robustly handle both fragments and overlaps.

The retrieval pipeline consists of three levels. First, query features are extracted for each proposal using SMFA, and the Top-\(K\) (\(K=25\)) nearest neighbors are retrieved from the feature bank based on cosine similarity. Second, hierarchical voting is performed: votes are first cast by category (majority wins), and ties are resolved by the cumulative similarity of supporting samples—prioritizing "consensus over peaks," which effectively suppresses misclassifications from single outlier high-similarity samples. Third, conflict resolution: intra-class NMS is first applied to eliminate duplicate detections, and all remaining proposals are sorted in descending order of semantic confidence (the average similarity of the winning category). These proposals then fill the segmentation map pixel-by-pixel on a "first-come, first-served" basis—where large proposals with high confidence occupy space first, and smaller proposals with lower confidence fill the remaining gaps. This rasterization strategy naturally handles occlusions and boundary overlaps.

A Complete Example¶

Take a test image from PASCAL VOC featuring "a person riding a bicycle" as an example. EntitySeg might produce \(30+\) proposals: the rider's entire body (large region), the bicycle (medium region), the rider's helmet (small region), wheels (fragments), background tree canopies (multiple fragments), the sky (large region), etc.

For each proposal, SMFA-extracted features are searched against the feature bank to retrieve the Top-25 nearest neighbors. For the "rider's body" proposal, its neighbors contain 22 votes for "person" and 3 votes for "bicycle" (derived from co-occurrence features in the training set), yielding a final vote of "person". For the "bicycle" proposal, its neighbors have 18 votes for "bicycle", 5 for "person", and 2 for "motorbike", voting "bicycle". For the "helmet" proposal, the neighbors yield 20 votes for "person" (since most people in training images include head regions) and 5 for "hat", voting "person". In the NMS phase, since the "rider's body" and "helmet" proposals both belong to the "person" class and their IoU exceeds the threshold, the "helmet" proposal is suppressed. Finally, confidence-prioritized rasterization takes place: the rider's body (confidence 0.92) is placed first, followed by the bicycle (0.88) occupying non-overlapping regions, with the background tree canopy and sky filling the rest. The final segmentation map clearly delineates the "person" and "bicycle" classes, with boundaries directly inherited from EntitySeg's geometric proposals, which is far more precise than conventional CAM-based methods.

Loss & Training¶

This method requires absolutely no training—both feature bank construction and inference are forward-only operations with zero backpropagation or parameter updates. The only parameters to adjust are hyperparameters, such as the number of erosion iterations (20), the outlier removal ratio (25%), the number of KNN nearest neighbors (25), and the target confidence threshold (0.5), all of which are tuned on the validation set.

Key Experimental Results¶

Main Results¶

Dataset	Metric	ModuSeg	Prev. SOTA	Gain
VOC 2012 val	mIoU	86.3	79.5 (SSR)	+6.8
VOC 2012 test	mIoU	86.6	79.6 (SSR)	+7.0
COCO 2014 val	mIoU	56.7	50.6 (SSR)	+6.1

Without any training, ModuSeg outperforms all prior methods that require multi-stage retraining or end-to-end learning. Compared to ExCEL (78.4) and SSR (79.5), which also utilize foundation models, ModuSeg delivers improvements of 7.9 and 6.8 percentage points, respectively.

Ablation Study¶

Configuration	VOC mIoU	Description
Baseline (C-RADIOv4 + EntitySeg)	84.3	Completely without SBP and SMFA
+ SMFA	84.6	Soft mask aggregation replacing hard quantization
+ SBP	85.2	Morphological erosion to remove boundary noise
+ SMFA + SBP (ModuSeg Full)	86.3	Synergy of both, 1.1% extra gain

Configuration	VOC mIoU	COCO mIoU
CorrCLIP w/o Label Filtering	68.7	42.5
CorrCLIP w/ Label Filtering	78.8	54.6
Mask Adapter w/o Label Filtering	76.6	55.1
Mask Adapter w/ Label Filtering	82.4	60.2

Backbone	VOC mIoU	COCO mIoU
DINOv2 ViT-B	82.9	51.0
DINOv3 ViT-B	84.7	53.5
DINOv3 ViT-L	85.3	55.8
C-RADIOv4 SO400M	86.3	56.7

Data Efficiency and Upper Bound Analysis¶

Extreme Data Efficiency: Using only 50 training images per class achieves 80.2 mIoU (93% of the full-dataset 86.3 mIoU); and only 20 images per class yields 69.8 mIoU, demonstrating remarkable efficacy in few-shot scenarios.
The Performance Ceiling Lies in the Proposer, Not the Feature Bank: Replacing pseudo-masks with Ground Truth (GT) masks during feature bank construction only slightly increases the mIoU from 86.3 to 86.7. However, replacing EntitySeg proposals with GT masks during inference boosts performance to 95.7. This indicates that the main bottleneck lies in the class-agnostic proposer's over-segmentation and semantic gap, rather than the quality of the feature bank.
Mask Proposer Selection: EntitySeg outperforms SAM 2 (86.3 vs 82.6 mIoU on VOC), mainly due to EntitySeg's superior instance consistency.

Training Efficiency¶

Method	Training Time (min)	GPU Memory (GB)	mIoU
CLIMS	1068	18.0	70.4
MCTformer+	1496	18.0	74.0
WeCLIP	270	6.2	76.4
ModuSeg	84	5.3	86.3

ModuSeg's execution time is only about 1/6 of the currently top-performing SSR method (estimated based on inference duration, as SSR did not directly report training time but requires multi-stage SFT).

Highlights & Insights¶

Plug-and-play Decoupling: The most prominent highlight of this work is architectural—revising WSSS from a "coupled optimization" paradigm to a modular "plug-and-play" paradigm. Any superior Mask Proposer or stronger semantic feature extractor can be directly swapped in to seamlessly boost performance, entirely bypassing network retraining. This is of immense practical value in the era of rapid foundation model iterations.
Asymmetric Boundary Handling between Offline/Online Stages: SBP is only utilized during feature bank construction (prioritizing precision), whereas raw proposals are used during inference (prioritizing recall). This asymmetric design is simple yet critical: if erosion were also applied during inference, fragmented proposals would be further degraded, causing a substantial drop in retrieval and matching accuracy.
Skipping Purification for Background Class: The clustering assumption is inapplicable to the highly multi-modal background class. Retaining all background embeddings ensures negative sample diversity. This subtle but correct decision not to apply filtration is far more logical than prior approaches that enforced blanket filtering across all classes.

Limitations & Future Work¶

Dependency on Pre-trained Proposer Quality: Upper-bound analysis indicates that the primary performance bottleneck lies in EntitySeg's over-segmentation (chopping contiguous objects into fragments). Employing more robust proposers (or integrating multiple proposers) could yield further performance gains.
Constraint of Closed-set Categories: This work follows the standard WSSS setup (operating on a pre-defined closed category set of the training dataset) and lacks open-vocabulary segmentation capabilities. Although the feature bank could theoretically be expanded into a cross-dataset shared prototype library, this direction is not explored.
Reliance on Image-level Labels: Although pixel-level annotations are avoided, the training set still requires image-level category labels (indicating which classes are present in each image). Hence, this falls strictly under "weakly supervised" rather than "unsupervised" setting.
Efficient Retrieval in Large-Scale Scenarios: Utilizing IVF indexing for acceleration on the COCO dataset is necessary. The choice of FAISS indexes and hyperparameters (e.g., nlist, nprobe) affects the trade-off between speed and accuracy.

vs CAM-based WSSS (ToCo, MCTformer+, WeCLIP, etc.): These methods essentially operate under the assumption that "since the overall framework is coupled, we should improve specific components within it" (e.g., refining attention mechanisms or incorporating CLIP priors). In contrast, ModuSeg reframes the problem at the architectural level: since decoupling completely sidesteps the key challenges, there is no need to refine a coupled paradigm.
vs SAM-based WSSS (S2C, SEPL, etc.): These methods employ SAM's mask proposals to refine CAMs or pseudo-labels but still require training to fuse masks with semantic information. ModuSeg directly utilizes mask proposals for localization, eliminating any need for training-based bridging.
vs Prototype-Learning WSSS (SSR, ExCEL, etc.): These methods utilize learnable text prototypes or trainable projection heads to align vision and language features, which inherently still requires backpropagation. ModuSeg's feature bank consists entirely of feed-forward outputs from frozen extractors, making it simpler and more scalable.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Completely shifts WSSS from a "coupled optimization" paradigm to a "decoupled plug-and-play" paradigm, representing a path-breaking architectural innovation.
Experimental Thoroughness: ⭐⭐⭐⭐ Comprehensive coverage of main experiments, ablations, and various analyses (ceiling analysis, data efficiency, generalization, and training efficiency), but lacks a more systematic visualization of hyperparameters.
Writing Quality: ⭐⭐⭐⭐⭐ Very clear motivation, well-structured methodology, complete ablation evidence, and rich illustrations that align well with the text.
Value: ⭐⭐⭐⭐⭐ Training-free, modular, and substantially outperforming state-of-the-art—this combination is highly attractive to both the research community and industry.