ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentation¶

Conference: ECCV2026
arXiv: 2604.07021
Code: https://github.com/Autumnair007/ModuSeg
Area: Semantic Segmentation
Keywords: Weakly Supervised Semantic Segmentation, Decoupling, Training-Free, Feature Retrieval, Foundation Models

TL;DR¶

ModuSeg explicitly decouples weakly supervised semantic segmentation into two independent stages: "class-agnostic object discovery" and "semantic retrieval." By utilizing a general Mask Proposer to extract geometric proposals and an offline feature bank for non-parametric retrieval, it achieves a new SOTA performance of 86.3 mIoU on VOC without any fine-tuning, using only image-level labels from the training set.

Background & Motivation¶

The goal of weakly supervised semantic segmentation (WSSS) is to achieve pixel-wise predictions using only image-level labels, thereby significantly reducing the cost of pixel-level annotation. Traditional methods almost all follow the same path: generating initial localization cues using Class Activation Maps (CAMs), and then expanding the segmented regions through multi-stage network retraining or end-to-end joint optimization. The fundamental issue with this approach is that semantic recognition and object localization are coupled into the same classification objective. Classification networks naturally focus only on the most discriminative local regions, causing CAM activation maps to cover only sparse fragments of objects (such as a bird's head rather than its entire body). Consequently, the generated foreground regions are inherently incomplete with ambiguous boundaries. To repair these incomplete seed regions, traditional paradigms either rely on multi-stage retraining (high computational overhead, complex pipelines) or end-to-end joint optimization (unstable training, difficult to suppress pseudo-label noise).

The rise of vision foundation models (DINOv2/v3, C-RADIOv4, SAM, EntitySeg, etc.) has introduced new possibilities to resolve this dilemma. Class-agnostic segmentation models exhibit exceptional boundary-aware capabilities, while vision-language models and self-supervised models learn highly potent dense semantic descriptions. However, many existing methods still follow the coupled optimization path—integrating foundation models for fine-tuning or end-to-end distillation—resulting in unavoidable interference from the inherent noise in weakly supervised pseudo-labels and failing to fully unleash the potential of foundation models.

The core insight of this paper is: since geometric localization and semantic classification are inherently two distinct tasks, why optimize them in a coupled manner? A general Mask Proposer (such as EntitySeg) is natively proficient at extracting clean target proposals without requiring any semantic information; a semantic foundation model (such as C-RADIOv4) can naturally provide high-quality dense features. Directly combining these two—using proposals for localization and feature retrieval for classification—bypasses all the trouble of coupled optimization. Core Idea: Explicitly decouple weakly supervised semantic segmentation into two independent stages: class-agnostic object discovery and semantic retrieval. A general Mask Proposer is utilized to extract geometric proposals, and an offline-constructed feature bank is employed for non-parametric KNN retrieval to assign semantics. The entire pipeline requires zero parameter fine-tuning.

Method¶

Overall Architecture¶

The core of ModuSeg is to split WSSS into two completely independent stages—offline feature bank construction and online inference retrieval—with only a non-parametric feature similarity calculation between them, free of any cross-stage gradient backpropagation or joint optimization.

Stage 1 (Offline): Building a robust feature bank based on training set image-level labels. This stage uses a pre-trained CorrCLIP to generate initial pseudo-masks for each training image. However, directly extracting features using these masks poses two problems: pseudo-masks suffer from severe feature mixing at object boundaries due to ViT patch alignment and weak supervision noise, and hard quantization sampling introduces quantization noise. This paper addresses these issues through Semantic Boundary Purification (SBP, discarding boundaries via morphological erosion) and Soft Mask Feature Aggregation (SMFA, weighted area interpolation) to extract high-quality class prototypes, followed by removing outliers based on clustering assumptions to obtain a clean feature bank.

Stage 2 (Online Inference): Extracting target proposals with a class-agnostic Mask Proposer, followed by retrieval voting in the feature bank to complete semantic classification. Given a test image, EntitySeg generates a set of binary Mask proposals (answering "there is an object here" without knowing what it is). The same SMFA is applied to each proposal to extract query features, retrieving the Top-K nearest neighbors in the feature bank. Majority voting determines the semantic category. Finally, intra-class NMS and confidence-prioritized rasterization are applied to overlapping proposals to obtain the final segmentation maps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training Images<br/>+ Image-Level Labels"] --> B["CorrCLIP<br/>Generates Pseudo-Masks"]
    B --> C["Semantic Boundary Purification<br/>Morphological Erosion Discards Boundaries"]
    C --> D["Soft Mask Feature Aggregation<br/>Weighted Area Interpolation"]
    D --> E["Prototype-Level Feature Purification<br/>Outlier Removal"]
    E --> F["Offline Feature Bank"]

    G["Test Image"] --> H["EntitySeg<br/>Mask Proposer"]
    H --> I["SMFA Extracts<br/>Query Features"]
    I --> J["Top-K Retrieval<br/>+ Majority Voting"]
    F --> J
    J --> K["Intra-class NMS<br/>+ Confidence Rasterization"]
    K --> L["Final Segmentation Map"]

Key Designs¶

1. Semantic Boundary Purification (SBP): Actively Discarding Unreliable Boundaries for Feature Purity

A structural mismatch inherently exists between ViT feature maps and images—if a \(16 \times 16\) patch happens to straddle an object boundary, it encodes a combined foreground and background signal. Moreover, weakly supervised pseudo-masks are inherently highly uncertain at object boundaries (where the classifier exhibits low confidence). If these "dirty" patches are used for feature bank construction, each class prototype becomes contaminated with noise, degrading the discriminative power of subsequent retrievals.

The solution of SBP is simple yet highly effective: use morphological erosion to "trim a ring" off the pseudo-masks. Specifically, for the binary mask of each class, a \(3 \times 3\) structuring element is applied with 20 erosion iterations. This actively abandons the "gray areas" near object boundaries, retaining only the core regions where the patches almost certainly belong to the target class, yielding the cleanest features. Although this sacrifices boundary geometric completeness, it provides highly semantically pure features. Note that this operation is only used during the offline feature bank construction (where accuracy is the goal), while the inference stage directly uses the raw proposals from EntitySeg without erosion (where recall and boundary fidelity are the goals), creating an intentional asymmetry between the training and inference stages.

2. Soft Mask Feature Aggregation (SMFA): Area Interpolation Replaces Hard Quantization to Suppress Quantization Noise

When mapping a high-resolution mask to a low-resolution ViT feature grid, naive nearest-neighbor downsampling assigns binary labels ("contains" or "does not contain") to each feature grid point. However, grid points at the mask boundaries are likely only partially covered by the object. Hard quantization either incorporates the grid point entirely (introducing background noise) or completely excludes it (losing valid information).

SMFA replaces hard quantization with area interpolation: the high-resolution purified mask is downsampled to the feature grid resolution, where each grid point receives a soft weight \(w \in [0, 1]\) representing the actual coverage ratio of the object. The feature prototype is then calculated as a weighted average:

\[v = \text{normalize}\left(\frac{\sum(w \cdot F)}{\sum w}\right)\]

Consequently, patches entirely within the object center contribute close to 1, while partially covered edge patches naturally decay proportionally, making the entire aggregation process differentiable and clean. This design also decouples the module from any specific ViT backbone—whether switching to DINOv2 or C-RADIOv4, the SMFA pipeline remains unchanged; only the feature extractor needs to be replaced.

3. Prototype-Level Feature Purification: Outlier Removal Driven by Clustering Assumptions

Even with SBP + SMFA, pseudo-label noise cannot be completely eliminated—some training samples have severely shifted pseudo-masks, or their feature vectors deviate from the intra-class distribution center. This paper performs a second round of filtering based on the clustering assumption (i.e., reliable features of the same class form compact clusters in high-dimensional space, whereas noisy samples are loosely distributed at the periphery).

For each foreground class, the mean of all candidate feature vectors is first computed as the global prototype \(\mu_c\), and the Euclidean distance \(d_v\) from each feature vector \(v\) to \(\mu_c\) is calculated. Outliers with the largest top \(\alpha\%\) (default 25%) distances are removed. A key design decision here is: the background class is not purified. The background is naturally multi-modal (sky, road, vegetation, buildings...), meaning a single-center assumption is entirely inapplicable. Enforcing filtering on the background would lead to a loss of background diversity, harming the model's discriminative ability in complex environments. Retaining all background embeddings ensures comprehensive coverage of negative samples.

4. Retrieval-Augmented Semantic Assignment: Top-K Search + Hierarchical Voting + Confidence Rasterization

Proposals provided by entity segmentation models (like EntitySeg) often suffer from over-segmentation—an object might be fragmented into multiple pieces, or a background region might be split. This requires the retrieval stage to robustly handle fragments and overlaps.

The retrieval pipeline is divided into three layers. First, SMFA is applied to each proposal to extract query features, retrieving the Top-K (\(K=25\)) nearest neighbors (based on cosine similarity) in the feature bank. Second, hierarchical voting is performed. Votes are first counted by class, applying majority voting. In case of a tie, the cumulative similarity of supporting samples determines the winner—prioritizing "consensus over peaks" successfully suppresses misjudgments caused by single outlier high-similarity samples. Third, conflict resolution: intra-class NMS is first applied to eliminate duplicate detections, and then all remaining proposals are sorted in descending order by semantic confidence (the average similarity of the winning class). They are then rasterized pixel-by-pixel in a first-come, first-served manner to populate the final segmentation map—large proposals with high confidence occupy space first, while smaller proposals with lower confidence fill the remaining gaps. This rasterization strategy naturally handles occlusions and boundary overlaps.

A Detailed Example¶

Take a test image from PASCAL VOC containing "a person riding a bicycle" as an example. EntitySeg might output over 30 proposals: the rider's body (large region), the bicycle (medium region), the rider's helmet (small region), the wheels (fragments), the background tree canopy (multiple fragments), the sky (large region), etc.

For each proposal, SMFA extracts features, and a Top-25 nearest neighbor retrieval is conducted in the feature bank. For the "rider's body" proposal, 22 of the nearest neighbors are "person", and 3 are "bicycle" (due to features from scenes where humans and bicycles co-occur in the training set), yielding a vote for "person". For the "bicycle" proposal, its neighbors vote 18 for bicycle, 5 for person, and 2 for motorbike, resulting in "bicycle". For the "helmet" proposal, the nearest neighbors assign 20 votes to person (as most "person" objects in training images include the head region) and 5 to hat, voting for "person". In the NMS phase, the "rider's body" and "helmet" proposals both belong to the "person" class, and because their IoU exceeds the threshold, the "helmet" is suppressed. Finally, during confidence rasterization: the rider's body (confidence 0.92) is placed first, then the bicycle (0.88) occupies the non-overlapping regions, and the background canopy and sky fill the rest. The final segmentation map clearly delineates the "person" and "bicycle" classes, with boundaries derived from EntitySeg's original geometric proposals, which are significantly sharper than those from traditional CAM-based methods.

Loss & Training¶

The proposed method is entirely training-free—both feature bank construction and inference are forward-pass operations with zero backpropagation or parameter updates. The only parts requiring adjustment are hyperparameters: morphological erosion iterations (20), outlier removal ratio (25%), the number of nearest neighbors \(K\) in KNN (25), object confidence threshold (0.5), etc., which can be determined on the validation set.

Key Experimental Results¶

Main Results¶

Dataset	Metric	ModuSeg	Prev. SOTA	Gain
VOC 2012 val	mIoU	86.3	79.5 (SSR)	+6.8
VOC 2012 test	mIoU	86.6	79.6 (SSR)	+7.0
COCO 2014 val	mIoU	56.7	50.6 (SSR)	+6.1

Without any training, ModuSeg outperforms all previous methods that require multi-stage retraining or end-to-end learning. Compared to ExCEL (78.4) and SSR (79.5), which also utilize foundation models, ModuSeg achieves gains of 7.9% and 6.8% respectively.

Ablation Study¶

Configuration	VOC mIoU	Description
Baseline (C-RADIOv4 + EntitySeg)	84.3	Completely without SBP and SMFA
+ SMFA	84.6	Soft Mask aggregation replaces hard quantization
+ SBP	85.2	Morphological erosion removes boundary noise
+ SMFA + SBP (ModuSeg Full)	86.3	Synergy between the two, 1.1% additional gain

Configuration	VOC mIoU	COCO mIoU
CorrCLIP w/o label filtering	68.7	42.5
CorrCLIP w/ label filtering	78.8	54.6
Mask Adapter w/o label filtering	76.6	55.1
Mask Adapter w/ label filtering	82.4	60.2

Backbone	VOC mIoU	COCO mIoU
DINOv2 ViT-B	82.9	51.0
DINOv3 ViT-B	84.7	53.5
DINOv3 ViT-L	85.3	55.8
C-RADIOv4 SO400M	86.3	56.7

Data Efficiency & Upper Bound Analysis¶

Extremely high data efficiency: ModuSeg achieves 80.2 mIoU using only 50 training images per class (93% of the 86.3 performance on the full dataset); using only 20 images per class yields 69.8 mIoU. It remains highly effective in few-shot scenarios.
Performance bottleneck lies in the Proposer rather than the feature bank: Replacing pseudo-masks with ground-truth (GT) masks to build the feature bank only minimally improves mIoU from 86.3 to 86.7. However, replacing EntitySeg proposals with GT masks during inference increases mIoU to 95.7—showing that the main bottleneck lies in the class-agnostic Proposer's over-segmentation and semantic gap, rather than the quality of the feature bank.
Mask Proposer Selection: EntitySeg outperforms SAM 2 (VOC 86.3 vs 82.6), primarily because EntitySeg exhibits stronger instance consistency.

Training Efficiency¶

Method	Training Time (min)	GPU Memory (G)	mIoU
CLIMS	1068	18.0	70.4
MCTformer+	1496	18.0	74.0
WeCLIP	270	6.2	76.4
ModuSeg	84	5.3	86.3

ModuSeg's training time is only about 1/6 of that of the current best method SSR (estimated based on inference time, as SSR did not directly report training time but requires multi-stage SFT).

Highlights & Insights¶

Decoupled plug-and-play capability: The biggest highlight of this work is structural—shifting the WSSS paradigm from "coupled optimization" to "modular assembly." Any improved Mask Proposer or stronger semantic feature extractor can be directly plugged in to organically boost performance without retraining networks, which is exceptionally valuable in the rapidly evolving era of foundation models.
Asymmetric training-inference boundary processing: SBP is used only when constructing the feature bank (prioritizing accuracy), while original proposals are used during inference (prioritizing recall)—this asymmetric design is simple yet crucial. If erosion were also applied during inference, fragmented proposals would be further weakened, causing a significant drop in retrieval matching accuracy.
Decision to skip background purification: The clustering assumption is inapplicable to multi-modal background classes. Retaining all background embeddings preserves negative sample diversity. This subtle but correct decision to "refrain from actions" is more rational than previous methods that blindly enforce filtering on all classes.

Limitations & Future Work¶

Dependency on pre-trained Proposer quality: Upper bound analysis reveals that the bottleneck lies in EntitySeg's over-segmentation (chopping continuous objects into fragments). Utilizing a stronger Proposer (or an ensemble of multiple proposers) offers considerable room for performance improvement.
Limitation to fixed category sets: This work still follows the standard WSSS setting (given a category set in the training set) and lacks open-vocabulary segmentation capabilities. The feature bank itself could be extended to a cross-dataset shared class prototype library, but this direction was not explored in this paper.
Requirement for image-level labels: Although pixel-level annotations are avoided, the training set still needs to include image-level category labels (annotated per image group), which strictly classifies it as "weakly supervised" rather than "unsupervised."
Efficient retrieval in large-scale scenarios: On the COCO dataset, using IVF index acceleration is necessary. The choice of FAISS index and hyperparameters (like nlist, nprobe, etc.) affects the accuracy-speed trade-off.

vs CAM-based WSSS (ToCo, MCTformer+, WeCLIP, etc.): These methods essentially operate under the mindset of "since the overall framework is coupled, try to improve individual links within the framework" (improving attention, introducing CLIP priors, etc.). ModuSeg reformulates the problem at the framework level—since decoupling bypasses the core conflict, there is no need to polish the coupling.
vs SAM-based WSSS (S2C, SEPL, etc.): These methods use SAM's Mask proposals to refine CAMs or pseudo-labels but still require training to integrate mask and semantic information. ModuSeg directly utilizes Mask proposals for localization without any training bridging them.
vs Prototype-Learning WSSS (SSR, ExCEL, etc.): These methods employ learnable text prototypes or trainable projection heads to align vision and language features, which inherently still requires backpropagation. ModuSeg's feature bank consists entirely of feedforward outputs from frozen extractors, making it simpler and more scalable.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Completely shifts the WSSS paradigm from "coupled optimization" to "decoupled plug-and-play," representing a path-level innovation.
Experimental Thoroughness: ⭐⭐⭐⭐ The main experiments, ablations, and various analyses (upper bound, data efficiency, generalization, training efficiency) are comprehensive, though more systematic hyperparameter visualizations are lacking.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, well-structured methodology, and complete evidence chains in the ablations, with abundant charts highly correlated with the text.
Value: ⭐⭐⭐⭐⭐ Training-free, modular, and substantially outperforming the SOTA—this combination is highly attractive to both the research community and industrial applications.