SGQA: Semantic-Geometric Quality Alignment for Training-Free Few-Shot Instance Segmentation

Conference: ECCV 2026
Paper: ECCV Official
Code: None
Area: Segmentation
Keywords: Training-Free, Few-Shot Instance Segmentation, Foundation Model Composition, Quality Alignment, Geometric Mean

TL;DR

Addressing the quality disconnect between semantic matching and mask geometry in training-free foundation model composition, this paper employs progressive oracle replacement to trace 64% of the performance gap to the scoring stage, and proposes SGQA—a hyperparameter-free geometric mean fusion that unifies semantic and geometric scores, boosting performance significantly without fine-tuning.

Background & Motivation

Composing frozen pre-trained vision foundation models, such as SAM/SAM2 for mask generation and DINOv2/CLIP for visual representation learning, has emerged as an attractive paradigm for training-free few-shot instance segmentation. By decoupling the workflow into a semantic encoder that specifies what to find and a mask generator that determines where to segment, this paradigm circumvents costly gradient updates and avoids the risk of severe overfitting to scarce novel-class annotations. Despite these advantages, existing compositions consistently exhibit a substantial performance gap compared to empirical upper bounds, and even recent techniques incorporating prototype refinement or prompt personalization fail to break this performance plateau.

A close inspection of the standard composition pipeline reveals that the primary bottleneck stems from "Quality Misalignment" at the scoring stage. Because the semantic matching confidence and the geometric mask fidelity are produced by two independently pre-trained foundation models without joint calibration, existing pipelines handle them in isolation. Typically, candidate masks are ranked and pruned via non-maximum suppression (NMS) using the mask generator's predicted IoU score, while the semantic similarity score is assigned as the reported detection confidence. Consequently, false positives that exhibit high semantic similarity but degraded, coarse mask geometry are frequently ranked above true positive candidates that possess well-balanced semantic and spatial qualities.

Addressing this defect purely through localized feature matching or prompt engineering fails to resolve the conflicting signals in the joint score space. Through a rigorous stage-by-stage progressive oracle replacement analysis, this work quantitatively reveals that the scoring stage alone is responsible for 64% of the performance gap between baseline compositions and the oracle ceiling, and that true positive instances fundamentally require mutual agreement between semantic relevance and geometric fidelity. Core Idea: Eliminate the decoupled dual-track scoring mechanism by fusing the semantic score and the geometric score via a hyperparameter-free geometric mean into a unified quality score, strictly penalizing one-sided false positives and enforcing consistent quality alignment across both NMS re-ranking and final confidence assignment.

Method

Overall Architecture

SGQA operates within the standard training-free few-shot instance segmentation framework. Given a few-shot support set with instance mask annotations, frozen visual features are extracted using a semantic encoder to construct class-wise memory banks and K-Means prototypes. During query inference, a frozen mask generator produces dense candidate masks along with their predicted geometric IoU scores, while the semantic encoder extracts average foreground features for each mask to obtain class assignment and cosine semantic similarity scores. SGQA subsequently fuses both signals via the geometric mean into a unified quality score that governs both NMS candidate suppression and final detection confidence evaluation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query Image and Few-Shot Support Set"] --> B["Prototype Construction & Dense Candidate Generation<br/>DINOv2 K-Means Prototypes + SAM2 Grid Prompted Masks"]
    B --> C["Progressive Oracle Replacement & Bottleneck Localization<br/>Stage-by-stage replacement reveals scoring accounts for 64% gap"]
    C --> D["Semantic-Geometric Quality Alignment Scoring<br/>Hyperparameter-free geometric mean penalizes unbalanced candidates"]
    D --> E["Dual-Role Cooperative Suppression<br/>Unified quality score governs NMS re-ranking and detection confidence"]
    E --> F["High-Precision Few-Shot Instance Segmentation Output"]

Key Designs

1. Progressive Oracle Replacement and Bottleneck Localization: Decoupled diagnosis of error sources across frozen composition stages

In a multi-model frozen pipeline, aggregate end-to-end evaluation metrics fail to pinpoint whether performance degradation originates from representational inadequacy or decision misalignment. To resolve this ambiguity, the authors introduce a progressive oracle replacement protocol that partitions the inference pipeline into candidate scoring, NMS suppression, and label classification. By systematically substituting each component with ground-truth oracle decisions, the analysis demonstrates that replacing the scoring function with oracle IoU eliminates 64% of the performance gap on COCO 30-shot, whereas oracle NMS or oracle classification provide negligible isolated gains. Furthermore, rank correlation analysis indicates that the Spearman rank correlations with ground-truth IoU are only \(\rho_{\text{sem}} = 0.36\) and \(\rho_{\text{geo}} = 0.39\) for semantic and geometric scores respectively, proving that false positive pairs are heavily concentrated in off-diagonal score spaces (e.g., high semantic but low geometric score). This diagnostic establishes that cross-model quality misalignment, rather than foundational model capacity, is the dominant system bottleneck.

2. Semantic-Geometric Quality Alignment Scoring: Imposing geometric mean penalty on asymmetric candidate scores

Having identified the scoring stage as the principal point of failure, the scoring function must enforce simultaneous satisfaction of semantic identity and spatial precision. The general weighted power-mean aggregation is formulated as:

\[Q(m_i) = (s_{\text{sem}, i})^\alpha \cdot (s_{\text{geo}, i})^{1 - \alpha}, \quad \alpha \in [0, 1]\]

Prior methods represent degenerate extremes of this formulation: DE-ViT relies primarily on semantic similarity (\(\alpha \to 1\)), whereas Matcher and PerSAM depend heavily on the mask generator's IoU head (\(\alpha \to 0\)). Given that the Spearman rank correlations indicate both signals possess comparable discriminative utility, symmetric weighting \(\alpha = 0.5\) is adopted, yielding the standard geometric mean:

\[Q(m_i) = \sqrt{s_{\text{sem}, i} \cdot s_{\text{geo}, i}}\]

Unlike the arithmetic mean whose linear iso-contours permit a high score in one dimension to offset a poor score in the other, the geometric mean features convex iso-value contours that aggressively penalize candidates with asymmetric scores. From a probabilistic perspective, treating both signals as conditionally independent likelihood ratios renders the product proportional to the Bayesian posterior quality. Additionally, taking the square root maintains identical ranking while appropriately rescaling confidence magnitudes for threshold-sensitive precision-recall evaluation, introducing zero additional hyperparameters.

3. Dual-Role Cooperative Suppression: Unifying NMS candidate ordering and detection confidence evaluation

A critical flaw in conventional pipelines is the divergence in scoring criteria across pipeline stages: geometric IoU scores are used to sort candidates during NMS, while raw semantic similarities are assigned to survivors as detection confidences. This separation allows false positives with high geometric quality to suppress semantically valid instances during NMS before semantic confidence is even considered. SGQA eliminates this structural disconnect by injecting the unified score \(Q(m_i)\) into both operational roles simultaneously. Using \(Q(m_i)\) for NMS ranking ensures that retained candidates satisfy both semantic and geometric criteria, while deploying it as the final confidence properly reflects true joint detection quality. Factorial ablation reveals that using \(Q\) solely for NMS yields a +1.7 AP gain and using it solely for confidence estimation yields +1.3 AP, while joint deployment achieves +2.6 AP, confirming the cooperative synergy between re-ranking and calibrated confidence assignment.

Key Experimental Results

Main Results

On the COCO-FSOD benchmark (10-shot and 30-shot novel classes) and PASCAL VOC across three splits, SGQA is evaluated against representative meta-learning, fine-tuning, and training-free approaches.

Method	Paradigm	Backbone	COCO 10-shot nAP	COCO 30-shot nAP	COCO 30-shot nAP50	COCO 30-shot nAP75
TFA	Fine-tuning	ResNet-101	10.0	13.5	24.9	13.2
FSCE	Fine-tuning	ResNet-101	11.9	16.4	-	16.2
DeFRCN	Fine-tuning	ResNet-101	18.8	20.9	-	-
CD-ViTO	Fine-tuning	ViT-L	35.3	35.9	54.5	38.0
DE-ViT	Training-free	ViT-L	34.0	34.0	52.9	37.2
NTT (Baseline)	Training-free	DINOv2-L + SAM2-L	36.6	36.8	54.5	38.7
SGQA (Ours)	Training-free	DINOv2-L + SAM2-L	39.3	39.4	57.3	41.7

Across low-shot scaling regimes on COCO novel categories, SGQA yields consistent and widening improvements as shots decrease:

Shot Count	Bbox nAP (NTT)	Bbox nAP (SGQA)	\(\Delta\) Bbox	Segm nAP (NTT)	Segm nAP (SGQA)	\(\Delta\) Segm
1-shot	33.5	36.8	+3.3	30.8	34.6	+3.8
5-shot	35.8	38.8	+3.0	33.2	36.7	+3.5
10-shot	36.6	39.3	+2.7	34.0	37.3	+3.3
30-shot	36.8	39.4	+2.6	34.2	37.5	+3.3

Ablation Study

Ablations on the generalized power-mean formulation \(M_p = (\frac{s_{\text{sem}}^p + s_{\text{geo}}^p}{2})^{1/p}\) and pipeline role decoupling are summarized below:

Aggregation / Configuration	Exponent \(p\) / Mode	COCO 30-shot nAP	Gain over Baseline	Mechanism & Behavior
NTT Baseline (Decoupled)	-	36.8	-	NMS ranked by \(s_{\text{geo}}\), confidence by \(s_{\text{sem}}\)
Harmonic Mean	\(p = -1\)	38.1	+1.3	Overly penalizes true positives with marginal geometry
Geometric Mean (SGQA)	\(p \to 0\)	39.4	+2.6	Optimally aligns ranking with ground-truth IoU
Arithmetic Mean	\(p = 1\)	37.9	+1.1	Linear iso-contours fail to suppress imbalanced FPs
Quadratic Mean	\(p = 2\)	37.2	+0.4	Relaxes penalty, retaining more spurious candidates
Unified \(Q\) in NMS Only	\(Q \to \text{NMS}\)	38.5	+1.7	Rectifies suppression but leaves confidences uncalibrated
Unified \(Q\) in Confidence Only	\(Q \to \text{Score}\)	38.1	+1.3	Improves PR sorting but cannot recover NMS-pruned TPs

Key Findings

• Larger Gains in Lower-Shot Regimes: Gains increase as reference shots decrease (+3.3 Bbox AP and +3.8 Segm AP at 1-shot vs. +2.6 and +3.3 AP at 30-shot). Remarkably, 1-shot SGQA achieves 36.8 Bbox AP, matching the 30-shot NTT baseline. When prototypes are noisy due to limited reference samples, geometric mean filtering effectively demotes pseudo-high semantic false positives.
• Cross-Encoder Generality: Across eight encoders spanning CLIP (B/32, B/16, L/14, L@336), DINOv2 (ViT-L), and DINOv3 (ViT-B, ViT-L, ViT-H), SGQA yields consistent gains between +1.6 and +2.6 AP. The optimal weighting parameter consistently peaks at \(\alpha = 0.5\) across all configurations.
• Substantial Oracle Gap Recovery: The theoretical score-level-recoverable ceiling is estimated at 6.2 AP; SGQA recovers 42% (2.6 AP) of this upper bound and 17% of the total 15.1 AP oracle gap without any training or parameter tuning.

Highlights & Insights

• Targeted Bottleneck Diagnosis: Rather than indiscriminately complicating network components or prompt generators, the authors employ progressive oracle replacement to rigorously demonstrate that over 60% of composition failure resides in decoupled scoring.
• Hyperparameter-Free Elegance: By grounding the choice of geometric mean in rank correlation analysis and Bayesian decision theory, the method avoids heuristic parameter tuning and offers clean plug-and-play integration.
• Broad Relevance for Modular AI: The quality misalignment identified here serves as a general diagnostic caveat for multi-foundation-model assemblies where independent frozen heads must cooperate without joint fine-tuning.

Limitations & Future Work

• Ceiling of Pure Score-Level Fusion: SGQA operates purely on scalar score post-processing. While it captures 42% of the score-level gap, bridging the remaining 58% will likely require bidirectional feature-level information exchange between the semantic encoder and mask generator.
• Dependence on Initial Prompt Grid Density: On extremely dense scenes with tiny objects, grid-prompted SAM2 may occasionally fail to produce valid candidates, which cannot be compensated by subsequent scoring adjustments.
• Future Directions: Exploring gradient-free, lightweight cross-attention feature prompting to extend scalar quality alignment into early-stage representation alignment.

Related Work & Insights

• vs. DE-ViT: DE-ViT focuses on feature prototype propagation but defaults to semantic matching similarity for ranking, neglecting mask fidelity; SGQA resolves this misalignment and outperforms DE-ViT by 5.4 nAP on COCO 30-shot.
• vs. Matcher / PerSAM: Matcher and PerSAM rely heavily on the mask generator's IoU prediction; SGQA demonstrates that semantic and geometric signals carry equal ranking value and should be symmetrically fused (\(\alpha = 0.5\)).
• vs. Supervised Quality Alignment (GFL / TOOD): While supervised detectors mitigate classification-localization misalignment using custom loss functions, SGQA translates this principle into frozen foundation model composition through a fully non-parametric, analytical formulation.

Rating

• Novelty: ⭐⭐⭐⭐☆ Pinpoints the critical quality misalignment bottleneck in frozen model compositions with elegant diagnostic tools.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 8 encoders, 4 SAM2 variants, multiple few-shot benchmarks, and detailed oracle/factorial ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from mathematical formulation and diagnostic motivation to empirical verification.
• Value: ⭐⭐⭐⭐⭐ Establishes a highly practical, parameter-free baseline for training-free few-shot vision systems.