This Looks Distinctly Like That: Grounding Interpretable Recognition in Stiefel Geometry against Neural Collapse¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Interpretability
Keywords: prototype networks, Stiefel manifold, neural collapse, dynamic rank calibration, part discovery

TL;DR¶

AMP replaces class prototypes with Stiefel-constrained orthonormal bases and uses sparse capacity and spatial constraints to select and localize distinct evidence, achieving 88.4% accuracy with ResNet50 on CUB, although its strict guarantee concerns basis rank rather than necessarily distinct semantic parts.

Background & Motivation¶

Models such as ProtoPNet turn recognition into matching local image regions against training exemplars, exposing the evidence behind a classification. Fine-grained recognition particularly benefits from this approach: the head, wings, and body texture may provide different clues for separating similar bird species. However, allocating multiple prototypes does not mean that the model uses multiple parts. When every prototype targets the most discriminative region, an explanation panel can appear rich while presenting highly redundant evidence. Such prototype collapse creates repetitive explanations and can leave other structural cues unused. Existing methods often penalize prototype similarity, but the optimizer can still accept similar prototypes when classification gains outweigh that penalty.

The authors connect this problem to neural collapse: late cross-entropy training tends to compress within-class variation, whereas part-based explanations require internal diversity. If same-class spatial features approach a common mean, freely learned prototypes may all follow that direction. This is the spatial-feature degeneration scenario underlying the paper's analysis, not an unconditional theorem about every convolutional network. AMP therefore excludes collinear basis vectors from the feasible parameter space instead of merely encouraging them to differ. Yet orthogonality specifies latent geometry, not that one direction must represent a bird's head and another its wing, so capacity selection and spatial localization remain necessary. Core Idea: separate prototype diversity into hard geometric independence, learnable effective capacity, and soft spatial-semantic constraints instead of asking a single diversity loss to handle all three.

Method¶

Overall Architecture¶

A convolutional backbone converts an input image into a spatial feature tensor, with a local feature vector at each position. AMP maintains orthogonal directions for each class and replaces ordinary prototype distances with the energy of local-feature projections onto those directions. Non-negative capacity weights then determine which directions contribute to classification, while spatial regularization makes retained responses concentrated and less overlapping. The outputs include a class prediction, heatmaps for active directions, and representative local exemplars retrieved from same-class training images. These exemplars visualize the evidence associated with a direction; they are not produced by a separate explanation-generating model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image and spatial features"] --> Basis["Stiefel Orthonormal Bases"]
    Basis --> Capacity["Dynamic Rank Calibration"]
    Capacity --> Maps["Active-direction response maps"]
    Maps -.->|Training constraint| Spatial["Spatial Gauge Fixing"]
    Spatial -.->|Gradient update| Basis
    Maps -->|Inference| Output["Class scores, heatmaps,<br/>same-class training exemplars"]

Spatial gauge fixing is a training objective, not an additional test-time post-processing step. Geometry and capacity parameters are updated during training; inference uses the learned bases and weights. Exemplar retrieval requires cached training features, so the complete explanation workflow also depends on an accessible training-example collection.

Key Designs¶

1. Stiefel Orthonormal Bases: exclude collinear prototypes from the feasible set

Each column of an ordinary prototype matrix can move freely, allowing several same-class columns to approach the same feature direction. AMP replaces this representation with \(U_c\), where \(D\) is the local-feature dimension and \(K\) is the number of basis directions per class. The following relation restates the explicit constraint in Section 4.1, retaining only what determines the feasible set:

\[ U_c\in\mathbb{R}^{D\times K},\qquad U_c^\top U_c=I_K. \]

This requires \(K\leq D\) and makes the directions unit length and mutually orthogonal. For \(K>1\), the complete basis matrix cannot become rank one, which distinguishes the hard constraint from a similarity penalty. For a local feature, the model computes squared inner products along the basis directions to measure how much energy the class subspace explains. Squaring removes the sign of a projection, so this evidence is directional energy rather than conventional signed template similarity.

Without capacity weighting, total projection energy depends on the subspace rather than the particular orthogonal coordinate axes chosen inside it. Consequently, equally predictive subspaces can admit different explanatory bases, motivating the subsequent spatial constraints. Equally importantly, full-rank \(U_c\) does not ensure that the backbone's within-class feature covariance avoids collapse. The guarantee concerns geometric independence of the prototype basis, not every possible degeneration of the representation-learning system. This boundary is essential for interpreting the phrase "against Neural Collapse."

2. Dynamic Rank Calibration: retain orthogonal candidate bases and disable surplus capacity

Using the same number of directions for every class can leave structurally simple categories with unnecessary prototypes. AMP introduces a non-negative diagonal capacity matrix \(\Sigma_c\), whose diagonal entries \(\sigma_{c,k}\) control individual contributions. Each direction's local projection energy is multiplied by its capacity before contributing to aggregated class evidence. Capacity and basis are separate parameter sets: one determines how many directions are used, and the other determines where they point. The paper describes the capacity-bearing representation as \(U_c\Sigma_c\) and defines its effective rank by the number of nonzero capacities. This must not be confused with the unweighted \(U_c\), which retains rank \(K\).

Training applies an \(\ell_1\) penalty to capacity, requiring additional directions to earn their place through classification utility. Ordinary gradient updates may produce small values without making them exactly zero. The authors instead follow a gradient step with proximal soft-thresholding and non-negative truncation, producing genuinely disabled directions. The active set contains positive-capacity directions; spatial losses operate only on that set without back-propagating through membership selection. The extracted symbols in Equation (8) are out of order, so the exact learning-rate and threshold formula is not reconstructed here. Deliberately pruning effective rank is also a design objective, distinct from the unconstrained prototype collinearity prohibited by the first design.

3. Spatial Gauge Fixing: associate independent directions with concentrated, different regions

Even orthogonal vectors may respond to the same image region or produce spatially diffuse heatmaps. Section 4.3 therefore computes a dense energy map for each direction and applies softmax across all spatial positions to obtain a normalized spatial distribution. Spatial entropy minimization averages distribution entropy over active directions, encouraging responses to concentrate at a small number of locations. Entropy here is the negative sum of probability times log probability; minimizing it discourages diffuse responses. Concentration alone cannot stop several directions from selecting the same part, so a spatial overlap penalty is also needed. That penalty averages the cosine similarity between different active heatmaps and minimizes their overlap.

When fewer than two directions are active, the overlap term is set to zero, avoiding normalization over nonexistent direction pairs. Together, the objectives express a preference for individually concentrated responses at different locations. The authors call this gauge fixing because it favors spatially more stable explanations among approximately classification-equivalent basis choices. Unequal capacities partially break rotational symmetry; the paper does not assert full rotational invariance for every weighted subspace. These spatial objectives require no part annotations, but remain soft inductive biases rather than guarantees of recovering real anatomical structure. In particular, excessive separation can impose unnatural spatial divisions when valid evidence overlaps.

Loss & Training¶

The total objective combines cross-entropy, spatial entropy, spatial overlap, and capacity sparsity under orthogonality and non-negativity constraints. Backbone parameters use ordinary SGD, basis matrices use Riemannian gradient updates, and capacities use a gradient step followed by proximal thresholding. For the bases, gradients are projected onto the tangent space and a QR-based retraction returns the updated matrices to the Stiefel manifold. Orthogonality is therefore maintained as part of each manifold update rather than checked only every few epochs. The main text assigns full derivations to an appendix, but the supplied full text ends with the references and does not include that appendix.

Implementation uses one NVIDIA RTX 4090, with bounding-box cropping and resizing to \(448\times448\). The initial per-class size is \(K=10\), the sparsity weight is \(\lambda=0.0001\), and both spatial weights are \(0.01\). Training lasts 100 epochs with batch size 32, and cosine scheduling reduces the learning rate from \(0.001\) to \(0.00001\). Backbones include VGG16, ResNet34, ResNet50, and DenseNet161, normally initialized with ImageNet weights. ResNet50 on CUB is the exception and uses iNaturalist pretraining, so that column must not be treated as an ordinary ImageNet-initialized result.

At inference, Section 4.4 describes forming class logits from maximum projection energy over spatial locations and selecting the highest-scoring class. Explanations retain only active directions of the predicted class and localize high-contribution regions in capacity-weighted energy maps. The system then searches cached features from same-class training images for the patch with maximum energy along the corresponding direction, providing a visible reference. This retrieves exemplars for explanation; it should not be confused with the ordinary prototype-projection procedure in Section 3, which would replace the basis vectors and disrupt orthogonality. The maximum and summation symbols in extracted Equation (14) are severely reordered, preventing verification of their exact nesting. It is therefore not established that summing each part's independently maximized response exactly reproduces the implemented logit; the original formula or code is needed to check this.

Key Experimental Results¶

Main Results¶

CUB-200-2011 contains 11,788 images from 200 bird species; Stanford Cars contains 16,185 images from 196 car classes. The table below selects ResNet50 results from original Table 1, page 11, using top-1 accuracy (%). CUB uses iNaturalist pretraining and Cars uses ImageNet pretraining; both follow the paper's bounding-box cropping setup.

Method	CUB-200-2011 accuracy (%)	Stanford Cars accuracy (%)	Model type
ProtoPNet	84.0 ± 0.5	88.7 ± 0.3	Intrinsically interpretable
MGProto	86.6 ± 0.4	90.5 ± 0.2	Intrinsically interpretable
AMP	88.4 ± 0.3	92.0 ± 0.2	Intrinsically interpretable
PMG	89.2 ± 0.3	93.6 ± 0.2	Black-box classifier

AMP improves over MGProto by 1.8 and 1.5 percentage points, respectively, but remains below PMG; it does not outperform every black-box model. Uncertainty terms are retained as reported; the supplied main text does not clearly specify repeat counts or their statistical definition.

The following table selects explanation metrics from original Table 2, page 12; its caption does not independently specify the backbone configuration. Higher Consistency and Stability and lower OIRR and DAUC are better, following the comparisons in Section 5.2. The main text does not fully define these evaluation procedures, including the expansion and normalization of OIRR or the deletion protocol for DAUC, so no formulas or units are invented.

Dataset	Method	Consistency	Stability	OIRR	DAUC
CUB-200-2011	MGProto	71.40 ± 1.90	45.80 ± 2.30	30.80 ± 2.10	4.02 ± 0.44
CUB-200-2011	AMP	76.80 ± 1.80	49.20 ± 2.10	28.10 ± 2.00	3.45 ± 0.41
Stanford Cars	MGProto	45.00 ± 2.40	74.20 ± 2.00	36.80 ± 2.10	6.05 ± 0.43
Stanford Cars	AMP	50.20 ± 2.20	76.40 ± 1.90	33.50 ± 2.00	5.65 ± 0.40

Ablation Study¶

The table below selects CUB accuracy, Consistency, and DAUC, plus Cars OIRR, from original Table 3, page 14. Full-model accuracy matches the ResNet50 column of Table 1, but Table 3 does not independently specify the backbone; matching numbers are not additional configuration evidence.

Config	CUB accuracy (%)	CUB Consistency	CUB DAUC	Cars OIRR
Without Stiefel constraint	85.2	65.0	4.95	40.8
Without capacity matrix	87.9	75.8	3.55	35.4
Without spatial entropy	87.0	73.2	3.92	35.0
Without spatial overlap	88.1	74.6	3.60	34.0
AMP full model	88.4	76.8	3.45	33.5

Key Findings¶

Removing the Stiefel constraint lowers CUB accuracy by 3.2 percentage points and raises DAUC from 3.45 to 4.95, the clearest degradation in Table 3.
Capacity has a smaller accuracy effect, but removing it raises Cars OIRR from 33.5 to 35.4, showing why redundancy reduction cannot be judged by accuracy alone.
Section 5.6 reports a human evaluation with 50 participants and states that CUB images typically activate three prototypes and Cars four; numerical figure ratings are not readable and are not supplied here.
The discussion of Figure 4 favors moderate sparsity and spatial weights; complete curve values are unavailable in the extracted text, so no new optimal parameter ranges are claimed.

Highlights & Insights¶

Diversity is no longer a single mixed objective: geometry prevents collinearity, capacity controls quantity, and spatial losses constrain evidence shape.
Proximal updates produce genuinely zero capacities rather than merely small contributions, giving the active-direction set a concrete discrete meaning.
Subspace classification and basis-direction explanation are different problems; recognizing that distinction is more informative than simply demanding sparser heatmaps.

Limitations & Future Work¶

Evidence covers two fine-grained datasets with bounding-box-cropped inputs, not full-scene recognition, medical diagnosis, or remote-sensing validation.
The argument from neural collapse to spatial-prototype collinearity depends on feature-degeneration assumptions; hard constraints do not replace direct measurements of backbone features and semantic parts.
Weighted effective rank can decrease while the orthogonal basis remains full column rank; claims of eliminating every kind of collapse would obscure this distinction.
The authors describe improved causal faithfulness, but better explanation metrics alone do not establish causal part semantics or immunity to out-of-distribution shortcuts.
Reproduction still needs an unambiguous inference aggregation formula, metric protocols, and appendix update details; future reporting should also quantify QR-update and exemplar-retrieval costs.

Compared with ProtoPNet: both support visual explanations with training exemplars, but AMP learns orthogonal directions, and exemplar retrieval must not be mistaken for reprojecting and replacing those directions.
Compared with diversity-constrained approaches such as TesNet: AMP emphasizes remaining in a geometric feasible set at every update, rather than merely adding another orthogonality loss.
Compared with MGProto: MGProto uses Gaussian-distributed prototypes and generative modeling, whereas AMP uses orthogonal subspaces and sparse capacity; Table 1 supports an accuracy advantage under the reported settings.
Transferable insight: applications to other concept models should separately test geometric independence, actual spatial responses, and task semantics; no two substitute for the third.

Rating¶

Novelty: 4/5. Hard orthogonality, proximal capacity, and spatial gauge fixing form a coherent response to prototype redundancy.
Experimental Thoroughness: 3/5. Multiple backbones, ablations, and human evaluation are included, but task coverage and reproducibility details remain limited.
Writing Quality: 3/5. Component roles are clear, but some guarantee language is strong and damaged formulas in the supplied text hinder verification.
Value: 4/5. The useful lesson is the division between geometric and semantic constraints, not equating orthogonality with trustworthy explanation.