Hyperbolic Hierarchical Clustering for Visual Representation Learning¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/weijianan1/HCFormer
Area: Segmentation
Keywords: Hyperbolic Geometry, Hierarchical Clustering, Vision Backbone, ClusterMixer, Semantic Segmentation
TL;DR¶
Addressing the black-box nature of conventional vision backbones and the quadratic complexity and Euclidean metric distortions of prior clustering backbones, this paper introduces an interpretable clustering token mixer (ClusterMixer) and the HCFormer architecture, which decouples local Euclidean patch clustering from global hyperbolic window clustering to achieve linear computational complexity, high accuracy, and strong interpretability.
Background & Motivation¶
Convolutional Networks (ConvNets) and Vision Transformers (ViTs) remain the predominant foundation backbones across computer vision tasks. ConvNets established the classic feature extraction paradigm through inductive biases such as spatial locality and translation equivariance, while ViTs and spatial MLP variants further broadened representation capacity by modeling long-range dependencies across image tokens. However, mainstream token mixers relying on convolution, self-attention, or spatial MLPs are inherently black-box models: their feature routing, spatial mixing, and internal reasoning mechanisms are opaque and largely unintuitive to human interpretation. Although recent efforts such as Context Cluster (CoC) and FEC explored clustering algorithms as a transparent alternative for token aggregation and dispatching, their task performance consistently lags behind mainstream ConvNet and Transformer baselines, leaving the full potential of clustering in representation learning untapped.
The core tension preventing clustering-based vision architectures from delivering both high efficiency and strong generalization lies in a dual dilemma. First is the rigid computational barrier: naive global clustering incurs quadratic computational complexity with respect to the number of data points and cluster centers. In dense prediction tasks like semantic segmentation and object detection that require high spatial resolutions, models cannot scale up the number of cluster centers without hitting severe computational bottlenecks, while artificially suppressing center counts severely degrades fine-grained visual details. Second is the geometric distortion in Euclidean metric modeling. Visual semantics naturally follow a tree-like hierarchy from local patterns to high-level semantic categories; however, zero-curvature Euclidean spaces cannot embed expanding hierarchical trees without substantial distortion, causing semantically distant entities across abstract hierarchies to appear spuriously close in Euclidean distance.
The paper resolves this impasse by orchestrating a dual-geometry hierarchical clustering strategy: leveraging Euclidean geometry locally within windows to preserve continuous flat textures, while projecting window-level abstractions onto hyperbolic space to exploit its constant negative curvature for tree-like hierarchical modeling with minimal distortion. Core idea: design an interpretable soft-assignment token mixer named ClusterMixer and assemble the HCFormer backbone, which decouples local Euclidean patch clustering from global hyperbolic window clustering to achieve linear-complexity, multi-scale visual representation learning.
Method¶
Overall Architecture¶
HCFormer adopts a multi-stage hierarchical pyramid design spanning 4 stages. An input image is initially partitioned by a patch embedding layer with a stride of 4 into a sequence of discrete visual tokens, with subsequent stages progressively halving the spatial token resolution and doubling the feature channel dimension via convolutional downsampling. Within each stage's residual blocks, conventional self-attention and spatial convolutions are replaced by ClusterMixer. To bypass the quadratic computational complexity of global clustering while thoroughly modeling multiscale semantic structures, HCFormer decouples the clustering process into two parallel branches: a patch-level local branch that performs soft clustering in Euclidean space across non-overlapping localized windows to capture fine details, and a window-level global branch that computes average-pooled window tokens and conducts global clustering on a Lorentzian hyperbolic manifold to capture abstract tree-structured context. The two aggregated feature representations are concatenated channel-wise, projected through a linear layer, and added to the residual pathway to update the visual tokens.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image & Patch Partition<br/>Token sequence generation"] --> B["Dual-granularity Feature Splitting<br/>Local window partitioning & pooling"]
B --> C["Local Soft ClusterMixer<br/>Parameter-free center pooling & soft assignment"]
B --> D["Global Hyperbolic Window Clustering<br/>Projection onto Lorentz manifold"]
C --> E["Dual-geometry Metric Evaluation<br/>Patch Euclidean distance + Window Lorentzian distance"]
D --> E
E --> F["Dual-branch Aggregation & Fusion<br/>Concatenation and linear projection"]
F --> G["Multi-stage Hierarchical Representation<br/>Output for classification & dense prediction"]
Key Designs¶
1. Local Soft ClusterMixer: Parameter-Free Center Estimation and Soft Assignment for White-Box Token Mixing Iterative clustering algorithms like K-Means or Sinkhorn-Knopp are computationally expensive and impractical for forward-backward token mixing. To overcome this limitation, ClusterMixer decouples center initialization from token propagation. Given input visual features \(X \in \mathbb{R}^{N \times C}\), cluster centers \(C \in \mathbb{R}^{M \times C}\) are precomputed via parameter-free \(K \times K\) average pooling over spatial neighborhoods, scaling strictly linearly with token sequence length. Unlike CoC, which relies on hard clustering where each patch strictly belongs to a single cluster, ClusterMixer adopts soft assignment to significantly increase representation capacity. The assignment probability \(a_{i,j}\) of token \(x_i\) to center \(c_j\) is given by: $\(a_i = \text{Softmax}_{c_j \in C} (\alpha \cdot D(x_i, c_j) + b_{i,j})\)$ where \(D(\cdot, \cdot)\) denotes the geometric similarity metric, \(\alpha\) is a learnable scaling factor, and \(b_{i,j} \in B\) represents a learnable relative positional bias. Centers aggregate assigned patch representations into updated centroids \(c'_j = (c_j + \sum_{i=1}^N A_{i,j} x_i) / (1 + \sum_{i=1}^N A_{i,j})\), which are subsequently broadcast back to data points based on the assignment matrix \(A\) to yield context features \(g_i = \sum_{j=1}^M A_{i,j} c'_j\). This pipeline ensures transparent, fully interpretable feature routing.
2. Hierarchical Decoupled Clustering: Taming Quadratic Complexity into Linear Scaling for Global Receptive Fields Pairing all \(N\) tokens with \(M\) global cluster centers generates an intractable \(N \times M\) assignment matrix for high-resolution images. To resolve this efficiency bottleneck, HCFormer splits the image into \(S\) non-overlapping windows containing \(K\) patches each and decomposes clustering into two complementary levels. In patch-level clustering, patches serve as data points and cluster centers are computed locally within each window, allowing fully parallelized execution with complexity \(S \times \frac{N}{S} \times \frac{M}{S}\). In window-level clustering, each window is modeled as an individual data point by averaging its internal patch embeddings, and cluster centers are formed globally across all windows. Compared to the shifted window mechanism of Swin Transformer, which restricts cross-window interaction to adjacent window boundaries, this hierarchical formulation grants full-image global receptive fields at negligible overhead, transforming the quadratic complexity of global clustering into strict linear complexity \(O(N)\).
3. Dual-Geometry Collaborative Metric: Optimal Alignment of Abstract Tree Semantics and Flat Local Geometry Different abstraction levels of visual representations demand distinct metric geometries. Local image textures and contours adhere closely to flat, zero-curvature manifolds, where Euclidean distance is numerically stable and computationally economical. In contrast, high-level visual concepts and cross-object relationships naturally form tree-like taxonomies; embedding such hierarchical graphs in Euclidean space inevitably causes geometric distortion due to bounded volume growth. To reconcile this conflict, HCFormer pairs Euclidean geometry with hyperbolic geometry. Fine-grained patch-level clustering uses standard Euclidean cosine similarity \(D_\mathbb{E}(x_\mathbb{E}, c_\mathbb{E}) = \frac{x_\mathbb{E} \cdot c_\mathbb{E}}{\|x_\mathbb{E}\| \|c_\mathbb{E}\|}\), whereas abstract window-level clustering projects tokens onto the Lorentz hyperboloid \(\mathbb{L}^n = \{x \in \mathbb{R}^{n+1} : \langle x, x \rangle_\mathbb{L} = -1/\kappa, x_{\text{time}} = \sqrt{1/\kappa + \|x_{\text{space}}\|^2}\}\) via the origin exponential map \(\text{expm}_0^\kappa(v)\). Hyperbolic distance is then measured via the Lorentzian geodesic distance: $\(D_\mathbb{L}^\kappa(x_\mathbb{L}, c_\mathbb{L}) = \sqrt{\frac{1}{\kappa} \text{arccosh}(-\kappa \langle x_\mathbb{L}, c_\mathbb{L} \rangle_\mathbb{L})}\)$ Finally, window-level feature \(g_W\) is upsampled to match the spatial resolution of patch feature \(g_P\), concatenated, normalized, and mapped through a linear projection layer: \(x' = x + \text{FC}(\text{Norm}([g_W, g_P]))\). This collaborative design avoids the numerical instability and high computational overhead of performing hyperbolic operations across all dense patches while harvesting the tree-embedding superiority of hyperbolic geometry at the semantic window level.
Loss & Training¶
To safeguard numerical stability in hyperbolic space, feature vectors are clipped to a bounded norm before exponential mapping, constraining the argument magnitude of \(\sinh\) and \(\cosh\) functions to prevent floating-point overflow. Furthermore, the learnable logit scale parameter \(\alpha\) absorbs variations across different curvature regimes. Training on ImageNet-1K utilizes the Timm codebase across 4 A100 GPUs with a batch size of 256 for 310 epochs under an AdamW optimizer (weight decay 0.05, momentum 0.9). The initial learning rate is 1e-3 governed by a cosine decay schedule with 5 warmup epochs. Standard data augmentations including Mixup, CutMix, CutOut, and RandAugment are applied. For downstream semantic segmentation and object detection, pretrained backbones are fine-tuned with task-specific loss functions under standard AdamW schedules.
Key Experimental Results¶
Main Results¶
HCFormer was benchmarked against representative ConvNet, Transformer, MLP, and clustering-based backbones on ImageNet-1K classification and ADE20K semantic segmentation (with Semantic FPN, 80k iterations).
| Architecture Type | Backbone | Params (M) | GFLOPs | Top-1 Acc (%) | ADE20K mIoU (%) |
|---|---|---|---|---|---|
| ConvNet | ResNet-18 | 12.0 / 15.5 | 1.8 | 69.8 | 32.9 |
| ConvNet | ResNet-50 | 26.0 / 28.5 | 4.1 | 79.8 | 36.7 |
| Transformer | PVT-Tiny | 13.2 / 17.0 | 1.9 | 75.1 | 35.7 |
| Transformer | PVT-Small | 24.5 / 28.2 | 3.8 | 79.8 | 39.8 |
| Transformer | Swin-Tiny | 29.0 / 31.9 | 4.5 | 81.3 | 41.5 |
| MLP | MLP-Mixer-B/16 | 59.0 | 12.7 | 76.4 | - |
| Clustering | CoC-Tiny | 5.3 / - | 1.0 | 71.8 | - |
| Clustering | CoC-Small (/4) | 14.0 / 17.6 | 2.6 | 77.5 | 36.6 |
| Clustering | CoC-Medium (/4) | 27.9 / 25.2 | 5.5 | 81.0 | 40.2 |
| Clustering | FEC-Small | 5.5 / 9.1 | 1.4 | 72.7 | 35.3 |
| Clustering | FEC-Base | 14.4 / 18.0 | 3.4 | 78.1 | 37.7 |
| Clustering | FEC-Large | 28.3 / 31.9 | 6.5 | 81.2 | 40.5 |
| Ours | HCFormer-Nano | 5.1 / 8.8 | 0.9 | 73.0 ± 0.29 | 36.8 |
| Ours | HCFormer-Tiny | 7.0 / 10.3 | 1.0 | 75.1 ± 0.14 | 37.3 |
| Ours | HCFormer-Small | 16.1 / 18.9 | 2.9 | 79.8 ± 0.06 | 40.4 |
| Ours | HCFormer-Medium | 33.7 / 35.7 | 6.4 | 82.4 ± 0.03 | 43.3 |
Note: Dual parameter numbers reflect ImageNet-1K classification backbone and ADE20K Semantic FPN configuration, respectively.
Ablation Study¶
Systematic ablations on HCFormer-Tiny analyze the contribution of core components, geometry configurations, and hyper-parameters on ImageNet-1K.
| Ablation Setting | Configuration | Top-1 Acc (%) | ADE20K mIoU (%) | COCO APbox | COCO APmask | Note |
|---|---|---|---|---|---|---|
| Baseline | w/o Hier. Clus. / w/o Hyp. Geo. / w/o Rel. Pos. | 71.9 | 35.1 | 34.3 | 32.7 | Plain flat local clustering baseline |
| Component Analysis | + Hyp. Geo. + Rel. Pos. (w/o Hier. Clus.) | 72.7 | 34.6 | 34.4 | 32.8 | Lacks decoupled multiscale interaction |
| Component Analysis | + Hier. Clus. + Rel. Pos. (w/o Hyp. Geo.) | 73.4 | 35.2 | 34.7 | 33.1 | Pure Euclidean hierarchical clustering |
| Component Analysis | + Hier. Clus. + Hyp. Geo. (w/o Rel. Pos.) | 74.4 | 36.4 | 35.9 | 34.2 | Missing explicit spatial bias |
| Full Model | HCFormer-Tiny (all components) | 75.1 | 37.3 | 36.4 | 34.5 | Full dual-geometry hierarchical architecture |
| Geometry Choice | Window Euc. + Patch Euc. | 73.4 | - | - | - | Fully Euclidean metric baseline |
| Geometry Choice | Window Hyp. + Patch Hyp. | 74.7 | - | - | - | Fully hyperbolic (throughput plummets to 319.2) |
| Geometry Choice | Window Hyp. + Patch Euc. (Ours) | 75.1 | - | - | - | Optimal tree modeling + high throughput (536.0) |
Key Findings¶
- Dual-Geometry Specialization Is Optimal: Assigning hyperbolic geometry to global window clustering and Euclidean geometry to dense patch clustering achieves superior accuracy and efficiency. Pure Euclidean clustering drops accuracy to 73.4% (-1.7%) due to tree distortion. Conversely, performing all operations in hyperbolic space yields 74.7% (-0.4%) because local neighborhoods are flat, while slashing throughput from 536.0 to 319.2 images/s.
- Hierarchical Clustering Outperforms Shifted Windows: Decoupling patch- and window-level clustering outperforms the Swin-style shifted window strategy (73.4% vs. 72.7%), showing that cross-scale clustering provides a more natural global context mechanism for cluster-based token mixers.
- Substantial Gains on Dense Prediction Tasks: On ADE20K semantic segmentation, HCFormer-Medium attains 43.3% mIoU, outperforming comparable clustering backbones CoC-Medium/4 (40.2%) and FEC-Large (40.5%) by nearly 3 mIoU points, verifying the strength of soft clustering and hyperbolic hierarchy in dense visual understanding.
Highlights & Insights¶
- White-Box Paradigm Shift: By substituting self-attention with parameter-free pooling centers and soft clustering assignment, HCFormer dispenses with black-box matrix multiplication while maintaining parity with or outperforming mainstream ViTs and ConvNets.
- Topological Alignment of Multiscale Semantics: Exploits the insight that visual representations exhibit flat geometry locally but tree-like hierarchical taxonomies globally, successfully deploying hyperbolic space at the window level to eliminate Euclidean metric distortion.
- Seamless Dense Prediction Backbone: With its standard 4-stage pyramid layout and strict linear complexity, HCFormer serves as a drop-in replacement backbone for Semantic FPN and Mask R-CNN.
Limitations & Future Work¶
- Author-Admitted Limitations: Hyperbolic operations rely on transcendental functions such as \(\text{arccosh}\), \(\cosh\), and \(\sinh\), which lack optimized low-level kernel acceleration compared to standard matrix multiplication on GPU Tensor Cores, leading to reduced throughput in fully hyperbolic regimes.
- Identified Limitations: The number of cluster centers and window dimensions across stages remain fixed hyper-parameters, lacking dynamic adaptation based on input image complexity.
- Future Directions: Exploring hardware-friendly hyperbolic approximation kernels and scaling up self-supervised pre-training to validate hyperbolic hierarchical clustering on massive uncurated visual datasets.
Related Work & Insights¶
- vs Context Cluster (CoC): CoC employs hard clustering where each token is strictly bound to a single center and lacks holistic cross-window context. HCFormer introduces soft continuous assignment and dual-geometry hierarchical clustering, substantially lifting representation capacity and downstream segmentation mIoU.
- vs FEC (Fully Clustering): FEC replaces both pooling downsampling and feature mixing with clustering algorithms, but operates entirely within Euclidean space. HCFormer introduces hyperbolic geometry to model semantic tree hierarchies, achieving a vastly superior accuracy-throughput trade-off.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering integration of hyperbolic geometry and hierarchical clustering into an interpretable general-purpose vision backbone.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation across ImageNet classification, ADE20K semantic segmentation, and COCO instance detection/segmentation with detailed ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulation, solid geometric motivation, and coherent experimental analysis.
- Value: ⭐⭐⭐⭐⭐ Sets a strong precedent for interpretable, non-black-box vision backbones challenging the dominance of attention and convolution.