TopoFuse: Topology-Aware Tri-Planar Fusion for 3D Cryo-Electron Tomography Segmentation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/rohitsalla/TopoFuse
Area: 3D Vision
Keywords: Cryo-ET, Persistent Homology, Differentiable Topology Projection, Tri-Planar Encoding, Topology Prior

TL;DR

Addressing fragmented membranes and collapsed organelles caused by ultra-low SNR and missing-wedge artifacts in 3D cryo-electron tomography, TopoFuse reframes topology enforcement from soft-loss penalties into a forward-pass differentiable projection operator coupled with tri-planar SAM encoders and FiLM fusion, achieving a 54% reduction in Betti Number Error while editing only 3.1% of voxels.

Background & Motivation

Cryo-electron tomography (cryo-ET) visualizes cellular ultrastructure at nanometer resolution, offering unprecedented views of macromolecules, membranes, and organelles in their native context. However, automated segmentation of cryo-ET volumes remains notoriously difficult due to extreme physical degradation: signal-to-noise ratios regularly fall below 0.1, and specimen tilt limitations induce severe missing-wedge artifacts that degrade axial resolution by a factor of 2 to 3. Standard volumetric deep learning models, such as 3D U-Net and nnU-Net, often achieve respectable voxel-wise overlap scores while producing structurally nonsensical masks—continuous mitochondrial membranes fragment into floating pieces, adjacent organelles merge together, and closed vesicle cavities collapse into solid blocks. These structural errors invalidate downstream morphometric analyses, such as surface area quantification and membrane contact-site identification.

To mitigate topological errors, existing topology-preserving frameworks incorporate soft regularizers into training objectives, leveraging persistent homology (PH), homotopy warping, or centerline Dice. Nevertheless, these methods inherently operate via gradient pressure during back-propagation. When gradients become attenuated across complex 3D volumes or when persistence pairings are suboptimal, topological violations survive the training process and inevitably persist at inference time. Soft losses can encourage topological fidelity, but they cannot structurally enforce it.

Crucially, persistent homology grounds every topological feature (such as 0-dimensional connected components or 2-dimensional enclosed voids) in exact spatial coordinates known as critical voxels, where features are born or die along the sublevel-set filtration. Because topological errors are localized to discrete, identifiable locations, one does not need to rely on diffuse gradient penalties across the entire volume; instead, one can directly manipulate the logits at these critical voxels. The core idea is to reframe topological regularization from a backward loss penalty into a forward-pass differentiable projection operator that performs PH-guided sparse edits on critical voxels, backed by tri-planar foundation features, topology-conditioned FiLM fusion, and a self-contained prior head that provides explicit repair certificates without requiring test-time ground-truth labels.

Method

Overall Architecture

TopoFuse is an end-to-end segmentation framework designed to eliminate topological violations directly during the forward pass. The workflow comprises four interconnected stages: first, an anisotropic tri-planar foundation encoder processes orthogonal slice stacks with a shared 2D SAM ViT-B model pretrained on natural images; second, a topology-conditioned FiLM fusion module dynamically weights the three planar representations based on a global topological health descriptor; third, a differentiable topology projection operator computes persistent homology on a downsampled grid to locate critical voxels of unmatched features via bottleneck matching and applies sparse logit updates while logging a repair certificate; and fourth, a topology prior head predicts expected Betti numbers and persistence lifetimes directly from fused features, enabling fully self-contained inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Cryo-ET Volume V"] --> B["Tri-Planar Foundation Encoder<br/>Orthogonal Slicing & Pretrained SAM ViT-B"]
    B --> C["Topology-Conditioned FiLM Fusion<br/>Global Topological Descriptor-Modulated 3D Weights"]
    C --> D["Topology Prior Head<br/>Predicts Betti Numbers & Lifetime Budgets"]
    C --> E["Differentiable Topology Projection<br/>Bottleneck-Matched Critical Voxel Sparse Edits"]
    D -->|Inferred Topology Target| E
    E --> F["Topology-Enforced Segmentation + Repair Certificate C"]

Key Designs

1. Tri-Planar Foundation Encoder: Bypassing Missing-Wedge Anisotropy via Pretrained 2D Representations Due to missing-wedge artifacts, axial resolution in cryo-ET is degraded by 2–3× compared to lateral dimensions. Training a 3D convolutional network from scratch tends to propagate directional blurring isotropically throughout the volume, exacerbated by the acute scarcity of annotated 3D cellular tomograms (typically only 40–70 annotated volumes per dataset). TopoFuse sidesteps this challenge through tri-planar orthogonal decomposition along the \(xy\), \(xz\), and \(yz\) canonical planes. Each orthogonal slice stack is processed by a shared 2D SAM ViT-B encoder. Pretrained on 11 million natural images from SA-1B, SAM delivers rich low-level contour and edge representations that far exceed the discriminative power of small scratch-trained 3D encoders. While an axial slice may suffer from severe missing-wedge blurring, orthogonal lateral slices preserve crisp boundary profiles. The slice representations are decoded through lightweight transpose convolutions and lifted back into 3D volumes \(\mathbf{Z}^{(p)}\) via inverse slicing.

2. Topology-Conditioned FiLM Fusion: Adaptive View Selection Driven by Global Connectedness Uniformly averaging three planar volumes fails to account for spatial variations in missing-wedge degradation. TopoFuse introduces an adaptive fusion mechanism conditioned on topological integrity. A preliminary probability volume \(\bar{P}\) is generated from averaged logits to extract a compact topological descriptor \(\mathbf{t} = \psi(\bar{P}) \in \mathbb{R}^{6C}\), which quantifies the count of \(d \in \{0, 2\}\) topological features across persistence thresholds as well as total persistence mass. A lightweight 3D CNN \(G(\cdot)\) computes voxel-wise fusion logits, which are then modulated via Feature-wise Linear Modulation (FiLM) using scale \(\gamma(\mathbf{t})\) and shift \(\beta(\mathbf{t})\) vectors:

\[\alpha^{(p)}(\mathbf{x}) = \mathrm{softmax}_{p}\Big(\gamma(\mathbf{t}) \odot G(\mathbf{x}; \{\mathbf{Z}^{(q)}\}) + \beta(\mathbf{t})\Big), \quad \mathbf{Z}(\mathbf{x}) = \sum_{p} \alpha^{(p)}(\mathbf{x})\mathbf{Z}^{(p)}(\mathbf{x})\]

When the descriptor indicates severe membrane fragmentation (manifested as an inflated 0-dimensional Betti count \(\hat{\beta}_0\)), the FiLM network adaptively shifts weighting toward the orthogonal planes that exhibit superior structural continuity, resolving potential discontinuities before explicit projection is initiated.

3. Differentiable Topology Projection: Sparse Critical-Voxel Edits via Bottleneck Matching Unlike conventional topology losses that rely on fragile gradient signals during back-propagation, TopoFuse formalizes topological compliance as an explicit forward-pass operator \(\mathrm{Proj}_T\). Formulated on an \(s=2\) downsampled grid (reducing \(128^3\) crops to \(64^3\) such that cubical persistent homology computes in roughly 8 ms), the operator calculates the bottleneck distance \(d_B\) between the predicted diagram and a target specification \(\Pi^\star_c\) for dimensions \(d \in \{0, 2\}\). By solving the optimal bottleneck bipartite matching, the operator pinpoints the exact critical voxel pairs \((v_{b_i}, v_{d_i})\) corresponding to unmatched spurious components or missing voids. Sparse logit perturbations are applied iteratively exclusively at these coordinates:

\[\mathbf{Z}_c(\mathbf{x}) \leftarrow \mathbf{Z}_c(\mathbf{x}) - \eta \cdot \nabla_{\mathbf{Z}_c} d_B\]

The step size \(\eta\) is dynamically verified via backtracking line search to ensure monotonic non-increase of \(d_B\). Crucially, this operation changes only a minimal fraction of the volume (median spatial sparsity of 3.1%) over at most \(T_{\max}=5\) steps. Every projection generates an algorithmic repair certificate \(\mathcal{C} = (I, V_{\mathcal{C}}, \Delta_{\mathcal{C}})\) detailing the iteration count, modified coordinates, and edit sparsity, providing transparent traceability for downstream biological validation.

4. Topology Prior Head: Self-Contained Inference via Direct Pseudo-Diagram Synthesis At inference time, ground-truth persistence diagrams are absent. TopoFuse bridges this gap using a three-layer MLP prior head \(H_\pi\) that takes global average pooled fused features \(\mathrm{GAP}(\mathbf{F}_{\mathrm{fused}})\) and predicts estimated Betti counts \((\hat{\beta}_0, \hat{\beta}_2)\) alongside persistence lifetime budgets across six bin thresholds. From these predictions, a typed pseudo-diagram \(\hat{\Pi}^\star_c\) is constructed: predicted Betti counts determine the number of target feature pairs, their birth values are anchored to unmatched critical values in the current filtration, and their death coordinates are set using the predicted persistence budget bounds. This enables autonomous structural enforcement without oracle reliance at test time.

Loss & Training

The overall network is trained end-to-end under a multi-task objective:

\[\mathcal{L} = \mathcal{L}_{\mathrm{Dice+CE}}(\hat{P}', Y) + \lambda_t \mathcal{L}_{\mathrm{topo}} + \lambda_\pi \mathcal{L}_{\mathrm{prior}} + \lambda_a \mathcal{L}_{\mathrm{Dice+CE}}(\hat{P}, Y)\]

with weights set to \(\lambda_t = 0.1\), \(\lambda_\pi = 0.05\), and \(\lambda_a = 0.5\). The primary term evaluates post-projection probability maps \(\hat{P}'\) against ground truth. The auxiliary topology loss \(\mathcal{L}_{\mathrm{topo}}\) uses a Wasserstein distance over pre-projection maps (linearly warmed up over 5,000 steps) to prime the encoder toward topologically plausible outputs. The prior loss \(\mathcal{L}_{\mathrm{prior}}\) performs \(\ell_1\) regression on ground-truth topological summary statistics, and the auxiliary term \(\lambda_a \mathcal{L}_{\mathrm{Dice+CE}}(\hat{P}, Y)\) ensures stable, direct gradient flow into the SAM encoder and fusion backbone.

Key Experimental Results

Main Results

TopoFuse was evaluated across three public cryo-ET datasets: EMD-0506 (ribosome, high SNR), EMPIAR-10499 (HeLa lamellae, moderate anisotropy), and EMPIAR-10045 (bacteria, severe anisotropy). All topology-loss baselines share the identical tri-planar backbone to isolate the mechanism.

Key comparisons from Table 2 in the paper are summarized below (\(D\): Dice, \(\mathrm{BE}_0\): 0-dimensional Betti Number Error, \(\mathrm{BME}\): Betti Matching Error):

Method	EMD-0506 (\(D \uparrow\))	EMD-0506 (\(\mathrm{BE}_0 \downarrow\))	EMPIAR-10499 (\(D \uparrow\))	EMPIAR-10499 (\(\mathrm{BE}_0 \downarrow\))	EMPIAR-10045 (\(D \uparrow\))	EMPIAR-10045 (\(\mathrm{BE}_0 \downarrow\))	Mean (\(D \uparrow\))	Mean (\(\mathrm{BE}_0 \downarrow\))
3D U-Net	70.2	3.84	69.8	4.12	67.3	5.21	69.1	4.39
nnU-Net	72.9	3.41	72.6	3.68	70.1	4.73	71.9	3.94
CryoSAM	75.6	3.07	75.2	3.24	72.8	4.31	74.5	3.54
Tri-planar (no topology)	76.4	2.94	76.0	3.10	73.5	4.18	75.3	3.41
+ clDice	78.1	2.61	78.4	2.73	76.2	3.72	77.6	3.02
+ HuTopo	79.2	2.48	79.6	2.59	77.3	3.53	78.7	2.87
+ BettiMatching	80.8	2.29	81.1	2.41	78.9	3.34	80.3	2.68
+ TopoPost (heuristic post-hoc)	79.0	2.17	79.3	2.28	77.1	3.19	78.5	2.55
TopoFuse (Learned Prior)	85.1	1.04	85.6	1.09	83.9	1.58	84.9	1.24
TopoFuse (Oracle Prior)	85.9	0.71	86.4	0.74	84.6	1.12	85.6	0.86

Note: Paired Wilcoxon signed-rank tests confirm \(p < 0.001\) over BettiMatching across all three benchmarks.

Ablation Study

Ablation analysis on EMPIAR-10499 (Table 3 in the paper) highlights the contribution of each architectural module and projection step budget:

Configuration	Dice (%) \(\uparrow\)	\(\mathrm{BE}_0 \downarrow\)	BME \(\downarrow\)	Edit Sparsity \(\Delta_{\mathcal{C}}\) (%)	Note
Tri-planar (uniform, no topology)	76.0	3.10	0.31	–	Plain tri-planar baseline
+ learned fusion (no FiLM)	77.8	2.88	0.29	–	Adaptive fusion baseline
+ \(\mathcal{L}_{\mathrm{topo}}\) (no projection)	81.1	2.41	0.24	–	Limit of soft regularizer
+ projection (\(T=5\))	85.2	1.11	0.11	3.1 ± 0.7	54% drop in \(\mathrm{BE}_0\); single largest gain
+ FiLM conditioning	85.4	1.10	0.11	3.1 ± 0.7	Improves boundary selection
TopoFuse (Learned Prior full model)	85.6	1.09	0.11	3.1 ± 0.7	Fully autonomous test-time system
TopoFuse (Oracle Prior)	86.4	0.74	0.07	3.0 ± 0.6	Upper bound using GT diagrams
Iteration budget: \(T=1\)	83.6	1.62	0.16	Convergence: 71%	Single edit step
Iteration budget: \(T=3\)	84.9	1.21	0.12	Convergence: 89%	Practical efficiency sweet spot
Iteration budget: \(T=5\)	85.2	1.11	0.11	Convergence: 96%	Primary operating setting
Iteration budget: \(T=10\)	85.3	1.09	0.11	Convergence: 98%	Diminishing marginal returns

Key Findings

• Projection Drives the Primary Breakthrough: Swapping out soft topology regularizers for the projection operator slashed \(\mathrm{BE}_0\) from 2.41 to 1.11 on EMPIAR-10499, representing a 54% reduction and verifying the hypothesis that direct structural enforcement vastly outperforms gradient encouragement.
• Voxel Accuracy Synergizes with Topology Enforcement: In cryo-ET, biological membranes are ultra-thin (2–4 nm). Sealing spurious micro-fractures inherently restores true voxel overlap, boosting mean Dice by 4.6 percentage points over the strongest baseline (80.3% \(\rightarrow\) 84.9%).
• Resilience Across Missing-Wedge Anisotropy: On EMPIAR-10045, where axial degradation is most extreme, TopoFuse achieves a 10.4 percentage point Dice gain over the tri-planar baseline (73.5% \(\rightarrow\) 83.9%), demonstrating that explicit geometric repair thrives precisely where directional artifacts are most disruptive.
• Sparsity and Convergence Saturation: TopoFuse edits only a median of 3.1% of spatial voxels to achieve topological feasibility, and convergence reaches 96% at \(T=5\). Setting \(T=3\) yields 89% convergence at 84 ms per crop, offering an attractive trade-off for latency-sensitive pipelines.

Highlights & Insights

• From Regularization to Direct Projection: TopoFuse fundamentally shifts topology-aware segmentation from indirect loss penalties to a forward-pass differentiable projection operator, systematically resolving topological defects at their geometric source.
• Verifiable Repair Certificates: By returning a certificate \(\mathcal{C}\) that tracks iteration counts, touched voxels, and spatial edit sparsity \(\Delta_{\mathcal{C}}\), TopoFuse provides a built-in audit trail. An edit sparsity exceeding 8% reliably flags instances of over-correction (such as artificial bridging of organelles) in 83% of failure cases.
• Broad Transferability Across Biomedical Volumetric Imaging: While validated on cryo-ET stress tests, the formulation of critical-voxel editing through persistent homology bottleneck matching is fundamentally general and directly applicable to vessel tracking, dendritic spine analysis, and cardiac lumen segmentation.

Limitations & Future Work

• Omission of 1-Dimensional Homology (\(d=1\)): Enforcing tunnel/loop constraints in 3D cubical complexes incurs an \(\mathcal{O}(n^{1.5})\) complexity cost. Consequently, TopoFuse omits \(d=1\) homology, leaving cristae perforations and sheet punctures (present in roughly 8.2% of volumes) uncorrected.
• Residual Gap from Learned Prior Estimation: A 0.35 \(\mathrm{BE}_0\) gap remains between the learned prior head and oracle GT diagrams. In approximately 4% of severely anisotropic volumes, an erroneous \(\hat{\beta}_0\) prediction can trigger over-correction by merging distinct neighboring organelles.
• Crop-Independent Processing: The projection layer operates locally on \(128^3\) crops. Although 95% of target structures fit within an individual crop, cross-tile boundary consistency across entire gigavoxel tomograms requires dedicated stitching strategies.

Related Work & Insights

• vs. 3D U-Net / nnU-Net: These conventional volumetric segmentors rely solely on local voxel losses; lacking structural awareness, they suffer severe membrane fragmentation under cryo-ET missing-wedge conditions. TopoFuse establishes substantial margins in both Dice and topological fidelity.
• vs. BettiMatching / HuTopo: Soft persistent-homology losses penalize errors via back-propagated gradients but fail to guarantee structural feasibility. TopoFuse demonstrates that forward critical-voxel editing cuts topological errors by an additional 54%.
• vs. TopoPost: Post-hoc non-differentiable topological heuristics cannot communicate with network features and degrade boundary fidelity (yielding only 78.5% Dice). TopoFuse provides differentiable, end-to-end forward corrections that actively harmonize with deep feature representations.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (Introduces the first forward-pass differentiable topology projection operator with repair certification for 3D segmentation)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Thorough validation across three public benchmarks and a controlled synthetic suite, including statistical significance tests, ablations, and overhead profiling)
• Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous mathematical formulation, clear problem progression, and transparent discussion of bounds and failure cases)
• Value: ⭐⭐⭐⭐⭐ (Delivers an auditable, state-of-the-art solution for cellular ultrastructure segmentation in cryo-ET with broad potential in biomedical 3D imaging)

Related Papers

• [ECCV 2026] NeuroRefiner: Morphology-Aware Multi-Agent Refinement for 3D Fluorescence Microscopy Neuron Segmentation
• [CVPR 2026] Spatial-SAM: Spatially Consistent 3D Electron Microscopy Segmentation with SDF Memory and Semi-Supervised Learning
• [ECCV 2026] Tri-Efficient Transfer Learning for Point Cloud Videos
• [ECCV 2026] G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Segmentation
• [CVPR 2026] TagSplat: Topology-Aware Gaussian Splatting for Dynamic Mesh Modeling and Tracking