Foundation-Guided Representation Alignment for Multimodal Medical Image Registration¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/MeggieGuo/FGRA-Reg
Area: Medical Imaging
Keywords: Multimodal Registration, Vision Foundation Models, Representation Alignment, Data ReAssembly, Hierarchical Feature Fusion
TL;DR¶
This paper introduces a foundation-guided representation alignment framework for multimodal medical image registration that transfers cross-modal priors from a 2D vision foundation model (DINOv3) to a lightweight 3D registration network via data reassembly and hierarchical feature fusion, eliminating foundation model dependencies during inference while substantially boosting MR-CT registration accuracy.
Background & Motivation¶
Multimodal medical image registration is essential for clinical diagnosis, radiotherapy planning, and image-guided interventions. In clinical practice, magnetic resonance (MR) and computed tomography (CT) offer highly complementary anatomical perspectives: CT highlights high-density osseous structures with superior spatial resolution, whereas MR provides exquisite soft-tissue contrast. Accurately aligning these complementary modalities facilitates comprehensive anatomical assessment. However, due to fundamentally disparate imaging physics, inter-modality appearance discrepancies are substantial—for instance, cortical bone appears intensely bright in CT but produces void-like dark signals in MR. Traditional registration relies on intensity similarity metrics such as Mutual Information (MI), which often suffer from non-convex optimization, local extrema, and weak gradient feedback. Image-to-image translation methods (e.g., CycleGAN or Invertible Neural Networks) frequently introduce hallucinated anatomical artifacts and fail to preserve 3D inter-slice volumetric coherence.
Deep learning-based deformable registration architectures (such as VoxelMorph, TransMorph, and ModeT) employ an encoder-decoder paradigm where the encoder is tasked with learning modality-invariant, anatomically discriminative representations. In multimodal regimes, however, encoders trained on limited paired cohorts struggle to discover true cross-modal anatomical correspondences. Meanwhile, emerging vision foundation models (VFMs, such as DINOv2 and DINOv3) exhibit exceptional cross-domain generalization and deep semantic alignment capabilities learned from web-scale self-supervised pretraining. Nonetheless, directly applying VFMs to 3D medical registration encounters three primary obstacles: prohibitive GPU memory and compute demands that preclude clinical deployment, a pronounced domain mismatch between 2D RGB natural images and 3D single-channel grayscale medical volumes, and an objective misalignment between global semantic extraction and fine-grained, localized coordinate deformation estimation.
Existing works integrating VFMs into medical image processing typically embed adapter modules or execute entire models during test time, incurring massive inference latency and memory footprints. The core idea of this paper is to utilize a pretrained 2D vision foundation model as a cross-modal representation guide during training via lightweight data reassembly and hierarchical feature fusion, while completely discarding the foundation model during inference to achieve highly accurate, generalizable MR-CT registration at zero extra test-time computational cost.
Method¶
Overall Architecture¶
The proposed framework comprises two cooperative pathways: a lightweight 3D dual-stream registration backbone (consisting of a shared encoder and a deformation flow decoder) and a frozen foundation feature guidance branch instantiated with DINOv3 (ViT-B/16). The inputs are 3D moving volume \(I_m\) (MR) and fixed volume \(I_f\) (CT). During training, the 3D registration backbone extracts multi-scale volumetric features, while the foundation guidance branch transforms 3D slices into 2.5D inputs via Data ReAssembly (DR) and dynamically aggregates multi-level representations via Hierarchical Representation Fusion (HRF). A feature-level cosine alignment loss then constrains the registration encoder to match the foundation representations. During inference, the foundation guidance branch is completely discarded, leaving only the compact 3D registration network to predict dense deformation fields with minimal latency.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: 3D Image Pair<br/>(Moving Im & Fixed If)"] --> B["Data ReAssembly<br/>Gather adjacent slices with 1x1 conv mapping"]
B --> C["Hierarchical Representation Fusion<br/>Multi-level VFM feature resampling & dynamic weighting"]
C --> D["Representation Alignment<br/>Feature distillation supervision & zero-overhead test decoupling"]
A --> E["Lightweight 3D Registration Encoder<br/>Dual-stream shared architecture extracting volumetric features"]
D -.->|Training-time feature alignment loss Lrep| E
E --> F["Deformation Flow Decoder<br/>Predict dense non-linear displacement field phi"]
Key Designs¶
1. Data ReAssembly: Eliminating slice redundancy while retaining local 3D spatial context
When deploying 2D three-channel foundation models to 3D single-channel medical volumes, conventional approaches extract individual 2D slices and naively replicate them across three channels (\(1 \times H \times W \to 3 \times H \times W\)). This repetition creates severe data redundancy, discards critical out-of-plane 3D spatial continuity along the z-axis, and produces inputs with unnatural cross-channel correlations that deviate from natural image statistics.
To resolve this issue, the Data ReAssembly (DR) strategy constructs an informative, non-redundant representation. For a given 2D slice \(S_n \in \mathbb{R}^{H \times W}\) at depth \(d = n\) from a 3D volume \(V \in \mathbb{R}^{H \times W \times D}\), it gathers the adjacent slices \(S_{n-1}\) and \(S_{n+1}\) to form a 2.5D tensor: $\(V_{2.5D} = [S_{n-1}, S_n, S_{n+1}] \in \mathbb{R}^{3 \times H \times W}\)$ Because feeding spatial neighbor slices directly into RGB channels causes a semantic distribution mismatch with VFM pretraining, DR introduces a lightweight context fusion module parameterized by a \(1 \times 1\) convolution. This convolution projects \(V_{2.5D}\) into two synthesized context channels \(S_{n'}\) and \(S_{n''}\): $\([S_{n'}, S_{n''}] = \text{Conv}_{1 \times 1}(V_{2.5D})\)$ These two synthesized channels are concatenated with the original slice \(S_n\) to yield the final three-channel input \(\tilde{I} = [S_{n'}, S_n, S_{n''}]\). This reassembly preserves local through-plane structural context, conforms to the input structure expected by 2D VFMs, and is restricted strictly to the foundation guidance branch, allowing the registration backbone to operate directly on the full 3D volume for global volumetric consistency.
2. Hierarchical Representation Fusion: Dynamically integrating multi-scale features for semantic and boundary awareness
Vision foundation models exhibit distinct hierarchical representation properties across their transformer blocks: earlier layers preserve high-resolution spatial details, local textures, and fine anatomical contours, whereas deeper layers synthesize high-level, invariant semantic abstractions. Deformable registration requires both macro-level organ semantic matching and micro-level boundary localization. Relying exclusively on the final layer representation discards low-level structural cues indispensable for fine deformation.
The Hierarchical Representation Fusion (HRF) module extracts multi-level feature maps \(\{F_1, F_2, \dots, F_L\}\) from layers 3, 6, 9, and 12 of the 12-layer DINOv3 ViT-B/16 encoder (\(L=4\)). Each feature \(F_l\) is first spatially resampled to match the 3D dimensions \(H \times W \times D\) of the registration encoder features: $\(\tilde{F}_l = \text{Resample}(F_l; H, W, D)\)$ To adaptively prioritize task-relevant features rather than relying on uniform averaging, HRF parameterizes a learnable layer weight vector \(w = (w_1, \dots, w_L) \in \mathbb{R}^L\) and applies Softmax normalization to dynamically aggregate representations: $\(\tilde{F}_{\text{fused}} = \sum_{l=1}^L \frac{\exp(w_l)}{\sum_{j=1}^L \exp(w_j)} \tilde{F}_l\)$ A subsequent 3D convolutional layer projects \(\tilde{F}_{\text{fused}}\) to match the channel dimensions of the registration encoder outputs, yielding target representations \(\alpha_T\) and \(\beta_T\). This dynamic multi-level integration provides balanced guidance spanning coarse semantics to fine anatomical boundaries while maintaining modular flexibility across diverse registration backbones.
3. Foundation-Guided Representation Alignment: Training-phase feature distillation with zero inference footprint
In standard unsupervised registration, encoders are optimized solely by backward gradients propagating from similarity or regularization losses. Under extreme cross-modal shifts between MR and CT, this optimization often converges to anatomically distorted local minima. The foundation-guided representation alignment mechanism introduces direct feature-level supervision during training, aligning the registration encoder features \((\alpha_S, \beta_S)\) with the rich representations \((\alpha_T, \beta_T)\) distilled from the foundation encoder.
The representation alignment loss optimizes cosine similarity across dense feature coordinates: $\(\mathcal{L}_{\text{rep}} = d(\alpha_S, \alpha_T) + d(\beta_S, \beta_T), \quad d(x_1, x_2) = 1 - \frac{\sum_i x_1^i x_2^i}{\sqrt{\sum_i (x_1^i)^2} \sqrt{\sum_i (x_2^i)^2}}\)$ By penalizing angular deviations in latent feature space, the registration encoder reshapes its embeddings to become modality-invariant and anatomically coherent. Crucially, this distillation constraint occurs entirely during training backpropagation; during testing, the VFM branch, DR layers, and HRF modules are completely eliminated, leaving a lightweight 3D registration backbone operating with zero extra parameters or computational overhead.
Loss & Training¶
The overall training loss optimizes three complementary objectives: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{rep}} + \mathcal{L}_{\text{sim}} + \mathcal{L}_{\text{smooth}}\)$ where \(\mathcal{L}_{\text{rep}}\) is the representation alignment loss defined above; \(\mathcal{L}_{\text{sim}}\) represents the cross-modal image similarity, combining Mutual Information loss \(\mathcal{L}_{\text{MI}}(I_f, I_m \circ \phi)\) with an optional auxiliary Dice loss \(\mathcal{L}_{\text{Dice}}(S_f, S_m \circ \phi)\) when training segmentations are available (segmentations are never required during test inference); and \(\mathcal{L}_{\text{smooth}}(\phi) = \sum_{p \in \Omega} \|\nabla u(p)\|^2\) is the diffusion regularization loss penalizing spatial displacement gradients to ensure topology preservation.
Models are trained using the Adam optimizer with an initial learning rate of \(2 \times 10^{-4}\) and a batch size of 1. Training runs for 400 epochs on the Learn2Reg dataset and 100 epochs on the SR-Reg dataset on a single 80GB NVIDIA H100 GPU.
Key Experimental Results¶
Main Results¶
The framework is evaluated on two standard MR-CT benchmarks: the Learn2Reg abdominal dataset (unpaired training with 8 paired test subjects, segmenting liver, spleen, and kidneys) and the SR-Reg brain dataset (180 subjects split 150/10/20 with synthetic deformations). Metrics include Dice Similarity Coefficient (DSC, % ↑), 95th percentile Hausdorff Distance (HD95, mm ↓), and the fraction of negative Jacobian determinants \(\%(|J_\phi| \le 0)\) (topology folding rate, % ↓).
On the challenging Learn2Reg abdominal MR-CT dataset, the framework is integrated into ModeT (OursM) and PiViT (OursP), outperforming traditional and recent deep learning registration methods as shown in Table 1:
| Method | DSC (%) ↑ | HD95 (mm) ↓ | %(|Jϕ| ≤ 0) ↓ | Params (M) |
|---|---|---|---|---|
| Initial | 57.24 ± 19.42 | 11.88 ± 7.44 | - | - |
| NiftyReg (Iterative) | 72.61 ± 22.89 | 8.54 ± 8.34 | 0.00 ± 0.00 | - |
| LapIRN | 70.91 ± 19.30 | 10.48 ± 6.05 | 0.10 ± 0.01 | 1.20 |
| VoxelMorph | 67.81 ± 18.50 | 10.23 ± 6.24 | 0.02 ± 0.01 | 0.30 |
| TransMorph | 70.97 ± 20.68 | 10.64 ± 7.96 | 0.07 ± 0.00 | 46.76 |
| TransMatch | 70.45 ± 20.74 | 11.12 ± 8.96 | 0.02 ± 0.01 | 112.38 |
| UTSRMorph | 75.22 ± 18.70 | 9.72 ± 7.77 | 0.03 ± 0.01 | 98.82 |
| PiViT | 67.15 ± 24.28 | 11.55 ± 8.26 | 0.01 ± 0.00 | 0.66 |
| OursP (PiViT+Ours) | 71.26 ± 18.82 | 10.47 ± 7.61 | 0.01 ± 0.00 | 0.66 |
| ModeT | 73.84 ± 23.64 | 7.80 ± 6.22 | 0.02 ± 0.02 | 1.03 |
| OursM (ModeT+Ours) | 76.16 ± 20.93 | 7.12 ± 5.74 | 0.03 ± 0.02 | 1.03 |
On the SR-Reg brain MR-CT dataset, evaluation includes image reconstruction quality metrics (MSE, SSIM, PSNR) against synthetic ground truth, detailed in Table 2:
| Methods | DSC (%) ↑ | HD95 (mm) ↓ | %(|Jϕ| ≤ 0) ↓ | MSE (%) ↓ | SSIM (%) ↑ | PSNR (dB) ↑ | Params (M) |
|---|---|---|---|---|---|---|---|
| Initial | 55.14 ± 4.81 | 4.33 ± 0.55 | - | 2.65 ± 0.25 | 66.58 ± 2.07 | 33.91 ± 0.28 | - |
| VoxelMorph | 69.79 ± 4.89 | 2.92 ± 0.56 | 0.01 ± 0.00 | 0.97 ± 0.15 | 76.65 ± 2.06 | 34.07 ± 0.33 | 0.30 |
| TransMorph | 71.16 ± 5.24 | 2.96 ± 0.67 | 0.49 ± 0.13 | 0.99 ± 0.19 | 77.15 ± 2.04 | 34.03 ± 0.34 | 46.76 |
| UTSRMorph | 75.20 ± 4.57 | 2.32 ± 0.58 | 0.05 ± 0.05 | 0.85 ± 0.13 | 78.53 ± 2.31 | 34.16 ± 0.38 | 98.82 |
| ModeT | 72.30 ± 3.63 | 2.58 ± 0.41 | 0.01 ± 0.01 | 0.88 ± 0.21 | 78.05 ± 2.00 | 34.15 ± 0.33 | 1.03 |
| OursM | 79.04 ± 2.37 | 1.97 ± 0.30 | 0.06 ± 0.02 | 0.83 ± 0.20 | 80.00 ± 2.27 | 34.19 ± 0.41 | 1.03 |
Ablation Study¶
A component-wise ablation study conducted on the SR-Reg dataset based on the ModeT backbone highlights the individual contributions of Representation Alignment (RA), Data ReAssembly (DR), and Hierarchical Representation Fusion (HRF), as shown in Table 3:
| Config | DSC (%) ↑ | HD95 (mm) ↓ | %(|Jϕ| ≤ 0) ↓ | Note |
|---|---|---|---|---|
| Base (ModeT) | 72.30 ± 3.63 | 2.58 ± 0.41 | 0.01 ± 0.01 | Baseline ModeT without foundation guidance |
| + RA | 76.83 ± 2.73 | 2.23 ± 0.37 | 0.04 ± 0.02 | Naive 3-channel slice repeat + final VFM layer alignment |
| + RA + DR | 77.26 ± 2.89 | 2.16 ± 0.37 | 0.07 ± 0.03 | Replacing slice repeat with adjacent slice 1x1 conv reassembly |
| + RA + HRF | 78.08 ± 2.47 | 2.09 ± 0.31 | 0.08 ± 0.03 | Dynamic attention fusion over layers 3, 6, 9, 12 |
| Ours (Full: RA + DR + HRF) | 79.04 ± 2.37 | 1.97 ± 0.30 | 0.06 ± 0.02 | Full integrated framework |
Key Findings¶
- Substantial Gains from Foundation Alignment: Adding representation alignment alone (+RA) to ModeT boosts brain DSC from 72.30% to 76.83% (+4.53%), showing that web-scale self-supervised features effectively alleviate cross-modal representation bottlenecks.
- Strong Synergy Between DR and HRF: Incorporating Data ReAssembly improves DSC to 77.26% by eliminating intra-slice redundancy, while Hierarchical Representation Fusion improves DSC to 78.08% by providing fine structural cues. Combining both yields 79.04% DSC and drops HD95 below 2 mm (1.97 mm).
- Superior Parameter Efficiency: OursM (1.03M parameters) significantly outperforms massive transformer baselines like UTSRMorph (98.82M parameters, 75.20% DSC) while maintaining strict topology preservation (\(0.06\%\) non-positive Jacobians).
- Cross-Dataset Generalizability: In cross-dataset transfer experiments where models trained on abdominal Learn2Reg are fine-tuned on 20 brain samples with frozen encoders, OursM achieves 64.83% DSC (vs. 62.89% for vanilla ModeT), demonstrating that foundation-guided training produces transferable, domain-robust representations.
Highlights & Insights¶
- Decoupled Distillation for Zero Inference Overhead: Rather than freezing large vision foundation models into the test pipeline, this work treats the VFM as a training-time geometric and semantic teacher, enabling compact registration models to achieve SOTA accuracy without test-time latency penalties.
- 2.5D Adaptive Reassembly Bridging Dimensional Gaps: The DR module effectively bridges 2D RGB models and 3D volumetric scans by fusing adjacent slices with learnable \(1 \times 1\) convolutions, encoding out-of-plane continuity while preserving natural image statistical characteristics.
- Hierarchical Feature Aggregation for Registration: Demonstrates that deformable registration benefits substantially from early-to-mid layer transformer representations (layers 3, 6, 9) rather than relying solely on the final semantic embedding.
Limitations & Future Work¶
- Domain Shift in Natural Foundation Models: Although DINOv3 exhibits impressive transferability, it is trained primarily on natural images whose noise characteristics and intensities differ from medical radiography. When anatomical structures lack natural visual analogues, suboptimal representations may be transferred.
- Fixed Inter-slice Context Window: The current DR module only considers immediate adjacent neighbors (\(S_{n-1}, S_{n+1}\)). In volumetric acquisitions with large slice thickness or severe anisotropy, multi-slice pooling or 3D foundation models could be explored to capture broader z-axis context.
Related Work & Insights¶
- vs DINOReg (Siebert et al., 2024): DINOReg feeds DINOv2 feature maps directly into ConvexAdam optimization during test time, requiring extensive GPU memory and multi-minute iterative optimization. In contrast, this paper uses DINOv3 for training-time feature distillation, preserving sub-second neural registration at inference.
- vs LLM-Morph (Ma et al., 2024): LLM-Morph incorporates large language model layers into the forward inference network, incurring severe computational costs. This work demonstrates that foundation model guidance can be entirely removed at test time.
- vs ModeT (Cheng et al., 2024): ModeT provides an efficient dual-stream 3D registration backbone but struggles with sharp MR-CT contrast inversions. Integrating the proposed foundation guidance lifts ModeT's brain registration DSC from 72.30% to 79.04% with zero extra inference parameters.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Clever decoupling of foundation model guidance during training from zero-overhead inference, complemented by 2.5D reassembly]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across abdominal and brain benchmarks, t-SNE feature visualizations, and cross-dataset transfer tests]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and well-structured empirical analyses]
- Value: ⭐⭐⭐⭐⭐ [Provides a practical blueprint for exploiting large vision foundation models in real-time clinical volumetric registration]