SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/JerrySongCST/SAM-Plus-D
Area: Medical Imaging
Keywords: Foundation Model Adaptation, Parameter-Efficient Fine-Tuning (PEFT), Dimensional Lifting, 3D/4D Segmentation, Segment Anything Model
TL;DR¶
SAM+D introduces a unified parameter-efficient dimensional lifting framework that inserts Depth-Routed LoRA (DRLoRA) experts and zero-parameter Depth Shift Modules (DSM) into frozen transformer blocks, tuning only ~2.8% to 3.7% of parameters to lift 2D SAM to 3D volumetric CT segmentation and 2D+T SAM2 to 4D spatiotemporal cell tracking.
Background & Motivation¶
Vision foundation models such as the Segment Anything Model (SAM) have demonstrated remarkable promptable segmentation capabilities on 2D images, and SAM2 extends this paradigm to video through streaming memory mechanisms. However, real-world biomedical imaging—such as 3D CT/MRI scans and 4D time-lapse fluorescence microscopy—relies heavily on volumetric depth continuity to disambiguate overlapping and low-contrast anatomical boundaries. Naive slice-by-slice 2D inference discards inter-slice dependencies and produces fragmented masks, whereas training full 3D networks from scratch requires prohibitive amounts of volumetric annotations and discards rich 2D pre-trained representations.
Adapting 2D foundation models to volumetric tasks currently suffers from three major bottlenecks. First, conventional Low-Rank Adaptation (LoRA) applies an identical, spatially uniform transformation across every slice, failing to account for dramatic anatomical variations along the depth axis (e.g., organ boundaries versus homogeneous interior parenchyma require distinct feature adaptations). Second, existing strategies for modeling inter-slice context introduce parameter-heavy 3D convolutions or cross-slice attention branches that inflate memory and computation, lacking a zero-overhead inter-slice communication mechanism. Third, existing adapters are tightly coupled to a single base architecture (either SAM or SAM2) and a single target dimensionality; no unified design lifts both 2D SAM to 3D and 2D+T SAM2 to 4D.
To overcome these limitations, the authors introduce a clean and parameter-efficient dimensional lifting strategy. Core idea: decouple inter-slice feature exchange from spatial depth adaptation by repurposing zero-parameter depth-wise channel shifting (DSM) for lossless adjacent-slice communication prior to attention, and routing lightweight low-rank experts (DRLoRA) via scalar depth coordinates for spatially adaptive updates, achieving seamless 3D and 4D dimensional lifting across SAM-family models while keeping over 96% of parameters frozen.
Method¶
Overall Architecture¶
SAM+D unifies 3D volumetric segmentation and 4D spatiotemporal tracking within a single modular framework. For 3D volumetric input, a target volume is uniformly resampled along the depth axis into \(D_s\) slices and fed as a batch into a frozen ViT-B encoder. Inside each transformer block, features first pass through DSM to exchange channel features bidirectionally between adjacent slices, followed by self-attention where \(Q\) and \(V\) projections are updated by DRLoRA experts routed by normalized slice depth coordinates. Per-slice features are projected through the neck, conditioned on relative depth prompt embeddings, and decoded via a LoRA-adapted prompt decoder with a lightweight 3D convolution smoothing head. For 4D spatiotemporal tasks, the identical DSM and DRLoRA modules are plugged into SAM2's Hiera backbone, with LoRA injected into memory cross-attention layers to enable robust volumetric spatiotemporal tracking.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input 3D Volume / 4D Microscopic Sequence<br/>Sample D_s depth slices into batch"] --> B["Stage 1: Depth Shift Module (DSM)<br/>Bidirectional adjacent slice channel shift (0 params/MAC)"]
B --> C["Stage 2: Depth-Routed LoRA (DRLoRA)<br/>Normalized depth z routes E experts via light MLP"]
C --> D["Stage 3: Frozen Attention & Feature Extraction<br/>Update Q/V projections while K and backbone stay frozen"]
D --> E["Stage 4: Relative Depth Prompt Embedding & Decoding<br/>Inject z_embed and refine volume via 3D conv head"]
E -->|4D Spatiotemporal Tasks| F["Stage 5: SAM2 Memory Attention LoRA Adaptation<br/>Cross-frame volumetric memory matching and tracking"]
Key Designs¶
1. Depth Shift Module (DSM): Zero-Parameter Cross-Slice Feature Exchange Processing volumes slice-by-slice blinds the model to inter-slice continuity, while standard 3D convolutions or cross-slice attention introduce substantial parameter and FLOP overheads. DSM resolves this dilemma by adapting the Temporal Shift Module (TSM) concept from video modeling to the spatial depth axis. Given a feature tensor \(\mathbf{X} \in \mathbb{R}^{D_s \times H \times W \times C}\), DSM splits channels along the embedding dimension by a shift ratio \(\alpha\) (defaulting to 25%): \(\alpha C\) channels are copied forward from slice \(i-1\), \(\alpha C\) channels are copied backward from slice \(i+1\), and the remaining \((1-2\alpha)C\) channels remain untouched: $\(\mathbf{X}' = \text{DSM}(\mathbf{X}) = \text{concat}(\mathbf{X}^{\text{fwd}}, \mathbf{X}^{\text{bwd}}, \mathbf{X}^{\text{id}})\)$ Boundary slices retain their own features to prevent boundary artifacts. Crucially, DSM is applied immediately before self-attention in each transformer block, ensuring that tokens entering query, key, and value projections already contain immediate neighboring-slice context. DSM introduces exactly 0 learnable parameters and 0 multiply-accumulate (MAC) operations, requiring only 0.34 ms memory-stride copy latency on an RTX 5090 GPU (1.7% of an encoder block's forward pass), yet allowing deep transformer layers to accumulate wide cross-slice receptive fields.
2. Depth-Routed LoRA Experts (DRLoRA): Spatially Adaptive Low-Rank Updates via Depth Coordinates Standard LoRA updates all slices identically regardless of their anatomical position, failing to address the substantial morphology differences between organ margins and interior regions. DRLoRA formulates adaptation as a mixture of \(E\) parallel low-rank experts \(\{\mathcal{E}_1, \dots, \mathcal{E}_E\}\), where each expert maintains down- and up-projection weights \(\mathbf{W}_{\text{down}}^{(i)} \in \mathbb{R}^{r \times d}\) and \(\mathbf{W}_{\text{up}}^{(i)} \in \mathbb{R}^{d \times r}\) with \(\mathbf{W}_{\text{up}}^{(i)}\) initialized to zero. Instead of routing on high-dimensional token representations as in NLP MoE models, DRLoRA routes on the normalized scalar depth position \(z = d / D_s \in [0, 1]\): $\(\mathbf{w}(z) = \text{softmax}(\mathcal{R}(z)), \quad \mathcal{R}: \mathbb{R} \xrightarrow{\text{Linear}} \mathbb{R}^h \xrightarrow{\text{GELU}} \mathbb{R}^h \xrightarrow{\text{Linear}} \mathbb{R}^E\)$ The combined output is given by \(\text{DRLoRA}(\mathbf{x}, z) = \sum_{i=1}^E w_i(z) \cdot \mathbf{W}_{\text{up}}^{(i)} \mathbf{W}_{\text{down}}^{(i)} \mathbf{x}\). Because slice depth \(z\) is inherently uniformly distributed across \([0, 1]\), DRLoRA fundamentally eliminates expert collapse without needing complex auxiliary load-balancing losses. The two-layer router requires merely \(\sim 0.14\text{K}\) parameters. Following optimal LoRA practices, DRLoRA is injected solely into the query (\(Q\)) and value (\(V\)) projections while keys (\(K\)) remain frozen, modulating both attention focus and propagated content while maintaining a stable pre-trained feature foundation.
3. Relative Depth Prompt Embedding and 3D Refinement Head In single-point interactive segmentation, SAM's native decoder lacks awareness of vertical 3D spatial distance relative to the user-provided prompt point. To resolve this, a two-layer MLP constructs a relative depth embedding \(z_{\text{embed}}\) that encodes the relative slice offset \((z_p - d) / D_s\) between any slice \(d\) and the prompted slice \(z_p\): $\(\mathbf{D}'_d = \mathbf{D}_{\text{prompt}} + z_{\text{embed}}\left( \frac{z_p - d}{D_s} \right)\)$ This enables the decoder to smoothly adapt its feature focus depending on whether a slice is near the center of the prompt or at the peripheral boundaries. Together with LoRA injected into the mask decoder attention layers and a lightweight 3D convolutional head on the stacked 2D logits, the network enforces volumetric smoothness and suppresses spurious slice-level false positives.
4. 4D Spatiotemporal Lifting and Memory Attention Adaptation When extending to 4D time-lapse microscopy sequences, the architecture must simultaneously process 3D spatial volumes and the temporal dimension. SAM+D strictly maintains modular separation: DSM and DRLoRA operate exclusively along the spatial depth dimension within each volume timestep, leaving temporal continuity to SAM2's native streaming memory architecture. Because SAM2's memory attention was originally trained on 2D video sequences, applying it directly to volumetric embeddings leads to suboptimal spatiotemporal matching. SAM+D injects LoRA (\(r=16\)) into all 4 layers of memory cross-attention, tailoring the memory bank to 3D volumetric token matching. Trained purely with per-frame DiceCE loss without explicit graph tracking heuristics, SAM2+D reliably performs end-to-end spatiotemporal cell segmentation and lineage tracking.
Loss & Training¶
The framework is optimized using a standard per-slice and per-frame Dice-CrossEntropy (DiceCE) loss. For 3D CT benchmarks, inputs are cropped into \(3 \times 128 \times 128 \times 128\) volumes around the prompt point and trained for 500 epochs using AdamW (learning rate \(4 \times 10^{-4}\), cosine decay). For 4D cell tracking, target cells are cropped into \(3 \times 32 \times 256 \times 256\) patches over clip lengths of \(T=16\) timesteps and trained for 500 epochs with a learning rate of \(1 \times 10^{-4}\). All models are trained across 3 NVIDIA RTX 6000 Pro GPUs with BF16 mixed precision and gradient checkpointing, processing a full \(128^3\) volume during inference in only ~228.5 ms.
Key Experimental Results¶
Main Results¶
SAM+D was comprehensively evaluated across four 3D CT tumor segmentation benchmarks (KiTS for kidney, Pancreas, LiTS for liver, Colon) and the Cell Tracking Challenge (Fluo-N3DH-SIM+). Under single-point prompt conditions, SAM+D matches or outperforms fully-supervised 3D networks and existing SAM adapters while requiring vastly fewer trainable parameters.
Table 1: Comparison with fully-supervised, interactive, and SAM-based methods on four 3D CT tumor segmentation datasets (single-point prompt 1 pt/vol; Dice (%) and NSD (%) at 2 mm tolerance)
| Category | Method | Trainable Params | KiTS (Kidney) Dice / NSD | Pancreas Dice / NSD | LiTS (Liver) Dice / NSD | Colon Dice / NSD |
|---|---|---|---|---|---|---|
| Fully-Supervised 3D | nnU-Net | 30.76 M | 73.09 / 77.42 | 41.70 / 62.92 | 60.27 / 75.60 | 44.38 / 53.66 |
| Fully-Supervised 3D | Swin-UNETR | 62.19 M | 66.48 / 72.44 | 39.66 / 58.33 | 52.16 / 65.71 | 33.53 / 41.16 |
| Interactive Baselines | Visual Sampler | 33.67 M | 72.47 / 78.25 | 49.48 / 69.49 | 52.51 / 64.29 | 53.49 / 68.05 |
| SAM Adaptation | SAM-B (1 pt/slice) | 0 M | 36.30 / 29.86 | 24.01 / 26.74 | 6.71 / 7.63 | 28.83 / 33.63 |
| SAM Adaptation | 3DSAM-adapter | 25.46 M | 80.16 / 87.40 | 53.69 / 76.44 | 58.02 / 71.55 | 59.29 / 76.10 |
| SAM Adaptation | Med-SA | 13.00 M | 70.73 / 82.41 | 43.59 / 73.35 | 51.61 / 72.04 | 53.56 / 74.85 |
| SAM Adaptation | MA-SAM (no prompt) | 63.04 M | 69.25 / 65.84 | 34.72 / 47.72 | 56.28 / 62.87 | 50.32 / 56.37 |
| Ours | SAM+D (SAM Dec.) | 2.57 M | 84.74 / 92.11 | 59.68 / 79.05 | 63.33 / 73.52 | 63.70 / 77.69 |
Table 2: 4D segmentation and tracking results on Fluo-N3DH-SIM+ from the Cell Tracking Challenge (Seq 02 test set; reporting DET, SEG, TRA, and OPCTB (%))
| Method | Seg. Source | Trainable Params | SEG ↑ | DET ↑ | TRA ↑ | OPCTB ↑ |
|---|---|---|---|---|---|---|
| BGU-IL | CellPose | 127.70 M | 53.80 | 85.20 | 84.20 | 69.00 |
| Ultrack | CellPose | 0 M | 54.51 | 85.43 | 85.17 | 69.84 |
| TrackStra | CellPose | 6.90 M | 54.15 | 84.86 | 84.16 | 69.16 |
| SAM2 (slice-based) | CellPose | 0 M | 54.19 | 84.84 | 84.14 | 69.16 |
| SAM2+D (Ours) | CellPose | 3.11 M | 56.79 | 87.32 | 87.15 | 71.97 |
| SAM2+D (Ours) | GT | 3.11 M | - | - | 99.31 | 99.47 |
Ablation Study¶
A systematic component and parameter sweep was conducted on the LiTS dataset with the ViT-B backbone (single-point prompt).
Table 3: Ablation study on LiTS dataset (default settings indicated by *)
| Group | Configuration | Dice (%) | NSD (%) | Note |
|---|---|---|---|---|
| (a) Component contribution | DRLoRA only | 56.41 | 65.28 | Depth-routed adaptation without inter-slice exchange |
| (a) Component contribution | + DSM | 59.17 | 68.74 | Adding depth shifting yields +2.76% Dice at zero param cost |
| (a) Component contribution | + z-embed | 61.05 | 71.03 | Adding relative prompt depth embedding gains +1.88% Dice |
| (a) Component contribution | + Dec. LoRA* (Full model) | 63.33 | 73.52 | Full model with LoRA on prompt decoder attention |
| (b) DSM shift ratio \(\alpha\) | \(\alpha = 0.125\) | 61.22 | 71.46 | Under-shifting leaves insufficient adjacent context |
| (b) DSM shift ratio \(\alpha\) | \(\alpha = 0.25\)* | 63.33 | 73.52 | Optimal balance preserving 50% identity channels |
| (b) DSM shift ratio \(\alpha\) | \(\alpha = 0.5\) | 62.08 | 72.15 | Over-shifting dilutes current-slice representations |
| (c) DRLoRA experts \(K\) | \(K = 1\) (std. LoRA) | 58.86 | 68.21 | Uniform LoRA lags behind 4 experts by 4.47% Dice |
| (c) DRLoRA experts \(K\) | \(K = 2\) | 61.47 | 71.30 | Binary expert partition provides baseline depth-awareness |
| (c) DRLoRA experts \(K\) | \(K = 4\)* | 63.33 | 73.52 | Best expert capacity-to-generalization trade-off |
| (c) DRLoRA experts \(K\) | \(K = 8\) | 63.10 | 73.18 | Marginal performance saturation with added parameter count |
| (d) LoRA rank \(r\) | \(r = 4\) / \(r = 8\) | 60.72 / 62.85 | 70.43 / 73.01 | Lower rank limits expressiveness |
| (d) LoRA rank \(r\) | \(r = 16\)* / \(r = 32\) | 63.33 / 62.19 | 73.52 / 72.34 | Rank 16 provides optimal adaptation performance |
Key Findings¶
- Depth-Conditioned Routing Outperforms Uniform LoRA and Content-Based MoE: Replacing a single uniform LoRA (\(K=1\), 58.86% Dice) with depth-routed \(K=4\) experts boosts Dice to 63.33% (+4.47%). Furthermore, supplementary comparisons against content-based MoE routers (MoLoRA, MixLoRA) under identical parameter budgets demonstrate a +7% to +9% Dice advantage for DRLoRA, proving that macroscopic depth coordinates provide a robust and non-collapsing routing signal for volumetric anatomy.
- Zero-Cost Inter-Slice Exchange via DSM: DSM contributes a +2.76% Dice improvement strictly through strided memory channel shifts without any parameter or multiply-accumulate overhead, outperforming computationally expensive 3D attention modules.
- Breaking Detection Ceilings in 4D Tracking: Under the Cellpose detector setting, all previous tracking baselines plateau at ~54% SEG due to detector errors. SAM2+D leverages its adapted memory attention to refine spatial segmentations across time, surpassing the detector ceiling to achieve 56.79% SEG and 71.97% OPCTB.
Highlights & Insights¶
- Repurposing Video TSM for Spatial Volumetric Depth: Transforming the Temporal Shift Module into a spatial Depth Shift Module proves that video temporal dynamics and medical volume slice sequences share structural continuity, enabling zero-parameter 3D feature communication.
- Clean Separation of Spatial Depth and Temporal Dynamics: For 4D spatiotemporal problems, confining DSM and DRLoRA strictly to the spatial depth axis while delegating temporal continuity to SAM2's pre-trained memory avoids dimensional entanglement and preserves pre-trained foundation knowledge.
- Scalar-Conditioned Non-Collapsing MoE Routing: By routing low-rank experts on normalized slice positions \(z \in [0, 1]\) rather than variable token content, DRLoRA guarantees uniform expert utilization, operates with an ultra-compact 0.14K parameter router, and eliminates auxiliary balancing losses.
Limitations & Future Work¶
- Single-Axis Depth Assumption: Current 3D implementations treat one anatomical axis as depth; isotropic multi-view fusion across axial, sagittal, and coronal planes could further refine boundary fidelity.
- Reliance on External Prompts for 4D Cell Division: While single cells track end-to-end from one prompt, dynamic cell division and immigration events still require external detection triggers for prompt re-initialization.
- Extension to Multimodal SAM3: The framework has not yet been extended to text-prompted multimodal architectures like SAM3, where text-visual alignment across spatial dimensions remains unexplored.
Related Work & Insights¶
- vs 3DSAM-adapter: 3DSAM-adapter inserts parameter-heavy 3D depthwise convolutions requiring 25.46 M trainable parameters; SAM+D attains superior accuracy (84.74% vs 80.16% on KiTS) using only 2.57 M parameters.
- vs Med-SA (SD-Trans): Med-SA doubles attention computation by splitting spatial and depth branches; SAM+D's DSM achieves equivalent cross-slice communication at zero FLOP and zero parameter overhead.
- vs Content-Based MoE-LoRA (MoLoRA / MixLoRA): Content-based routing suffers from token routing instability and requires auxiliary balancing losses; DRLoRA leverages normalized depth coordinates to ensure balanced, stable expert dispatch.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant transposition of channel shifting to spatial depth and robust scalar-routed MoE for dimensional lifting across SAM and SAM2.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across four 3D CT benchmarks and a 4D cell tracking benchmark with detailed ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured presentation with clear mathematical definitions and self-consistent claims.
- Value: ⭐⭐⭐⭐⭐ Establishes a highly generalizable and lightweight recipe for scaling 2D foundation models to higher-dimensional 3D and 4D vision tasks.