ReliefSAM: A Geometry-Augmented Multi-Prior Adapter for Bas-Relief Segmentation¶
Conference: ECCV 2026
Paper: ECCV Official Page
Area: Segmentation
Keywords: Bas-Relief Segmentation, Segment Anything Model, Parameter-Efficient Fine-Tuning, Geometric Priors, Cultural Heritage
TL;DR¶
Addressing the severe lack of chromatic and geometric contrast in monocular bas-relief archival photographs, ReliefSAM freezes the SAM backbone and explicitly injects image-aligned depth and soft-edge priors via a lightweight multi-prior feature encoder and bidirectional interaction adapters, establishing accurate prompt-free human figure segmentation.
Background & Motivation¶
Bas-relief sculptures are vital historical records that preserve complex religious narratives, historical chronicles, and societal life in densely carved scenes, such as the extensive narrative panels of the Borobudur temple in Central Java (notably the 160 panels of the "Hidden Relief" Karmawibhangga). In these carved compositions, human figures serve as the primary semantic carriers of the historical narrative. Accurate human figure segmentation is therefore an indispensable prerequisite for scalable digital documentation, cross-modal archive indexing, and downstream computational art-historical analysis. However, due to late 19th-century structural reinforcement, erosion, and strict conservation restrictions, many reliefs remain permanently reburied or physically inaccessible for contemporary 3D laser scanning or LiDAR acquisition. Consequently, researchers must rely solely on archival monocular photographs taken more than a century ago as their sole visual record.
Segmenting human figures from monocular bas-relief photographs presents severe visual ambiguities that defeat conventional segmentation paradigms. Unlike natural scene imagery, bas-reliefs are carved from completely homogeneous stone; foreground figures and background stone substrates share identical color, texture, and reflectance, causing appearance-based chromatic contrast to collapse entirely. Foreground delineation depends almost exclusively on subtle 2.5D geometric relief variations and soft transition boundaries, which are themselves severely degraded by weathering, surface erosion, and physical fractures. Pretrained foundation models such as the Segment Anything Model (SAM) demonstrate exceptional zero-shot generalization on natural imagery, but their representations are heavily biased toward RGB appearance textures. When applied to bas-reliefs, SAM either overlooks shallow relief figures or mistakenly latches onto high-frequency weathering cracks as semantic boundaries. Moreover, naively fine-tuning or feeding a single geometric prior into foundation models often induces severe modality interference and systematic bias.
Faced with the strict monocular setting and the inability of foundation models to perceive shallow geometric cues, this work introduces ReliefSAM, a prompt-free, parameter-efficient framework operating strictly on a single RGB photograph. Core idea: without requiring external 3D sensing, derive image-aligned depth and soft-edge priors directly from the monocular RGB photograph, encode and re-weight them via a lightweight Multi-Prior Feature Encoder (MFE), and inject them into a frozen SAM backbone through bidirectional extract-then-inject cross-attention adapters to form a geometric consensus that counterbalances individual prior biases and breaks the RGB-only performance ceiling.
Method¶
Overall Architecture¶
ReliefSAM takes a single monocular RGB bas-relief photograph \(\mathbf{I}_{\mathrm{rgb}} \in \mathbb{R}^{B \times 3 \times H \times W}\) as its only input and produces a full-resolution binary human figure probability map \(\hat{\mathbf{Y}} \in [0, 1]^{B \times 1 \times H \times W}\). The framework proceeds through three cohesive stages: first, image-aligned depth \(\mathbf{D}\) and soft-edge \(\mathbf{E}\) priors are derived from the input photograph; second, a lightweight Multi-Prior Feature Encoder (MFE) extracts spatial representations from both priors and performs channel recalibration to output compact geometric tokens \(\mathbf{Z}\); third, relief-specific geometric adapters inserted into designated Transformer blocks of a frozen SAM ViT image encoder execute an extract-then-inject bidirectional cross-attention routine to enrich visual tokens with filtered geometry. Finally, SAM's mask decoder operates without manual prompts to predict foreground probabilities, and an overlapped sliding-window scheme with Gaussian-weighted blending generates seamless, full-resolution segmentation masks.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Monocular Bas-Relief Input<br/>I_rgb"] --> B["Image-Aligned Multi-Prior Extraction<br/>Derive Depth D and Soft-Edge E"]
B --> C["Multi-Prior Feature Encoder MFE<br/>Dual-Branch CNN + SE Channel Recalibration"]
A --> D["Frozen SAM ViT Image Encoder"]
C --> E["Relief-Specific Geometric Adapters<br/>Extract-Then-Inject Bidirectional CA"]
D <--> E
D --> F["Prompt-Free Mask Decoding<br/>Default No-Mask Embedding + Single Output"]
F --> G["Gaussian-Weighted Sliding-Window Inference<br/>Overlapped Patch Prediction and Seamless Merging"]
G --> H["Seamless High-Resolution Mask"]
Key Designs¶
1. Image-Aligned Multi-Prior Extraction and Lightweight Encoding: Complementary Inherent Geometry
To overcome the severe chromatic homogeneity of relief surfaces without relying on external 3D scanners, the network must recover the 2.5D structure embedded within the monocular photograph. Standard edge detectors like Canny fail because weathered stone surfaces produce massive noisy edges, while general-purpose depth estimators suffer from depth compression on shallow relief surfaces. The authors adopt a relief-dedicated framework to derive an image-aligned depth prior \(\mathbf{D} \in \mathbb{R}^{B \times 1 \times H \times W}\) (preserving facial and anatomical curvatures) and an edge confidence map \(\mathbf{E} \in \mathbb{R}^{B \times 1 \times H \times W}\) (emulating soft edges from stochastic point-based rendering). Both priors are linearly scaled to \([0, 1]\) and processed by the Multi-Prior Feature Encoder (MFE). The MFE employs two parallel 5-layer \(3\times3\) Conv-BN-ReLU branches to yield intermediate feature maps \(\mathbf{F}_d\) and \(\mathbf{F}_e\) (each with dimension \(\lfloor C/2 \rfloor\) and \(C - \lfloor C/2 \rfloor\)). The features are concatenated along the channel dimension (\(\mathbf{F} = [\mathbf{F}_d; \mathbf{F}_e] \in \mathbb{R}^{B \times C \times H_g \times W_g}\)) and recalibrated using a Squeeze-and-Excitation (SE) module with channel descriptors \(\mathbf{a} \in \mathbb{R}^{B \times C}\):
$\(\mathbf{s} = \sigma\left(W_2 \delta(W_1 \mathbf{a})\right) \in (0, 1)^{B \times C}\)$
The spatially re-weighted feature \(\tilde{\mathbf{F}} = \mathbf{F} \odot \mathbf{s}\) is flattened and transposed into geometric tokens \(\mathbf{Z} \in \mathbb{R}^{B \times N_g \times C}\), matching the embedding dimension \(C\) of SAM image tokens for subsequent seamless cross-attention fusion.
2. Relief-Specific Geometric Adapters: Extract-Then-Inject Bidirectional Interaction
Directly fusing geometric priors into frozen foundation features via naive addition or concatenation corrupts pretrained representations with estimation artifacts, rock fracture noise, and weathering cracks. To solve this, the authors introduce relief-specific geometric adapters at Transformer blocks \(S = \{2, 4, 6, 8, 10\}\) and after the final block of the ViT-B backbone. The adapter executes an extract-then-inject bidirectional cross-attention mechanism across two pre-normalized stages. In the extraction stage, the geometric tokens \(\mathbf{Z}\) act as queries to attend to image tokens \(\mathbf{X}^{(l)}\):
$\(\mathbf{Z} \leftarrow \mathrm{CA}\left(\mathrm{LN}(\mathbf{Z}), \mathrm{LN}(\mathbf{X}^{(l)})\right)\)$
This enables the geometric tokens to absorb image-level contextual and instance-aware cues, effectively suppressing noise and spurious fractures unaligned with the true figures. In the subsequent injection stage, image tokens act as queries to retrieve the purified geometric representation:
$\(\mathbf{X}^{(l)} \leftarrow \mathbf{X}^{(l)} + \mathrm{CA}\left(\mathrm{LN}(\mathbf{X}^{(l)}), \mathrm{LN}(\mathbf{Z})\right)\)$
This two-stage interaction ensures that only contextually verified, high-confidence geometric boundaries and surface contours are injected into the visual feature stream.
3. Prompt-Free Decoding and Gaussian-Weighted High-Resolution Inference: Seamless Automation
Real-world archaeological documentation involves archival panels of massive native dimensions (e.g., \(3200 \times 1024\) pixels) and demands batch processing without manual prompt intervention. At the decoding stage, SAM's prompt encoder is frozen without point or box prompts; the model relies exclusively on SAM's learned default no-mask embeddings and fixes multimask_output=False to predict a single foreground logit map. For high-resolution inference, the input photograph and its geometric priors are synchronously tiled into \(1024 \times 1024\) windows with a stride of 64. To eliminate seam artifacts along overlapping tile boundaries, the method accumulates local probability maps using a spatial Gaussian weighting kernel \(\mathbf{G}\):
$\(\hat{\mathbf{Y}} = \frac{\mathbf{S}}{\mathbf{W} + \epsilon} = \frac{\sum_k \mathbf{G}_k \odot \hat{\mathbf{Y}}_k}{\sum_k \mathbf{G}_k + \epsilon}\)$
A standard threshold of 0.5 is applied to generate seamless, continuous full-resolution binary masks across the entire carved frieze.
Loss & Training¶
The model is optimized using a balanced Dice-Focal compound loss function: $\(\mathcal{L} = \lambda_{\mathrm{dice}} \mathcal{L}_{\mathrm{dice}} + \lambda_{\mathrm{focal}} \mathcal{L}_{\mathrm{focal}}\)$ where \(\mathcal{L}_{\mathrm{dice}} = 1 - \frac{2\langle\hat{\mathbf{Y}}, \mathbf{Y}\rangle + \epsilon}{\|\hat{\mathbf{Y}}\|_1 + \|\mathbf{Y}\|_1 + \epsilon}\) enforces global spatial overlap, while the pixel-wise Focal loss (\(\alpha = 0.25\), \(\gamma = 2.0\)) penalizes challenging boundary misclassifications. The balancing factors are set to \(\lambda_{\mathrm{dice}} = \lambda_{\mathrm{focal}} = 1.0\). During training, the SAM ViT-B image encoder and prompt encoder remain strictly frozen; only the MFE, geometric adapters, and SAM mask decoder are updated. Optimization is conducted on a single NVIDIA GeForce RTX 5090 GPU (32 GB VRAM) using AdamW for 50 epochs with batch size 2, learning rate \(1 \times 10^{-4}\), weight decay \(1 \times 10^{-4}\), and a cosine annealing schedule on \(1024 \times 1024\) patches (stride 128).
Key Experimental Results¶
Main Results¶
Evaluation is performed on the newly established Borobudur "Hidden Relief" (Karmawibhangga) benchmark consisting of 156 archival monocular photographs (\(3200 \times 1024\) resolution) divided at the image level into 116 training, 20 validation, and 20 test panels. All baselines are evaluated with full-resolution sliding-window inference.
| Method | Backbone | IoU (%) ↑ | Dice (%) ↑ | Notes |
|---|---|---|---|---|
| SegFormer | MiT-B3 | 87.86 | 93.45 | Generic transformer baseline |
| DeepLabV3+ | ResNet-50 | 89.32 | 94.27 | Generic CNN baseline |
| SAM-DecoderFT | ViT-B | 83.82 | 90.95 | Frozen encoder, decoder fine-tuning |
| SAM-Adapter (Ours, RGB-only) | ViT-B | 91.25 | 95.37 | Pure-RGB adapter baseline |
| ReliefSAM (Ours) | ViT-B | 91.78 | 95.65 | Multi-prior geometry adapter |
Ablation Study¶
To isolate the effects of architectural adaptation from the contributions of geometric priors, the authors evaluate single-prior and multi-prior variants on top of the fixed SAM-Adapter backbone (Tables 2 & 3 in the paper):
| Configuration | Depth Prior | Edge Prior | IoU (%) ↑ | Dice (%) ↑ | Precision (%) ↑ | Recall (%) ↑ |
|---|---|---|---|---|---|---|
| SAM-Adapter (RGB-only) | ✗ | ✗ | 91.25 | 95.37 | 94.50 | 96.31 |
| + Depth prior | ✓ | ✗ | 89.97 | 94.57 | 95.37 | 93.89 |
| + Edge prior | ✗ | ✓ | 90.52 | 94.95 | 92.65 | 97.43 |
| ReliefSAM (Ours) | ✓ | ✓ | 91.78 | 95.65 | 94.26 | 97.14 |
Key Findings¶
- Modality Interference Induced by Isolated Priors: Naively introducing a single geometric prior into the strong pure-RGB SAM-Adapter baseline causes performance degradation (IoU drops by -1.28% with depth alone and by -0.73% with edge alone). Precision-Recall analysis reveals that each isolated prior introduces opposite biases: the depth prior acts conservatively, eliminating false positives to achieve the highest Precision (95.37%) but missing shallow boundaries due to low relief contrast (Recall drops to 93.89%); conversely, the soft-edge prior acts aggressively, recovering faint contours to reach the highest Recall (97.43%) while mistaking stone fractures for figure contours (Precision drops to 92.65%).
- Geometric Consensus Through Synergistic Dual Priors: ReliefSAM jointly encodes depth and soft-edge priors through the MFE, allowing the conservative depth structure to suppress edge-induced false positives while the edge prior recovers faint contours missed by depth. This mutual counterbalancing yields a geometric consensus that breaks the RGB-only ceiling (+0.53% IoU and +0.28% Dice over SAM-Adapter).
- Encoder Adaptation vs. Decoder-Only Fine-Tuning: Tuning only the mask decoder of SAM (SAM-DecoderFT) achieves merely 83.82% IoU, severely underperforming generic CNN/Transformer baselines. Injecting adapters into the image encoder (SAM-Adapter) brings a massive +7.43% IoU boost, confirming that domain adaptation in the feature representation stage is crucial when dealing with severe material homogeneity.
Highlights & Insights¶
- Self-Contained 2.5D Recovery Under Monocular Constraints: The method completely bypasses the need for modern on-site 3D scanning equipment, successfully unlocking scalable digital analysis for reburied or damaged heritage assets where archival monocular photographs are the only surviving visual evidence.
- Empirical Demonstration of Modality Interference and Prior Counterbalancing: The paper provides valuable empirical evidence that multi-modal fusion is not a simple additive process; single geometric priors degrade strong foundation models through specific biases, and true gains require complementary structural counterbalancing.
- Extract-Then-Inject Attentive Filtering: The two-stage cross-attention design filters geometric priors through visual context before injecting them into the backbone, offering a robust paradigm for preventing artifact leakage in low-contrast, highly textured environments.
Limitations & Future Work¶
- Binary Segmentation Scope: The current benchmark and framework are restricted to binary human figure segmentation; extending the architecture to multi-class semantic segmentation (distinguishing garments, deities, animals, and architectural components) remains an open challenge.
- Dependency on Upstream Prior Estimation Quality: The performance of the adapter relies on the fidelity of the monocular depth and soft-edge estimators; severe optical degradation, lens distortion, or chemical damage on historical film negatives could degrade prior quality and cascade downstream.
- Future Directions: End-to-end co-training of prior generation and segmentation adapters, frequency-domain decoupling of weathering cracks from carving edges, and broader evaluations on other world heritage sites (e.g., Egyptian bas-reliefs, Greco-Roman friezes).
Related Work & Insights¶
- vs SAM-DecoderFT [13]: Standard PEFT recipes often freeze SAM's image encoder and only train the mask decoder; however, on bas-reliefs, appearance features lack chromatic contrast. ReliefSAM modifies the image representation directly via adapters, outperforming decoder-only tuning by +7.96% IoU.
- vs Ji et al. [10]: While Ji et al. explored soft-edge guidance for Borobudur reliefs, their pipeline required explicit on-site 3D scans and task-specific datasets; ReliefSAM operates under a strictly monocular setting, making it directly applicable to archival photographs of reburied assets.
- vs Heritage Adapters like EdgeSAM-CASD [29]: Prior adapters for cultural heritage (e.g., mural damage segmentation) use simple convolutional bottlenecks; ReliefSAM's extract-then-inject cross-attention dynamically suppresses weathering artifacts by cross-referencing visual and geometric streams.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Pioneers an image-aligned dual-prior geometric adaptation scheme for archival bas-relief segmentation and systematically uncovers the prior counterbalancing consensus mechanism.
- Experimental Thoroughness: ⭐⭐⭐⭐☆ Thorough evaluation on 156 high-resolution panels of the UNESCO Borobudur "Hidden Relief" with rigorous image-level splits and insightful Precision-Recall ablation.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp technical clarity, solid motivation, and insightful ablation analysis.
- Value: ⭐⭐⭐⭐⭐ Delivers an actionable, non-invasive digital documentation pathway for globally significant, physically inaccessible cultural heritage.