HIDA: A Human-Intuition-Guided Depth-Aware Framework for Zero-Shot Amodal Segmentation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/lipenghuai/HIDA
Area: Segmentation
Keywords: amodal segmentation / zero-shot learning / pseudo-depth prior / soft MoE / vision foundation models

TL;DR

HIDA integrates frozen SAM and Depth Anything V2 as plug-and-play prior generators, leveraging depth-polarity adaptive cropping to focus on occluded regions and an occlusion-aware token-routing soft MoE to empower a lightweight ViT, achieving state-of-the-art zero-shot and fully supervised amodal segmentation using only visible bounding boxes.

Background & Motivation

Human vision demonstrates an instinctive capability for amodal perception, naturally deducing and hallucinating the complete shape of an object from its partially visible fragments based on physical common sense and geometric relations. In computer vision, amodal segmentation aims to delineate the full silhouette of occluded targets, playing an indispensable role in downstream scenarios such as autonomous driving scene parsing, robotic manipulation and obstacle avoidance, image de-occlusion, and 3D scene reconstruction. However, the vast majority of conventional approaches remain constrained to in-domain supervised training on closed-world benchmarks, incurring prohibitive manual annotation expenses while suffering from severe generalization degradation when encountering novel categories or open-set environments.

With the emergence of vision foundation models, recent efforts have explored leveraging large-scale pretrained priors for zero-shot amodal completion. For instance, SAMBA introduces additional structural layers and fine-tunes the SAM encoder to establish zero-shot amodal capabilities, whereas Pix2Gestalt trains a diffusion model for generative mask recovery. Nonetheless, these paradigms either suffer from excessive training overhead and architectural lock-in stemming from foundation model fine-tuning, or rely on heavy generative pipelines while neglecting the underlying 3D continuous geometry of physical scenes. Human visual intuition dictates that occlusion relationships adhere to an elementary depth rule: only objects positioned closer to the imaging sensor in 3D space can occlude targets located further behind. When monocular depth estimation and accurate visible contours are explicitly leveraged, amodal completion transitions from an ill-posed global hallucination into a well-anchored local boundary extrapolation.

Motivated by this intuition, this paper introduces HIDA, a two-phase, tuning-free, plug-and-play amodal segmentation framework. By keeping both SAM and Depth Anything V2 strictly frozen as offline prior generators, HIDA formulates depth-polarity adaptive image cropping alongside a lightweight ViT encoder driven by an occlusion-aware token-routing soft MoE module to dynamically model invisible regions. Core idea: decouple amodal segmentation into frozen foundation model 3D geometric prior extraction, depth-polarity-guided asymmetric adaptive cropping, and occlusion-aware token-routing soft MoE feature learning in a lightweight network for reliable contour extrapolation.

Method

Overall Architecture

Conditioned solely on an input RGB image \(I\) and a modal (visible) bounding box \(B_m\), HIDA predicts a binary amodal mask \(\tilde{M}^a \in \{0, 1\}^{H \times W}\) covering both visible and occluded components in a category-agnostic manner. The architecture is decomposed into a non-trainable prior generation phase and a trainable lightweight segmentation network: Depth Anything V2 first computes a normalized pseudo-depth map (PDM) \(D\) in parallel with SAM extracting a high-precision visible mask \(\tilde{M}^m\); subsequently, boundary occlusion scores evaluate depth polarity across all four directions to adaptively expand the bounding box towards genuinely occluded edges, cropping \(I\), \(D\), and \(\tilde{M}^m\); finally, these cropped multimodal inputs are forwarded to a lightweight ViT encoder whose internal representations are dynamically enriched by an occlusion-aware soft MoE router, before being concatenated with frozen SAM features to decode the complete amodal mask.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: RGB Image I + Modal Box Bm"] --> B["Stage 1: Frozen Dual Foundation Model Prior Extraction<br/>SAM extracts visible mask / Depth-Anything-V2 yields depth map"]
    B --> C["Stage 2: Depth-Polarity-Guided Adaptive Target Cropping<br/>Score outer border depth polarity to asymmetrically expand towards occluders"]
    C --> D["Stage 3: Occlusion-Aware Token-Routing Soft MoE Encoding<br/>Lightweight ViT dynamically routes tokens via occlusion signal occ"]
    D --> E["Stage 4: Dual Feature Concatenation & Lightweight Decoding<br/>Fuse lightweight ViT features with frozen SAM features to decode full mask"]
    E --> F["Output: Final Full-Image Amodal Mask Ma"]

Key Designs

1. Depth-Polarity-Guided Adaptive Target Cropping: concentrating receptive fields on occluded context

Conventional bounding box expansions rely on uniform, symmetric scaling, which introduces excessive uninformative background clutter, degrades signal-to-noise ratios, and incurs massive blank space for elongated items. To address this, HIDA translates the physical premise that "foreground occluders exhibit larger depth values (positioned closer to the camera)" into an directional occlusion scoring mechanism across four sides \(s \in \{l, r, t, b\}\). Along each side outside visible mask \(\tilde{M}^m\), an irregular stripe \(\Omega_s\) is constructed with perpendicular linearly decaying weights \(w(p)\). By eroding \(\tilde{M}^m\) into a 5-pixel inner border band \(band_{in}\), the occlusion score is computed as: $\(score_s = \frac{\sum_{p \in \Omega_s} \mathbf{1}[D(p) \ge \text{med}(D(band_{in})) + \Delta] w(p)}{\sum_{p \in \Omega_s} w(p)}\)$ where \(\Delta\) denotes a depth margin threshold. If outer pixels are significantly closer than the median depth of the inner contour, foreground occlusion is recognized. Within the expansion range \([E_{r,\min}, E_{r,\max}] = [0.2, 0.6]\), horizontal and vertical expansion scales are allocated asymmetrically according to relative score ratios: $\(E_{r,left} = \min\left\{\max\left(E_{r,\min}, \frac{score_{left}}{score_{left} + score_{right} + \varepsilon} \times E_{r,\max}\right), E_{r,\max}\right\}\)$ This directional expansion guarantees that the cropped window selectively stretches toward occluded regions while discarding uninformative distant background, maintaining consistent object centering and eliminating scale jitter.

2. Occlusion-Aware Token-Routing Soft MoE: routing feature pathways conditioned on occlusion intensity

Amodal segmentation exhibits a severe representation gap because visible and occluded regions differ drastically in local texture and depth characteristics. Conventional hard-routing MoE selects discrete experts and causes boundary instability, whereas continuous soft routing provides smooth blending across transitions. HIDA extracts three low-resolution spatial cues: a Sobel-filtered normalized depth gradient magnitude \(g\) capturing depth discontinuities, visible mask boundary \(d\), and downsampled mask \(m\), forming an occlusion-sensitive routing signal: $\(occ = [\sigma(g + d) \parallel d \parallel m]\)$ Within each lightweight ViT layer, token feature \(x_{i,j}\) at position \((i, j)\) is reduced via a \(1 \times 1\) pointwise convolution to \(z\), concatenated with \(occ\), and fed into router \(R\) to produce continuous softmax gating weights \(g_{k,i,j} = \text{softmax}(R([z \parallel occ]))_k\). The outputs of \(K\) lightweight convolutional experts \(E_k\) are blended into a convex combination and added via residual connection: $\(y_{i,j} = \sum_{k=1}^K g_{k,i,j} E_k(x_{i,j}) \quad \text{s.t.} \quad g_{k,i,j} \ge 0, \sum_{k=1}^K g_{k,i,j} = 1\)$ Homogeneous interior and distinct background regions are handled by dominant single experts, whereas occlusion boundaries and ambiguous hidden zones dynamically engage multiple complementary experts to ensure smooth feature extrapolation across occlusion interfaces.

3. Dual Foundation Model Integration with Lightweight Network: complementary decoding without fine-tuning

To reconcile open-world generalization with training tractability, HIDA keeps both SAM (ViT-H) and Depth Anything V2 (Large) completely frozen without back-propagating gradients into their weights. All foundation priors are precomputed and cached offline. Only the lightweight ViT encoder (4–8 layers), soft MoE modules, and a lightweight mask decoder participate in gradient updates. Cropped RGB images are forwarded through the frozen SAM encoder, and the resulting rich semantic representations are concatenated channel-wise with the geometry-aware features from the lightweight ViT encoder. This design harnesses SAM's open-set representation while compensating for its lack of invisible completion capabilities, constraining trainable parameters to only 77.86M.

Loss & Training

The framework is trained on the synthetic occlusion benchmark pix2gestalt (849,667 instances derived from SA-1B) in a class-agnostic manner. Cropped images are resized to \(256 \times 256\) without data augmentation. The total loss comprises amodal segmentation loss \(\mathcal{L}_{seg}\) and a load-balancing regularizer \(\mathcal{L}_{LB}\): $\(\mathcal{L} = \mathcal{L}_{seg} + \mathcal{L}_{LB}\)$ The segmentation objective combines binary cross-entropy loss and Dice loss: $\(\mathcal{L}_{seg} = \mathcal{L}_{BCE}(\tilde{m}^a, m^a) + \mathcal{L}_{Dice}(\tilde{m}^a, m^a)\)$ To prevent gating collapse toward a degenerate expert subset, a load-balancing term is applied using batch- and spatially-averaged gating coefficients \(\bar{g}_k = \frac{1}{B \times H \times W} \sum_{b,i,j} g_{b,i,j,k}\): $\(\mathcal{L}_{LB} = -\sum_{k=1}^K \bar{g}_k \log(\bar{g}_k + \epsilon)\)$ Optimization is performed using AdamW on a single NVIDIA RTX 4090 for 15 epochs with a batch size of 32, an initial learning rate of \(1 \times 10^{-4}\), weight decay of \(1 \times 10^{-4}\), and a cosine annealing schedule. Zero-shot evaluation directly uses these weights, whereas supervised benchmarks initialize from them and perform in-domain fine-tuning.

Key Experimental Results

Main Results

Evaluations are conducted on three established benchmarks: the COCOA-cls validation set, D2SA validation set, and KINS test set (evaluating vehicles with height \(> 50\text{px}\)). Performance is measured via mean Intersection-over-Union (mean IoU, %).

Method	Type / Venue	COCOA-cls (val)	D2SA (val)	KINS (test)
GIN	Supervised (TMM 2023)	72.46	78.23	68.31
SAM	Zero-shot visible baseline (ICCV 2023)	73.10	84.65	75.88
C2F-Seg	Supervised (ICCV 2023)	80.28	89.10	82.22
SDXL-Inpainting†	Zero-shot inpainting (ICLR 2024)	73.65	80.53	76.19
Pix2Gestalt†	Zero-shot generative (CVPR 2024)	79.08	81.82	81.45
SDAmodal	Supervised (CVPR 2024)	80.01	-	-
SAMBA†	Zero-shot tuned SAM (CVPR 2025)	81.82	90.98	88.47
HIDA† (Ours Zero-Shot)	Zero-shot	83.16	91.11	86.71
HIDA (Ours Supervised)	Supervised	85.56	93.33	91.50

Note: † designates zero-shot evaluation methods.

Ablation Study

1. Decoder feature sources and pseudo-depth contribution (COCOA-cls val, zero-shot)

Index	SAM Enc.	ViT Enc.	Pseudo-depth (PDM)	IoU (%)	Training Time / Epoch (h)
1	✓	✗	–	60.90	3.5
2	✗	✓	✓	81.56	5.3
3	✓	✓	✗	81.78	6.1
4 (Full)	✓	✓	✓	83.16	6.1

2. Number of experts in soft MoE and ViT encoder depth (COCOA-cls val)

Ablation Factor	Setting	Enc. Params (M)	Enc. FLOPs (G)	IoU (%)	Train Time / Epoch (h)
No. of Experts \(K\)	\(K=1\)	60.29	17.68	81.98	-
	\(K=2\)	61.93	18.10	82.69	-
	\(K=4\) (Default)	65.23	18.95	83.16	6.1
	\(K=8\)	71.87	20.65	82.97	-
ViT Depth	4 layers	6.43	3.89	82.18	4.5
	6 layers	24.81	8.58	83.05	5.1
	8 layers (Default)	65.23	18.95	83.16	6.1

Key Findings

• Crucial role of pseudo-depth geometry: Zeroing out the pseudo-depth input drops zero-shot IoU from 83.16% to 81.78% (-1.38%), confirming that relative front-back depth ordering provides essential geometric guidance for amodal contour extrapolation.
• Optimal capacity in soft MoE: Increasing expert count from 1 to 4 steadily boosts IoU from 81.98% to 83.16%. Scaling further to 8 experts causes performance to saturate and slightly decline to 82.97%, indicating that redundant experts complicate routing optimization and induce overfitting.
• Pronounced supervised gains in domain-specific scenes: Fine-tuning HIDA achieves 93.33% on D2SA (+4.23% over C2F-Seg) and 91.50% on KINS (+9.28% over C2F-Seg), illustrating that depth-guided completion is exceptionally powerful in structured, heavy-occlusion scenarios.

Highlights & Insights

• Tuning-free foundation model reuse: Unlike SAMBA which alters SAM's internal structure or Pix2Gestalt which fine-tunes an 860M diffusion backbone, HIDA keeps SAM ViT-H and Depth Anything V2 Large completely frozen. Training only 77.86M lightweight parameters on a single RTX 4090 yields competitive performance without catastrophic forgetting.
• Depth-polarity asymmetric cropping: Rather than applying naive uniform bounding box enlargement, HIDA uses directional depth contrast to expand specifically toward the occluding side, maximizing signal-to-noise ratio and foreground alignment.
• Occlusion-aware soft MoE dispatch: Injecting depth gradients and contour boundaries into the router enables shallow layers to establish sharp boundary-versus-interior specialization while deeper layers achieve continuous structural aggregation.

Limitations & Future Work

• Depth aliasing under tight contact: When elongated objects closely contact nearer occluders, monocular depth models may produce blurry discontinuities or even inverted depth ordering, causing HIDA to over-extend into occluders or under-complete hidden silhouettes. Incorporating explicit physical boundary reasoning could alleviate this issue.
• Inference latency bottleneck: Running full-scale SAM ViT-H and Depth Anything V2 in serial requires ~0.8s per instance on an RTX 4090, limiting real-time deployment in high-frequency autonomous perception pipelines. Future iterations could explore distilled, lightweight depth and segmentation backbones.

Related Work & Insights

• vs SAMBA (CVPR 2025): SAMBA fine-tunes the heavy SAM backbone end-to-end, incurring massive computational costs; HIDA freezes both foundation models and achieves superior zero-shot performance on COCOA-cls and D2SA with a lightweight MoE network.
• vs Pix2Gestalt (CVPR 2024): Pix2Gestalt employs an 860M diffusion model that suffers from high latency and potential geometric hallucination; HIDA uses deterministic depth polarity and soft MoE to achieve sharper, geometrically consistent masks.
• vs C2F-Seg (ICCV 2023): C2F-Seg relies on domain-specific coarse-to-fine priors; HIDA generalizes seamlessly across open-world settings by leveraging universal monocular depth cues.

Rating

• Novelty: ⭐⭐⭐⭐ [Ingenious combination of depth polarity with soft MoE routing for amodal completion]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive evaluation across three benchmarks under both zero-shot and supervised settings]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation grounded in human visual intuition, well-structured method formulation]
• Value: ⭐⭐⭐⭐ [Demonstrates an efficient, plug-and-play paradigm for deploying frozen foundation models in specialized vision tasks]

Related Papers

• [ECCV 2026] XSemanticFlow: Cross Object Semantic Alignment for Zero-shot Manipulation
• [CVPR 2026] Learning and Aligning Click-Aware Shape Prior for Interactive Amodal Instance Segmentation
• [ICLR 2026] Object-Centric Refinement for Enhanced Zero-Shot Segmentation
• [AAAI 2026] RSVG-ZeroOV: Exploring a Training-Free Framework for Zero-Shot Open-Vocabulary Visual Grounding in Remote Sensing Images
• [CVPR 2026] DSS: Discover, Segment, and Select for Zero-shot Camouflaged Object Segmentation