FoundYou: A Unified Model for Personalized Segmentation and Retrieval¶

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://ga1i13o.github.io/FoundYou/
Code: https://ga1i13o.github.io/FoundYou/
Area: Segmentation
Keywords: Personalized Segmentation, Personalized Retrieval, Instance Alignment, SAM 2, Parameter-Efficient Fine-Tuning

TL;DR¶

FoundYou repurposes the memory attention mechanism of a frozen SAM 2 for instance alignment across independent images, achieving a unified 52M-parameter personalized segmentation and retrieval model with only 5.9M trainable parameters running at 90 FPS (over 75x faster than PDM) while outperforming prior unified and task-specific baselines by +18.4 mIoU on PerMIS and +17.8 mAP on ILIAS.

Background & Motivation¶

Personalized visual understanding aims to locate and identify a specific physical instance—such as a user's own handbag, personal pet, or particular household object—across completely independent images and unrelated scenes. This overarching objective naturally manifests as two distinct problems: personalized segmentation requires predicting a dense, pixel-accurate binary mask for the target instance within a query image, whereas personalized retrieval requires ranking gallery images based on whether they depict that exact physical object. Despite sharing the identical underlying challenge of instance-level discrimination, the two tasks have historically evolved in silos. Segmentation-focused approaches such as PerSAM or Matcher rely on correspondence matching via DINOv2 or diffusion features followed by SAM prompting, yet they struggle to separate identical semantic classes with fine visual differences. Conversely, retrieval-specific methods such as AMES and RoMav2 optimize for feature re-ranking and geometric verification, but completely lack dense spatial localization capabilities. PDM, the only prior unified attempt, chains multi-timestep Stable Diffusion features with SAM mask generation, leading to an unwieldy 1200M-parameter pipeline running at a sluggish 1.2 FPS.

The root tension behind this divide lies in the limitations of available foundation models: general vision models either abstract toward high-level semantic categories (collapsing fine instance identities) or rely on rigid geometric overlap across views. In contrast, Segment Anything 2 (SAM 2), trained to track objects continuously across video frames, implicitly learns rich, instance-discriminative representations within its memory bank and cross-attention blocks. However, SAM 2 was trained under a strict spatio-temporal continuity prior: frame-to-frame motions are small, and background context remains stable. When presented with distinct, independent photographs lacking contextual or spatial continuity, SAM 2's native representations suffer noticeable degradation. Furthermore, SAM 2 has no built-in mechanism to assign a global, image-level instance presence score suitable for candidate ranking.

This paper's angle of attack is to unlock and adapt SAM 2's memory mechanism so that it bridges independent images rather than consecutive video frames. Core idea: keep the SAM 2-small backbone and segmentation decoder frozen, insert lightweight AdaptFormer blocks in the late image encoder stages to remove the spatio-temporal continuity bias, and append a compact bidirectional cross-attention retrieval head to aggregate dense memory features into global similarity scores; regularized by offline DINOv2 semantic hard-negative mining and mask self-distillation, FoundYou achieves unified personalized segmentation and retrieval with only 5.9M trainable parameters.

Method¶

Overall Architecture¶

FoundYou takes as input a visual prompt \(V_r = (I_r, A_r)\), where \(I_r\) is a reference image and \(A_r\) specifies the instance via a mask, bounding box, or single point, alongside a target image \(I_t\). For personalized segmentation, it predicts a binary mask \(M_t \in \{0, 1\}^{H \times W}\); for personalized retrieval, it computes an instance existence probability score \(s_t \in [0, 1]\).

The architecture is built upon a frozen SAM 2-small foundation. An adapted Image Encoder extracts reference features \(F_r\) and target features \(F_t\). The Memory Encoder combines \(F_r\) with the reference mask (either provided directly or predicted via prompt) into an instance memory representation \(S_r\), stored in the Memory Bank. Target features \(F_t\) attend to \(S_r\) through Multi-Head Cross Attention (Memory Attention), generating dense memory-conditioned features \(F_t^{\text{mem}}\). These conditioned features are then dispatched to two distinct heads: the frozen SAM 2 Segmentation Decoder to produce fine masks, and a lightweight Retrieval Decoder to aggregate spatial correspondences into a scalar similarity score.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Inputs: Visual Prompt Vr & Target Image It"] --> B["Feature Adaptation Mechanism<br/>AdaptFormer removes temporal continuity bias"]
    B --> C["Memory Encoding & Memory Attention<br/>Produces memory-conditioned features Ft_mem"]
    C --> D["Bidirectional Cross-Attention Retrieval Head<br/>Aggregates global similarity score st"]
    C --> E["Frozen Segmentation Decoder<br/>Outputs target instance binary mask Mt"]
    F["Mask Self-Distillation & Hard-Negative Mining"] -.->|Regularization & Intra-class contrast| B
    F -.->|BCE supervision| D

Key Designs¶

1. Feature Adaptation Mechanism: Mitigating the Spatio-Temporal Continuity Bias

Because SAM 2 is pretrained on continuous video sequences, its internal representations encode a strong positional and contextual continuity bias, making its feature similarity degrade rapidly under substantial spatial translation or abrupt background shifts. To adapt these features for matching across independent scenes without corrupting their rich pretrained geometric knowledge, FoundYou inserts AdaptFormer residual bottleneck blocks into the final two transformer stages (blocks 2–3) of the Image Encoder. For each intermediate token vector \(x_{\text{self}}\) output by the self-attention layer, the AdaptFormer branch applies a down-projection \(W_{\text{down}} \in \mathbb{R}^{d \times \tilde{d}}\), a ReLU non-linearity, and an up-projection \(W_{\text{up}} \in \mathbb{R}^{\tilde{d} \times d}\) back to the residual pathway:

\[x' = x_{\text{self}} + \text{MLP}(x_{\text{self}}) + \text{ReLU}(x_{\text{self}} W_{\text{down}}) W_{\text{up}}\]

With the bottleneck dimension \(\tilde{d}\) set to \(0.5 \times d\), this module alters only the high-level semantic token interactions while keeping all original backbone weights frozen. This selective adaptation effectively insulates the representation against arbitrary viewpoint variations, background replacement, and object displacement across distinct images.

2. Bidirectional Cross-Attention Retrieval Head: Aggregating Dense Matches into Global Scores

Although SAM 2 outputs an occlusion score to indicate object visibility during video tracking, this prediction assumes temporal smoothness and collapses when applied directly to independent image retrieval (yielding an unusable 3.5 mAP). Nevertheless, the spatial memory features \(F_t^{\text{mem}} = \text{MHCA}(Q(F_t), K(S_r), V(S_r))\) already encode localized evidence of instance correspondence. FoundYou introduces a streamlined Retrieval Decoder that discards the mask upsampling convolutional layers and mask-specific tokens of the original decoder. It introduces a single learnable retrieval query token \(z^{(0)} \in \mathbb{R}^{1 \times D}\), which conducts two alternating bidirectional cross-attention exchanges with the flattened memory feature sequence \(F^{(0)} \in \mathbb{R}^{L \times D}\) (\(L = \frac{H}{16} \cdot \frac{W}{16}\)):

\[\begin{aligned} F^{(k)} &= \text{MHCA}(F^{(k-1)}, z^{(k-1)}) \\ z^{(k)} &= \text{MLP}(\text{MHCA}(z^{(k-1)}, F^{(k)})), \quad k \in \{1, 2\} \end{aligned}\]

The refined token \(z^{(2)}\) passes through a shallow projection layer and a sigmoid activation to yield the final instance presence probability \(s_t = \text{sigmoid}(\text{MLP}_{D \to 1}(z^{(2)}))\). This design requires negligible computation while effectively filtering background distractors. Furthermore, because reference features reside in the Memory Bank, providing multiple reference views in a few-shot retrieval scenario requires only concatenating memory tokens along the sequence dimension, allowing candidate targets to cross-attend to multi-view evidence simultaneously at inference time without architectural changes.

3. Mask Self-Distillation and Semantic Hard-Negative Mining: Balancing Discrimination and Mask Precision

Training the model with random negative pairs (such as pairing a handbag with an automobile) provides trivial learning signals that fail to teach the model how to distinguish between different instances within the same semantic category. Performing online hard-negative mining during training, on the other hand, imposes prohibitive latency overheads. FoundYou solves this by leveraging a frozen DINOv2 backbone \(\phi(\cdot)\) offline: for each reference image \(I_r\), it pre-selects the top-\(K\) non-matching candidates exhibiting the highest global feature similarity as hard negatives:

\[N_r = \text{argmax top-}K_{I_k \notin P_r} (\phi(I_r)^\top \phi(I_k))\]

This ensures that the binary cross-entropy retrieval loss \(\mathcal{L}_{\text{ret}}\) directly targets fine-grained intra-class discrepancies.

Simultaneously, optimizing adapters exclusively with an image-level retrieval loss can erode the fine spatial localization properties needed for segmentation. To prevent this degradation, FoundYou introduces a mask-based self-distillation objective. Treating the unmodified, frozen SAM 2 as a teacher and the adapted FoundYou as a student, the model computes foreground probability maps \(P^T, P^S \in \mathbb{R}^{H \times W}\) for each image and enforces consistency using a pixel-wise Kullback-Leibler divergence:

\[\mathcal{L}_{\text{dist}} = \frac{1}{HW} \sum_{h, w} \text{KL}\big(P^T(h, w) \parallel P^S(h, w)\big)\]

The joint optimization objective \(\mathcal{L} = \mathcal{L}_{\text{ret}} + \lambda_{\text{dist}} \mathcal{L}_{\text{dist}}\) (with \(\lambda_{\text{dist}} = 0.5\)) harmonizes strong instance-level retrieval ranking with pristine boundary segmentation accuracy.

Loss & Training¶

The framework is trained on the UnED dataset using the Adam optimizer with a learning rate of \(1 \times 10^{-4}\) and batch size 16 for 150k iterations. The SAM 2-small backbone (46M parameters) remains fully frozen, with only the 5.9M adapter and retrieval head parameters updated. Each query instance is paired with 4 positives and 8 offline-mined hard negatives.

Key Experimental Results¶

Main Results¶

FoundYou is evaluated across personalized segmentation (PerSeg, PerMIS) and personalized retrieval benchmarks (PerMIR, ILIAS). On ILIAS (100M distractor pool), candidate re-ranking is applied to the top-1k candidates retrieved by SigLIP (SigLIP alone scores 19.6 mAP).

Category	Model	PerSeg (mIoU / bIoU)	PerMIS (mIoU / bIoU)	PerMIR (mAP)	ILIAS (mAP)	Speed (FPS)	Params (M)
Segmentation-only	SEEM (NeurIPS'23)	87.1 / 55.7	34.2 / 31.8	n.a.	n.a.	8.5	341
Segmentation-only	SegGPT (ICCV'23)	94.3 / 76.5	38.7 / 35.5	n.a.	n.a.	12.7	354
Segmentation-only	PerSAM (ICLR'24)	89.3 / 71.7	42.1 / 34.4	n.a.	n.a.	2.5	945
Segmentation-only	PerSAM-F (ICLR'24)	95.3 / 77.9	47.8 / 34.5	n.a.	n.a.	2.5	945
Segmentation-only	GF-SAM (NeurIPS'24)	91.7 / 74.2	45.6 / 38.1	n.a.	n.a.	1.5	945
Retrieval-only	RRT (ICCV'21)	n.a.	n.a.	41.7	9.2	31.0	50
Retrieval-only	MatchAnything (ArXiv'25)	n.a.	n.a.	19.7	23.2	3.4	111
Retrieval-only	MASt3R (ECCV'24)	n.a.	n.a.	37.7	25.3	1.7	688
Retrieval-only	RoMav2 (ArXiv'25)	n.a.	n.a.	32.3	26.6	2.7	425
Retrieval-only	AMES (ECCV'24)	n.a.	n.a.	38.2	26.4	90.9	89
Segmentation + Retrieval	PDM (NeurIPS'24)	89.5 / 72.0	44.2 / 38.0	65.3	14.7	1.2	1200
Segmentation + Retrieval	FoundYou (Ours)	96.4 / 85.6	62.6 / 57.4	92.1	32.5	90.2	52

In the proposed few-shot personalized retrieval setting on ILIAS, FoundYou aggregates multiple reference views directly via its memory bank, delivering substantial gains compared to score-averaging baselines:

Model	1-shot (mAP@1k)	2-shot (mAP@1k)	3-shot (mAP@1k)	4-shot (mAP@1k)
SigLIP Baseline	22.9	22.9	22.9	22.9
XFeat	15.5	16.5	17.4	17.1
RoMa	22.0	21.9	22.1	21.9
SP-LightGlue	22.9	23.0	23.3	22.4
ELoFTR	23.8	24.6	25.7	25.0
MASt3R	24.3	26.0	27.2	27.5
MatchAnything	25.2	26.3	27.2	26.8
RoMa v2	27.5	28.2	28.4	27.7
AMES	27.6	28.2	29.0	30.2
FoundYou (Ours)	30.1	31.8	32.5	34.2
Oracle	38.9	38.9	38.9	38.9

Ablation Study¶

The following table summarizes the ablation analysis across fine-tuning schemes, adapter architectures, and adapter insertion stages evaluated on PerMIS segmentation (mIoU, %) and ILIAS retrieval (mAP, %):

Investigation	Configuration	PerMIS (mIoU)	ILIAS (mAP)	Note
Tuning Strategy	Frozen SAM 2 (Occlusion Head)	60.2	3.5	Baseline occlusion score fails on non-adjacent images
Tuning Strategy	Full Fine-Tuning	27.8	13.8	Severe catastrophic forgetting of pretrained weights
Tuning Strategy	Backbone Fine-Tuning	57.2	20.5	Degrades both segmentation and retrieval accuracy
Tuning Strategy	QKV Projection Fine-Tuning	59.8	27.2	Underperforms lightweight modular adapter
Tuning Strategy	Adaptation w/o Distillation	56.7	32.7	Lacks spatial regularizer; segmentation drops -5.9 mIoU
Tuning Strategy	FoundYou (Adapt + Distill)	62.6	32.5	Best overall trade-off across both modalities
Adapter Architecture	LoRA	-	28.9	Suboptimal adaptation capacity
Adapter Architecture	Standard Adapter	-	30.2	Moderate improvement
Adapter Architecture	AdaptFormer (0.3x Bottleneck)	-	29.8	Capacity slightly constrained
Adapter Architecture	AdaptFormer (0.8x Bottleneck)	-	30.5	Marginal overfitting
Adapter Architecture	AdaptFormer (0.5x Bottleneck)	-	32.5	Optimal retrieval performance
Adapter Placement	Early Blocks (0–1)	-	21.7	Premature alteration of low-level geometric tokens
Adapter Placement	Middle Blocks (1–2)	-	29.6	Noticeable retrieval gain
Adapter Placement	All Blocks (0–3)	-	31.8	Suboptimal compared to targeted late adaptation
Adapter Placement	Late Blocks (2–3)	-	32.5	Focuses on high-level semantics without harming spatial cues

Key Findings¶

Distillation is the Linchpin of Multi-Task Coexistence: Omitting mask self-distillation preserves retrieval accuracy (32.7 mAP) but causes segmentation to plummet to 56.7 mIoU on PerMIS. Restoring the pixel-wise KL divergence regularizer recovers segmentation performance to 62.6 mIoU (+5.9 gain) with negligible retrieval impact (32.5 mAP).
Targeted Late Adaptation Outperforms Full Tuning: Full fine-tuning disrupts the delicate instance-tracking structures in SAM 2, yielding a meager 27.8 mIoU. Adapting only stages 2–3 with AdaptFormer achieves superior identity discrimination while preserving fine boundary localization.
Robustness to Flexible Prompt Types: On PerMIS, transitioning from full masks to bounding boxes or single point prompts drops performance by only 4.0 mIoU (62.6 to 58.6) for FoundYou, compared to severe degradations in PDM (-9.4 mIoU) and PerSAM-F (-6.2 mIoU). On PerSeg, FoundYou achieves 95.2 mIoU using only single-point visual prompts.

Highlights & Insights¶

Repurposing Video Tracking for Cross-Image Instance Alignment: Rather than treating video segmentation and image retrieval as distinct technical problems, FoundYou demonstrates that video tracking foundation models naturally encode cross-image instance identities once their temporal continuity bias is lifted.
Zero-Overhead Decoupled Negative Mining: By using a frozen DINOv2 model offline to extract intra-category hard negatives, the framework enforces fine-grained instance discrimination during training without requiring slow, periodic online index updates.
Native Few-Shot Multi-View Synergy: Leveraging SAM 2's native multi-token memory bank enables seamless inference-time few-shot aggregation, allowing users to input multiple images of an object to boost retrieval accuracy to 34.2 mAP (+4.1 gain over 1-shot) without model re-training.

Limitations & Future Work¶

Dependency on Initial Coarse Candidates: For multi-million-scale retrieval (e.g., ILIAS 100M), FoundYou operates as a second-stage re-ranker over SigLIP top-1k candidates, meaning that any instance missed in the initial coarse pool cannot be recovered.
Severe Non-Rigid Deformations: While robust to viewpoint and lighting variations, objects undergoing severe physical tearing, extreme non-rigid morphing, or extreme occlusions remain challenging when prompted with a single reference view.
Future Directions: Integrating multimodal LLMs to allow hybrid text-plus-image prompts (e.g., "retrieve this specific handbag, but in a blue variant") and distilling the cross-attention retrieval head into a sub-millisecond vector indexing format.

vs PDM: PDM relies on multi-timestep Stable Diffusion features and cascades with SAM, leading to 1.2B parameters and 1.2 FPS. FoundYou builds directly on SAM 2, shrinking parameters to 52M, accelerating inference by 75x (90.2 FPS), and outperforming PDM by +18.4 mIoU on PerMIS and +17.8 mAP on ILIAS.
vs PerSAM / PerSAM-F: PerSAM relies on superficial feature cosine matching to prompt SAM and lacks global scoring mechanisms or hard intra-class discrimination. FoundYou provides a unified retrieval-segmentation capability with a +14.8 mIoU advantage on crowded scenes.
vs AMES / RoMa v2: Specialized retrieval and matching methods focus on local geometric keypoints or DINOv2 re-ranking, which suffer from perceptual aliasing in cluttered environments and cannot segment targets. FoundYou leverages SAM 2's instance tracking awareness, yielding both superior retrieval (+6.1 mAP over AMES) and dense masks.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant reinterpretation of video memory attention for static cross-image instance alignment.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across segmentation, retrieval, few-shot scaling, prompt flexibility, and category-level transfer.
Writing Quality: ⭐⭐⭐⭐⭐ Clear logical progression, crisp formulation of motivation and trade-offs, and informative figures and tables.
Value: ⭐⭐⭐⭐⭐ Exceptional efficiency (52M parameters, 90 FPS) coupled with state-of-the-art accuracy establishes a practical foundation for real-world personal visual search and robotics.