SARIF: Segment Anything for Robust Image Forensics¶

Conference: ECCV 2026
arXiv: 2606.21108
Code: https://github.com/...
Area: Semantic Segmentation
Keywords: Image Forensics, Forgery Localization, SAM, Dual-Encoder Residual, Feedback-Guided Decoding

TL;DR¶

SARIF simultaneously runs a frozen original SAM encoder and a LoRA fine-tuned encoder, calculating the residuals between them at each global attention layer as forgery-specific cues. These are then fused with the predicted mask from the previous round into prompts, driving SAM's original lightweight decoder to perform 5-step progressive refinement, achieving the strongest average performance in cross-dataset forgery localization.

Background & Motivation¶

The ubiquity of image editing tools and generative models has made it easy for ordinary users to produce highly realistic manipulated images, posing severe challenges to misinformation identification, judicial forensics, and public trust. Early image forensics relied on hand-crafted features (such as illumination inconsistencies, JPEG compression artifacts, sensor pattern noise, etc.), but was limited to image-level detection and exhibited extremely poor generalization capabilities when facing diverse forgery types. With the rise of deep learning, models like MantraNet and SPAN achieved pixel-level forgery localization but suffered from high computational overhead and limited representation capacity. Subsequent Transformer-based models (such as TransForensics and M2SFormer) and attention mechanisms improved efficiency and accuracy, yet their robustness under cross-domain generalization and image degradation (noise, blur, JPEG compression) remains highly unsatisfactory. Meanwhile, real-world forged images originate from highly diverse sources—ranging from simple copy-move tampering to AI-generated content—posing a generalization gap that remains a core bottleneck for practical applications.

Recently, vision foundation models like Segment Anything Model (SAM), with their strong generalization capabilities trained on billion-scale mask annotations, have introduced new directions for forgery localization. Existing works such as IMDPrompter, SAMIF, and SAFIRE have attempted to introduce SAM into this task. However, two common issues persist: first, they still rely on manual prompts or grid points (confining localization to object boundaries rather than actual forgery traces); second, they only extract output from SAM's final embedding layer, ignoring different granularities of forgery cues potentially contained in intermediate hierarchical layers. A deeper problem is that while single-branch LoRA fine-tuning is parameter-efficient, it cannot explicitly answer "what forgery-related information the adapter has actually learned"—meaning semantic content and forgery traces remain entangled in the encoder's output, making them difficult to separate.

The key insight of SARIF is that if two copies (pre- and post-fine-tuning) are run simultaneously, the difference between them naturally represents the "incremental knowledge learned by fine-tuning," which points to forgery patterns more easily than a single feature. This design is inspired by the "opponent-channel" mechanism in biological vision—where the visual system encodes relative contrast rather than absolute intensity. Core Idea: Simultaneously run both a frozen and a LoRA-tuned SAM encoder, isolating the forgery cues learned by the adapter through layer-by-layer residual features. These cues are then fused with the predicted mask from the previous round to form SAM prompts, driving a lightweight decoder to perform progressive refinement, achieving fully automatic and prompt-free forgery localization.

Method¶

Overall Architecture¶

The architecture of SARIF consists of three main pipelines: a completely frozen original SAM image encoder, an encoder copy embedded with LoRA adapters for parameter-efficient fine-tuning, and a feedback-guided decoder that reuses SAM's native prompt interface. The input image is simultaneously fed into both encoders, yielding two parallel sets of features. At four global attention blocks of the ViT (the 5th, 11th, 17th, and 23rd layers) and the final embedding layer, FSIE (Forgery Specific Information Extractor) calculates the residuals between the two feature sets layer by layer—these residuals represent the incremental information learned by LoRA for the forgery task. The residual features of each layer are sequentially injected into FGMD (Feedback-Guided Mask Decoder): FGMD encodes the predicted mask from the previous round via the SAM prompt encoder as a mask prompt, which is element-wise added to the current layer's forgery feature to obtain a task prompt. This task prompt, along with the final embedding of the fine-tuned encoder, is fed into SAM's original lightweight decoder to produce a more refined prediction for the current round. This process is repeated for 5 steps, progressively converging from coarse to fine to yield the final localization mask.

flowchart TD
    A["Input Image"] --> B1["Frozen SAM Encoder"]
    A --> B2["LoRA-adapted SAM Encoder"]
    B1 --> C["FSIE<br/>5+1 Hierarchical Residual Extraction"]
    B2 --> C
    C -->|"5th/11th/17th/23rd Block<br/>+ Final Embedding"| D["FGMD<br/>Prompt Encoding + Decoding<br/>×5-step Refinement"]
    D -->|"Previous Mask<br/>as Prompt"| D
    D --> E["Final Forgery Mask"]

Key Designs¶

1. Dual-Encoder Residuals: Isolating Forgery Cues via Feature Differences Before and After Fine-Tuning

Although a single-branch LoRA adapter can achieve domain adaptation with extremely minimal parameter overhead, its output entangles semantic content and forgery traces, making it impossible to separate "which specific part of the features is activated strictly due to the forgery task." SARIF's solution is to treat the frozen original encoder as a baseline reference and compare the fine-tuned encoder against it layer by layer—the difference between them naturally represents the incremental knowledge learned by LoRA, which points more directly to forgery-related change patterns than a single feature. This design is inspired by the "opponent channel" and "center-surround" mechanisms in biological vision: the visual system encodes relative contrast rather than absolute intensity.

Specifically, FSIE extracts two sets of features, \(O_t\) (frozen branch) and \(F_t\) (fine-tuned branch), at each selected layer \(t\). It first computes the position-wise cosine similarity map between them as a coarse-grained measurement of the discrepancy. Then, a \(1 \times 1\) convolution compresses each set of features into a lower dimension \(C'\) to reduce computational load. The two compressed features are concatenated, and channel attention (global average pooling and global max pooling, each followed by two \(1 \times 1\) convolutions, summed and passed through a sigmoid to generate channel weights) is applied to the concatenated result to emphasize channels with significant discrepancies. Finally, the cosine similarity map is concatenated with the channel-weighted features and fused through a lightweight Conv-BN-GeLU residual block to form a compact forgery feature \(T_{fs}^t\). FSIE outputs such a feature set at each of the 5th, 11th, 17th, and 23rd blocks, as well as the final embedding layer, establishing hierarchical cues ranging from large-scale semantic inconsistencies to local texture anomalies, providing multi-granularity feature inputs for subsequent refinement.

2. Feedback-Guided Decoding: Progressive Refinement Driven by Historical Masks as SAM Prompts

Conventional SAM variants for forgery localization often perform only one-time predictions without utilizing SAM’s own prompt interface for iterative refinement. SARIF's design is to take the predicted mask from the previous round as input to the SAM prompt encoder in each iteration, generating a mask prompt. This is element-wise added to FSIE's current-layer forgery feature to form a task prompt, which is then fed into SAM's original lightweight decoder to predict the current round's mask.

The entire process starts with an initial mask: the output of the decoder using only the final embedding of the fine-tuned encoder under no external prompts. The inputs for the \(t\)-th refinement round include: (i) the FSIE output at the \(t\)-th selected global attention block, and (ii) the prompt-encoded embedding of the predicted mask from the \((t-1)\)-th round. These two are added and input into the SAM mask decoder alongside the final embedding of the fine-tuned encoder to yield the \(t\)-th round prediction. This process is executed in 5 steps sequentially along the hierarchical order of FSIE (5th \(\rightarrow\) 11th \(\rightarrow\) 17th \(\rightarrow\) 23rd block \(\rightarrow\) final embedding), with each round's output serving as the mask prompt for the next round without any manual intervention. Experiments show that mask quality steadily improves with the number of steps: on the seen domain CASIAv2, the initial IoU is 53.3%, which reaches 56.7% after 5-step refinement; a consistent improvement trend is also observed on unseen domains, where earlier steps yield more pronounced gains while later stages tend to saturate. In addition, the output of each round is supervised by a BCE loss (deep supervision), which not only accelerates convergence but also ensures stable propagation of gradients for refinement.

Loss & Training¶

SARIF is based on the SAM ViT-L pretrained weights, with LoRA adapters (rank=32) inserted at the selected global attention blocks. During training, only the LoRA parameters and the mask decoder are updated, while the remaining backbone is completely frozen. Training uses the CASIAv2 dataset, with the Adam optimizer (initial learning rate of \(1e-4\), cosine annealing to \(1e-6\)), a batch size of 32, for 100 epochs, and inputs uniformly scaled to \(256 \times 256\). The loss function is the sum of BCE losses across 6 stages (initial + 5-step refinement).

Key Experimental Results¶

Main Results¶

DSC comparison on the seen domain and 7 unseen domains after training on CASIAv2 (part of representative methods):

Dataset	SAM	AutoSAM	M2SFormer	SAFIRE	SARIF
CASIAv2 (Seen)	27.1	49.0	58.8	56.8	63.1
CASIAv1 (Unseen)	33.4	45.1	58.4	61.6	58.4
Columbia (Unseen)	24.6	28.2	42.4	53.1	58.4
CoMoFoD (Unseen)	19.7	23.8	24.9	39.5	65.4
DIS25k (Unseen)	18.7	31.4	38.5	50.7	47.8
IMD2020 (Unseen)	22.3	34.0	32.6	53.1	48.4
In the Wild (Unseen)	29.6	36.5	35.0	63.3	55.7
MISD (Unseen)	31.3	61.3	69.1	48.3	66.2

SARIF achieves the best results on the seen domain, Columbia, and CoMoFoD, with its lead on CoMoFoD being particularly remarkable (+25.9 DSC), achieving the highest overall average DSC. On the latest challenging datasets (CocoGlide inpainting and TGIF AI-generated forgery), SARIF achieves DSC scores of 34.0 and 30.8, respectively, comprehensively outperforming comparative methods.

Ablation Study¶

Configuration	Seen Domain DSC	Unseen Domain Avg. DSC	Parameters	FLOPs
SAM only (No LoRA / No Refinement)	23.6	23.3	13.90M	490.65G
+LoRA	41.2	33.0	34.90M	503.79G
+LoRA+FSIE	62.4	44.7	41.04M	994.88G
+LoRA+FGMD	62.0	42.6	41.34M	506.13G
SARIF (Ours)	63.1	57.9	52.62M	998.43G

Key Findings¶

The LoRA adapter itself brings substantial improvements (seen domain DSC +17.6), showing that parameter-efficient fine-tuning is an effective baseline for cross-domain forgery localization.
FSIE and FGMD yield comparable gains on unseen domains when introduced individually (+11.7 and +9.6 DSC), but their combination triggers a leap in unseen domain DSC from 44.7/42.6 to 57.9 (+13.2/+15.3), demonstrating that "high-quality features" and "iterative refinement" form a strong complementary positive feedback loop.
The false positive rate (FPR) on authentic images is 5.1%, which is the lowest among SAM-based methods, indicating that SARIF does not cause extensive false activations in untampered regions.
The refinement curve (Table 8) shows that earlier steps yield larger gains while later stages tend to saturate, allowing real-world deployment to truncate steps as needed to trade off speed.

Highlights & Insights¶

Dual-encoder residuals can generalize to other "pre-training + fine-tuning" paradigms: Any task requiring the separation of incremental knowledge from fine-tuning (e.g., domain adaptation, anomaly detection, fine-grained classification) can replicate this design—using the frozen original model as a baseline and highlighting incremental signals via residuals.
Reusing instead of rewriting SAM's prompt interface: Rather than designing an entirely new and complex decoder, SARIF integrates the minimalist modification of "wrapping the previous round's mask as a feedback prompt" directly into SAM's native architectural loop, incurring negligible overhead yet contributing significantly to accuracy.
Coarse-to-fine injection of hierarchical cues: Sequentially injecting residual features into the 5th \(\rightarrow\) 11th \(\rightarrow\) 17th \(\rightarrow\) 23rd \(\rightarrow\) final layers naturally constructs a progressive localization pathway from large-scale semantic inconsistency to local texture anomalies, utilizing SAM's hierarchical representation more thoroughly than using only the final embedding.
Massive lead on CoMoFoD (copy-move forgery): SARIF achieves a high DSC of 65.4 on CoMoFoD, far exceeding SAFIRE's 39.5 and the runner-up M2SFormer's 24.9. The hypothesized reason is that inter-region similarities in copy-move forgery are highly discernable under the dual-encoder residual mechanism.

Limitations & Future Work¶

Initial mask quality is a bottleneck: If the initial mask locates an incorrect region, subsequent refinements will magnify the error—turning the advantage of feedback into a disadvantage under faulty initialization. A potential remedy is introducing confidence validation to switch to a "re-initialization" branch when initial mask confidence is low.
High dual-encoder inference overhead: Running two ViT-L encoders simultaneously along with 5-step decoding results in approximately 998G FLOPs and an inference time of 47.5ms, which is nearly 20 times that of M2SFormer (2.5ms), limiting its adoption in real-time scenarios.
Staircase artifacts: The SAM prompt encoder projects maps to low resolution and then upsamples them via transposed convolutions, yielding blocky staircase artifacts that affect fine boundaries. Introducing a learnable upsampling layer could improve this.
Weak response on authentic images: On completely unhampered authentic images, SARIF still produces sparse false activations. Although its false positive rate is the lowest among SAM-based methods (5.1%), it still warrants caution for strict forensic applications.

vs IMDPrompter: IMDPrompter learns a multi-view prompt pool and automatically generates prompts using an optimal selection mechanism, but requires a multi-view encoder and auxiliary modules. SARIF directly derives prompts from the residuals of the frozen/fine-tuned encoders, featuring a simpler structure.
vs SAFIRE: SAFIRE uses grid-point prompts for source region segmentation and aggregates predictions, making its localization granularity constrained by grid density. SARIF's continuous mask refinement offers distinct advantages in scenarios requiring fine boundaries (such as CoMoFoD).
vs SAMIF: SAMIF introduces a high-frequency branch and SRM filters to enhance high-frequency cues but only utilizes SAM's final embedding. SARIF leverages hierarchical residuals to provide a coarse-to-fine feature flow, thereby enabling stronger generalization on unseen domains.

Rating¶

Novelty: ⭐⭐⭐⭐ The combination of dual-encoder residuals and feedback-guided refinement shows clear novelty in image forensics. The motivation is inspired by visual neuroscience, and the design implementation is clean.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 8 datasets (1 seen + 7 unseen) + various degradation conditions (noise/blur/JPEG/mixed) + latest challenging datasets (CocoGlide/TGIF) + exhaustive ablations + false positive analysis + refinement stage analysis; the experiments are highly thorough.
Writing Quality: ⭐⭐⭐⭐ The motivational chain and design motivation are clearly articulated, and diagrams facilitate understanding; however, the formulaic descriptions (specifically the channel attention sequence) are slightly dense and could be further streamlined.
Value: ⭐⭐⭐⭐ It provides an effective solution for fully automated localization using SAM in the forensic domain. The concept of dual-encoder residuals could be generalized to other downstream tasks requiring the extraction of incremental knowledge from pretrained models.