Trustworthy Image Authentication using Forensic Knowledge Graphs¶

Conference: ECCV 2026
arXiv: 2606.23917
Code: None
Area: AIGC Detection / Image Forensics
Keywords: Image Forensics, Forgery Detection, Forensic Knowledge Graph, Self-Supervised Fingerprint, VLM Explainability

TL;DR¶

This work combines the "evidence-seeking" capability of image forensics with the natural language generation of VLMs into a single system: first, a self-supervised forensic fingerprint backbone is used to segment the image into forensically consistent regions and predict the source identity, post-processing history, and compression parameters for each region to construct a structured "Forensic Knowledge Graph (FKG)"; subsequently, a VLM is guided to generate step-by-step verifiable forensic explanations based strictly on the graph, consistently outperforming pure forensic models and pure VLMs in terms of detection, forgery typing/localization, and forensic explanation.

Background & Motivation¶

Generative AI and modern image editors make forgeries extremely realistic, ranging from whole-image AI generation to local splicing and local AI inpainting, making it increasingly difficult for even professional analysts to distinguish them. A truly trustworthy authentication system must simultaneously satisfy three criteria: accurate detection across all forgery types, generating natural language explanations of "what was changed and how," and grounding each conclusion on verifiable evidence. However, existing systems can only meet some of these requirements. Traditional forensicators (such as CAT-Net, TruFor, and HiFi-IFDL) rely on content-agnostic statistical fingerprints (e.g., sensor noise, demosaicing residuals, and compression artifacts) left by the imaging pipeline. Although reliable for detection, each method only targets a specific forgery type: splicing detectors perform well on spliced images but completely miss AI generation, while AI generation detectors spot whole-image forgeries but fail to detect traditional manipulations. Moreover, their output is usually limited to binary "real/fake" labels or a heatmap, without providing any reasoning.

On the other hand, VLMs (such as GPT-5, Gemini, Qwen3-VL, etc.) can generate natural language explanations but rely on "visible semantic cues" like unnatural lighting or texture patterns, which quickly disappear as generators become more realistic. Crucially, VLMs struggle to leverage invisible statistical artifacts. Recent benchmarks consistently prove that even the strongest VLMs perform barely better than random guessing in forgery detection. In real-world scenarios, some chatbots have confidently misclassified AI-generated images as authentic, further eroding public trust in digital media and AI verification. This forms a Core Problem: the forensic evidence required for reliable detection (source identity, compression history, modification lineage) is structured and causally dependent, while the VLMs capable of translating evidence into natural language lack access to such evidence, leading to hallucinations or false negatives.

The Key Insight of this work is: instead of forcing the VLM to "infer forensic conclusions from pixels," a deterministic forensic system is first employed to extract and organize evidence into an explicit structured graph, after which the VLM is solely tasked with "describing the graph." Core Idea: By using a "Forensic Knowledge Graph (FKG)," the source, post-processing, compression, modification lineage, and causal dependencies of each region are explicitly encoded into triplets. All forensic decisions originate deterministically from the proposed forensic network (rather than VLM inference), and the VLM only generates step-by-step verifiable explanations under the constraints of the graph, thereby unifying reliable detection, structured reasoning, and interpretable output into a single system.

Method¶

Overall Architecture¶

The system aims to determine whether an input image is real or fake, identify the forgery type and location, and provide the underlying forensic evidence for each conclusion. The pipeline consists of two main components. The first is the FKG Generation System: an input image is first partitioned into patches, and a forensic backbone network extracts the forensic fingerprint of each patch; then, the Forensic Region Proposal Network (FRPN) clusters patches with similar fingerprints into "Forensic Regions"; a set of "Task Expert Networks" combined with a "Transformer Inference Module" then determines the source identity, post-processing operations, and compression attributes of each region, and deterministically instantiates them into an FKG according to a predefined ontology. In this graph, each region is represented as a Region node, connected to source, post-processing, and compression nodes via typed edges such as produced_by, modified_by, and compress_by. Crucially, these edges are deterministically filled by the system's region-wise predictions rather than guessed by the VLM, ensuring that each relationship is anchored to a concrete forensic prediction. The second component is Explanation: the FKG is serialized into subject-verb-object triplets and fed to the VLM along with Set-of-Mark visual annotations. An "Iterative Contextual Refinement" (ICR) strategy then guides the VLM to generate faithful, complete, and verifiable forensic explanations grounded step-by-step in the graph evidence.

The ontology of FKG defines six core entity types: Image (the root), Region (a subset of pixels sharing the same forensic artifact, forming the center of all relationships), Source (pixel source, with subclasses Camera Model and AI Generator), Post-Processing (post-generation operations such as resampling, blurring, or noise addition), Compression (compression traces, recording whether uncompressed, final compression parameters, and double-compression traces), and Content (semantic scene elements). When different regions connect to different sources, the graph structure inherently exposes the forgery—for instance, if one region links to a Nikon camera and another to a FLUX generator, with the camera region showing double compression and the FLUX region only single compression, it clearly points to "post-hoc content insertion."

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image<br/>split into patches"] --> B["Self-Supervised Forensic Backbone<br/>extracts content-agnostic forensic fingerprints"]
    B --> C["Forensic Region Proposal Network FRPN<br/>clusters patches with similar fingerprints into regions"]
    C --> D["Task Experts + Transformer Inference<br/>predicts source/post-processing/compression and builds FKG"]
    D --> E["Forensic Knowledge Graph FKG<br/>region-source-operation triplets"]
    E --> F["Iterative Contextual Refinement ICR<br/>guides VLM to explain step-by-step based on the graph"]
    F --> G["Output: Real/Fake / Type / Localization<br/>+ Verifiable Forensic Evidence"]

Key Designs¶

1. Self-Supervised Forensic Fingerprint Backbone: Leverages "patches within the same image share the same fingerprint" to turn unannotated images into free training samples

The foundation of the forensic system is a backbone network capable of capturing statistical fingerprints of the imaging pipeline, which FRPN relies on for partitioning and task experts rely on for predicting attributes. Previous methods learned these fingerprints using task labels (such as camera model classification), which tightly tied the backbone to specific sources, resulted in poor generalization, and naturally failed to cover "new sources" like AI generators. This work leverages a simple yet crucial property of the imaging pipeline: local patches within the same image share the same forensic fingerprint, whereas patches from different images do not. Based solely on this property, every unannotated image becomes a free supervisory signal. The backbone can learn general forensic fingerprints on massive datasets without needing forgery labels or source annotations. To prevent the network from learning "content shortcuts" (mapping semantically similar patches to the same image), a learnable high-pass filter is first applied to suppress the image content. Then, two complementary targets are used for self-supervised learning: a pairwise similarity term forces representation alignment for patches from the same image and separation for different images, while a contrastive term sharpens patch-wise discriminativeness:

\[\mathcal{L}_{B}=\frac{1}{|\mathcal{X}|^{2}}\sum_{i,j}w_{ij}(Y_{ij}-z_{i}^{\top}z_{j})^{2}-\lambda\sum_{i}\log\frac{e^{\cos(z_{i},z_{i^{+}})/\tau}}{\sum_{j}e^{\cos(z_{i},z_{j})/\tau}}\]

where \(z_i\) is the L2-normalized patch embedding, \(Y_{ij}\in\{0,1\}\) indicates whether two patches belong to the same image, \(w_{ij}\) balances positive and negative pairs, and \(\tau\) is the temperature. Because this objective learns "inherent fingerprints of the imaging pipeline" rather than content or specific forgery characteristics, it treats AI generators equally—the fingerprint of an AI-generated image is simply identified as "different from any camera source," making it inherently detectable. Ablation studies show that removing self-supervision (relying only on task labels) causes all three key metrics to collapse (detection AUC drops from 0.94 to 0.72, localization F1 drops from 0.94 to 0.44), demonstrating that this is the foundation for the entire system's generalization.

2. Hybrid Graph Attention Transformer (FRPN): Global self-attention to find similarities, local graph attention to suppress false positives

After obtaining patch fingerprints, the image must be segmented into "forensically homogeneous" regions. Standard visual segmentation or region proposal models fail here because they look at visual features, whereas forensic regions are defined by invisible statistical properties (a spliced area may look visually seamless but carry entirely different fingerprints). The authors observe that partitioning an image into forensically homogeneous regions is essentially a "pairwise comparison between every patch and all other patches," which is precisely the all-to-all operation that Transformer self-attention naturally computes. However, pure global self-attention suffers from "attention dilution": the accumulated contributions of distant, unrelated patches can bridge forensic regions that should be separated, leading to false positives and misclassification. Therefore, FRPN alternately stacks global self-attention (to capture long-range forensic similarities) and local graph attention (which aggregates only within the k-nearest neighbors of each token and uses additive scoring to suppress false positives):

\[\alpha_{ij}^{\text{global}}=\frac{\exp(\mathbf{w}_{qi}^{\top}\mathbf{w}_{kj}/\sqrt{d})}{\sum_{m}\exp(\mathbf{w}_{qi}^{\top}\mathbf{w}_{km}/\sqrt{d})},\quad \alpha_{ij}^{\text{local}}=\frac{\exp(\sigma(\mathbf{a}_{s}^{\top}\mathbf{w}_{gi}+\mathbf{a}_{d}^{\top}\mathbf{w}_{gj}))}{\sum_{m\in\mathcal{N}(i)}\exp(\sigma(\mathbf{a}_{s}^{\top}\mathbf{w}_{gi}+\mathbf{a}_{d}^{\top}\mathbf{w}_{gm}))}\]

where \(\mathcal{N}(i)\) is the k-nearest neighbors dynamically selected based on cosine similarity. The refined embeddings are then fed into a classification head to predict cluster assignments for each patch. Ablation studies confirm that both attention mechanisms are indispensable: keeping only local attention drops the typing accuracy from 0.87 to 0.57 (loss of long-range dependencies), while keeping only global attention drops it from 0.87 to 0.55 (loss of local connectivity and destruction of region homogeneity)—perfectly validating the division of labor: "global for similarity search, local for false positive suppression."

3. Hungarian Matching for Region Assignment Training: Aligning predicted cluster IDs with ground-truth regions

The cluster IDs predicted by FRPN are arbitrary integers that do not directly correspond to ground-truth region labels (e.g., from a known forgery mask), preventing direct cross-entropy calculation. This work uses Hungarian matching to find the optimal mapping \(\hat{\sigma}\) from predictions to ground truths, then combines it with a pairwise similarity term—which preserves the embedding structure, keeping patches of the same region close in the embedding space:

\[\mathcal{L}_{F}=\frac{1}{|\mathcal{P}|^{2}}\sum_{i,j}w_{ij}(Y_{ij}-\tilde{z}_{i}^{\top}\tilde{z}_{j})^{2}+\alpha\sum_{i}\ell_{\text{CE}}(y_{i},\hat{\sigma}(\hat{c}_{i}))\]

Neither term can be omitted: keeping only the contrastive term (removing Hungarian matching) yields unstable and inconsistent region assignments (as contrastive learning alone cannot generate discrete, semantically aligned regions); keeping only the Hungarian matching (removing continuous similarity constraints) degrades both detection and classification metrics (as continuous similarity is crucial for coherent grouping). Patches in the same cluster finally form a forensic region, which becomes a Region node in the FKG.

4. Task Experts + Transformer Inference Module: Modeling cross-task dependencies to predict attributes and deterministically mapping them to FKG

With the regions established, this step determines "exactly what happened in each region" and assembles the FKG. The refined patch embeddings of each region are first average-pooled into a region-level vector \(\psi_k\) and fed to a set of task expert networks (MLPs specialized in source identification, post-processing classification, and compression analysis, respectively) to produce task-specific embeddings. Crucially, these forensic tasks are not independent—for instance, compression traces can conversely assist in source identification. Thus, a Transformer Inference Module is employed to model cross-task dependencies and refine the features of each task. Finally, task heads predict binary, categorical, or continuous outputs. The training objective is a weighted sum of individual task losses: \(\mathcal{L}_{T}=\sum_{t}\lambda_{t}\mathcal{L}_{t}\) (using cross-entropy or mean squared error). The predictions are deterministically instantiated according to the ontology: each region becomes a Region node, and the predicted attributes become entity nodes and produced_by/modified_by/compress_by edges. This cross-region relational structure is exactly what makes "distinguishing forgery types" possible—different camera sources reveal splicing, missing camera sources reveal GAN/diffusion-based whole-image synthesis, and mixed real/synthetic traces reveal AI inpainting (AI-Edit). In ablation studies, replacing the inference module with linear classifiers causes the typing accuracy to drop by 0.18, proving its necessity in integrating cross-region evidence to infer forgery semantics.

5. Iterative Contextual Refinement (ICR): Boosting VLM fidelity solely through curated in-context examples without fine-tuning

Even after serializing the FKG into triplets to feed to the VLM, two hurdles remain: the FKG is multimodal, mixing symbolic attributes with spatial-visual content, which standard VLMs cannot easily digest from pure graph text; and VLMs lack forensic domain knowledge, making them prone to hallucinating or omitting evidence. The first hurdle is resolved using Set-of-Mark prompting—overlaying numbered markers on each Region in the image and referencing them symbolically in the triplets. This allows the VLM to align spatial scene content with forensic evidence. The second hurdle is tackled by ICR. A naive in-context learning approach with manually selected "FKG-report" examples is impractical, while existing prompt optimization methods are designed to "answer questions correctly" rather than "adhere strictly to structured evidence." ICR treats each triplet as an atomic fact and defines two metrics—completeness (the ratio of ground-truth triplets covered by the report) and correctness (the ratio of correctly described triplets among those mentioned):

\[M_{\text{comp}}=\frac{1}{|F|}\sum_{j}\tau(m_j,R),\quad M_{\text{corr}}=\frac{\sum_{j}Q(m_j,R)\tau(m_j,R)}{\sum_{j}\tau(m_j,R)}\]

where \(\tau(m_j,R)\in\{0,1\}\) indicates whether fact \(m_j\) is mentioned in report \(R\), and \(Q(m_j,R)\in\{0,1\}\) indicates whether it is described correctly. ICR works by "directly addressing the VLM's failure modes": starting from an empty context, an evaluator LM evaluates the completeness and correctness of each FKG in the training set. The FKGs with the most severe hallucinations or omissions are paired with their errors as "correction demonstrations" and appended to the context. This context is prepended to the prompt in the next round, iterating until the step-wise change in both metrics is less than a threshold \(\epsilon\). This process operates entirely at the prompt level without modifying the VLM's parameters, yet pushes completeness from 0.55 to 0.85.

Loss & Training¶

The three components—backbone, FRPN, and task networks—are trained sequentially (see Appendix D of the original paper for details). The backbone is trained self-supervised using \(\mathcal{L}_B\) (pairwise similarity + contrastive); the FRPN is trained using \(\mathcal{L}_F\) (pairwise similarity + Hungarian matching cross-entropy); the task experts/inference module are trained using the weighted multi-task loss \(\mathcal{L}_T\). No fine-tuning is performed on the VLM; its contextual demos are optimized at the prompt level using ICR.

Key Experimental Results¶

Evaluation is conducted across three dimensions (detection, typing/localization, and forensic explanation) on the self-built FKG-50K dataset (40k training + 10k evaluation, covering splicing, traditional editing, AI inpainting, and whole-image synthesis) and four OOD public datasets (DSO-1, CASIAv2, GenImage, Synthbuster). Baselines include state-of-the-art forensicators and VLMs.

Main Results¶

Forgery Detection (ACC / AUC, showing Overall and representative OOD):

Dataset	Metric	Ours (FKG)	Best Forensicator	Best VLM
FKG-50K Overall	ACC/AUC	0.92 / 0.94	TruFor 0.70/0.81	GPT-5 0.66/0.72
AI-Edit (Hardest)	AUC	0.88	TruFor 0.76	GPT-5 0.60
DSO-1 (OOD)	ACC/AUC	0.92 / 0.96	CAT-Net 0.99/1.00	GPT-5 0.59/0.61
GenImage (OOD)	ACC/AUC	0.92 / 0.96	NPR 0.92/0.97	GPT-5 0.93/0.88

Forensicators only perform well on the forgery types they were trained on (e.g., CAT-Net achieves an AUC of 1.00 on DSO-1 splicing but only 0.37 on whole-image AI; NPR achieves 0.98 on Synthbuster but 0.50 on splicing). No single forensicator covers all genres. Meanwhile, VLMs can identify whole-image synthesis but struggle to catch local manipulations that maintain global visual consistency. By learning general forensic fingerprints, the proposed method is the only system that performs strongly across all categories.

Forgery Type Identification / Localization (Ours uses no oracle; VLM numbers are shown with oracle assistance in parentheses):

Task	Dataset	Ours (FKG)	Best VLM (No Oracle)	Best Forensicator
Type ACC	FKG-50K Overall	0.87	0.31 (Qwen3 [T])	—
Loc F1	FKG-50K Overall	0.94	0.39 (Qwen3 [T])	CAT-Net 0.47
Loc F1	DSO-1 (OOD)	0.95	—	TruFor 0.93

Forensic Explanation (correctness / completeness, evaluated on FKG-50K; VLMs are provided with oracle assistance):

Method	Overall COR	Overall COM
Ours (FKG) (No Oracle)	0.75	0.85
Best VLM (With Oracle)	0.24 (Gemini)	0.10 (Sonnet 4)

Even when provided with the ground-truth forgery type and precise location, standard VLMs' explanations cover only about 10% of the forensic facts and mostly consist of visual speculations like "inconsistent lighting," whereas the proposed method covers 85% of the facts and anchors them to verifiable artifacts such as camera model, compression, and modification genealogy.

Ablation Study¶

Configuration	Detect AUC	Type ACC	Loc F1	Explanation / Description
Ours (Full)	0.94	0.87	0.94	—
Backbone w/o self-supervision	0.72 (-0.22)	0.66 (-0.21)	0.44 (-0.50)	Relies only on task labels; fingerprints fail to generalize, leading to global collapse.
Backbone w/o pairwise similarity	0.90 (-0.04)	0.63 (-0.24)	0.86 (-0.08)	Weakened structural alignment.
FRPN w/ only local graph attention	0.85 (-0.09)	0.57 (-0.30)	0.93 (-0.01)	Lacks long-range dependencies; typing drops the most.
FRPN w/ only global self-attention	0.92 (-0.02)	0.55 (-0.32)	0.93 (-0.01)	Attention dilution; region homogeneity is destroyed.
Only contrastive loss (w/o Hungarian)	0.89 (-0.05)	0.52 (-0.35)	0.72 (-0.22)	Unstable region assignment.
Only Hungarian (w/o contrastive)	0.81 (-0.13)	0.45 (-0.42)	0.59 (-0.35)	Lacks continuous similarity; incoherent grouping.
w/o Transformer Inferer	0.94 (±0.00)	0.69 (-0.18)	0.94 (±0.00)	Lacks integration of cross-region evidence; typing drops by 0.18.

ICR Ablation (on the same batch of FKG inputs): Zero-shot completeness is only 0.32; random examples improve completeness to 0.51 (teaching output format); adding CoT increases correctness to 0.70; ICR nearly doubles completeness from 0.55 to 0.85, without any VLM fine-tuning.

Key Findings¶

Self-supervised backbone is the key to generalization: Removing it halves the localization F1 (-0.50), indicating it is the root of OOD generalization because it learns imaging pipeline properties rather than image content.
Two attention mechanisms have distinct roles: Global attention handles "finding similarities," while local attention handles "suppressing false positives." Removing either drops typing accuracy by around 0.30.
Graph topology is robust to parameter errors: While the perfect-match rate of the entire graph is only 0.65, detection, localization, and explanations remain highly accurate. This is because a correct graph topology is sufficient for verification; most errors reside in fine-grained parameters (e.g., specific generator models, exact compression quality) rather than structural relations. AI-Edit is the hardest task (perfect match is 0.58) because AI editors generate pixels based on real content, partially inheriting the original image's forensic properties.
Human trust evaluation: 30 participants rated 5 images. Providing only the final decision yielded an average trust score of 3.39, which increased to 4.53 when parenthesized with the structured graph-grounded forensic explanation, proving that an auditable chain of evidence significantly boosts trust.

Highlights & Insights¶

Complete decoupling of "evidence extraction" and "linguistic explanation": Entrusting the deterministic forensic network with decision-making while restricting the VLM to describing the graph completely eliminates the unreliable nature of VLMs "hallucinating forensic conclusions from raw pixels"—which remains the primary reason pure VLM forensics perform barely better than random guessing. This paradigm of "construct structured evidence graph first, then ground the LLM step-by-step" is directly transferable to other high-risk decision-making tasks requiring verifiable explanations.
"Intra-image patch fingerprint sharing" turns unannotated images into free supervision: This simple observation bypasses the bottleneck where task labels tie the backbone to specific sources, allowing the backbone to treat AI generators equally as "different from any camera source," serving as the pivot for the entire system's generalization.
Framing region partitioning as an "all-to-all comparison" to leverage self-attention: The motivation for FRPN is not merely to use a Transformer, but because "partitioning forensically homogeneous regions is a pairwise comparison" which mathematically aligns with the self-attention operator. Pairing this with local graph attention resolves the attention dilution bottleneck, presenting a highly coherent and task-specific design.
ICR doubles completeness purely via prompt-level refinement without fine-tuning: By employing an evaluator LM to systematically identify the VLM's hallucination and omission blind spots and appending these worst-case failures as corrective demonstrations in context, the authors provide a reusable path for utilizing black-box VLMs in structured grounding tasks.

Limitations & Future Work¶

The authors acknowledge that predicting fine-grained parameters (e.g., generator models, compression quality) and error propagation from imperfect region proposal networks represent the main bottlenecks for perfect-FKG reconstruction. However, they emphasize that both issues can be resolved within the modular framework by swapping in stronger task experts or region proposal networks without altering the ontology, inference pipeline, or ICR.
The evaluation of forensic explanations relies on an LLM-as-judge framework (evaluating whether triplets are correctly mentioned), meaning the metrics are fundamentally bound to the evaluator's reliability, acting as an automated approximation.
Most VLM baselines are evaluated with oracle assistance for typing, localization, and explanation, whereas the proposed system operates completely without an oracle. This disparity serves as a caveat (though the proposed system still outperforms them).
The system relies on original camera photos with signature EXIF metadata to construct the FKG-50K training data. Whether the fingerprints remain reliable for wild, real-world images where metadata is erased or which have undergone multiple social media re-compressions remains to be verified.

vs Traditional Forensicators (TruFor / CAT-Net / HiFi-IFDL / MVSS-Net): These approaches also leverage forensic micro-structures, but as binary detectors they cannot reason about the causal relationships among multiple forensic clues, each covers only a single class of forgery, and they do not output explanations. The proposed method unifies multi-type detection and structured reasoning through a self-supervised general fingerprint scheme and an FKG ontology.
vs AI Generation Detectors (NPR / UFD / DE-FAKE): These methods are limited to binary classification without localization or description. Moreover, UFD and DE-FAKE rely on CLIP/BLIP embeddings rather than low-level forensic features. The proposed method covers all four major forgery types and provides verifiable explanations.
vs Forensic VLMs (FakeShield / ForgeryGPT / SIDA): These models fine-tune VLMs but still rely primarily on visible visual cues rather than low-level forensic microstructures, and their coverage is narrow (e.g., ForgeryGPT only handles faces). In contrast, the proposed method structures the forensic evidence first before passing it to the VLM, ensuring the criteria are anchored on signal-level traces.
vs General-purpose VLMs (GPT-5 / Gemini / Qwen3-VL): While they excel at fluent text generation, their performance on forensic tasks is barely better than random guessing, often entirely failing to catch local manipulations. The proposed method delegates the "detection" entirely to the forensic network, leaving the VLM to solely describe the FKG.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Serves as the first to bridge VLMs and forensicators through a structured forensic evidence graph. The three design components—self-supervised fingerprints, hybrid graph attention, and ICR—all possess highly concrete and sound motivations.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluation across three dimensions on 5 datasets against over ten forensicator and VLM baselines. Comprehensive component-wise ablations, ICR ablation, and a human trust study are provided, where the proposed method consistently leads despite operating without oracle assistance.
Writing Quality: ⭐⭐⭐⭐ Exceptionally clear flow with logically unfolding motivations. The abundance of details on the ontology and the multi-staged pipeline can be overwhelming, requiring the reader to cross-reference the appendix.
Value: ⭐⭐⭐⭐⭐ Directly addresses the critical real-world pain point of "trustworthy media authentication." The auditable chain of evidence substantially enhances human trust, and the proposed paradigm holds great transfer value for high-risk explainable decision-making.