Efficient Document Tampering Localization with Multi-Level Discrepancy Features and Unified DCT-Quantization Embedding

Conference: ECCV 2026
arXiv: 2606.22285
Code: Yes (https://github.com/Mohamed-Dhouib/DiffNet)
Area: AIGC Detection
Keywords: Document Tampering Localization, RGB-DCT Fusion, Discrepancy Features, Zero-Sum Filtering, Cross-Domain Generalization

TL;DR

DiffNet introduces two complementary designs: a multi-level discrepancy transform that converts the feature pyramid from content signals into sign-invariant discrepancy intensity signals, and a unified DCT-quantization joint embedding that replaces the traditional, high-overhead Frequency Perception Head (FPH) with discrete embeddings. This achieves approximately a 30% F1 improvement on cross-domain and human-manipulated document localization at a lower computational cost, with up to a 7-fold increase in throughput.

Background & Motivation

Document images form the core of financial, administrative, and identity authentication workflows, making them prime targets for tampering attacks. If tampering goes undetected, it can lead to fraud, identity theft, and cascading errors in automated decision-making systems. However, current state-of-the-art models still exhibit poor generalization in cross-domain and human-manipulated scenarios. A primary reason is that evaluation is mostly conducted on synthetic benchmarks, where synthetic data generation pipelines introduce systematic biases: inserted text often comes from a limited set of fonts, overwriting processes reuse a small set of algorithms that leave characteristic boundary artifacts, and copy-move or splicing operations share similarly repeatable patterns. This causes models to learn to rely on pipeline-specific "fingerprints" rather than the inconsistencies caused by the tampering itself to localize tampered regions.

The key challenge of this problem lies in the extreme scarcity and high acquisition cost of human-annotated tampered data, which forces reliance on large-scale synthetic pre-training, but synthetic pipelines encourage shortcut learning. To make matters worse, the source documents for synthetic datasets are often highly restricted (for instance, IIT-CDIP tobacco litigation documents constitute a large portion), allowing the model to minimize training loss by memorizing domain-specific clues of the source documents rather than truly capturing the underlying inconsistencies introduced by tampering. Even in human-manipulated datasets, similar issues persist—these datasets are usually small and constructed by a few experts, and thus might still suffer from annotation biases.

This paper takes a design path different from previous works: rather than further increasing architectural complexity, it constrains the representations of the feature pyramid to force the decoder to be driven primarily by inconsistency clues instead of content signals. Specifically, the authors introduce a lightweight multi-level discrepancy transform at the output of each backbone stage, using a set of learnable zero-sum filters to convert features into magnitude responses, obtaining a sign-invariant discrepancy representation across the feature pyramid. Meanwhile, for the frequency branch, they design an efficient DCT-domain backbone. Exploiting the fact that both quantized DCT coefficients and quantization table entries are discrete integers, they utilize an embedding layer to achieve a compact joint embedding, replacing the traditional high-overhead frequency perception head. Core Idea: Use learnable zero-sum convolutional kernels to transform features from content signals to discrepancy signals, while using discrete embeddings to achieve efficient joint DCT-quantization encoding, enabling the model to achieve significant improvements in cross-domain and human-manipulated scenarios.

Method

Overall Architecture

DiffNet is a two-branch early-fusion architecture. The input image is simultaneously fed into the RGB and DCT branches: the RGB branch uses a four-stage ConvNeXt-V2 backbone to extract multi-scale features; the DCT branch transforms the quantized DCT coefficients and the quantization table into a compact feature map via a joint embedding layer, processes it through several ConvNeXt-V2 blocks, and then blends it into the RGB stream at the output of RGB stage 1 using concatenation followed by linear projection. The fused representation is further encoded by stages 2-4. At the output of each backbone stage (including the pure RGB features of stage 1 and the fused features of stages 2-4), a multi-level discrepancy transform is applied to obtain a discrepancy-feature pyramid across four scales. Finally, an FPN-style top-down decoder performs multi-scale fusion of the discrepancy features, followed by 6 layers of ConvNeXt-V2 refinement, and a lightweight segmentation head outputs the tampering mask.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image"] --> B["RGB Branch<br/>ConvNeXt-V2 Stage 1"]
    A --> C["DCT Branch<br/>Quantized Coefficients → Joint Embedding → ConvNeXt Blocks"]
    B --> D["Early Fusion<br/>Concatenation + Linear Projection"]
    C --> D
    D --> E["RGB Branch<br/>Stage 2 → 3 → 4"]
    B --> F["Discrepancy Transform ϕ<br/>Zero-Sum Filtering → |·| → Projection"]
    D --> G["Discrepancy Transform ϕ"]
    E --> H["Discrepancy Transform ϕ"]
    F --> I["Discrepancy Feature Pyramid<br/>{F̃1, F̃2, F̃3, F̃4}"]
    G --> I
    H --> I
    I --> J["FPN-style Multi-scale Fusion<br/>+ 6-layer ConvNeXt Refinement"]
    J --> K["Segmentation Head → Tampering Mask"]

Key Designs

1. Multi-Level Discrepancy Transform: Transforming Features from Content to Discrepancy Signals via Zero-Sum Filters

Synthetic pre-training data biases models toward relying on pipeline-specific, repeatable artifacts (such as rendering traces from fixed font sets or boundary features from specific blending algorithms) rather than genuine inconsistencies introduced by tampering. To break this shortcut learning, the key mechanism is to design a transform that forces features to focus on local discrepancies rather than absolute intensities. The authors apply a set of depth-wise separable zero-sum filters at the output of each backbone stage. The core constraint is that the sum of all filter weights is zero—which automatically suppresses constant bias components of each channel because zero-sum filters produce zero response to homogeneous regions. On top of this, taking the absolute value of the filtered responses ensures that the representation only cares about the "intensity" of the discrepancy, regardless of its "direction" (either positive or negative difference signifies an inconsistency). The combination of these two elements naturally focuses the filter output on local inconsistent regions rather than the document content itself.

Specifically, for each input channel \(c\), let \(F_c\) be its feature map at the \(i\)-th stage. The model learns \(M\) depth-wise separable \(K \times K\) zero-sum filters \(k_{c,m}\) (\(m=1,\dots,M\)). The response is given by \(u_{c,m}(p) = \sum_{\Delta} k_{c,m}(\Delta) F_c(p+\Delta)\) under the constraint \(\sum_{\Delta} k_{c,m}(\Delta)=0\). Taking the absolute value yields the discrepancy intensity signal \(v_{c,m}(p) = |u_{c,m}(p)|\). The authors further design two complementary classes of zero-sum filters: the "free" family learns all parameters freely under the zero-sum constraint; the "center-anchored" family binds the center weight to the negative sum of the neighborhood weights (with neighborhood weights restricted to be non-negative), acting as a measurement of deviation from "neighborhood-predicted center"—the larger the difference between the neighborhood weighted average and the center value, the stronger the response. Two filters (\(M=2\)) are assigned per channel \(c\), one free and one center-anchored.

Visualizations demonstrate that after applying zero-sum filtering, peak activation regions of the features shift significantly from following the document content (stroke of characters, table lines, background textures) to clustering around tampering boundaries, while responses in clean background regions are highly suppressed. This indicates that the discrepancy transform successfully plays the key role of transforming the representation from "what is the content here" to "is there an inconsistency here."

2. Unified DCT-Quantization Joint Embedding: Replacing Dilated Convolutions with Discrete Embeddings

Frequency evidence in the DCT domain has proven highly effective in document tampering detection, but prior mainstream solutions—namely the Frequency Perception Head (FPH)—suffer from massive computational overhead: they first expand 2D DCT coefficients of each image into a 64-channel full-resolution feature map via dilated convolutions, which is then projected and expanded to 256 channels through multiple modules like MBConv, materializing a huge volume of intermediate activations during the process. The key insight of this paper is that quantized DCT coefficients generally fall into discrete integers between 0 and 20, and quantization table entries are discrete integers between 1 and 255. Both are essentially discrete tokens, naturally suited for direct mapping via embedding layers.

In concrete implementation, for each \(8 \times 8\) block \(p\) and frequency index \(k\), the quantized DCT coefficient is first mapped into a \(d_{\text{dct}}=4\) dimensional vector \(v_{p,k}\) using a learned embedding. Meanwhile, the corresponding quantization table entry \(Q_k\) is transformed via another embedding into FiLM-style modulation parameters \((\gamma_k, \beta_k)\) to element-wise modulate \(v_{p,k}\):

\[e_{p,k} = (1 + \gamma_k) \odot v_{p,k} + \beta_k + f_k + t\]

Where \(f_k\) is the frequency index embedding (indicating to the network which DCT frequency index this vector originates from), and \(t\) is a global table bias generated by a lightweight MLP on the entire quantization table (capturing global quantization intensity patterns beyond individual \(Q_k\)). Concatenating all embeddings across the 64 frequency indices of each block along the channel dimension yields a \(64 \times 4 = 256\) dimensional block-level representation, which is processed by several ConvNeXt-V2 blocks before being fed into the fusion layer.

Compared to FPH, this design offers two key advantages: first, it avoids early expansion of high-dimensional features—the embedding dimension is only 4, yet it achieves or surpasses the performance of FPH at 256 channels; second, the combination of FiLM modulation, frequency indices, and global table biases provides richer frequency-aware expression capability than FPH (ablation experiments show that removing any of these components leads to performance degradation, with removing the FiLM \(\beta\) term having the biggest impact). This allows the DCT branch to provide stronger frequency clues with fewer parameters and less computational cost.

Loss & Training

Training employs Focal Loss (\(\gamma=2\)) to jointly optimize the segmentation head and the document-level classification head. Under the Syn2Real-TDoc protocol, the learning rate is set to \(1\times10^{-4}\) with a cosine annealing schedule, using a batch size of 64 for 2 epochs. The data augmentation pipeline follows the setup of [leveragingcontrastive]. During inference, a sliding window strategy (with a window size of 1024x1024) is used to stitch global predictions. Zero-sum filters are implemented using custom CUDA operators for both forward and backward passes, avoiding the explicit construction of dense zero-sum kernels, which improves end-to-end training throughput by approximately 12% compared to the torch.compile baseline.

Key Experimental Results

Main Results

Cross-domain results under the Doc protocol (trained on DocTamper, tested on four cross-domain datasets):

Dataset	Metric	ADCD-Net (Prev. SOTA)	DiffNet (Ours)	Gain
T-SROIE	F1	0.623	0.745	+0.122
OSTF	F1	0.441	0.495	+0.054
Tampered-IC13	F1	0.579	0.647	+0.068
RTM	F1	0.159	0.173	+0.014
Cross-domain Avg	F1	0.450	0.515	+0.065

Results under the Syn2Real-TDoc protocol (trained on TDoc-2.8M, tested on three human-manipulated datasets):

Dataset	Metric	FFDN (Prev. SOTA)	DiffNet (Ours)	Gain
RTM	Pix F1	0.238	0.255	+0.017
FindItAgain	Pix F1	0.255	0.307	+0.052
FindIt	Pix F1	0.318	0.350	+0.032
Avg Pix F1	F1	0.270	0.304	+0.034

Efficiency comparison (throughput measured at a resolution of 768x768):

Model	Params	Inference Throughput (it/s)	Training Throughput (it/s)	Avg F1
ADCD-Net	45.7M	5.7	4.4	0.312
FFDN	140.0M	33.1	20.8	0.286
DiffNet	113.0M	40.1	41.9	0.410

Ablation Study

Configuration	Doc Pix F1	TDoc Pix F1	Avg	Description
Full model	0.515	0.303	0.409	Full DiffNet
w/o zero-sum constraint (A0)	0.496	0.291	0.393	Removes zero-sum constraint, verifying the effectiveness of the filter constraint
Replacing FPH (A1)	0.493	0.290	0.391	Replaces DCT embedding with traditional FPH; yields lower performance under the same compute budget
w/o global table bias (A3)	0.513	—	—	Removes t, minor drop
w/o frequency index (A4)	0.512	—	—	Removes f_k, minor drop
w/o FiLM γ (A5)	0.509	—	—	Removes modulation γ, notable drop
w/o FiLM β (A6)	0.504	—	—	Removes modulation β, largest drop
Replacing embedding with linear mapping (A7)	0.494	—	—	Replaces embedding with a linear layer, significant drop
Fixed SRM kernels (A14)	0.511	—	—	Replaces with fixed high-pass filtering (SRM); underperforms learned kernels
Replacing with Bayar convolution (A18)	0.506	—	—	Replaces with Bayar-constrained convolutions, performance degrades
Discrepancy transform as an extra modality (A19)	0.499	—	—	Used as an extra input modality instead of feature transform
Single filter family (A17)	0.513	—	—	Uses only either free or center-anchored family

Key Findings

• The two core design elements of the discrepancy transform—zero-sum constraint and taking absolute value—are both indispensable: removing the zero-sum constraint (A0) results in the largest drop, while omitting the absolute value or replacing it with tanh also degrades performance, indicating that a sign-invariant discrepancy intensity representation is critical.
• FiLM modulation has the largest impact on performance in the DCT embedding, particularly the \(\beta\) item. This suggests that conditioning coefficient embeddings on the quantization table is the crucial mechanism for capturing frequency clues. Without \(\beta\), performance drops to 0.504, which is close to replacing the embedding with a linear mapping (0.494).
• Applying the discrepancy transform as a "feature transform" is more effective than using it as an "extra modality" (A19 drops to 0.499 vs. 0.515 for the baseline). This demonstrates that imposing constraints within the feature space is more direct and effective than adding an extra input branch.
• The free and center-anchored filter families are complementary: utilizing only one of them drops the performance to 0.513, which is lower than the 0.515 achieved by using both families together.

Highlights & Insights

• Transforming features into discrepancy signals using zero-sum filters is the most ingenious design in this paper. Unlike conventional methods that apply high-pass filtering at the input as an extra modality, it extracts discrepancies at every level of the feature space. This idea can be readily transferred to other tasks that require detecting "local inconsistencies," such as deepfake detection, splicing detection, and image integrity verification.
• Replacing dilated convolutions with discrete embedding layers to handle DCT coefficients is a simple yet elegant insight. The discrete integer nature of quantized DCT coefficients makes embeddings more natural than continuous transformations. This suggests that when handling JPEG-domain information, embedding-based representations might be universally superior to convolutional expansions.
• The center-bound parameterization of the zero-sum filter not only guarantees that the constraint is automatically satisfied, but also yields high computational efficiency. A custom CUDA operator fuses zero-sum filtering, absolute value calculation, and channel projection into a single kernel, avoiding intermediate activation materialization. This approach of co-optimizing mathematical constraints with low-level implementation is highly exemplary.
• The counter-intuitive phenomenon where training throughput (41.9 it/s) exceeds inference throughput (40.1 it/s) stems from the fused design of the custom CUDA kernel. During training, a single forward-backward pass shares compact representations, whereas inference requires materializing the fixed kernel for standard depth-wise convolutions.

Limitations & Future Work

• The absolute performance on human-manipulated datasets remains relatively low (Pixel F1 ≈ 0.3). The authors point out two main reasons: first, human tampering often occurs in homogeneous background areas and is paired with advanced concealment techniques (e.g., local post-processing to fade traces), leaving very few clues; second, the benchmarks contain a high proportion of images with zero positive pixels (reflecting sparse tampering in real scenarios), which limits recall. Improving the realism and diversity of concealment strategies in synthetic data is a crucial direction for future work.
• The parameter count of DiffNet is 113M. While its efficiency is far superior to ADCD-Net (45.7M) and FFDN (140M), there remains room for further lightweighting. Having two 7x7 filters per channel in the discrepancy transform exerts significant shared-memory pressure under high-resolution inputs. Multi-group or dimension-reduction schemes could be considered.
• Although the custom CUDA kernel brings throughput benefits, it increases engineering complexity and cross-platform deployment difficulty (currently, only CUDA is supported). Porting to mobile or edge devices requires additional development.
• The current method is tailored specifically for document images (structured text layouts), and its generalization capability on natural scene images has not yet been evaluated. Although zero-sum filtering itself does not rely on domain priors and the ConvNeXt-V2 backbone is generic, validation on broader image forensics benchmarks is warranted.

Related Work & Insights

• vs. DTD / FFDN / ADCD-Net: These methods all inherit the Frequency Perception Head (FPH) to process DCT-domain features, focusing architectural evolution on fusion strategies and auxiliary enhancement modules. In contrast, DiffNet directly improves the DCT backbone itself by replacing FPH with embeddings and imposing a discrepancy transform on the feature pyramid, rather than stacking post-processing modules.
• vs. SRM / Bayar Convolutions: Traditional image forensics methods employ fixed high-pass filters (SRM) or constrained convolutions (Bayar) to extract residual signals from the input image as an extra modality. DiffNet's discrepancy transform differs fundamentally: instead of adding an input-level modality, it extracts discrepancies at every level of the feature space and takes absolute values for sign-invariant representations, yielding superior performance (as verified by A14/A18 ablations).
• vs. Synthetic-to-Real Transfer: [leveragingcontrastive] improves the quality of synthetic data from a data generation perspective, whereas DiffNet reduces reliance on synthetic data artifacts via architectural constraints. The two are orthogonal and can be combined in the future.

Rating

• Novelty: ⭐⭐⭐⭐☆ The zero-sum filtering-driven discrepancy transform is novel, and using embeddings to replace FPH is a clever insight, though the overall architecture is an improvement on mainstream two-branch early-fusion models.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across two protocols and seven datasets, with ablation experiments covering over 20 design choices, providing robust formulation and quantitative proof.
• Writing Quality: ⭐⭐⭐⭐⭐ The motivation is lucidly presented, the logical chain of design is complete (pain points \(\to\) design \(\to\) verification), mathematical formulas and visualizations are well-coordinated, and the appendix provides thorough details on the CUDA implementation.
• Value: ⭐⭐⭐⭐⭐ Achieves a ~30% F1 improvement and up to a 7x throughput speedup on cross-domain and human-manipulated localization, offering high practical value.

Related Papers

• [ICLR 2026] Omni-IML: Towards Unified Interpretable Image Manipulation Localization
• [ECCV 2026] Multi-Task Bayesian In-Context Learning
• [ICLR 2026] RelayFormer: A Unified Local-Global Attention Framework for Scalable Image and Video Manipulation Localization
• [CVPR 2026] Inconsistency-aware Multimodal Schrodinger Bridge for Deepfake Localization
• [NeurIPS 2025] Reasoning Compiler: LLM-Guided Optimizations for Efficient Model Serving