Skip to content

MG-RWKV: Multi-Grained Context-Aware RWKV for Temporal Forgery Localization

Conference: ECCV 2026
arXiv: 2607.00902
Area: Multimedia Security / Video Understanding
Keywords: Temporal Forgery Localization, RWKV, Multi-Grained Mixture of Experts, Bidirectional Recurrent Modeling, Cross-Grained Consistency

TL;DR

MG-RWKV proposes a temporal forgery localization (TFL) framework based on the linear-complexity recurrent architecture of RWKV. By capturing global temporal context via bidirectional RWKV, adaptively selecting explicit temporal receptive fields through a Multi-Grained Mixture of Experts (MG-MoE), and eliminating contradictory predictions across multi-scale features using a Cross-Grained Consistency constraint (CGC), it comprehensively outperforms previous state-of-the-art methods on four benchmarks: Lav-DF, TVIL, Psynd, and AV-Deepfake1M, while maintaining \(O(T)\) linear complexity with an inference time of only 73.4ms.

Background & Motivation

AIGC-driven audio-visual deepfakes are becoming increasingly realistic. Attackers often alter only specific segments of a video (e.g., face-swapping, voice cloning), making traditional binary real-vs-fake detection insufficient. Consequently, the task has evolved into temporal forgery localization (TFL): pinpointing exactly which temporal intervals in an untrimmed long video have been manipulated. This requires models to identify subtle manipulation clues, such as semantic replacement, emotional inconsistency, and object inpainting traces, across hundreds to thousands of frames.

Existing methods face fundamental architectural bottlenecks. CNN-based approaches are limited by local receptive fields, making it difficult to capture global inconsistencies across time—yet forgery detection precisely requires contrasting "distant" normal frames with suspicious ones. Transformer-based approaches offer a global view, but the \(O(T^2)\) complexity of self-attention leads to excessive computation and memory overhead when processing long videos of thousands of frames, forcing methods to use local window attention and thereby compromising global modeling capability. Emerging linear-complexity models (such as State Space Models like Mamba) are highly efficient but face a unique dilemma in TFL: the model must efficiently compress the global authentic context (smooth accumulation) while remaining highly sensitive to millisecond-level abrupt mutational signals at forgery boundaries (rapid response). The state transition mechanisms of traditional linear models are often fixed or data-independent, making it difficult to achieve an optimal balance between these two needs.

The key insight of this paper is that the "data-dependent decay" and dynamic state evolution mechanisms in the RWKV architecture naturally resolve this conflict. In RWKV, the decay rate \(d_t\) at each step is computed from the current input via a quadratic function. When the input frame contains a forgery mutation, \(d_t\) increases, causing the old state to be rapidly flushed and the model to immediately perceive the anomaly and reset the context. In authentic background regions, \(d_t\) decreases, allowing the state to smoothly accumulate global information. This dynamic property of "sensitivity to change, stability to consistency" makes RWKV an ideal foundation for TFL. Based on this, this paper proposes MG-RWKV, systematically adapting RWKV into a linear-complexity framework for TFL. The core idea is to replace the quadratic self-attention of Transformers with data-dependent linear recurrent state evolution, achieve explainable scale adaptation through explicit multi-grained receptive field routing, and eliminate contradictory predictions naturally introduced by multi-branch structures via cross-scale consistency constraints. These three components form a closed loop: BiDir establishes the global context, MG-MoE performs adaptive multi-scale routing on top of it, and CGC eliminates cross-scale inconsistencies introduced by routing.

Method

Overall Architecture

The goal of MG-RWKV is to take a feature sequence \(X \in \mathbb{R}^{T \times D}\) (where \(T\) is the number of time steps and \(D\) is the feature dimension) of an untrimmed video as input, and output classification scores and boundary offsets for each time step. After Soft-NMS post-processing, the top-100 manipulated segment proposals are obtained. The overall pipeline is divided into four stages.

The first stage is feature extraction: utilizing pre-trained TSN (visual) and BYOL-A (audio) to extract features, which are then fused and projected to obtain the input sequence \(X\). The second stage is the core stacking of \(L\) MG-RWKV blocks: in each layer, local context is injected via gated multi-scale dilated convolutions, followed by bidirectional RWKV scanning and dynamic routing via MG-MoE to fuse expert outputs from branches with different dilation rates, producing hierarchical multi-scale features \(\{H^{(l)}\}\). The third stage is top-down FPN fusion, refining multi-scale features into \(\{F^{(l)}\}\), with CGC constraints applied during training to align features of adjacent scales in real regions. The fourth stage is the dense prediction head: the classification head outputs frame-level scores belonging to each category \(P \in \mathbb{R}^{T \times N_c}\), and the regression head outputs boundary offsets \(O \in \mathbb{R}^{T \times 2}\). These are combined and filtered via Soft-NMS to yield the Top-100 proposals.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Video + Audio"] --> B["Feature Extraction<br/>TSN / BYOL-A"]
    B --> C["Feature Fusion & Projection → X"]
    C --> D["MG-RWKV Block ×L<br/>Multi-Scale Dilated Conv"]
    D --> E["Bidirectional RWKV Scanning<br/>Forward + Backward"]
    E --> F["MG-MoE Dynamic Routing<br/>Position-Adaptive Dilation Rate Selection"]
    F --> G["FPN Feature Pyramid<br/>+ CGC Cross-Grained Consistency"]
    G --> H["Classification Head + Regression Head"]
    H --> I["Soft-NMS → Top-100 Proposals"]

Key Designs

1. Bidirectional RWKV (BiDir): Achieving Global Bidirectional Temporal Context with Linear Complexity

Boundary localization in TFL inherently requires both preceding and succeeding context. To predict the start time of a forged segment, the model needs to know what the normal frames "afterward" look like. Relying solely on unidirectional scanning systematically leads to blurred boundaries. Standard bidirectional Transformers solve this through full self-attention, but at the cost of \(O(T^2 \cdot d)\) complexity. For instance, with \(T \approx 1500\) frames in Lav-DF, each layer requires about \(2.25 \times 10^6\) pairwise computations, which is unscalable.

This paper builds a bidirectional extension based on RWKV-7. The core of RWKV-7 is data-dependent state evolution: for the input \(x_t\) at each time step, token shift is first applied to blend adjacent frames' information to obtain \(x_t'\) and \(x_t''\); then, adaptive parameters are generated through input-dependent quadratic functions—the decay rate \(d_t = w_0 + w_1 \odot x_t' + w_2 \odot (x_t')^2\) and the context modulation \(a_t = a_0 + a_1 \odot x_t'' + a_2 \odot (x_t'')^2\). The state evolves according to the recursive formula \(s_t = e^{-e^{d_t}} \odot s_{t-1} + k_t \odot v_t + a_t \odot s_{t-1}\), producing the final output \(o_t = r_t \odot s_t \cdot \sigma(g_t)\). Notably, \(d_t\) is determined by the current input \(x_t'\): when encountering a forgery mutation, \(d_t\) increases, \(e^{-e^{d_t}} \to 0\), the old state is rapidly flushed, and the model immediately "forgets" the past and perceives the anomaly. In authentic background regions, \(d_t\) decreases, and the state smoothly accumulates global context. Meanwhile, the \(a_t\) term implements in-context state modulation, allowing the model to autonomously decide "how much historical state to retain" based on the current input. The entire computation relies only on the current input and the state from the previous time step, with a complexity of \(O(T \cdot d^2)\), which is linear with respect to the sequence length.

The bidirectional extension is simple and efficient: running forward and backward scans using independent parameter sets \(\theta_{fwd}\) and \(\theta_{bwd}\), respectively. The backward scan is equivalent to reversing the input sequence, running the forward RWKV, and then reversing the output back, yielding two sets of direction-specific features \(\{F_k^{fwd}\}\) and \(\{F_k^{bwd}\}\). Since the parameters of both directions are completely independent, the forward scan learns "history \(\to\) current" causal dependencies (suitable for detecting contextual anomalies after a forgery ends), while the backward scan learns "future \(\to\) current" anti-causal dependencies (suitable for detecting signs before a forgery starts). They complement each other to cover the complete temporal context, with MG-MoE responsible for fusing them.

2. Multi-Grained Mixture of Experts (MG-MoE): Explainable Adaptive Scale Selection via Explicit Receptive Field Routing

The temporal scales of video forgeries vary drastically: frame-level flickering (e.g., single-frame face-swapping) requires frame-by-frame analysis, whereas long-term scene synthesis (e.g., complete voice cloning) requires a coarse-grained global view. The key observation in this paper is that forgery scales are not on a completely random continuous spectrum but are instead structurally distributed like object sizes in object detection. Therefore, the receptive field is designed as explicit discrete levels—each "expert" corresponds to a specific dilation rate, and its temporal receptive field is a calculable physical quantity \(((w-1) \times d_k + 1 \text{ frames})\) rather than a black-box representation implicitly emerging from unconstrained learnable weights.

The specific design comprises expert library construction and position-adaptive routing. Expert library construction: the scale spectrum is discretized into \(K\) representative dilation rates \(\mathcal{D} = \{d_1, d_2, \dots, d_K\}\) (this paper uses \(\{1, 2, 4\}\)). The input feature \(X\) first goes through gated depthwise separable dilated convolutions \(X_{ms} = X + \gamma \cdot \text{MSConv}_{\mathcal{D}}(X)\) to inject multi-scale local context (where \(\gamma\) is a learnable gating coefficient controlling the intensity of multi-scale information injection), and then runs bidirectional RWKV along \(K\) paths, each with a different dilation rate \(d_k\). For instance, with dilation rate \(d_k=2\) and kernel size \(w=3\), the effective temporal receptive field is \((3-1) \times 2 + 1 = 5\) frames, nearly doubling the field of view compared to \(3\) frames when \(d_k=1\). This yields \(2K\) expert representations—\(K\) forward and \(K\) backward—where each expert encodes forgery evidence at a specific temporal resolution along a specific direction.

Position-adaptive routing: the input to the router comes from two complementary poolings of each expert's features—Channel Mean Pooling (\(\text{Mean}_C\)) to capture overall energy response and statistical distribution, and Channel Max Pooling (\(\text{Max}_C\)) to preserve the most prominent anomalous peak signals. These are concatenated, passed through a lightweight 1D convolution, and scaled by a temperature \(\tau\)-scaled softmax to generate position-wise \(K\)-dimensional routing weights \(W^b\). To prevent expert collapse (soft mixtures might degrade all experts into a similar average representation), a sparse Top-\(K\) gating is introduced—keeping only the \(K_{top}\) experts with the largest weights at each position, re-normalizing them, and setting the rest to zero to force experts to specialize. This paper sets \(K_{top}=2\), allowing adjacent granularities to activate simultaneously at boundaries for smooth scale transition. Finally, the fused representations \(H_{fwd}\) and \(H_{bwd}\) are merged via linear projection \(H = W_{\text{fusion}}[H_{fwd} \oplus H_{bwd}]\) to form the output of a block.

The two poolings in the router provide indispensable information: mean pooling indicates "whether coarse-grained or fine-grained features dominate in this region," while max pooling ensures "even if most frames are normal, an anomalous peak in a single frame can still be captured by routing." They complement each other; using both in experiments (\(85.91\) mAP) significantly outperforms using mean alone (\(84.71\)) or max alone (\(84.29\)).

3. Cross-Grained Consistency (CGC): Eliminating Contradictory Predictions of Multi-Scale Features in Authentic Regions

While MG-MoE effectively captures multi-scale forgery patterns, parallel heterogeneous receptive field branches naturally generate inconsistent feature representations in authentic (non-forged) regions. Coarse-grained branches might mix in distant forgery information due to excessively large fields of view, while fine-grained branches might be overly sensitive to normal texture fluctuations due to overly small fields of view. This results in contradictory feature activations at the same authentic position, directly leading to false positives. CGC enforces feature alignment between adjacent FPN scales strictly in authentic regions while preserving scale-specific discriminative capabilities in forged regions.

CGC achieves precise alignment through three interlinked design dimensions. Structurally, it adopts hierarchical pairwise pairing—applying the cosine similarity constraint \(\mathcal{L}_{CGC}\) only to adjacent FPN levels \((l, l+1)\) rather than aligning all scales at once. This is more refined than "pulling all layers together": adjacent layers are naturally more similar, minimizing resistance to alignment, and layer-by-layer propagation indirectly achieves full-scale consistency without hard compression. Spatially, it introduces a boundary-aware weight \(W_b\): the ground-truth forgery mask \(M_{gt}\) is first dilated via maximum pooling with radius \(r\) to obtain \(M_{dilate}\) (marking transition frames near boundaries as "suspected"), which is inverted to yield the negative sample mask \(M_{neg} = M_{valid} \wedge \neg M_{dilate}\). Then, within a range of \(r_b\) frames on both sides of the boundaries inside the negative sample mask (i.e., the edge areas of \(M_{dilate}\)), the constraint intensity is reduced from \(1.0\) to \(0.5\), acknowledging that these transition frames indeed have scale-dependent differences in authentic semantics (e.g., right after a forgery ends, the model's judgment on "whether it has returned to normal" naturally differs across scales) and should not be forced to align. Temporally, a progressive warmup schedule \(\lambda_{CGC}(e) = \lambda_0 \cdot e/E_w\) is employed (linearly increasing during the first \(E_w=5\) epochs, and then maintaining \(\lambda_0=0.01\)), allowing each scale to first develop independent discriminative representations before gradually imposing consistency constraints. Among these three, the warmup contributes the most (adding \(+0.87\) mAP in ablation), validating that a "diverge first, converge later" strategy is crucial to prevent premature compression of multi-scale diversity.

The final loss \(\mathcal{L}_{CGC} = \frac{1}{|\mathcal{P}|} \sum_{(i,j)\in\mathcal{P}} \left[ \frac{\sum_t M_{neg}(t) \cdot W_b(t) \cdot \left(1 - \frac{F_t^{(i)} \cdot F_t^{(j)}}{\|F_t^{(i)}\| \|F_t^{(j)}\|} \right)}{\sum_t M_{neg}(t)} \right]\) is computed only during training, incurring zero inference overhead.

Loss & Training

The total loss is formulated as \(\mathcal{L}_{total} = \mathcal{L}_{cls} + \lambda_{reg} \mathcal{L}_{reg} + \mathcal{L}_{reco} + \lambda_{CGC}(e) \mathcal{L}_{CGC}\), with clear division of labor among the four terms: \(\mathcal{L}_{cls}\) is the Focal Loss to address severe positive-negative sample imbalance (forgery frames are typically far fewer than authentic ones); \(\mathcal{L}_{reg}\) is the DIoU Loss to optimize the regression accuracy of boundary offsets; \(\mathcal{L}_{reco}\) is an auxiliary reconstruction loss providing additional representation learning signals; and \(\mathcal{L}_{CGC}\) is the cross-grained consistency loss with a progressive warmup schedule. Training uses the AdamW optimizer with an initial learning rate of \(10^{-4}\) and a cosine annealing schedule, training for 45 epochs on Lav-DF and TVIL, and 30 epochs on Psynd. The loss weights are set as \(\lambda_{reg}=2.0\), and the target CGC weight \(\lambda_0=0.01\) with a warmup duration of \(E_w=5\) epochs. Data augmentations include random cropping, label smoothing, and drop path. During inference, Soft-NMS is used to retain the top-100 proposals with a threshold of \(\theta=0.01\). All experiments are completed on a single RTX 3090 GPU.

Key Experimental Results

Main Results

MG-RWKV comprehensively reaches SOTA on three benchmarks: Lav-DF, TVIL, and Psynd. The table below shows the core comparison on Lav-DF:

Method Modality [email protected] [email protected] [email protected] AR@100
BA-TFD V+A 76.90 38.50 0.25 58.42
ActionFormer V 95.34 90.20 23.73 90.41
UMMAFormer V+A 98.83 95.54 37.61 92.48
TriDet V+A 96.29 86.84 23.64 91.00
MFMS V+A 98.47 94.15 27.80 90.69
MG-RWKV V 96.73 92.36 26.60 92.17
MG-RWKV V+A 98.92 94.81 38.47 93.41

On TVIL (visual-only), MG-RWKV comprehensively surpasses UMMAFormer (\(88.68 / 84.70 / 62.43\)) with \(91.22 / 87.44 / 71.31\) ([email protected]/0.75/0.95). Notably, the strict threshold [email protected] improves by 8.88 percentage points, alleviating the systematic shortcoming of prior methods in precise boundary localization. On Psynd (audio-only), [email protected] surges from \(79.87\) of UMMAFormer to \(90.09\) (\(+10.22\%\)), while [email protected] is \(100.00\) for both and AR is near saturation (\(98.61\)), validating that multi-grained temporal modeling is also effective for non-visual modalities. The three datasets cover multi-modal forgery, video inpainting, and voice cloning, representing three highly distinct forgery types; the consistent and significant improvements indicate that MG-RWKV addresses the fundamental bottlenecks of TFL rather than tuning parameters for specific forgery styles.

On the larger-scale AV-Deepfake1M (over one million LLM-driven audio-visual forgery segments), MG-RWKV leads DiMoDif (\(86.93 / 5.43\)) with \(87.60\) [email protected] and \(24.53\) [email protected]. More importantly, regarding the accuracy decay ratio—the degradation multiplier from [email protected] to [email protected]—UMMAFormer is \(32.7\times\), DiMoDif is \(16.0\times\), and MG-RWKV is only \(3.57\times\). This demonstrates that its boundary estimation is not a "coincidental overlap under loose thresholds" but is structurally regularized by bidirectional recurrence and CGC into precise temporal ranges.

Ablation Study

Progressive component ablation on TVIL (Baseline = Unidirectional RWKV-7 + FPN, without BiDir/MG-MoE/CGC):

Configuration BiDir MG-MoE CGC mAP [email protected] AR@100
Baseline × × × 83.32 63.37 90.86
+BiDir ✓ × × 83.08 66.58 91.01
+MG-MoE ✓ ✓ × 84.35 65.87 91.57
Full (MG-RWKV) ✓ ✓ ✓ 85.91 71.31 92.24

The three components emphasize different aspects. BiDir mainly improves the strict threshold [email protected] (\(+3.21\)), because its bidirectional context directly improves boundary localization precision, though mAP on TVIL declines slightly by 0.24 (Note: BiDir yields positive mAP gains on both Lav-DF and Psynd; the slight drop on TVIL is due to the dataset's specific forgery pattern characteristics). MG-MoE contributes \(+1.27\) mAP through adaptive granularity selection. CGC yields the largest contribution (\(+1.56\) mAP, \(+5.44\) [email protected]), because cross-scale consistency directly suppresses false positives from multi-scale features in authentic regions, which is especially pronounced under strict boundary thresholds.

MG-MoE sub-component ablation (TVIL): The dilation rate combination \(\{1, 2, 4\}\) achieves the optimal \(85.91\) mAP, surpassing both single-scale (\(83.65\)) and four-scale \(\{1, 2, 4, 8\}\) (\(85.10\)). This indicates that 3 moderately spaced scales are optimal, as too many scales introduce excessive temporal smoothing that blurs boundaries. \(K_{top}=2\) is optimal (\(85.91\)), while \(K=1\) (\(84.43\)) is too rigid for smooth transitions at boundaries, and \(K=3\) (\(85.66\)) introduces redundant noise. The router input with aggregated mean and max pooling (\(85.91\)) outperforms using mean alone (\(84.71\)) or max alone (\(84.29\)), validating that both global statistical patterns and local anomalous peaks are indispensable for scale selection.

CGC sub-component ablation (TVIL): The basic \(\mathcal{L}_{CGC}\) brings \(+0.42\) mAP; adding the boundary-aware weight \(W_b\) yields another \(+0.27\) mAP; and incorporating the warmup schedule \(\lambda(e)\) contributes the most with \(+0.87\) mAP, accumulating to \(+1.56\) mAP. The dominant contribution of warmup validates the core design intuition: if cross-scale consistency constraints are applied immediately in the early stages of training, different scales are forced to align before they can form independent discriminative representations, thereby squeezing out multi-scale diversity. Letting each scale develop freely for 5 epochs before gradually converging is a necessary condition for CGC to be effective.

Key Findings

  • The warmup of CGC is the single design choice contributing the most in the ablation study (\(+0.87\) mAP), indicating that "diverge first and converge later" is a key prerequisite for multi-scale consistency constraints to be effective. This lesson is applicable to any task requiring alignment constraints across multiple branches.
  • The RWKV-7 backbone significantly outperforms Mamba: Replacing the backbone with Mamba under the same setting drops the mAP from \(82.43\) to \(80.15\). This validates that the advantages of RWKV's data-dependent decay and in-context state modulation mechanisms in perceiving local forgery mutations are substantial, rather than merely being a consequence of it being a linear model.
  • MG-MoE routing visualization reveals clear semantic patterns: Coarse-grained experts dominate the weights in core forged regions (requiring a wide context to capture manipulation patterns), whereas fine-grained experts activate at boundaries and in authentic regions (requiring precise frame-by-frame localization), with smooth scale transitions at boundaries. This proves the router learns position-adaptive temporal properties rather than simply overfitting to discrete labels. This explainability is a direct benefit of assigning explicit physical meanings (dilation rates) to the experts.
  • Inference Efficiency: The full model runs inference in 73.4ms, uses 274MB GPU memory, and contains 56.2M parameters on an RTX 3090, with CGC introducing zero inference overhead. Compared to the Baseline's 34.3ms / 199MB / 36.7M, it trades approximately \(2\times\) inference time and \(1.5\times\) parameters for a substantial gain of \(+4.86\) mAP, presenting a reasonable efficiency-accuracy trade-off.

Highlights & Insights

  • RWKV's data-dependent decay naturally fits the "global smoothness vs. local mutation" conflict of TFL: This is the deepest insight of this paper. The decay rate \(d_t\) at each step in RWKV is computed from the current input via a quadratic function \(w_0 + w_1 \odot x_t' + w_2 \odot (x_t')^2\). When an input frame contains a forgery mutation, \(d_t\) increases, the state is rapidly flushed, and the model immediately "resets" the context and perceives the anomaly. In authentic background regions, \(d_t\) decreases, and the state accumulates smoothly. This dynamic property of "sensitivity to change, stability to consistency" is not an after-the-fact regularization or gating, but an inherent property of RWKV's recurrent structure, perfectly resolving the core challenge of TFL. This paper is not a simple "swap Transformer for RWKV," but a realization of the deep alignment between TFL's task requirements and the architectural properties of RWKV.
  • MG-MoE assigns explicit physical meanings to experts with dilation rates: Traditional MoE experts are black-box FFNs, and selecting which expert lacks semantic explanation. This paper maps each expert to an explicit dilation rate \(d_k\), whose temporal receptive field is a precisely calculable physical quantity (e.g., \(d_k=4, w=3\) corresponds to a 9-frame view). The routing weights directly reflect "how much temporal view is required at this position." The visualization presents clear patterns: coarse-grained \(\to\) forgery core, and fine-grained \(\to\) boundaries and authentic regions. This is particularly valuable in the security domain where deep learning explainability is scarce. This paradigm of "assigning physical meaning to experts" can be transferred to any temporal task requiring multi-scale perception (e.g., temporal action detection, video anomaly detection).
  • CGC's three-dimensional joint design embodies the philosophy that "constraints are not the tighter, the better": Hierarchical pairwise pairing (avoiding global compression), boundary-aware weighting (acknowledging the rationality of transitions), and warmup scheduling (leaving room for diversity to develop)—none of these three dimensions are entirely novel on their own, but their synergy transforms CGC from "aligning all scales everywhere" to "aligning at the right place, at the right time, with the right intensity." This philosophy of leaving room in constraints is highly reusable for future multi-branch/multi-scale architectural designs.
  • The accuracy decay ratio analysis on AV-Deepfake1M is highly insightful: By measuring boundary stability through the ratio of [email protected] / [email protected], MG-RWKV's \(3.57\times\) vs. DiMoDif's \(16.0\times\) vs. UMMAFormer's \(32.7\times\) proves much more convincingly than absolute precision values that the CGC constraint indeed "structurally improves boundary quality" rather than simply offering numerical fine-tuning gains. This analytical method is worth promoting in other localization/detection tasks.

Limitations & Future Work

  • Multi-modal fusion is relatively preliminary: Currently, TSN and BYOL-A features are simply concatenated and projected before being fed into RWKV, lacking dedicated cross-modal interaction mechanisms (such as cross-attention or cross-modal gating). In audio-visual joint forgery scenarios like AV-Deepfake1M, manipulations in both modalities often occur simultaneously, and their inconsistencies themselves serve as strong detection signals, which the current framework fails to exploit explicitly. One could consider introducing cross-modal gating within the MG-RWKV block—allowing the visual RWKV state and audio RWKV state to modulate each other—or designing cross-scanning paths between modalities.
  • The combination of dilation rates is manually set: While \(\{1, 2, 4\}\) is optimal in experiments, this discretization is essentially an a priori choice. For entirely new types of forgery (such as extremely long-term scene replacement or extremely short frame-level injections), the fixed discrete scale spectrum might be insufficient. One could explore content-based dynamic dilation rate generation (e.g., using a small network to automatically predict the optimal dilation rate based on input statistics) or continuous scale parameterization (treating dilation rates as differentiable parameters optimized end-to-end).
  • CGC hyperparameters require fine-tuning for target domains: Although the paper demonstrates that \(\lambda\) and \(r\) are stable within a wide range (\(\lambda \in [0.01, 0.03], r \in [6, 10]\)), optimal values still vary across datasets. In practical deployment, a small amount of target-domain labeled data might be needed to tune these two hyperparameters. Meta-learning or adaptive scheduling (automatically adjusting \(\lambda\) based on the feature dispersion of each scale during training) could be explored to reduce manual intervention.
  • The recurrent inference property of RWKV is underutilized: The recurrent form of RWKV naturally supports frame-by-frame incremental inference (similar to hidden state passing in RNNs), but this paper still processes videos in batch as a whole segment. For real-time video stream monitoring scenarios, one could process only a few newly arrived frames at each step while reusing previous state caches, achieving true streaming TFL. This offers a massive speed advantage over Transformers, which require recomputing the entire history. This paper does not explore this direction, leaving it as a clear unexploited potential.
  • Evaluations are only on medium-scale datasets, lacking broader cross-domain evaluation: Although covering three mainstream benchmarks and AV-Deepfake1M in the appendix, the diversity of forgery types remains limited (content-driven face-swapping, video inpainting, and voice cloning). More complex scenarios such as partial frame replacement, progressive manipulation, and multi-segment interleaved forgeries have not yet been addressed. Evaluating generalization performance under broader cross-domain settings (e.g., training on Lav-DF and testing on TVIL) would be much more convincing.
  • vs. UMMAFormer (prior TFL SOTA): UMMAFormer utilizes a Transformer backbone + temporal anomaly attention + cross-attention FPN, achieving \(37.61\) on Lav-DF for [email protected] with a complexity of \(O(T^2)\). MG-RWKV replaces the backbone with RWKV and changes the modeling from attention to recurrence, increasing [email protected] to \(38.47\) while reducing complexity to \(O(T)\). The core difference lies not in the FPN or the prediction head (their structures are similar), but in the modeling approach of the backbone—UMMAFormer's self-attention computes interactions indiscriminately across all positions, whereas MG-RWKV's data-dependent decay allows historical information with low similarity to the current frame to be forgotten quickly while high-similarity information is retained, which is a more temporally causal context modeling approach.
  • vs. State Space Models like Mamba: Mamba's state transition mechanism is data-selective (utilizing input-dependent \(\Delta\) to control discretization step size), but its state update formula remains essentially linear, lacking explicit in-context state modulation terms like \(a_t \odot s_{t-1}\) in RWKV. On TFL, Mamba's mAP is \(80.15\) vs. RWKV-7's \(82.43\) (a \(2.28\) point gap), confirming that explicit state modulation mechanisms are more effective than implicit step-size control in tasks requiring "sensitiveness to sudden mutations." This finding is valuable for backbone selection in general anomaly detection tasks.
  • vs. DiMoDif (strong baseline on AV-Deepfake1M): DiMoDif achieves higher AR under loose thresholds (AR@100: \(76.64\) vs. \(69.03\)), but its accuracy drops sharply under strict thresholds ([email protected]: \(5.43\) vs. \(24.53\)). This asymmetry in accuracy and recall reflects different implicit preferences—DiMoDif tends to generate broad coarse-grained proposals (high recall but inaccurate boundaries), whereas MG-RWKV's CGC constraint biases it toward more precise boundaries (high accuracy but potentially missing some marginal proposals). For forensics, the importance of precise boundaries typically outweighs recall (mislabeled boundaries are more dangerous than missed detections), making MG-RWKV's trade-off more reasonable. However, if downstream tasks prioritize recall (such as initial screening), DiMoDif's approach remains valuable.

Rating

  • Novelty: ⭐⭐⭐⭐ Using RWKV for TFL is itself novel, but the greater highlight lies in uncovering the deep alignment between RWKV's data-dependent decay and TFL's task requirements (rather than a simple architecture replacement). In addition, MG-MoE's design of assigning explicit physical meaning to experts via dilation rates and the three-dimensional joint design of CGC form a systematic new framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Four datasets (including the large-scale AV-Deepfake1M) + progressive component ablation + fine-grained sub-component ablations for both MG-MoE and CGC + hyperparameter sensitivity analysis + efficiency/memory comparison + routing visualization + qualitative comparison. The coverage is comprehensive, and each ablation is backed by insightful analysis rather than a simple layout of numbers.
  • Writing Quality: ⭐⭐⭐⭐ The motivation chain is complete (challenge \(\to\) insight \(\to\) design \(\to\) validation), core formulas correspond well with text descriptions, and the tables and figures are high in information density. The accuracy decay ratio analysis and progressive visualization in the appendix further strengthen the persuasiveness of the arguments. A minor flaw is that the specific formulation of the auxiliary reconstruction loss \(\mathcal{L}_{reco}\) is not elaborated.
  • Value: ⭐⭐⭐⭐ TFL is a core sub-problem in the AIGC security domain, with growing practical demand as deepfake technology proliferates. This paper provides a linear-complexity, high-precision solution runnable on consumer-grade GPUs. Furthermore, MG-MoE's explainable routing and CGC's constraint design can be directly transferred to general temporal detection/localization tasks (temporal action detection, video anomaly detection, audio event localization, etc.).