Following Motion for Sequential Modeling in Video Frame Interpolation¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Video Understanding
Keywords: Video Frame Interpolation, State Space Models, Mamba, Motion-Guided Serialization, Contextual Synthesis
TL;DR¶
Addressing the semantic fragmentation caused by fixed scanning orders in conventional State Space Models for video frame interpolation, MGMVFI reformulates optical flow motion trajectories into a 1D serialization mechanism for Mamba, combined with hollow-convolution contextual synthesis to restore flow failure zones, achieving state-of-the-art interpolation with linear computational complexity.
Background & Motivation¶
Video Frame Interpolation (VFI) aims to synthesize intermediate frames at arbitrary timestamps between consecutive input frames, playing a crucial role in video frame rate up-conversion, slow-motion generation, and novel view synthesis. Deep learning-based VFI approaches have predominantly relied on Convolutional Neural Networks (CNNs) and Vision Transformers. However, CNNs suffer from restricted receptive fields, making them prone to severe ghosting and blur when encountering large-magnitude or fast-moving motions. Conversely, Transformers capture global long-range dependencies via attention mechanisms, but their quadratic computational complexity \(O(N^2)\) becomes prohibitively expensive on high-resolution formats such as 2K and 4K video.
Recently, selective State Space Models (SSMs), particularly S6 / Mamba, have emerged as promising alternatives offering linear \(O(N)\) computational complexity alongside dynamic recurrent modeling. Nevertheless, SSMs are inherently causal sequence models whose hidden state transitions depend strictly on the scanning trajectory chosen to flatten 2D/3D grids into 1D sequences. Existing visual Mamba interpolation methods (such as the interleaved scanning in VFIMamba or shifted-window 3D Hilbert curve scanning in LC-Mamba) apply fixed, motion-agnostic scanning orders. Under dynamic and rapid motion, features of the same moving object across consecutive frames are separated by long sequential distances in the serialized 1D token stream. This forces the SSM hidden states to simultaneously track long-range motion paths and model fine-grained inter-frame residuals, leading to significant degradation in challenging dynamic scenes.
To overcome this core tension, this paper reinterprets motion alignment not as mere spatial warping or image blending, but as a serialization problem for sequence models. Core Idea: utilize bi-directional optical flow trajectories to dynamically determine the 1D token ordering of the state space model (Motion-Guided Serialization, MGS), ensuring that Mamba causal state transitions occur along semantically consistent motion paths, while employing a hollow-convolution contextual synthesis module to reconstruct regions corrupted by optical flow failure.
Method¶
Overall Architecture¶
The overall MGMVFI architecture comprises three cascaded stages: multi-scale feature extraction and optical flow estimation, motion-guided feature processing (MGM block), and intermediate frame synthesis. First, a shared CNN encoder extracts multi-scale deep features \(F_0, F_1\) from input frames \(I_0, I_1\), while a pre-trained optical flow network computes bi-directional flow fields \(f_{0 \to 1}, f_{1 \to 0}\). Next, the MGM block applies forward splatting and backward bilinear warping to align features to the target timestamp, detects occlusion and brightness-failure regions via bi-directional flow consistency checks, and invokes a masked contextual adapter. The refined features are then interleaved into a 1D token sequence and processed by a 2D selective state space block (SS2D) paired with an Efficient Discriminative Frequency-domain Feed-Forward Network (EDFFN) for residual modeling. Finally, a lightweight synthesis head reconstructs the target RGB frame \(I_t\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input Frames I0, I1 and Bi-directional Flow f01, f10"] --> MGS["Motion-Guided Serialization<br/>Forward splatting intermediate flow & backward resampling"]
MGS --> CS["Contextual Synthesis<br/>Consistency mask + hollow convolution repair"]
CS --> Fusion["Sequence Interleaving & Fusion<br/>Residual addition and spatial token concatenation"]
SS2D --> EDFFN["Frequency Feed-Forward Network EDFFN<br/>Frequency-domain gating for high-frequency details"]
Fusion --> SS2D["SS2D State Space Modeling<br/>4-direction causal updates for spatiotemporal dynamics"]
EDFFN --> Out["Frame Synthesis Output It"]
Key Designs¶
1. Motion-Guided Serialization: reformulating motion alignment as an SSM token sequence ordering mechanism While classical flow-based VFI models warp features to the target grid for direct spatial weighting, this work introduces the perspective that motion trajectories should dictate the 1D token sequence order fed into the SSM. Assuming locally linear motion between frames, intermediate displacement fields \(f_{t \to 0}\) and \(f_{t \to 1}\) are projected onto target grid \(p_t\) via forward splatting \(\mathcal{S}\): \(f_{t \to 0} = \mathcal{S}(-t \cdot f_{0 \to 1})\) and \(f_{t \to 1} = \mathcal{S}(-(1-t) \cdot f_{1 \to 0})\). Source sampling coordinates \(p_0 = p_t + f_{t \to 0}\) and \(p_1 = p_t + f_{t \to 1}\) are calculated, followed by bilinear backward warping \(\mathcal{B}\) to fetch motion-aligned feature maps \(F'_0 = \mathcal{B}(F_0, p_0)\) and \(F'_1 = \mathcal{B}(F_1, p_1)\). Applying a standard raster scan to these motion-aligned maps guarantees that tokens from the same moving entity across time are placed immediately adjacent in the 1D sequence, vastly simplifying the SSM learning objective to focus on local inter-frame residual variations.
To quantitatively measure token semantic coherence, the authors define the Sequential Feature Similarity metric \(S\): $\(S = \frac{1}{N-1} \sum_{n=1}^{N-1} \cos(\phi(z_{n+1}), \phi(z_n))\)$ where \(N\) denotes sequence length and \(\phi(z_n)\) is the feature representation of the \(n\)-th token. MGS achieves \(S = 0.46\), significantly exceeding the 0.35 of interleaved scanning.
2. Contextual Synthesis via Masked Adapter: repairing corrupted features in flow-unreliable zones While backward warping populates the feature grid, optical flow estimates frequently break down in regions subject to physical occlusions, disocclusions, or abrupt illumination changes that violate brightness constancy. In such failure zones, optical flow models often predict zero vectors, sampling background features instead of dynamic foreground and causing severe ghosting. Rather than simply zeroing out corrupted features—which leaves single-frame dependencies and causes blur—the paper introduces a flow consistency check coupled with a masked adapter. Forward-backward round-trip error is computed: $\(\epsilon_{0 \to 1}(p) = \left\| f_{0 \to 1}(p) + f_{1 \to 0}(p + f_{0 \to 1}(p)) \right\|_2^2\)$ A binary reliability mask \(M_{0 \to 1}(p)\) is obtained by comparing against a threshold proportional to the average flow magnitude (\(\alpha=1.1\)). In masked failure regions, a masked adapter \(\mathcal{A}_1\) uses a \(7 \times 7\) hollow convolution to explicitly exclude the corrupted center token and synthesize replacement features purely from surrounding valid spatial context: $\(F^c_1 = \text{Conv}_{1 \times 1}(\text{GELU}(\text{ConvHollow}_{7 \times 7}(F'_1))) + F'_1\)$ The refined feature is gated by \(F^{\text{ref}}_1 = (1 - M_{0 \to 1}) \cdot F'_1 + M_{0 \to 1} \cdot F^c_1\), with a symmetric module handling \(F'_0\) and \(M_{1 \to 0}\).
3. Frequency-Domain Feed-Forward Network (EDFFN): compensating for high-frequency attenuation in Mamba Because visual Mamba blocks inherently focus on low-frequency global structures, they risk over-smoothing fine textures and sharp edges. Additionally, motion-guided serialization introduces modest computational overhead. To mitigate this trade-off, a modified Efficient Discriminative Frequency-domain FFN (EDFFN) is placed after the SS2D block. By performing frequency-domain gating and filtering on downscaled channel dimensions, EDFFN selectively restores fine spatial details and high-frequency textures with minimal extra parameter and compute costs.
A Worked Example¶
Consider a moving vehicle situated at the left border (Patch 7) in frame \(I_0\) and traveling to the right (Patch 9) in frame \(I_1\): Under standard interleaved scanning, the serialized 1D sequence places Patch 7 and Patch 9 dozens or hundreds of positions apart; the causal hidden states must carry information over a vast temporal gap while tracking complex trajectories. Under MGMVFI: 1. The optical flow model estimates the large displacement vector of the car, which forward splatting maps to the intermediate grid at \(t=0.5\); 2. Backward resampling pulls Patch 7 from \(F_0\) and Patch 9 from \(F_1\) to the identical spatial location on the target grid; 3. Flow consistency detects that the background tree occluded by the moving vehicle has invalid flow, flagging it for \(7 \times 7\) hollow convolution context synthesis from surrounding uncorrupted foliage; 4. Raster scanning over the aligned target grid places Patch 7 and Patch 9 as adjacent tokens in the 1D sequence, enabling the SS2D block to perform instant causal state fusion with minimal overhead before EDFFN sharpens textural details.
Loss & Training¶
The entire framework is trained end-to-end using a composite loss function \(\mathcal{L}_{\text{total}}\): $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{lap}}(I_t, I_t^{\text{GT}}) + \lambda \mathcal{L}_{\text{warp}}\)$ where \(\mathcal{L}_{\text{lap}}\) is the Laplacian pyramid L1 loss between synthesized frame \(I_t\) and ground truth \(I_t^{\text{GT}}\), penalizing multiscale structural errors; \(\mathcal{L}_{\text{warp}} = \sum_s \mathcal{L}_{\text{lap}}(I_t^s, I_t^{\text{GT}})\) provides intermediate supervision across feature pyramid scales \(s\), weighted by \(\lambda = 0.5\). The model is trained on Vimeo-90K for 150 epochs using \(256 \times 256\) crops, AdamW optimizer, and cosine learning rate decay from \(2 \times 10^{-4}\) down to \(2 \times 10^{-5}\). Bi-directional flow is pre-computed using a frozen RAFT-Large network.
Key Experimental Results¶
Main Results¶
Quantitative comparisons against representative CNN-, Transformer-, and SSM-based VFI models across Vimeo-90K, UCF101, and SNU-FILM (PSNR / SSIM):
| Dataset | Metric | Ours (MGMVFI) | Prev. SOTA (VFIMamba) | Baseline (EMA-VFI) | Gain |
|---|---|---|---|---|---|
| Vimeo-90K | PSNR / SSIM | 36.67 / 0.9820 | 36.64 / 0.9819 | 36.64 / 0.9819 | +0.03 dB / +0.0001 |
| UCF101 | PSNR / SSIM | 35.53 / 0.9710 | 35.45 / 0.9702 | 35.48 / 0.9701 | +0.08 dB / +0.0008 |
| SNU-FILM (Easy) | PSNR / SSIM | 40.45 / 0.9910 | 40.51 / 0.9912 | 39.98 / 0.9910 | -0.06 dB / -0.0002 |
| SNU-FILM (Medium) | PSNR / SSIM | 36.38 / 0.9804 | 36.40 / 0.9805 | 36.09 / 0.9801 | -0.02 dB / -0.0001 |
| SNU-FILM (Hard) | PSNR / SSIM | 31.04 / 0.9411 | 30.99 / 0.9401 | 30.94 / 0.9392 | +0.05 dB / +0.0010 |
| SNU-FILM (Extreme) | PSNR / SSIM | 25.86 / 0.8732 | 25.79 / 0.8682 | 25.69 / 0.8661 | +0.07 dB / +0.0050 |
| Xiph 2K | PSNR / SSIM | 37.15 / 0.9454 | 37.13 / 0.9451 | 36.74 / 0.9445 | +0.02 dB / +0.0003 |
| Xiph 4K | PSNR / SSIM | 34.65 / 0.9059 | 34.62 / 0.9059 | 34.54 / 0.9054 | +0.03 dB / 0.0000 |
Ablation Study¶
1. Progressive Module Ablation on Vimeo-90K and SNU-FILM Hard:
| Config | Vimeo-90K PSNR / SSIM | SNU-FILM Hard PSNR / SSIM | Note |
|---|---|---|---|
| Baseline pure SSM (SS2D) | 35.96 / 0.9793 | 29.71 / 0.9293 | Motion-agnostic raster scan, struggles with large motions |
| + EDFFN | 36.10 / 0.9797 | 29.80 / 0.9300 | Frequency gating enhances fine textures (+0.14 / +0.09 dB) |
| + MGS (Motion Serialization) | 36.26 / 0.9802 | 30.24 / 0.9345 | Trajectory alignment brings dramatic gain on Hard (+0.44 dB) |
| + Contextual Synthesis (Full) | 36.67 / 0.9820 | 31.04 / 0.9411 | Hollow conv restores occlusion failure (+0.80 dB jump on Hard) |
2. Comparison of Serialization Scanning Strategies:
| Scanning Method | Sequential Similarity \(S \uparrow\) | Vimeo-90K PSNR / SSIM | SNU-FILM Hard PSNR / SSIM |
|---|---|---|---|
| Sequential Raster | 0.15 | 35.39 / 0.9791 | 29.45 / 0.9336 |
| Spiral Scan | 0.12 | 35.17 / 0.9781 | 28.83 / 0.9302 |
| Z-scan | 0.29 | 35.50 / 0.9792 | 29.79 / 0.9335 |
| Hilbert Curve | 0.33 | 36.45 / 0.9813 | 30.57 / 0.9376 |
| Interleaved Scan | 0.35 | 36.56 / 0.9817 | 30.88 / 0.9390 |
| Ours (MGS) | 0.46 | 36.67 / 0.9820 | 31.04 / 0.9411 |
Key Findings¶
- Superiority amplifies under extreme motion: As the motion magnitude increases across SNU-FILM splits, MGMVFI's advantage grows continuously, outperforming VFIMamba by 0.07 dB in PSNR and 0.0050 in SSIM on the Extreme split.
- Contextual synthesis is crucial for occlusion handling: Integrating contextual synthesis lifts SNU-FILM Hard performance from 30.24 dB to 31.04 dB (+0.80 dB), proving that repairing flow-corrupted tokens is critical to preventing error propagation in SSM causal states.
- Robust flow backbone elasticity: Switching the flow estimator from RAFT-Large (5.26M params, 415.9 ms) to RAFT-Small (0.99M params, 277.9 ms) incurs only a slight 0.12 dB drop (30.92 dB on SNU-FILM Hard), offering favorable latency-accuracy trade-offs for resource-constrained deployments.
Highlights & Insights¶
- Conceptual Shift in Sequence Modeling: Instead of treating optical flow purely as an image-warping tool, the authors elevate it to a topological serializer for 1D causal models, bridging the dimensional mismatch between physical motion and state space transitions.
- Hollow Convolution for Clean Inpainting: The \(7 \times 7\) hollow convolution explicitly masks the center corrupted token, ensuring that feature repairs only absorb reliable surrounding spatial information without diffusing bad tokens.
- Frequency-Domain Complementarity: Incorporating EDFFN addresses the low-frequency bias of Mamba architectures, demonstrating that lightweight frequency-domain gating is an effective counterpart to spatial state space layers.
Limitations & Future Work¶
- Cascade Error from Flow Estimation: The serialization order is strictly conditioned on optical flow. Under severe motion blur or non-rigid physics where flow collapses, erroneous token ordering can propagate into the SSM. Joint end-to-end flow and state space refinement is an important next step.
- Single-Trajectory Representation Limit: The current MGS assumes a dominant intermediate flow field per pixel, which struggles to separate multi-layer semi-transparent motions or overlapping objects with conflicting velocities. Exploring multi-trajectory serialization remains an open direction.
Related Work & Insights¶
- vs VFIMamba [47]: VFIMamba established SSMs in VFI using static interleaved column scanning. MGMVFI replaces heuristic scans with motion-adaptive serialization, increasing 1D token similarity by 31% and setting new state-of-the-art results on large-motion benchmarks.
- vs LC-Mamba [18]: LC-Mamba employed 3D Hilbert curve scans within local windows to preserve spatiotemporal continuity. MGMVFI explicitly grounds sequence paths on true motion fields, achieving superior reconstruction fidelity on 4K resolution.
- vs EMA-VFI [48]: EMA-VFI utilizes Transformer attention across frames with quadratic computational complexity at higher resolutions. MGMVFI delivers superior large-motion accuracy while preserving linear scalability with respect to resolution.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Reformulates optical flow as a motion-adaptive serialization operator for SSMs]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 4 benchmarks, 4 motion tiers, 4K video, and detailed scanning ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear motivation, and strong narrative continuity]
- Value: ⭐⭐⭐⭐⭐ [Provides a foundational paradigm for applying state space models to dynamic video and low-level vision tasks]