Aligning Anything: Hierarchical Motion Estimation for Video Frame Interpolation¶

Conference: ECCV 2026
Paper: Official paper page
PDF: Full paper
Code: https://github.com/hhhhhumengshun/SAM-VFI
Area: Video Understanding / Video Frame Interpolation
Keywords: Object priors, hybrid context, flow distillation, intra-object feature correlations, SAM

TL;DR¶

The paper injects SAM object regions into contextual feature extraction, flow distillation, and feature supervision, retaining pixel-level motion flexibility while improving object-level coherence; under matched retraining settings, AMT-G improves from 36.42 to 36.62 dB on Vimeo90K.

Background & Motivation¶

Video frame interpolation must determine where the content at an unseen intermediate time should come from in two neighboring frames. Motion-based systems such as AMT and IFRNet extract multiscale features, estimate intermediate flows toward the input frames, and warp and fuse their content. Similar-looking pixels can correspond to different objects, however, and separate estimates within one object can become inconsistent. A wider attention window exposes more candidates without necessarily identifying which candidates belong to the same moving entity.

Object-level information can reduce this ambiguity, but imposing one displacement on an entire object would remove genuine articulation and deformation. Meanwhile, a pretrained optical-flow teacher is not an infallible substitute for missing motion supervision: its pseudo labels can be unreliable at occlusions, boundaries, and large displacements, and optical-flow accuracy is not identical to interpolation quality. The useful role for object structure is therefore to guide pixel estimates and their supervision, rather than replace a dense flow field with rigid region motion.

SAM makes this possible without a fixed semantic category vocabulary. Core idea: use object masks as the support for contextual aggregation, the reference for assessing pseudo-flow reliability, and the boundary for feature correlations, while keeping the interpolation network's pixel-level motion representation.

Method¶

Overall Architecture¶

This is an enhancement to existing motion-based interpolators, not a replacement synthesis architecture. Two input frames and a target time enter the original multiscale encoder, motion decoder, and warping/fusion pipeline. The full configuration adds Hybrid Contextual Feature Extraction (HCE) to the feature path and Hierarchical Motion Loss (HML) plus Hierarchical Feature Loss (HFL) during training. AMT's all-pairs correlation machinery is retained rather than redesigned.

The training triplet includes a real intermediate frame. Its SAM masks define the supervision regions, while LiteFlow provides pseudo flows from that frame to both input frames. At inference, the real intermediate frame and both loss branches disappear. HCE, when enabled, still requires masks of the input frames. The dashed connections below are training supervision, not postprocessing stages executed after synthesis at deployment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input frames and<br/>multiscale features"] --> B["Hybrid Contextual<br/>Feature Extraction (HCE)"]
    S["Input-frame SAM masks<br/>unique region assignment"] --> B
    B --> C["Original motion decoder<br/>and frame synthesizer"]
    C --> D["Predicted intermediate frame"]
    C -.-> E["Hierarchical Motion<br/>Loss (HML)"]
    T["Ground-truth frame and masks<br/>LiteFlow pseudo flows"] -.-> E
    D -.-> F["Hierarchical Feature<br/>Loss (HFL)"]
    T -.-> F

Key Designs¶

1. Hybrid Contextual Feature Extraction: give each local pixel a representation of its object

SAM can produce overlapping or nested masks and leave some pixels uncovered. The method first processes masks in descending area order so that each pixel receives exactly one region assignment, placing uncovered pixels in a new background mask. The cached text does not specify the overlap overwrite implementation beyond that ordering and uniqueness requirement, so it does not justify inventing a more detailed priority algorithm. Nor does the framework introduce an explicit cross-frame object-ID matcher: the masks primarily organize features within each frame.

For each region, HCE indexes the interpolation encoder's pixel features, averages them, and copies the resulting region vector back to the region's positions. Each pixel thus receives both its local representation and an object-wide summary. These are concatenated across channels and processed with convolution, PReLU, and channel attention, while the architecture diagram preserves the original pixel features through a residual path. The region summary supplies context rather than forcing a constant output flow. Retaining local features is important because different parts of one object can move differently. In particular, these object representations are pooled interpolation features, not SAM image embeddings directly substituted for the encoder features.

2. Hierarchical Motion Loss: assess pseudo-flow reliability at pixel and object levels

HML obtains bidirectional pseudo labels from pretrained LiteFlow using the real intermediate frame available in training. Its pixel-level discrepancy is the endpoint error between a pseudo flow and the student's predicted flow. For the object-level discrepancy, the method averages pseudo-flow vectors inside each intermediate-frame mask, copies the mean vector throughout that region, and measures each pseudo vector's endpoint error relative to this object mean. One discrepancy captures teacher-student disagreement; the other captures how far a teacher vector departs from the regional motion reference. The mean is a reliability reference, not a hard target requiring the student's object motion to be rigid.

Equation (10) combines the two discrepancies into a reliability quantity. Here \(P_l\) denotes teacher-student endpoint error, \(O_l\) denotes pseudo-flow deviation from the object mean, and \(l\) indexes the direction:

\[ H_l=\exp\left[-\beta(P_l+O_l)\right]. \]

Larger discrepancies lower reliability. The important detail is that this is not merely a scalar multiplying an otherwise unchanged loss. Following task-oriented distillation, reliability controls the robust Charbonnier parameters through \(\epsilon=10^{(H_l-1)/3}\) and \(\alpha=H_l/2\). Distillation then supervises upsampled lower-level decoder flows, reducing the influence of questionable teacher knowledge. The cached extraction of Equation (11) loses operators, so an exact full loss is not reconstructed here. Its accompanying text and visible summation establish three lower decoder scales and two directions; the prose does not report a numerical value for \(\beta\).

3. Hierarchical Feature Loss: constrain appearance relationships within each object

Suppressing unreliable flow supervision does not by itself remove broken texture or incoherent structure inside a synthesized object. HFL extracts VGG19 features from the predicted and real intermediate frames and models spatial feature relationships using self-attention. The intermediate-frame masks restrict these relationships to individual objects, preventing unrelated objects from interfering. The loss compares the resulting object-conditioned representations between the prediction and ground truth, rather than only comparing colors or supervising a separate tracking result.

This turns the desired visual coordination of an object's parts into a feature-level training constraint, complementing HML's dependence on pseudo flows. Any benefit to motion continuity is indirect, through interpolation training; the loss does not mathematically guarantee a continuous flow field. Equations (12)-(14) in the cached extraction contain damaged brackets and operators, and the prose describes the attention scaling variable as the number of attention heads. Consequently, this note retains the identifiable mechanism without silently replacing the paper's expressions with a standard attention implementation. VGG19 feature extraction, object-restricted correlations, and comparison of real and predicted branches are the clearly supported elements.

A Worked Example¶

Consider the articulated-motion situations discussed in the paper. This is a conceptual walkthrough, not an additional measured experiment. Similar local textures in the two inputs can leave the motion decoder uncertain about correspondences. HCE augments each position with the summary of its assigned object region, helping distinguish object context without requiring all of that object's pixels to share a displacement.

During training, a boundary vector from LiteFlow that disagrees strongly with regional motion can be downweighted through HML's combined object discrepancy and teacher-student disagreement. If the synthesized intermediate object still contains a broken local structure, HFL compares object-restricted feature relationships with the real intermediate frame, without mixing in the neighboring background. At deployment, those two training constraints are already reflected in the model parameters. Choosing the configuration without HCE also removes the need to segment the input frames.

Loss & Training¶

Image reconstruction uses a Charbonnier penalty with \(\alpha=0.5\) and \(\epsilon=10^{-3}\). HML uses its adaptive robustness parameters, and HFL adds supervision for intra-object perceptual structure. Reconstruction, motion distillation, and feature supervision have distinct roles; HFL does not replace the full reconstruction objective. The main text does not specify the complete set of total-loss coefficients, so none are invented here.

Training uses 51,312 Vimeo90K triplets cropped to 224 by 224, AdamW, batch size 48, and 300 epochs on eight RTX 4090 GPUs. The learning rate follows cosine annealing from \(10^{-4}\) to \(10^{-5}\). The reported HCE count is six for large interpolators and one for small ones. The full configuration favors quality; HML plus HFL alone retains the original inference structure but still requires segmentation and teacher supervision during training.

Key Experimental Results¶

Main Results¶

The following matched comparison comes from Table 1's retrained AMT-G rows. Each metric cell lists PSNR in dB, SSIM, and LPIPS; higher PSNR/SSIM and lower LPIPS are better. The original table marks the retrained models with a double dagger, which matters when isolating the contribution of the proposed adaptations.

Test set	Retrained AMT-G baseline	Full method on AMT-G	PSNR gain
Vimeo90K	36.42 / 0.9816 / 0.0174	36.62 / 0.9820 / 0.0168	+0.20 dB
SNU-FILM Medium	36.07 / 0.9798 / 0.0376	36.33 / 0.9802 / 0.0341	+0.26 dB
SNU-FILM Hard	30.75 / 0.9377 / 0.0614	30.97 / 0.9384 / 0.0583	+0.22 dB
SNU-FILM Extreme	25.38 / 0.8637 / 0.1080	25.61 / 0.8647 / 0.1036	+0.23 dB
Xiph 4K	34.73 / 0.9489 / 0.1339	34.87 / 0.9490 / 0.1295	+0.14 dB

The Vimeo90K test set contains 3,782 triplets. SNU-FILM contains 1,240 triplets across four difficulty levels. Xiph uses eight 4K videos and constructs 2K/4K evaluations using the established protocol. The Xiph-4K name should not automatically be read as native full-frame 4K processing by every network: the described protocol includes center cropping to 2K resolution.

Ablation Study¶

Table 2 evaluates components on IFRNet-M, a different backbone from the AMT-G comparison above. Reported timings use 1280 by 720 inputs on an RTX 3090.

Configuration	Vimeo90K PSNR / SSIM / LPIPS	Medium PSNR / SSIM / LPIPS	Time	Parameters / FLOPs
Baseline	35.70 / 0.9789 / 0.0176	35.84 / 0.9796 / 0.0327	30 ms	5.0M / 0.21T
HCE only	35.85 / 0.9793 / 0.0171	36.01 / 0.9798 / 0.0325	38 ms	5.5M / 0.24T
HML only	35.87 / 0.9795 / 0.0175	35.96 / 0.9798 / 0.0330	30 ms	5.0M / 0.21T
HCE + HML	35.96 / 0.9800 / 0.0172	36.03 / 0.9799 / 0.0325	38 ms	5.5M / 0.24T
HML + HFL	35.84 / 0.9793 / 0.0172	35.92 / 0.9797 / 0.0327	30 ms	5.0M / 0.21T
HCE + HML + HFL	35.98 / 0.9802 / 0.0169	36.01 / 0.9800 / 0.0320	38 ms	5.5M / 0.24T

The table does not separately itemize SAM segmentation latency, parameters, or FLOPs. Its 38 ms and 5.5M figures should therefore not be treated as a fully documented end-to-end deployment budget including segmentation. The paper explicitly states that HCE and SAM add inference computation; the no-extra-inference-overhead claim applies only to the loss-only configuration without HCE.

Key Findings¶

HCE alone and HML alone improve Vimeo90K PSNR by 0.15 and 0.17 dB, respectively. Combining them gives 0.26 dB, and the full configuration gives 0.28 dB; the benefits are complementary rather than additive.
Table 3 reports 35.76 dB for pixel-only HCE and 35.85 dB after adding object context. Some benefit comes from the extra feature processing itself, so not all HCE gains can be attributed to the masks.
Table 4 reports 35.79, 35.80, and 35.87 dB for pixel-only, object-only, and hierarchical motion distillation. This supports using both levels to assess teacher reliability.
HFL introduces a perceptual-quality trade-off. Adding it to HML-only lowers Vimeo90K PSNR from 35.87 to 35.84 while improving LPIPS from 0.0175 to 0.0172. On Medium, the full configuration also has lower PSNR than HCE plus HML, so improvements are not monotonic across all metrics.
Cross-backbone transfer is not uniformly positive either. For retrained IFRNet-L on Extreme, Table 1 shows PSNR improving from 25.25 to 25.33 while LPIPS worsens from 0.1057 to 0.1072.

Highlights & Insights¶

Object priors have three explicit roles: representation support, teacher-reliability reference, and perceptual correlation boundary. This is more targeted than simply adding another pretrained encoder's output to the input.
Region-mean motion is a reference for deciding how to learn, not a rigid output template. That distinction explains how the method can preserve pixel-level nonrigid motion while benefiting from object structure.
Part of the object knowledge can be delivered through training losses alone. This makes the efficient variant practically relevant, but it should be evaluated separately from the stronger and more expensive full configuration.

Limitations & Future Work¶

The authors identify segmentation quality as a dependency. Severe occlusion or ambiguous boundaries can produce incorrect object priors and degrade interpolation; stronger segmentation and boundary localization are direct improvement directions.
Accurate masks do not solve extreme displacement or complex nonrigid motion. Object membership cannot replace long-range correspondence modeling or resolve every velocity ambiguity.
As a methodological concern, valid deformation can also depart from region-mean flow, so object discrepancy is not a perfect indicator of teacher error. The paper does not systematically isolate these potential cases of excessive suppression.
As an experimental concern, controlled mask-quality perturbations, a segmentation-cost breakdown, and complete main-text loss coefficients are missing. These gaps limit reproducibility of the robustness and deployment-cost claims.
The authors propose combining the approach with Velocity Disambiguation to move from passive object regularization to user-controlled trajectories or nonrigid object motion. That remains future work, not a capability established by the reported experiments.

AMT and IFRNet supply the underlying pixel-motion estimation and synthesis networks. The contribution is a set of contextual and supervisory adaptations rather than a new base interpolation architecture.
Task-oriented flow distillation already filters harmful teacher knowledge in IFRNet. HML extends the reliability assessment with within-object motion references instead of depending only on pixelwise disagreement.
SAM region-prior interpolation is already cited in the paper's related work. It would therefore be inaccurate to call this the first use of SAM in interpolation without qualification; the more precise contribution is the joint design of explicit object context and two hierarchical supervision mechanisms.
Global-attention interpolation broadens the correspondence search. HFL addresses a different question: which object's features should participate in perceptual correlations, using masks to suppress cross-object interference.

Rating¶

Novelty: 4/5. A clear combination of region context, robust distillation, and feature constraints, but neither SAM priors nor semantic motion guidance are new in isolation.
Experimental Thoroughness: 4/5. Multiple benchmarks, two backbone families, and component ablations are useful; controlled mask robustness and segmentation-cost accounting remain limited.
Writing Quality: 3/5. The main pipeline is understandable, but some formulas and implementation details cannot be recovered reliably from the current cached text and require checking the implementation.
Value: 4/5. The training-only HML plus HFL configuration offers a transferable way to improve existing interpolators without changing their inference structure.