MegaFlow: Zero-Shot Large Displacement Optical Flow

Conference: ECCV2026
Authors: Dingxi Zhang, Fangjinhua Wang, Marc Pollefeys, Haofei Xu
Paper: ECCV Paper
PDF: Full Paper
Project: MegaFlow
Area: Video Understanding
Keywords: optical flow, large displacement, global matching, zero-shot generalization, point tracking

TL;DR

MegaFlow builds cross-frame features from pretrained visual geometry priors, locates large motions through global matching, and refines them with local and temporal evidence, achieving 1.83 EPE in four-frame zero-shot Sintel Final evaluation while supporting transfer to point tracking without tracking supervision.

Background & Motivation

Optical flow must locate every source pixel in another frame, so the challenge extends beyond motion direction to distant correspondences, occlusion, appearance changes, and subpixel detail. RAFT-style methods refine flow iteratively, but correlation queries are centered on the current estimate; constructing all-pairs correlations does not itself perform explicit matching across the entire target image. When initialization is far from the correct destination, repeated textures or occlusion can mislead local evidence, and additional iterations do not guarantee recovery of the true match. This explains why high accuracy on ordinary motion does not automatically extend to the large-displacement regime above 40 pixels.

A separate bottleneck is the training distribution: features learned for particular flow datasets often require further adaptation to domains such as Sintel or KITTI. Pretrained vision models provide broader semantic and geometric representations, but direct displacement regression or exclusively local querying does not necessarily turn those priors into long-range correspondence capability. Models such as VGGT already organize static-scene geometry through alternating frame-wise and cross-frame attention, motivating their transfer to videos containing dynamic objects. Here, zero-shot flow means evaluation without adapting to the target domain after source-domain flow training; zero-shot point tracking instead means transfer without tracking supervision.

MegaFlow first resolves where a correspondence lies and then how precisely it can be localized: global matching crosses large spatial gaps, while recurrent updates handle residual errors. Patch representations nevertheless lose thin structures and boundaries, so a trainable CNN must restore local structure in both matching features and refinement. Multi-frame context supplies additional motion evidence, but longer context is not always better when objects rapidly leave the field of view and introduce further occlusion. Core Idea: turn transferable visual geometry priors into explicit global correspondences, then refine them with local detail and temporal context, assigning distinct roles to large-displacement initialization and precise correction.

Method

Overall Architecture

The input is a video of \(T\) frames; flow mode outputs \(T-1\) dense two-dimensional displacement fields between consecutive frames. Prior Feature Fusion combines frozen DINOv2 patch tokens, a trainable cross-frame Transformer, and local CNN features into multi-frame matching representations. Global Matching searches all target locations for each adjacent frame pair to produce low-resolution initial flow. Spatial-Temporal Recurrent Refinement upsamples that initialization to the CNN feature scale and improves it through local correlations, spatial convolutions, and temporal attention. Point-tracking mode reuses the modules, changing pairing from adjacent frames to the query frame and each target frame rather than adding a separate tracking network.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Video frames and frame pairs"] --> Fusion["Prior Feature Fusion"]
    Fusion --> Matching["Global Matching"]
    Matching --> Refinement["Spatial-Temporal<br/>Recurrent Refinement"]
    Fusion -->|Local features and cross-frame context| Refinement
    Refinement --> Output["Adjacent flow or query trajectories"]
    Truth["Training: flow or trajectory labels"] -.->|Initial prediction supervision| Matching
    Truth -.->|Iterative prediction supervision| Refinement

Key Designs

1. Prior Feature Fusion: bring geometric knowledge and pixel detail into one matching space

DINOv2 generates patch tokens independently for each frame, and its image encoder remains frozen during flow training. The tokens pass through 24 layers of alternating frame-wise and global attention following VGGT, organizing within-frame structure and cross-frame relationships respectively. Freezing DINOv2 does not mean freezing the entire Transformer: the subsequent attention backbone is updated to adapt static geometry priors to dynamic motion. VGGT weights initialize the image encoder, Transformer, and parts of the fusion module instead of learning all correspondence representations from scratch. This initialization provides a stronger starting point for frame comparison and reduces the difficulty of training a large backbone from limited flow data alone.

The local branch uses the first two blocks of an ImageNet-pretrained ResNet to provide \(1/2\)- and \(1/4\)-resolution features. Pixel unshuffle reorganizes spatial information in the \(1/4\) features into channels before concatenation with intermediate Transformer tokens. A DPT-style fusion head aligns both representations and produces matching features at approximately \(H/7 \times W/7\) resolution with 128 channels. The resulting descriptors retain local cues such as boundaries and thin structures alongside broader object and scene relationships. Local CNN features also enter refinement, so texture information is not supplied only once at the input. The feature-fusion removal and frozen-Transformer experiments in Table 6 support adapting both components, but do not establish that a particular spatial scale is optimal.

2. Global Matching: let initial displacement cover the entire target image

For source location \(u\), the model computes the dot product between its fused feature and every target location \(v\), followed by softmax over target positions. The result is a distribution over correspondence candidates, not a displacement restricted to a small window. Its probability-weighted target coordinate gives the match, and subtracting the source coordinate produces initial flow. The following summarizes the mechanism from Section 3.2, Equations (1)–(3), and their accompanying prose; the cached equation layout is corrupted, so this is not a character-for-character transcription.

\[ f_i^{\mathrm{init}}(u)=\sum_v M_i(u,v)G(v)-G(u),\qquad M_i(u,v)=\operatorname{softmax}_v\!\left(\langle F_i(u),F_{i+1}(v)\rangle\right). \]

Here, \(G\) is the feature-grid coordinate map, \(F\) denotes fused features, and \(M\) contains normalized matching weights. Displacement magnitude does not determine whether a target enters the candidate set, unlike repeated local queries around an incorrect initialization. Global candidates nevertheless do not guarantee a unique correct match: repeated textures or occlusion can spread the distribution, and the weighted mean can fall between competing candidates. The operation therefore supplies an initialization for correction rather than independently guaranteeing final pixel-level precision. The contribution concerns using pretrained representations for explicit matching, not inventing global matching itself; this operation builds on GMFlow and related work.

3. Spatial-Temporal Recurrent Refinement: recover detail near the global estimate and use other frames to resolve ambiguity

Initial flow is bilinearly upsampled to the CNN feature scale at \(1/4\) resolution, after which target features are queried around the current correspondence. Local correlation compares source features with candidates around the predicted destination; the paper writes the offset range as \([-r,r]^2\) but does not specify the radius numerically in the supplied main text. This stage retains local search, but its center comes from global matching, so it no longer has to traverse a large displacement from scratch. The update network contains two ConvNeXt blocks and two temporal attention blocks: the spatial branch aggregates correlation and CNN detail, while the temporal branch combines motion evidence across the sequence. Each iteration adds a predicted correction to the current flow; training uses 4 updates and evaluation uses 8.

Temporal attention uses other frames to reduce uncertainty caused by blur or partial occlusion rather than simply averaging neighboring flow fields. The architecture accepts variable frame counts, with four-frame inference by default and random sampling of 2–6 frames during training, without rebuilding the network for each window size. Rapid forward ego-motion can instead push objects outside later frames, making extra context uninformative; the KITTI frame-count ablation demonstrates this limitation. For point tracking, both global matching and refinement directly estimate query-to-target displacement rather than accumulating adjacent flow fields. This reduces error propagation from flow composition within a window, but cross-window propagation still depends on previous predictions and is not drift-free. Long sequences use sliding windows of length 8, with current trajectories initializing the next window; Section 3.5 explicitly describes propagation without visibility heuristics or confidence scores.

A Worked Example

Consider four input frames in which a pixel's object moves far from its original position in the next frame; this illustrates the pipeline rather than adding an experiment. Prior Feature Fusion processes all four frames, encoding semantics, local outlines, and cross-frame relationships in each frame's features. Global Matching compares the pixel against the entire target image to obtain an approximate destination; a displacement above 40 pixels does not exclude it from the candidate set. Spatial-Temporal Recurrent Refinement then checks local texture around that destination and uses other frames to correct errors from blurred boundaries or partial occlusion. Flow mode produces three adjacent displacement fields, whereas tracking mode fixes the query frame and matches the pixel to each target frame. Section 3.5 converts displacement to a trajectory using the following equation, where \(x\) is the query coordinate and \(p_t\) the target-frame coordinate.

\[ p_t(x)=x+f_{0\to t}(x). \]

Loss & Training

Flow training uses smooth L1 for the global initialization and L1 for iterative predictions, with exponential weighting controlled by \(\gamma=0.9\) that emphasizes later refinements. Cached Equations (6) and (7) have severe character displacement, so their summation indices are not guessed; only the mechanism confirmed by neighboring prose is retained here. Tracking fine-tuning replaces dense flow supervision with trajectory-coordinate errors at labeled query points, still supervising initialization and refinements. Source-domain training consists of 20K iterations on FlyingChairs, 30K on TartanAirV1, and 30K on FlyingThings; the last comprises 15K two-frame and 15K multi-frame iterations. Table 1 evaluates the resulting model zero-shot, without the subsequent mixed-domain stage. Online benchmarks and tracking transfer additionally use 30K mixed-flow iterations on FlyingThings, HD1K, Sintel, and KITTI; this is distinct from separately fine-tuning for each benchmark. The text says “3 stages” but enumerates four items; this note separates source-domain and mixed-domain stages according to the listed procedure instead of treating the numbering error as another configuration. Optimization uses AdamW, batch size 128, gradient-norm clipping at 1.0, bfloat16, gradient checkpointing, and FlashAttention-3. The full model has 936M parameters and reportedly trains on 64 NVIDIA GH200 GPUs for four days; “lightweight” primarily describes refinement, not the complete system. The tracking-supervised variant adds 20K iterations of Kubric fine-tuning after mixed-flow training and must be distinguished from zero-shot tracking.

Key Experimental Results

Main Results

Table 1 on page 8 evaluates cross-domain generalization on Sintel train and KITTI train; EPE is the mean Euclidean distance between predicted and ground-truth displacement, with lower values better. KITTI Fl-all is the percentage of flow outliers; methods do not use identical training data, so their differences cannot be attributed entirely to architecture.

Method	Frames	Sintel Clean EPE	Sintel Final EPE	KITTI Fl-epe	KITTI Fl-all (%)
RAFT	2	1.43	2.71	5.04	17.4
UFM	2	1.15	2.01	2.96	11.0
MemFlow-T	3	0.85	2.06	3.38	12.8
MegaFlow	2	0.89	2.07	3.00	10.6
MegaFlow	4	0.85	1.83	3.20	10.7

Four-frame MegaFlow achieves 1.83 Sintel Final EPE versus 2.07 for two frames, but KITTI favors the two-frame 3.00/10.6 over the four-frame 3.20/10.7. The prose on page 8 combines four-frame Fl-all 10.7 with two-frame Fl-epe 3.00; this note preserves the separate configurations in Table 1. Table 2 on page 9 groups ground-truth displacement into \(s_{0-10}\), \(s_{10-40}\), and \(s_{40+}\) for 0–10, 10–40, and over 40 pixels respectively; every entry is EPE.

Method	Clean \(s_{40+}\)	Clean \(s_{10-40}\)	Clean \(s_{0-10}\)	Final \(s_{40+}\)	Final \(s_{10-40}\)	Final \(s_{0-10}\)
MemFlow-T	5.239	0.980	0.211	13.670	2.224	0.371
UFM	6.836	1.209	0.259	12.963	2.129	0.399
WAFT-DINOv3-a2	8.870	1.218	0.217	13.192	1.966	0.324
MegaFlow	4.729	0.909	0.314	11.175	1.941	0.480

MegaFlow's advantage is concentrated in medium and large motion, not small motion; its Final small-displacement EPE of 0.480 exceeds WAFT's 0.324. Table 3 on page 10 reports mean zero-shot tracking accuracy of 73.6%, increasing to 79.6% after Kubric fine-tuning; the latter is not zero-shot. At \(384\times512\) input resolution, tracking metric \(\delta_{\mathrm{avg}}\) averages the accuracy of positions within \(k\in\{1,2,4,8,16\}\) pixels, rather than measuring visibility prediction quality. Table 4 on page 12 reports four-frame EPE of 0.349 without Spring-specific fine-tuning; this transfer result follows mixed-flow training and is not the same checkpoint protocol as Table 1. Table 5 on the same page reports Sintel test Final EPE of 2.43, behind VideoFlow-MOF's 1.65, so zero-shot leadership does not imply leadership on every test metric.

Ablation Study

Table 7 on page 14 varies temporal attention and input frame count under the zero-shot training and evaluation setup; lower is better for all three metrics.

Config	Frames	Sintel Clean EPE	Sintel Final EPE	KITTI Fl-epe
Without temporal attention	2	0.88	2.09	3.12
Without temporal attention	4	0.95	1.99	4.22
Without temporal attention	6	0.98	2.04	4.56
With temporal attention	2	0.89	2.07	3.00
With temporal attention	4	0.85	1.83	3.20
With temporal attention	6	0.94	1.92	3.67

At four frames, temporal attention lowers Sintel Final EPE from 1.99 to 1.83; extending context to six frames increases it to 1.92. Table 6 on page 13 reports KITTI Fl-all of 14.7 with a frozen Transformer, 13.9 without feature fusion, and 10.9 for the full model, supporting adaptation and local structural fusion. Its full-model two-frame Sintel Final/KITTI Fl-all values are 2.08/10.9, whereas Tables 1 and 7 give 2.07/10.6 or report only 2.07; the text does not explain this cross-table discrepancy, so the values are not merged.

Key Findings

• Large-displacement gains fit the purpose of global initialization, but the listed ablations do not independently remove global matching and therefore do not quantify its isolated contribution.
• Multi-frame gains depend on motion patterns: Sintel favors four frames and KITTI two, making frame count a deployment choice rather than a setting to maximize unconditionally.
• Table 6 measures 327.9 ms latency and 6.08 GB peak memory on an RTX 3090 at \(540\times960\) with two frames; these costs should not be extrapolated to long videos or full HD.

Highlights & Insights

• Where a prior enters the pipeline matters more than merely including a large model. It first shapes globally matchable representations and then guides local refinement, giving geometry transfer an explicit intermediate interface.
• Flow and point tracking share displacement fields while changing frame pairing and supervision. Their unity comes from task parameterization rather than placing two independent prediction heads in one system.
• Reader interpretation: when transferring to other dense correspondence tasks, first check whether initialization covers the true match before introducing a more complex update module; the paper does not experimentally establish this transfer.

Limitations & Future Work

• The authors acknowledge rising computational cost for dense long-sequence modeling and propose better sequence efficiency and unified pretraining; 936M parameters also constrain lightweight deployment.
• Small-motion accuracy, degradation with longer context under rapid ego-motion, and Sintel Final test results establish meaningful boundaries that should accompany large-displacement claims.
• The authors attribute some Sintel Final errors to Ambush 1 and place comparisons excluding that sequence in supplementary material; the supplied main-text cache does not contain that evidence, so this note retains the full-test ranking.
• Omitting explicit visibility prediction simplifies tracking, but reported positional accuracy does not establish occlusion-detection ability; future evaluation could jointly examine visibility and trajectory error.
• Corrupted extracted equations and cross-table numerical discrepancies are flagged above; the main text also leaves reproduction details such as the local search radius unresolved.

Related Work & Insights

• RAFT, paper reference [54]: uses recurrent correlation queries and updates. MegaFlow retains refinement but supplies a nonzero initialization through explicit global matching; RAFT should not be mischaracterized as having no global correlation representation.
• GMFlow / UniMatch, references [65,66]: establish global matching and unified correspondence estimation. MegaFlow adds VGGT priors, local fusion, and multi-frame refinement rather than introducing softmax coordinate matching for the first time.
• VGGT / DINOv2, references [56,38]: provide geometry initialization and visual representations. Freezing the image encoder while training the later backbone separates preservation of general knowledge from adaptation to dynamic tasks.
• AllTracker, reference [14]: addresses dense long-range tracking. MegaFlow follows its sliding-window propagation approach while demonstrating transfer from flow training; a mean accuracy difference of 79.6 versus 79.5 should not be described as overwhelming superiority.
• Classification: the main outputs are inter-frame optical flow and two-dimensional point trajectories, supporting video_understanding; using geometry pretraining does not make this a 3D reconstruction paper.

Rating

• Novelty: 4/5. Individual components have precedents; the contribution is their integration and cross-task unification of priors, global correspondence, and multi-frame refinement.
• Experimental Thoroughness: 4/5. Cross-domain tests, displacement bins, tracking, and architectural ablations are included, but isolated global-matching ablation and statistical uncertainty are missing.
• Writing Quality: 3/5. The method is coherent, but training-stage numbering, cross-table values, and strong comparative language require careful reading.
• Value: 4/5. Useful for large-displacement estimation and general motion representations, with computational cost and small-motion performance relevant to deployment choices.