Seeing Fast and Slow: Learning the Flow of Time in Videos¶

Conference: ECCV2026
Paper: ECCV Paper
Project: Seeing Fast and Slow
Area: Video Generation / Video Understanding
Keywords: playback-speed estimation, slow-motion dataset, speed-conditioned generation, temporal super-resolution, self-supervised learning

TL;DR¶

The paper learns visual speed perception from audio pitch changes and temporal resampling, annotates the SloMo-44K dataset, and uses it for explicit speed-controlled generation and temporal super-resolution with deblurring; on blurred-input 8-fold reconstruction, FVD decreases from the base model's 436.1 to 134.3.

Background & Motivation¶

Recognizing what happens in a video does not imply knowing how fast it happens in the physical world. An object falling at normal playback speed and in extreme slow motion can follow the same correct direction while representing entirely different temporal scales. Common training videos primarily provide imagery at 24–60 FPS, encouraging models to learn a familiar cadence without experiencing the same kinds of dynamics across speeds. Consequently, asking a video-language model for a playback multiplier or adding an “ultra-slow motion” prompt to a generator may not yield accurate temporal judgments or control. Optical flow does not solve this directly: displacement also depends on object size, camera motion, and the action itself, so it is not an absolute playback-speed measurement.

High-frame-rate slow-motion footage can provide this missing experience, but online videos often lack playback multipliers and capture frame rates, and may change speed internally. Simply repeating frames or stretching ordinary footage adds none of the intermediate motion details lost during exposure and can instead teach a generator to stutter. Existing high-frame-rate datasets also face limits in scale, content diversity, or maximum frame rate, restricting their usefulness for broad generative training. The paper therefore places temporal annotation before temporal control: it first learns speed from naturally available signals, then converts poorly documented videos into training resources. The supervision does not require people to measure physical processes frame by frame; it uses observable audio changes and known, deliberately applied resampling factors.

Audio can locate when speed changes, while resampling specifies the relative speed relationship that should hold between two versions of a clip. A small amount of footage with known capture conditions supplies an absolute scale, allowing the model to predict multipliers relative to normal speed rather than merely rank clips. These perception tools ultimately support data curation rather than running as additional detectors during every video generation request. Core Idea: use cross-modal cues, speed equivariance, and limited absolute calibration to annotate time in genuine slow-motion footage, then train generators on its real temporal detail to control temporal scale explicitly.

Method¶

Overall Architecture¶

The method connects four designs: speed perception, slow-motion data curation, explicit speed-conditioned generation, and joint deblurring temporal super-resolution. The first two turn online footage into SloMo-44K with speed labels and captions; the latter two are separately trained downstream branches, not consecutive stages of one generator. The generation branch takes an image, text, and target playback speed and produces a new video unfolding at the requested pace. The super-resolution branch takes low-frame-rate footage and a VACE conditioning mask and recovers a denser, sharper frame sequence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Online videos"] --> Perception["Speed perception"]
    Supervision["Original audio, resampling,<br/>and known-speed samples"] -.->|Training supervision| Perception
    Perception --> Dataset["Slow-motion data curation"]
    Dataset -->|Training data| Generation["Explicit speed-conditioned<br/>generation"]
    Dataset -->|Synthetic paired training data| Reconstruction["Joint deblurring temporal<br/>super-resolution"]
    Conditions["Image, text, target speed"] --> Generation
    LowFPS["Low-frame-rate video and mask"] --> Reconstruction
    Generation --> Generated["New video with controlled pace"]
    Reconstruction --> Restored["Sharp high-frame-rate video"]

Key Designs¶

1. Speed perception: separate change localization, relative multipliers, and absolute scale

The system first needs to identify when a video changes speed; assigning one multiplier to footage containing both normal and slow motion would introduce incorrect supervision. The authors exploit the fact that temporal compression raises original-audio pitch and temporal stretching lowers it, automatically obtaining over 8K speed-change labels. This cue requires original audio corresponding to the visual content and does not apply to arbitrary music, dubbing, or pitch-preserving edits. The labels fine-tune a VideoMAEv2 visual speed-change detector with binary cross-entropy. At deployment, the detector uses visual input alone, so audio supplies training supervision rather than a required inference input.

Localizing a change does not reveal its magnitude, so a second model estimates the playback multiplier. For an unlabeled clip, the authors construct an original and an accelerated version and constrain the same estimator's outputs to follow the known multiplier relationship. This provides consistency supervision without first knowing whether the original clip plays at normal speed or in slow motion. The following is the equivariance relationship explicitly described in Section 3.2, not a reconstruction of the damaged loss equation:

\[ f_{\theta}(V^k) \approx k f_{\theta}(V). \]

Here \(V^k\) is the \(k\)-times accelerated version of \(V\), and \(f_{\theta}\) predicts the playback multiplier relative to real-world normal speed. The authors constrain this relationship in log space because speed is multiplicative, making ratio errors more appropriate than linear differences across extreme slow motion and normal footage. Relative relationships alone can leave predictions globally mis-scaled; known-speed samples such as Adobe240fps therefore provide supervised calibration in log space. For ultra-slow clips with barely visible motion differences, the current estimate is used to accelerate playback toward normal speed, followed by re-estimation and refinement accounting for the resampling multiplier, unrolled 3 times. This iterative prediction magnifies subtle changes into a motion range familiar to the estimator, but does not recover visual information that has already been lost.

2. Slow-motion data curation: turn online footage into speed-consistent training clips

The sources are YouTube, Vimeo, and Flickr, searched using terms related to high frame rates, high-speed cameras, and slow motion. Conventional cleaning first segments shots with TransNetv2, removes excessive text overlays using OCR, excludes CGI and screen recordings with Qwen2.5-VL, and filters by video quality scores. These steps address shot boundaries, content origin, and visible quality; they do not replace playback-speed annotation. The visual speed-change detector then creates more speed-homogeneous clips, the speed estimator assigns a multiplier to each, and InternVL3 supplies detailed captions. This aligns speed labels, visual dynamics, and text conditions for subsequent generative training, avoiding a fixed-speed label on samples that cross a speed transition.

Table 1 distinguishes 44,632 clips from 18,235 source videos, totaling approximately 18M frames, with maximum raw capture rates above 10,000 FPS. These figures do not mean every clip reaches the maximum frame rate or every automatically estimated multiplier has human-verified ground truth. The footage spans varied content and motion, adding not only more samples but also fine dynamics unavailable under ordinary exposure conditions. The authors call this collection SloMo-44K, use it for speed prediction and generation, and compare against standard-video training to test the contribution of temporal-scale coverage. At the system level, this is a perception-driven data curation process, not a loop without any human-provided information: known-speed samples still anchor absolute calibration.

3. Explicit speed-conditioned generation: supply both a global multiplier and per-frame temporal coordinates

The generator builds on Wan2.1-I2V, retaining image-to-video inputs while adding a target playback speed. The first conditioning path divides the range from 0.01× to 1.0× into 10 logarithmically spaced buckets, giving the extreme-slow-motion range meaningful condition resolution. The bucket index passes through sinusoidal positional encoding and an MLP, then is added to the diffusion timestep embedding to provide a global speed condition throughout denoising. The diffusion timestep is a coordinate of the generation algorithm, not the video's physical time; the additional embedding connects these distinct notions. The second path adds an encoding derived from frame index multiplied by target speed to each latent video frame, allowing local temporal features to reflect the requested progression.

The readable per-frame conditioning relation in Section 4.1 is given below, with \(i\) indexing the latent tensor in time:

\[ \operatorname{latent\_vid}[i] \leftarrow \operatorname{latent\_vid}[i] + \operatorname{MLP}_{\psi}(\phi(i\cdot\operatorname{speed})). \]

The two signals respectively specify the overall speed range and the temporal positions that frames should represent at that speed. They provide more direct control than adding “slightly slow” or “ultra-slow” to a prompt, since the motion multiplier no longer depends entirely on semantic text. Training updates the linear projection layers and LoRA adapters on the Transformer backbone rather than training from scratch or fully fine-tuning the base model. Inference requires only the image, text, and speed condition; it does not first generate a normal-speed video, extract trajectories, and synthesize another version. The method therefore controls the temporal pace of newly generated events rather than merely replaying an existing sequence on a rescaled timeline.

4. Joint deblurring temporal super-resolution: simulate low-frame-rate exposure during training instead of only dropping frames

Conventional interpolation evaluations often subsample sharp high-frame-rate footage while assuming that the remaining input frames are still clear. Real low-frame-rate capture often involves longer exposures, so a single input frame already mixes multiple motion states; interpolation between blurred endpoints can preserve that blur. Starting from slow-motion ground truth, the authors average a temporally centered window of 8 frames, then subsample in time to synthesize low-frame-rate inputs with exposure blur. The resulting pairs provide both blurred observations and sharp high-frame-rate targets, teaching deblurring and intermediate-motion recovery rather than merely increasing the frame count. The base model is Wan2.1-VACE, which uses a reference video and binary mask to express conditioning and the frames or regions to generate, again trained with LoRA.

Computational constraints limit the paper to 8-fold temporal upsampling; arbitrary factors are not claimed to have been validated. Training blur is approximated by temporal-window averaging, not a complete imaging model covering arbitrary exposure, rolling shutter, and compression artifacts. At inference, the fine-tuned generative prior reconstructs dense sequences from actual low-frame-rate footage; this branch shares data with speed-conditioned generation but uses a different backbone and input interface. Without a genuine high-frame-rate reference, improved appearance does not automatically establish exact recovery of every intermediate physical state.

Loss & Training¶

Speed-change detection uses binary cross-entropy; speed estimation combines log-space resampling consistency with supervised regression on limited known-speed data. The two generation branches fine-tune Wan2.1-I2V and Wan2.1-VACE separately, using annotations or synthetic paired data from SloMo-44K. The cached extraction damages the end of loss Equation (1), the resampling-distribution notation, and the bucket expression in Equation (2), preventing reliable verification of the loss power, sampling details, or bucket endpoint handling. This note therefore retains only the explicitly described target relationship, 10-bucket range, and readable per-frame conditioning, without guessing exact implementation details, learning rates, loss weights, or training steps.

Key Experimental Results¶

Main Results¶

Speed estimation is evaluated on 111 videos with verified playback multipliers; Table 2 appears on paper page 9, with the protocol in Section 5.2 on pages 11–12. Pearson \(\rho\) measures linear association between predictions and ground truth, Spearman \(r_s\) measures rank association, and RMSE measures error; these three use log speeds. eRMSE is the exponentiated log-space RMSE, interpreting deviation on a multiplicative scale rather than as an arithmetic mean of per-sample relative errors.

Method	\(\rho\) ↑	\(r_s\) ↑	RMSE ↓	eRMSE ↓
Human expert	0.880	0.783	0.492	1.636
Gemini 2.5 / VideoLLM	0.426	0.308	1.568	4.796
SpeedNet	0.476	0.331	1.261	3.529
Pulse-of-Motion	0.508	0.525	1.181	3.258
Ours	0.735	0.706	0.649	1.913

The method substantially exceeds the listed automated approaches, but its \(\rho\) and RMSE still trail human experts; “near-human” should not be read as human-level performance. Speed-change detection is evaluated separately: Section 5.1 on page 11 reports 92.4% accuracy, compared with 82.5% for optical flow and 59.5% for Gemini 2.5. Its test labels are reviewed by 4 annotators, retaining only cases where human and audio-derived labels agree; this is not accuracy on unfiltered random online footage.

The next table selects results from Table 3 on paper page 9, using synthetically blurred SloMo-44K-Test inputs and 8-fold temporal super-resolution; lower is better for all metrics. FloLPIPS measures interpolation-oriented temporal perceptual differences, LPIPS measures image perceptual differences, and FID/FVD compare image/video feature distributions, respectively.

Method	FloLPIPS ↓	LPIPS ↓	FID ↓	FVD ↓
FILM	0.099	0.080	20.3	250.0
LDMVFI	0.136	0.117	31.4	340.4
Generative Inbetween	0.123	0.108	26.1	439.4
Wan2.1-VACE	0.126	0.095	28.5	436.1
Ours	0.067	0.058	12.4	134.3

Speed-conditioned generation separately checks condition ordering and average optical flow on 56 image–prompt pairs, using condition values 1, 4, 7, and 10, with larger values indicating faster motion. FID/FVD use the 48 text–image–speed triplets with corresponding ground truth; the 56 pairs should not all be treated as having paired references. Table 4 on page 12 reports FID/FVD of 68.4/1114.1 for the proposed model, 72.2/1266.8 for Wan2.1, and 73.4/1473.5 for ATI. Monotonic optical-flow changes support relative controllability, but do not directly establish that every generated clip's real playback multiplier precisely matches the requested value.

Ablation Study¶

The following results come from Table 7 on paper page 14 and test iterative prediction (IP) on the same speed-estimation task, not generation quality.

Model	IP	\(\rho\) ↑	\(r_s\) ↑	RMSE ↓
VideoLLM	No	0.426	0.308	1.568
VideoLLM	Yes	0.552	0.479	1.221
Ours	No	0.680	0.684	0.917
Ours	Yes	0.735	0.706	0.649

Adding IP reduces the proposed model's RMSE from 0.917 to 0.649, showing that re-observation after resampling aids estimation rather than merely tuning one particular backbone. In Table 8 on page 14, with IP enabled for both, standard-data training yields RMSE 0.737 versus 0.649 for SloMo-44K training, supporting broader speed coverage. In Table 5 on page 12, speed-controlled generation trained on standard videos has FVD 1392.9 versus 1114.1 with SloMo-44K; qualitative Figure 8 also identifies stuttering from artificially slowed training footage.

Key Findings¶

Improvements extend beyond synthetic blurred inputs: in the clean-input setting of Table 6 on page 13, the proposed model achieves FVD 394.0 on DAVIS and 182.2 on SloMo-44K-Test.
Not every single-frame metric is best on clean inputs: DAVIS LPIPS is 0.203 for the proposed method versus 0.200 for FILM, while SloMo-44K-Test FID is 10.9 versus 10.7.
Real inputs lack slow-motion ground truth; the text on page 14 reports human preference above 90% in this setting, which must not be interpreted as objective reconstruction accuracy.

Highlights & Insights¶

Temporal supervision comes from controlled transformations rather than dense manual labels: a known resampling factor constrains predictions even when the original speed is unknown. Limited absolute calibration resolves the scale ambiguity left by relative learning.
Perception tools improve training-data quality: detecting speed changes before assigning clip-level multipliers reduces ambiguity from mixed temporal scales. Data curation therefore directly supports the downstream training objective.
Genuine high-frame-rate data and simple slow playback are not interchangeable: the former contains extra motion detail, whereas the latter only changes playback duration. The generation training-data comparison provides experimental support for this distinction.

Limitations & Future Work¶

The authors acknowledge that limited-motion footage and people deliberately moving slowly can mislead speed perception. Distinguishing an intrinsically slow action from slowed playback is not always uniquely possible from visual evidence.
The authors also note dependence on Wan backbones; training primarily uses LoRA, so the results should not be attributed to a new full generative architecture, and super-resolution evaluation is limited to 8-fold upsampling.
From an evaluation perspective, requiring agreement between audio and human labels may favor more recognizable speed changes, and synthetic blur does not cover every real imaging degradation. These are reader interpretations of the generalization limits.
For reproducibility, the cached extraction damages Equations (1), (2), and Table 9; this note does not transcribe Table 9's interleaved values or guess implementation details, and unavailable supplementary material is not treated as reviewed evidence.

Compared with SpeedNet and Pulse-of-Motion: these also use temporal pace or frame-rate cues to understand motion, while this paper emphasizes broad speed coverage, absolute calibration, and iterative prediction beyond discrete fast/slow classification.
Compared with ATI, BulletTime, and SpaceTimePilot: trajectory control or existing-video editing can alter temporal progression, whereas this work aims to generate new dynamics at a specified speed from an image and text without requiring a complete event video first.
Compared with FILM, LDMVFI, and Generative Inbetween: this work includes exposure blur when constructing low-frame-rate inputs and jointly learns sharpening and interpolation. The transferable lesson is to match training degradation to real acquisition, not just to output resolution.

Rating¶

Novelty: 4/5. Connects speed perception, annotation, and explicit temporal control, with principal contributions in supervision design and genuine slow-motion data.
Experimental Thoroughness: 4/5. Covers perception, generation, super-resolution, and data ablations, but absolute generated speed and real reconstruction fidelity need more direct evaluation.
Writing Quality: 4/5. Presents a clear problem chain and close alignment between training and applications; extraction damage prevents complete verification of a few formulas.
Value: 4/5. Provides reusable data and methods for cross-scale temporal learning while demonstrating why ordinary slow playback cannot replace high-frame-rate detail.