ECCV 2026 Human Understanding Video Understanding gaze trajectory generation autoregressive diffusion video saliency long-video modeling eye-movement prediction

Infinite Gaze Generation for Videos with Autoregressive Diffusion¶

Conference: ECCV 2026
Paper: ECCV 2026 official page / project page
Area: Human Understanding / Video Understanding
Keywords: gaze trajectory generation, autoregressive diffusion, video saliency, long-video modeling, eye-movement prediction

TL;DR¶

This paper is the first to cast video gaze prediction as raw gaze trajectory generation over arbitrarily long videos: a compression bottleneck is inserted into UNISAL so that each video frame is reduced to a compact saliency latent injected into the diffusion backbone every 5 frames, and the last 90 gaze coordinates (3 s) serve as a clean history prefix while the model autoregressively denoises the next 45 future coordinates (continuous x,y at 30 samples per second) in a sliding window, beating six image-domain gaze/scanpath models adapted to video on Levenshtein distance, DTW and Maximum Temporal Correlation on DIEM and DHF1K (Discrete Fréchet Distance excepted).

Background & Motivation¶

Human gaze modeling has long followed two routes: saliency maps, which give a time-independent spatial probability distribution (video saliency models such as DeepVS and UNISAL), and scanpaths, which discretize eye movements into an ordered sequence of fixations (DeepGaze III, GazeFormer, HAT, TPP-Gaze and others). Both abstractions work well on static images, but they flatten the high-frequency temporal dynamics of raw eye-tracking data — smooth pursuit, the most common eye movement in video, is barely expressible as a sequence of fixations. Worse, existing models are evaluated within 3–5 second windows, whereas viewing behavior in real content depends on much longer time scales: an observer may change strategy only after an object reappears tens of seconds later, and a short-window model never sees such long-range dependencies, so it cannot synthesize genuinely long-horizon gaze behavior.

At the same time, collecting high-fidelity gaze data is limited by expensive hardware, cumbersome calibration and privacy constraints, which makes generating gaze data rather than collecting it a practical necessity. Recent video world models have shown that content can be generated causally, frame by frame, which suggests a new route: treat human gaze itself as the object being generated, so that the model keeps producing where to look next while it watches a dynamic world. The difficulty is that video frames carry an order of magnitude more information than a single image, making per-frame RGB conditioning both expensive and redundant, while fixed-length full-sequence diffusion models only ever emit a clip of fixed duration — generating arbitrarily long trajectories requires a model that consumes a visual stream and can roll its own outputs forward as conditioning.

The approach here feeds video frames in one at a time and, after predicting each segment, writes the newly generated gaze coordinates back into a history cache; a diffusion model then denoises future gaze coordinates inside this autoregressive window. The key observation is that for "what induces eye movements," saliency is both a stronger and a cheaper prior than generic visual features — the value of a saliency map is literally the probability of being attended, living in the same semantic space as the coordinates to be predicted. Conditioning therefore uses compressed saliency latents rather than RGB or generic visual tokens. Core idea: combine saliency-latent conditioning with an autoregressive diffusion objective built on a clean history prefix and a noised future window, so that a diffusion model can stream out raw gaze trajectories of arbitrary length over videos of arbitrary duration.

Method¶

Overall Architecture¶

Formally, the video stimulus is \(V=\{I_t\}_{t=1}^{T}\) with \(I_t\in\mathbb{R}^{H\times W\times 3}\), and the goal is to predict a gaze trajectory \(R\in\mathbb{R}^{T\times 2}\) where each timestep is an in-frame 2D coordinate \((x,y)\) sampled as densely as 30 points per second — the output is continuous raw eye movement rather than a handful of discrete fixations. Following DiffEye, gaze prediction is written as a conditional generative problem \(p_\theta(R\mid V)\), approximated with a DDPM over gaze trajectories; the noise-prediction network is a U-Net of hierarchical 1D convolutional blocks with multi-scale self-attention, conditioned on the diffusion timestep through sinusoidal embeddings. On top of this backbone the paper makes exactly two changes: the generation process is reformulated autoregressively, and the video-stimulus conditioning mechanism is redesigned.

End to end the pipeline is streaming: video frames enter the saliency branch one at a time and produce compressed saliency latents; each round the diffusion backbone consumes three things — a clean gaze-history prefix, the future window to be denoised, and the saliency latent injected every 5 frames; after sampling, the new coordinates are appended to the history and the whole window shifts right. The trajectory length is limited only by the video duration.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["video frame stream + gaze history cache"] --> B["saliency latent conditioning<br/>per-frame extraction, injected every 5 frames"]
    B --> C["autoregressive diffusion denoising<br/>90 clean history + 45 noised future"]
    C --> D["sliding-window rolling inference<br/>sampled coordinates written back"]
    D -->|window shifts right, reusing the newest coordinates| C
    C --> E["raw gaze trajectory of arbitrary length"]

Key Designs¶

1. Saliency latent conditioning: compress the video stimulus into a compact encoding of "where people are likely to look"

The first difficulty with video conditioning is sheer volume: per-frame RGB (or per-frame visual tokens) is slow and redundant, and the discrete global summary produced by a visual tokenizer sits one implicit mapping away from gaze behavior. The paper's second observation matters more — for inducing eye movements, pixel appearance is not the most relevant signal; saliency is both a stronger and a more direct prior. Conditioning therefore uses video saliency maps: single-channel, semantically meaningful, decoupled from surface appearance, and interpretable as the probability that a given spatial location is attended.

Concretely, the video saliency model UNISAL serves as the backbone, but existing saliency models are designed for prediction accuracy rather than compact representation, so a compression bottleneck is inserted between its encoder and decoder: the RNN channel count drops from 256 to 64, and the latent produced by the encoder is further compressed spatially by 2D average pooling. The compressed saliency latent is injected into the diffusion backbone through cross-attention at every block. Saliency features are extracted per frame and concatenated along time, but conditioning is applied at a reduced temporal frequency (every 5 frames) to cut computation further. The payoff is quantified against raw-pixel conditioning: parameters fall from 32.33M to 8.27M, forward FLOPs from 71.65G to 1.18G, peak memory from 974.90MB to 49.50MB, while throughput rises from 17.61 to 49.19 samples/s. The ablation supports the claim that shape matters more than size: swapping in MAGVIT2 generic token conditioning fails to converge outright, degrading DTW from 12.33 to 54.92. ⚠️ The paper does not state whether the saliency branch (UNISAL) is trained jointly with the diffusion backbone or reused frozen — refer to the original paper.

2. Autoregressive diffusion denoising: anchor on a clean history prefix and denoise only the future window

Fixed-length full-sequence diffusion cannot extrapolate to variable-length trajectories, while naive point-by-point recursion invites error feedback and drift in the conditioning distribution. The paper adopts the autoregressive diffusion paradigm (in the spirit of Diffusion Forcing) and splits a gaze segment of length \(n\), as represented in the input vector, into two halves: the first \(k\) coordinates are a conditional prefix standing for the past, and the remaining \(n-k\) coordinates are the future to be generated. During training, history windows of length \(k\) are randomly sampled from the ground-truth segments, and the model predicts \(R_{k+1:n}\) given the history \(R_{1:k}\) and the video condition \(S\); the diffusion loss is computed only on the predicted segment:

\[\mathcal{L}=\mathbb{E}_{t,\,\epsilon}\Big[\big\|\epsilon-\epsilon_\theta\big(R^{(t)}_{k+1:n},\,t \mid R_{1:k},\,S\big)\big\|_2^2\Big]\]

The history segment is neither noised nor supervised — it is an already-observed fact, not a variable to be denoised. The history length is set to \(k=90\) frames (about 3 s), chosen as the midpoint of the 1–6 s range over which visual working memory decays, balancing temporal context, noise and compute.

The benefit lies in training/inference alignment: because the history is always clean, the model never has to guess the trajectory it has "already seen," which keeps the conditional distribution stable; and because supervision lands only on the future segment, what the model learns is the mapping from "past plus current scene" to "where to look next," exactly matching the inference-time usage of rolling its own generated coordinates forward, with no train/test distribution mismatch in between.

3. Sliding-window rolling inference: streaming generation of arbitrarily long trajectories

At inference there is no ground truth to lean on, so the model iteratively conditions on the most recent \(k\) generated coordinates to predict subsequent gaze points, appends the new coordinates to the history cache, and shifts the whole window right, letting the trajectory extend indefinitely. Because the conditioning window has a constant length, the per-step compute is independent of total video length, which is what allows the method to handle videos of arbitrary duration — this is what "infinite" actually means in the title: not training on extremely long sequences, but rolling a 3-second window without bound.

The paper also probes how much the initialization of that window matters. The main experiment primes the model with 90 frames of ground-truth history; replacing the initial buffer with a single ground-truth coordinate replicated 90 times (a zero-velocity cold start that removes all speed and direction cues) leaves Levenshtein distance (3.41 vs. 3.40) and Maximum Temporal Correlation (0.235, unchanged) essentially identical, with only DTW rising slightly from 12.33 to 12.39, which the authors attribute to initial misalignment before the history is built up through autoregression. The model therefore does not depend on explicit motion cues in the history; gaze dynamics can be recovered from the saliency latents and the autoregressive process itself. The window setting also sets the ceiling on long-range consistency: lengthening the prediction segment to 90 frames raises Levenshtein distance from 3.40 to 3.77 and DTW from 12.33 to 14.65, making the accumulation of autoregressive error plainly visible — exactly the "a fixed-length sliding window cannot carry longer-term memory" limitation the authors concede.

Loss & Training¶

The objective is the DDPM ε-prediction loss above, applied to the noised and supervised future segment only. The noise schedule is linear, from \(1\times10^{-4}\) to \(2\times10^{-2}\); training uses 1000 diffusion steps and sampling switches to DDIM with 50 steps for speed at nearly no quality cost. Optimization uses Adam with a constant learning rate of \(1\times10^{-4}\) for 70 epochs. Key hyper-parameters: 90-frame history, 45-frame prediction, saliency conditioning injected every 5 frames, and a gaze sampling rate of 30 points per second; training uses the 80% split of DIEM.

Key Experimental Results¶

Main Results¶

Experiments run on DIEM (the in-domain benchmark: 84 videos at 1280×720, 30–209 s long, 67 observers on average, 553,116 s of total viewing time) and DHF1K (generalization check: 1,000 videos, 17–42 s, 640×360, 17 observers; 16 videos are evaluated, four from each of the four motion categories — slow/fast camera motion and slow/fast scene motion), with an 80/20 split on DIEM. All metrics are trajectory-level: Levenshtein distance measures sequence ordering structure, Discrete Fréchet Distance measures the maximal pointwise deviation along the trajectory, DTW measures the optimal alignment allowing speed and length differences, and Maximum Temporal Correlation measures the highest cross-correlation between two temporal signals. For each video, 10 trajectories are generated and compared pairwise against multiple ground-truth paths, reporting both best (the best of the 10 comparisons) and mean (average over the 10), where best reflects peak capability and mean reflects consistency. Since this is the first video gaze trajectory generation model with no direct baseline, six image-domain methods are adapted to video: generation in 3-second chunks that are then concatenated, only the first frame of each chunk as input (the baselines are single-image models), the last predicted position of the previous chunk as the start for models that allow control of the starting gaze position (HAT, DeepGaze III), and re-sampling of all baselines to the paper's 30 points per second density (DiffEye is evenly downsampled).

Mean values are shown below (best scores lead to the same conclusion; the paper's Levenshtein best is 3.31 on DIEM and 1.50 on DHF1K):

Dataset	Method	Levenshtein ↓	Disc. Fréchet ↓	DTW ↓	Max Temp Corr. ↑
DIEM	DeepGaze III	4.36	4.78	20.07	0.149
DIEM	DiffEye	3.78	3.82	12.47	0.176
DIEM	GazeFormer	4.26	4.17	15.64	0.188
DIEM	HAT	4.45	5.16	23.44	0.131
DIEM	Chen et al.	4.23	4.36	15.97	0.175
DIEM	TPP-Gaze	4.26	3.60	15.17	0.131
DIEM	Ours	3.40	4.14	12.33	0.235
DHF1K	DeepGaze III	3.61	3.12	11.12	0.136
DHF1K	DiffEye	1.76	2.34	4.24	0.232
DHF1K	GazeFormer	1.76	2.49	4.43	0.258
DHF1K	HAT	1.84	3.12	6.08	0.179
DHF1K	Chen et al.	1.77	2.54	4.70	0.265
DHF1K	TPP-Gaze	1.78	2.34	4.30	0.225
DHF1K	Ours	1.57	2.50	4.01	0.290

On efficiency, replacing the compressed saliency-latent conditioning with raw-pixel conditioning costs: 8.27M vs. 32.33M parameters (3.91× fewer), 37.89MB vs. 348.54MB checkpoints (9.20× smaller), 0.0085s vs. 0.6353s data loading (74.85× faster), 1.18G vs. 71.65G forward FLOPs (60.95× fewer), 49.19 vs. 17.61 samples/s throughput (2.79× higher), and 49.50MB vs. 974.90MB peak memory (19.69× lower).

Ablation Study¶

Ablations on DIEM (default setting: 90-frame history, 45-frame prediction):

Config	Levenshtein ↓	Disc. Fréchet ↓	DTW ↓	Max Temp Corr. ↑	Note
Ours regular	3.40	4.14	12.33	0.235	90-frame history + 45-frame prediction
10-frame prediction	3.97	3.67	17.31	0.048	short horizons are simpler with less drift, but lose robustness over full sequences and temporal correlation collapses
90-frame prediction	3.77	4.38	14.65	0.230	autoregressive error accumulates, long-horizon uncertainty rises
No initial history (cold start)	3.41	4.13	12.39	0.235	a single coordinate replicated to fill the window; Levenshtein and temporal correlation unchanged
MAGVIT2 generic token conditioning	3.82	7.97	54.92	0.054	generic-tokenizer conditioning fails to converge and degrades across the board

As a further control on the conditioning mechanism, the authors retrain a modular baseline — DiffEye with the same saliency latents but no autoregression — and in its native 3-second window it reaches DTW 2.224 and Maximum Temporal Correlation 0.037, against 1.599 and 0.156 for this method: the gain comes from combining saliency conditioning with autoregressive diffusion, not from saliency conditioning alone.

Key Findings¶

The shape of the conditioning signal matters far more than its compression rate: MAGVIT2 generic token conditioning pushes DTW from 12.33 to 54.92 and Maximum Temporal Correlation from 0.235 to 0.054, failing to converge at all, whereas the saliency latent trains stably with 3.91× fewer parameters and 60.95× fewer FLOPs. This backs the paper's central claim that a probability-like representation in the same semantic space as gaze (saliency) suits gaze generation better than a generic visual summary.
Prediction length involves a clear trade-off: 10-frame short prediction achieves a lower mean Discrete Fréchet (3.67 vs. 4.14), showing that short-horizon modeling is simpler with less drift, but its DTW degrades to 17.31 and temporal correlation collapses to 0.048, so it is unreliable over whole videos. Stretching to 90 frames does the opposite, with accumulated error worsening both Levenshtein and DTW. 45 frames is the best balance of near-term accuracy and long-term stability.
The history prefix matters less than one might expect, though it is not useless: in the cold-start experiment Levenshtein and temporal correlation are essentially unchanged while only DTW rises slightly, indicating that motion dynamics can also be recovered from the saliency latents; these results nonetheless hold under randomly sampled training windows and an always-clean history, and long-range coherence still rests on rolling the window.
Discrete Fréchet Distance disagrees with human perception: this is the one metric where the method is not best (TPP-Gaze reaches a lower mean of 3.60 on DIEM). The authors run a user study: six videos (three per dataset) with the predictions of the two methods closest to the ground truth under Discrete Fréchet, rendered as moving dots overlaid on the video, with 12 participants asked to pick the trajectory that looks closer to ground truth after being allowed to replay. This method is selected in 85% of cases overall (per video 67% / 83% / 83% / 92% / 100% / 83%, higher in every single instance). The authors conclude that the metric emphasizes the maximal pointwise deviation along a trajectory and is over-sensitive to the brief saccades and minor timing shifts common in video, making it unsuited to evaluating the global temporal structure of video gaze.
The method wins under both long-window and native 3-second evaluation: over long windows it is best on every metric except Discrete Fréchet, and the native 3-second setting shows the same trend, with a mean Levenshtein of 0.447 against 0.461 for the second best.

Highlights & Insights¶

Human gaze as the generation target of a world model: previous autoregressive diffusion and world models generate video frames or actions; this paper redirects the paradigm to a human's behavioral response to a dynamic scene, producing continuous eye-movement trajectories — an approach transferable to any continuous human-behavior data such as mouse trajectories, driving behavior or robot manipulation.
Conditioning that shares a semantic space with the prediction target: saliency values already are attention probabilities, so the condition needs no translation into gaze coordinates. The underlying principle — feed a prior that already points at the target quantity instead of a generic representation — is useful for any conditional generation task.
The compression bottleneck is an explicit design principle, not a post-hoc optimization: conditioning volume is the main bottleneck of streaming generation, and the paper cuts it from 71.65G to 1.18G FLOPs via smaller RNN channels, spatial pooling and a reduced injection frequency without losing accuracy. Deciding what a condition actually needs to retain is more interpretable than simply shrinking the encoder.
Clean history plus supervision on the future only: treating history as a condition rather than a variable to denoise aligns the training distribution with the inference-time usage of extrapolating from one's own outputs — a trick directly portable to any autoregressive diffusion system.
Questioning the metric before adding human evaluation: facing the single metric it loses, the paper neither hides nor games it but demonstrates with a user study that the metric disagrees with perception, and points out that all trajectory metrics compare pairwise against a single ground-truth path and thus ignore the multi-modality of gaze. This is more valuable than tuning for a leaderboard.

Limitations & Future Work¶

Missing long-range memory: the model relies only on a fixed-length (90-frame, 3 s) sliding window and cannot carry behavioral dependencies over longer horizons; in human viewing, scene understanding from tens of seconds earlier shapes current choices, especially with static cameras and persistent background structure. The authors suggest borrowing mechanisms from video world models that maintain long-term spatial memory.
Inherent flaws of the evaluation metrics: existing trajectory metrics all compare pairwise against a single ground-truth path, so they cannot reflect the multi-modality of human gaze (qualitatively different trajectories can be equally plausible), and Discrete Fréchet is highly sensitive to local outliers. The authors call for metrics designed to match perceptual validity.
Head movements are not modeled: gaze trajectories in the training data were acquired under fixed head poses, which is acceptable for desktop viewing but not for the physical world or wide-field displays (VR/AR), where head motion dominates. Extending generation to joint head orientation and relative gaze is the natural next step toward imitation learning and VR/AR applications.
Limitations I see: first, "infinite" rests on rolling the window, yet the ablation shows that lengthening the prediction segment already causes error accumulation; the paper reports only the 3-second and full-length regimes and never breaks results down by video duration, so cumulative drift over very long videos remains unquantified. Second, inference is primed with 90 frames of ground-truth history; although the cold start is shown to be robust, the fully unsupervised start with no ground truth at all is not evaluated. Third, the capacity of the saliency branch bounds generation quality, and no sensitivity analysis is done with a weaker or stronger saliency model. Fourth, all metrics are trajectory distances; spatial overlap metrics common in saliency work (AUC/CC/NSS) are absent — a natural consequence of outputting coordinate sequences instead of heatmaps — so cross-comparison with the saliency prediction literature should be made cautiously.

vs DiffEye: the two share the same DDPM formulation and U-Net 1D-convolution backbone, but DiffEye conditions on image patch features and generates fixed-length (3 s) sequences, while this paper uses video input, saliency-latent conditioning and an autoregressive sliding window. The "DiffEye retrained with saliency latents but no autoregression" ablation is decisive: it decomposes the gain and shows that saliency conditioning alone is insufficient, the two parts being complementary.
vs autoregressive diffusion and video world models (Diffusion Forcing, long-term spatial memory video world models, EPONA, etc.): the paradigm is the same — denoise the future conditioned on a clean past and support variable-length rollouts — but the generated quantity changes from video frames or driving actions to human gaze coordinates, and the video condition is compressed into saliency latents rather than RGB. This also means the method can directly absorb progress on long-horizon memory from that line of work.
vs scanpath generation on 360° content (ScanDMM, ScanGAN360, ScanTD, VPT360, etc.): these handle gaze under spherical geometry, but mostly model viewports or head orientation rather than high-frequency 2D eye movements, so they are not directly comparable to datasets that record fine-grained eye movements in conventional RGB video. This explains the paper's evaluation design of adapting image-domain methods in the absence of a direct baseline.
vs video saliency prediction (including UNISAL, the conditioning backbone here): those methods output time-independent or weakly temporal heatmaps without ordering or timestamps; this paper inverts the role of the saliency model, using its latent to drive the generation of a coordinate sequence — a suggestive role reversal.
vs TPP-Gaze: it models gaze dynamics with a neural temporal point process and is the only method that beats this one on Discrete Fréchet, but it lags on sequence structure (Levenshtein), temporal alignment (DTW) and temporal correlation, and is judged inferior in the user study (only 15%). The comparison illustrates the gap between a point-process view and a trajectory-similarity view.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The first framework for raw gaze trajectory generation over videos of arbitrary length, successfully transferring autoregressive diffusion and the world-model paradigm to human gaze modeling.
Experimental Thoroughness: ⭐⭐⭐⭐ Two datasets, six adapted baselines, five ablation settings, an efficiency comparison and a 12-participant user study; however, in-domain training uses DIEM only, DHF1K evaluation covers just 16 videos, and there is no duration-stratified analysis of long-horizon drift.
Writing Quality: ⭐⭐⭐⭐ The method is clearly described and the motivation concrete; the handling of the one losing metric (acknowledgment plus a user study plus a critique of the metric) is notably honest, though some implementation details (whether the saliency branch is frozen, the exact UNISAL configuration) are thin due to space limits.
Value: ⭐⭐⭐⭐ Offers a reusable paradigm for synthetic human-behavior data — saliency-latent conditioning plus a clean-history autoregressive diffusion objective — transferable to other continuous behavior generation tasks, and argues that video gaze evaluation metrics need to be redesigned.