Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: autoregressive diffusion, recurrent memory, overlapping attention, hidden-state prefetching, world models
TL;DR¶
RAD adds recurrent memory after local attention in a video DiT, using overlapping frame-wise windows for detail transfer and clean hidden-state prefetching for efficient training; on Minecraft, LSTM-f improves PSNR from 14.24 for LSTM-c to 16.59, although gains do not extend to every dataset and metric.
Background & Motivation¶
Interactive video world models must do more than produce a convincing next frame: when a camera leaves a location and later returns, they should recover the previously observed scene. A standard video diffusion transformer can attend directly to recent frames, but retaining the entire history in its attention context increases storage and computation with sequence length. Sliding windows control this cost while discarding older observations, so local continuity does not guarantee long-term recall.
Recurrent neural networks offer a fixed-size memory channel, but inserting one into a DiT is insufficient. With non-overlapping video chunks, wall textures, object boundaries, and scene layouts can reach the next chunk only through compressed states. The memory module must therefore carry both long-term information and high-frequency pixel detail. Training introduces another problem: if each window must wait for the preceding window's attention and state update, recurrence serializes otherwise parallel attention. Different memory inputs during training and inference can further introduce a procedural mismatch.
Rather than simply seeking a stronger RNN, the paper reassigns information to suitable channels. Highly overlapping attention windows transfer recent details directly, while recurrent states compress more distant history, with persistent memory always updated from clean frames. Core Idea: separate local continuity from long-term memory, then precompute clean historical states so training attention can run in parallel while memory updates follow consistent rules at inference time.
Method¶
Overall Architecture¶
RAD performs action-conditioned video prediction: it takes observed video context and an action sequence and generates subsequent frames. A pretrained VAE first encodes video into latent space. Each DiT layer then applies spatial attention, temporal attention, and a temporal RNN, before the generated latents are decoded into video. Spatial attention models within-frame relationships, temporal attention accesses recent frames inside the window, and the additional RNN supplies history across windows.
The main frame-wise variant generates only one new frame at a time and advances the window by one frame. Training adds a clean-sequence prefetching branch: historical states are computed at each layer and temporal position, then supplied to diffusion training with the normal window size. Solid arrows below indicate the main data flow; dashed arrows indicate training-state provision and supervision. The prefetching branch is not an additional inference condition that reads future ground-truth frames.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
I["Recent frames, actions,<br/>and new-frame noise"] --> A["Frame-wise Overlapping Attention"]
A --> R["Action-conditioned Recurrent Memory"]
R --> O["Iterative denoising<br/>and current-frame output"]
O -->|Advance window by one frame| I
T["Clean training sequence"] --> P["Clean Hidden-state Prefetching"]
P -.->|Position-matched history during training| R
O -.->|Training prediction| L["Diffusion supervision loss"]
T -.->|Construct supervision targets| L
The model retains both explicit local context and a compressed global state; it does not replace attention entirely with an RNN. At inference time, recurrent state size does not continually grow with generated duration, but the recent window still consumes storage. Training also stores a state bank for the sample sequence, so fixed-size inference memory should not be confused with training memory independent of sequence length.
Key Designs¶
1. Frame-wise Overlapping Attention: bypassing compression for recent details
The paper compares two autoregressive granularities. Chunk-wise processing partitions video into non-overlapping windows, applies a causal temporal attention mask within each chunk, and passes only recurrent states between chunks. Frame-wise processing instead uses a window stride of 1, so adjacent predictions share most of their context. Each prediction takes recent clean frames and noise for the current frame, generates the new frame through iterative denoising, and advances the window. Within-window attention does not grant access to future ground-truth frames: available information remains constrained by autoregressive generation order.
Overlap lets recent textures, contours, and motion cues enter the next prediction directly. The RNN no longer needs to reconstruct every pixel-level detail across chunk boundaries and can devote more capacity to distant history. On Minecraft, all chunk-wise RNN variants underperform Diffusion Forcing without an RNN, whereas all improve substantially with frame-wise processing. However, on the simpler Maze dataset, LSTM-c still exceeds LSTM-f in PSNR and SSIM. The evidence therefore supports the importance of a direct local pathway in visually dense scenes, not a universal rule that smaller strides improve every metric.
2. Action-conditioned Recurrent Memory: adding history after attention
Each RNN receives features already processed by spatial and temporal attention, updating and retrieving states along the temporal axis rather than treating all patches within a frame as additional autoregressive timesteps. Both attention modules use RoPE. The diffusion timestep and action condition pass through an MLP and modulate the modules through adaptive Layer Normalization, including the RNN. Memory access is thus associated with the current interaction rather than supplying a history summary insensitive to the requested movement.
RAD compares three memory implementations: LSTM, Mamba2, and TTT. LSTM outputs and cell states provide a relatively direct short-term pathway and a long-term state; Mamba2 uses state-space recurrence; TTT writes history into memory weights through test-time-learning-style updates. The authors argue that LSTM's two pathways help when attention windows do not overlap, while differences shrink once attention handles local transfer. This interpretation fits the reported trends, but it is not a direct measurement of internal memory representations and does not establish general LSTM superiority in video generation.
3. Clean Hidden-state Prefetching: building historical states before parallel attention training
Naive frame-wise training on a sequence of length \(L\) with a window of size \(l\) requires \(L-l+1\) sequential window-attention computations because later windows depend on preceding recurrent states. RAD first scans the clean training sequence with the same model using a window size of 1, recording states at every layer and timestep. It then applies frame-wise noise to the diffusion inputs and supplies the prefetched states at corresponding historical positions, allowing attention with the normal window size to run in parallel. Recurrence itself remains; the method removes the serial chain of repeatedly waiting for full sliding-window attention.
Training prefetching admits only clean frames into persistent memory, and inference follows the same rule: only the first frame in each window updates persistent state, with that update committed only at the final DDIM denoising step. Writing every intermediate noisy version into history would repeatedly contaminate memory with different noise levels of the same content. The paper's equivalence argument assumes that the preceding state has accumulated sufficient context. It is therefore more precise to describe the method as aligning memory inputs and operating rules than as rigorously proving identical training and free-running generation distributions.
A Worked Example¶
Consider the 20-frame window used in the Maze stride ablation as a mechanism illustration. After observing a corridor, the model needs to reconstruct a previously seen wall when turning back. A frame-wise prediction uses the latest 19 frames and noise for 1 new frame. Attention first accesses recent visual details, and recurrent memory supplies older scene information. Action conditioning specifies the requested viewpoint change, and denoising produces the new frame.
The window then advances by 1 frame. Adjacent windows share 19 positions, and the generated image becomes local context for the next prediction. The clean state committed under the first-frame update rule also carries forward. Content that leaves the explicit window must subsequently rely on compressed memory, so a fixed state can still forget. This example explains information flow rather than introducing an additional experimental trajectory or a measurement of exact memory capacity.
During training, the corresponding procedure first scans the clean ground-truth video with window size 1 to build historical states, then samples normal-length windows for diffusion learning. Prefetched states must match their historical positions rather than condition a prediction on the entire ground-truth future. The correspondence with inference is specifically that clean frames update history while the local window handles the current prediction.
Loss & Training¶
RAD follows Diffusion Forcing: frames independently sample diffusion noise levels, and the model learns denoising conditioned on actions and noisy latents using a mean-squared diffusion prediction loss. There is no separate long-term memory reconstruction loss. Mathematical expressions in the cached paper have extraction damage, and Algorithm 2 denotes its target as vgt. This note therefore retains the verifiable training mechanism without reconstructing damaged expressions as the authors' exact prediction parameterization.
The second stage randomly samples a subset of subsequences instead of computing losses for every window in a long sample. The Maze RNN comparison trains from scratch for 3 epochs. On Minecraft, a standard Diffusion Forcing base is first trained on the common training set, and each RAD variant is then fine-tuned for 5000 steps. These controls help isolate architectural and windowing effects, but RNN parameter counts differ, so the comparison is not strictly parameter-matched.
Key Experimental Results¶
Main Results¶
Memory Maze contains approximately 30000 training videos of 1000 frames in 15×15 mazes. Evaluation uses another 200 out-and-back trajectories of 100–300 frames, with the initial 60% as context and the remaining 40% as prediction targets. Minecraft uses 20000 MineRL-generated training videos of 1200 frames and 60 memory-test sequences, including actions such as rotation in place. Test sizes and environments differ, so scores should not be compared directly across datasets.
Table 1 selects results from the paper's Tables 2 and 3. PSNR measures pixel reconstruction fidelity in dB, with higher values better; SSIM measures structural similarity, with higher values better; LPIPS measures perceptual difference, with lower values better. All compare predictions against ground-truth videos. They suit the scene-revisitation setting but do not establish general open-domain generation quality or physical correctness.
| Dataset | Model | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| Maze | Diffusion Forcing | 14.73 | 0.32 | 0.51 |
| Maze | VRAG | 15.55 | 0.41 | 0.43 |
| Maze | RAD LSTM-c | 15.64 | 0.43 | 0.47 |
| Maze | RAD LSTM-f | 15.50 | 0.41 | 0.45 |
| Minecraft | Diffusion Forcing | 15.65 | 0.45 | 0.53 |
| Minecraft | VRAG | 16.11 | 0.49 | 0.50 |
| Minecraft | RAD LSTM-c | 14.24 | 0.39 | 0.55 |
| Minecraft | RAD LSTM-f | 16.59 | 0.46 | 0.46 |
| Minecraft | RAD TTT-f | 16.72 | 0.46 | 0.47 |
On Minecraft, LSTM-f gains 2.35 dB over its chunk-wise counterpart and reduces LPIPS by 0.09, but its SSIM of 0.46 is below VRAG's 0.49. On Maze, LSTM-f also does not exceed VRAG in PSNR or LPIPS, so the results should not be summarized as universal superiority.
Ablation Study¶
Table 2 combines the memory-noise experiment in the paper's Table 7 and the Maze stride experiment in Table 8. The text adjacent to Table 7 does not explicitly specify its dataset and complete training budget, and its clean-memory results differ from the main LSTM-f results. Comparisons should therefore remain within that experiment rather than treating the absolute values from both groups as one shared setting.
| Experiment | Config | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| Memory noise, paper Table 7 | LSTM-f, noisy memory | 14.70 | 0.38 | 0.52 |
| Memory noise, paper Table 7 | LSTM-f, clean memory | 15.30 | 0.41 | 0.50 |
| Maze stride, paper Table 8 | LSTM-c, stride 20 | 15.64 | 0.43 | 0.47 |
| Maze stride, paper Table 8 | LSTM-f, stride 1 | 15.50 | 0.41 | 0.45 |
| Maze stride, paper Table 8 | LSTM, partial overlap, stride 10 | 15.06 | 0.40 | 0.55 |
Table 3 selects the paper's Table 5: Minecraft, the same DiT, 8 NVIDIA L40 GPUs, and a reported batch size of 3. The paper does not clarify whether this batch size is per GPU or global. Memory is the reported GPU Mem, time is one RAD training step, and parameter counts cover only the RNN module rather than the full model.
| RNN | GPU Memory GB | Seconds/Training Step | RNN Parameters M |
|---|---|---|---|
| LSTM | 10.0 | 3.7 | 195 |
| Mamba2 | 11.1 | 3.6 | 153 |
| TTT | 16.0 | 12.9 | 118 |
Key Findings¶
- Clean memory improves PSNR by 0.60 dB within Table 7, showing that noise in persistent-state inputs materially affects training rather than being merely an implementation detail.
- Local continuity and long-term memory require more than one aggregate score: in the paper's Table 4, Minecraft LSTM-f achieves motion smoothness of 95.3 versus 95.12 for LSTM-c, but imaging quality falls to 66.72 from 69.34.
- Prefetching is not free acceleration. In the paper's Table 6, first-stage RNN computation accounts for 35.7% of total forward FLOPs, first-stage attention for 17.5%, and the diffusion stage for 46.9%. The statement that additional attention costs less than 20% does not describe the entire prefetching overhead.
Highlights & Insights¶
- The main contribution is the division of information channels rather than a new recurrent cell. Providing a direct path for recent details before comparing memory architectures better explains the performance trends than forcing all historical information into a fixed state.
- The aligned component is the memory-writing rule. Both training and inference commit states from clean frames, a principle potentially useful for other memory-augmented generators with noisy iterative processes.
- A simple LSTM is a competitive engineering reference. In this implementation, it has more parameters but lower memory use and nearly Mamba2-level speed, demonstrating that operator optimization also matters when choosing an architecture.
Limitations & Future Work¶
- Author-stated limitations: RNN blocks increase training and inference costs, and fixed-size states have finite storage capacity, potentially limiting ultra-long-context performance.
- Note author's observation: evidence concentrates on Maze and Minecraft, without establishing the same benefits for open-domain real videos, large text-conditioned models, or arbitrary generation lengths; the reported tables also lack error bars.
- Note author's observation: the claimed absence of a training-inference gap primarily concerns architecture and state updates. Generated histories can still differ from ground-truth training histories, and prefetching equivalence assumes sufficient state information.
- Further evaluation could measure memory decay over longer scene-revisitation intervals, compare architectures under a fixed total compute budget, and separately report total prefetching cost, training throughput, and online generation latency.
Related Work & Insights¶
- vs Diffusion Forcing: RAD retains independently sampled frame noise levels and autoregressive generation, adding recurrent history beyond the window rather than replacing the diffusion objective with a new world-model loss.
- vs VRAG / WorldMem: these approaches supply explicit memory through historical-frame retrieval, whereas RAD compresses history into fixed-size states. Its different storage representation offers potential cost benefits but risks losing detail.
- vs TTT / LaCT: the paper treats TTT-c as a LaCT-style implementation and shows substantial improvement with the same memory type under frame-wise overlap, suggesting that cross-chunk information flow can be the primary bottleneck.
- vs Long-context State-space Video World Models: related work integrates state-space layers into diffusion models. RAD particularly emphasizes recurrence after attention and keeping all persistent-memory inputs clean rather than only an initial part of the sequence.
Rating¶
- Novelty: 4/5, for jointly designing memory, sliding windows, and parallel training rather than introducing a new recurrent cell.
- Experimental Thoroughness: 3/5, with two environments, three RNN types, and several ablations, but limited scale, uncertainty analysis, and compute-matched comparisons.
- Writing Quality: 3/5, with a clear central mechanism, although some claims of being best or gap-free require qualification by the tables and assumptions.
- Value: 4/5, as an interpretable architectural reference for interactive long-video modeling under bounded memory.