MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens¶
Conference: ECCV2026
Paper: ECCV Paper
Project: https://memrope.github.io
Area: Video Generation
Keywords: autoregressive diffusion, dual-rate memory, RoPE, KV cache, training-free inference
Date: 2026-09-17
TL;DR¶
Without updating generator weights, MemRoPE continuously folds evicted context into long- and short-term EMA memory and applies RoPE to unrotated keys only at attention time, enabling hour-scale generation with a fixed cache and a VBench-Long average of 87.99 versus 87.01 for the comparison method on one-hour LongLive videos.
Background & Motivation¶
Autoregressive video diffusion divides a video into successive chunks, denoises only the current chunk, and conditions on cached keys and values from earlier frames. This removes the need to generate the entire video at once, but does not automatically preserve its history: a sliding window repeatedly evicts old frames, gradually removing evidence about the original subject and scene. Self-Forcing and LongLive improve the relationship between training and generation, yet extended rollouts still face finite context and positional encoding limits.
Keeping initial frames as attention sinks provides a relatively high-quality visual anchor, but cannot reflect later changes. Deep Forcing selects historical tokens through cumulative attention in its Participative Compression mechanism; this paper observes that long-retained tokens accumulate an advantage, causing the selected set to stagnate, while occasional admission of high-scoring new tokens can produce abrupt shifts. RoPE introduces another obstacle: even if bounded relative positions prevent index extrapolation, keys carrying different rotary phases cannot simply be averaged into a memory with one shared position.
Memory compression and positional encoding therefore need joint treatment: content must first be separated from positional phases before historical aggregation becomes well-defined. Core Idea: cache unrotated keys, continuously summarize history through two EMA streams with different update rates, and reassign bounded positions to the finite context at each attention call.
Method¶
Overall Architecture¶
The inputs are a text prompt and a pretrained autoregressive video diffusion model; the output is a video extended chunk by chunk. MemRoPE's persistent cache contains a static initial sink, long- and short-term memory, and a recent local window. The current noisy chunk reads this context during denoising. There is no separately trained memory network or growing database containing every previous frame.
The two contributions are Dual-Rate Memory and Online RoPE Indexing. The former determines how evicted content survives; the latter allows keys to be merged without positional-phase interference and restores positional information for the actual attention computation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Text and denoised chunks"] --> Memory["Dual-Rate Memory"]
Memory --> Cache["Three-tier unrotated cache<br/>sink, memory, local window"]
Cache --> Position["Online RoPE Indexing"]
Noise["Current noisy chunk"] --> Position
Position --> Denoise["Frozen generator denoising"]
Denoise --> Output["Append output and local cache"]
Output -->|"Absorb old content on overflow"| Memory
This loop spans generated chunks rather than updating memory at every denoising substep. A chunk's clean keys and values enter the persistent cache only after all its denoising steps finish, preventing intermediate noisy states from contaminating memory.
Key Designs¶
1. Dual-Rate Memory: smoothly absorb content leaving the window
The three cache tiers serve different roles: the sink anchors the opening appearance, the local window preserves recent detail, and memory between them summarizes distant history. When the window exceeds capacity, its oldest chunk is not simply deleted. Spatial average pooling over each frame's token grid first produces compressed keys and values, which then update both memory streams. Keys and values follow the same update rule; spatial pooling also means the retained representation is a summary, not a reconstructable copy of old frames.
Each stream adds the old memory multiplied by one minus its update rate to the newly pooled content multiplied by that rate. The long-term stream uses \(\alpha_L=0.01\), while the short-term stream uses \(\alpha_S=0.1\): the former is less affected by any individual update and can preserve stable appearance; the latter absorbs recent changes faster, avoiding a cache tied entirely to the opening scene. These are two temporal aggregations of the same input, not a hard assignment of frames to long-term or short-term categories.
Only after both streams are updated is the old chunk removed from the local window. Long-term memory does not preserve history losslessly: earlier contributions weaken over time, but unlike FIFO, eviction does not immediately erase the chunk's entire influence on future generation. Compared with discrete selection by cumulative attention, small continuous updates also reduce the risk of abruptly replacing the cache with a different set of strong conditioning signals.
The implementation uses a sink setting of \(S=3\), \(M=1\) memory token per stream, and a local window of \(L=4\) frames. The source mixes token and frame terminology for these settings, so these temporal-slot counts should not be interpreted as the network's total number of scalar KV elements; actual tensor sizes also involve spatial tokens, layers, and attention heads.
2. Online RoPE Indexing: aggregate content before assigning context positions
Conventional caches store keys after applying RoPE. Keys from different times carry different rotation matrices, so averaging them mixes positional phases even when their contents are similar. The issue is not that rotation itself is nonlinear: different position-specific rotations generally cannot be factored into one shared rotation. MemRoPE stores every cached key unrotated, including sinks, memory, and the local window, allowing EMA to operate in a consistent content representation space.
At each attention call, the system orders the context as sink, long-term memory, short-term memory, local window, and current chunk. It assigns contiguous block-relative indices starting at zero, then rotates keys and current queries on demand. Values are not rotated. The next step reuses the unrotated persistent cache rather than writing these rotated keys back, avoiding repeated corrective re-rotation of old keys.
Positions now describe relative order within the current finite context, not how long the video has been running. The index budget for historical context and the current chunk together must remain within the model's training range; with bounded attention context, advancing generation time does not make indices grow without bound. A summary memory has no single original frame timestamp, so assigning it a stable relative slot is more appropriate than treating it as one particular old frame.
Unlike this design, ∞-RoPE re-anchors already rotated cached keys, primarily addressing positional extrapolation. Delaying positional encoding also removes the barrier to historical aggregation. Rolling Forcing already applies RoPE dynamically to sink keys, but MemRoPE extends unrotated storage to the entire cache, allowing arbitrary content leaving the local window to enter memory.
A Worked Example¶
Consider the paper's example of a person continuously walking along a street. The following illustrates the mechanism rather than introducing an additional experiment.
- After the opening chunk is denoised, its visual content becomes the static sink; memory is initialized as in the algorithm, and the local window begins accepting subsequent chunks.
- For the next chunk, concatenate the three-tier cache with the current noisy state. Each attention call assigns block-relative positions, and the frozen model completes denoising.
- Append the finished chunk's clean KV states to the local window. On overflow, spatially pool the outgoing chunk and update both memory streams.
- As the person enters another neighborhood, the short-term stream adapts more quickly to the new appearance, while the long-term stream changes slowly. This does not guarantee identity preservation, but makes distant context less dependent on the static opening anchor.
- Repeating this procedure keeps the persistent cache independent of total video length, while output storage and cumulative generation time still grow.
Loss & Training¶
MemRoPE introduces no training loss and does not fine-tune the generator. Training-free refers to this cache and positional mechanism, not to the base model having never been trained. Experiments use Self-Forcing and LongLive built on the Wan2.1-T2V-1.3B architecture.
Each chunk contains 3 latent frames and uses 4 denoising steps at timesteps 1000, 750, 500, and 250. Persistent cache updates occur only after the final denoising step. Cache updates, spatial pooling, and online positional encoding are inference-time operations.
Several displayed equations are corrupted in the extracted source. This note therefore explains the update order and mechanism using verifiable method prose and algorithm text, without reconstructing damaged equations.
Key Experimental Results¶
Main Results¶
Prompts come from MovieGenBench and are refined with Qwen2.5-7B-Instruct. Outputs have resolution 480×832 at 16 fps. The 120-second and 240-second settings each use 128 prompts; 480-second generation uses 20 randomly sampled prompts, and one-hour generation uses 10 randomly sampled prompts.
The following extracts VBench-Long averages from Tables 1 and 2, with higher scores preferred. The average summarizes aesthetic quality, background consistency, imaging quality, motion smoothness, subject consistency, and temporal flickering scores. Methods should be compared within the same base model and duration.
| Base Model | Duration | Original Base | Deep Forcing | ∞-RoPE | MemRoPE |
|---|---|---|---|---|---|
| Self-Forcing | 120 seconds | 84.25 | 83.84 | 83.64 | 85.23 |
| LongLive | 120 seconds | 85.51 | 86.01 | 85.58 | 86.45 |
| Self-Forcing | 240 seconds | 83.04 | 81.66 | 82.84 | 84.89 |
| LongLive | 240 seconds | 85.33 | 85.48 | 85.21 | 86.22 |
| Self-Forcing | 480 seconds | Not reported | Not reported | 82.09 | 85.21 |
| LongLive | 480 seconds | Not reported | Not reported | 84.75 | 85.81 |
| LongLive | 1 hour | Not reported | Not reported | 87.01 | 87.99 |
At one hour, MemRoPE leads ∞-RoPE on all six metrics, but this does not show that longer videos have higher quality: prompt samples differ between duration settings. A separate duration analysis uses the same 20 prompts throughout; both methods degrade, but MemRoPE declines more slowly.
Ablation Study¶
The second numerical table uses the stability analysis in Table 3(b), rather than reconstructing the damaged Table 4. Gemini 3.1-Pro scores exposure stability and visual degradation in 120-second videos. The main text does not provide the full scoring scale or prompt, so these values should not be interpreted as percentages or assigned an assumed scale maximum.
| Config | Visual Stability Score | Note |
|---|---|---|
| Self-Forcing | 1.55 | Standalone baseline in the table |
| Rolling Forcing | 3.40 | Standalone baseline in the table |
| LongLive | 4.10 | Original base in the LongLive group |
| LongLive + Deep Forcing | 3.90 | LongLive group |
| LongLive + ∞-RoPE | 4.05 | LongLive group |
| LongLive + MemRoPE | 4.15 | LongLive group |
Actual component ablations are summarized qualitatively from the prose: Online RoPE Indexing alone improves subject consistency, Memory Tokens provide stronger aesthetic and imaging-quality gains, and combining them gives the best average. Stream ablations show that dual memory improves imaging quality, but its advantage over short-term memory alone is small; the evidence does not support attributing all gains to the long-term stream.
Key Findings¶
- MemRoPE has the highest average for every base-model and duration combination in the main table, not necessarily the best individual metrics. At shorter durations, ∞-RoPE often scores higher on motion smoothness and temporal flickering.
- The parameter analysis tests 12 update-rate combinations with an average-score variation below 0.7. This supports robustness within the tested range, not insensitivity to every possible setting.
- The user study includes 30 participants, each comparing 20 pairs of 120-second videos across six perceptual dimensions. It complements automated metrics but does not guarantee indefinite quality.
Highlights & Insights¶
- Cache management and positional encoding are co-designed. Removing the representation barrier to temporal aggregation explains why solving positional extrapolation alone can still lose identity and fidelity.
- Dual-rate EMA assigns stable retention and fast adaptation to different update scales. Attention lets the model use both streams without training an additional router.
- Fixed capacity need not imply static content. Smooth updates can avoid stagnation and abrupt changes caused by discrete token selection, at the cost of compressing historical detail.
Limitations & Future Work¶
- Infinite describes a mechanism unconstrained by growing position indices and historical cache size, not proof of correct content at arbitrary duration. The longest evaluated setting is 1 hour with only 10 prompts.
- Spatial average pooling and EMA lose fine-grained layout and rare-event information. This is a limitation inferred from the mechanism; multiple summary slots or content-selective aggregation merit testing rather than assuming that all history remains recoverable.
- The main-text evaluation lacks full stability-scale details and confidence intervals. The difference between 4.15 and 4.10 cannot be called statistically significant from these results alone.
- Fixed KV capacity is not fixed total cost: longer outputs require more cumulative computation and storage. Architectural generality also needs validation beyond the two evaluated base models.
- Table 4 is corrupted in the extracted text. This note neither quotes its numerical entries nor guesses component-ablation differences; damaged equations are replaced with reliable prose descriptions.
Related Work & Insights¶
- vs Self-Forcing / LongLive: These provide trained autoregressive generators, while MemRoPE is an inference-time addition. Gains should be measured against the same base, not attributed to an entirely new generator.
- vs Deep Forcing: Participative Compression selects discrete historical tokens by attention, whereas MemRoPE continuously aggregates outgoing window content. The latter is smoother but cannot preserve the same explicit token-level detail.
- vs ∞-RoPE / Rolling Forcing: The former emphasizes positional re-anchoring, while the latter dynamically handles positions for static sinks. MemRoPE extends unrotated caching across all tiers to enable evolving memory.
- Transferable insight: Before introducing cache summaries in another streaming model, check whether merged representations contain incompatible position-specific transformations. This is a design suggestion, not a cross-task result validated by the paper.
Rating¶
- Novelty: 4/5. The contribution lies chiefly in jointly designing memory updates and positional decoupling, not EMA or RoPE alone.
- Experimental Thoroughness: 4/5. Two base models, multiple durations, and perceptual evaluations are covered, but the hour-scale sample is small.
- Writing Quality: 4/5. Failure modes and mechanisms are clearly explained; extraction damage limits equation and ablation verification.
- Value: 4/5. A practical long-video inference approach for frozen autoregressive generators, without a guarantee of indefinite stability.