Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editing

Conference: ECCV2026
Paper: ECCV Paper
Project: Memory-V2V
Area: Video Generation
Keywords: multi-turn video editing, cross-turn consistency, external memory, dynamic tokenization, adaptive token merging

TL;DR

Memory-V2V adds retrievable, compressible visual constraints from previous edits to video diffusion models, reducing average MEt3R from independent ReCam generation's 0.1892 to 0.1357 in three-turn novel-view synthesis while supporting appearance-consistent editing across long-video segments.

Background & Motivation

Single-turn video editors can follow camera trajectories or text instructions, but practical creation rarely ends after one generation. For example, a user may pan left through a dynamic scene, orbit to another side, and revisit an earlier region, expecting previously synthesized backgrounds to remain the same. When ReCamMaster generates every turn from the original video alone, regions outside that input have no shared constraint: individual outputs can be plausible yet mutually incompatible. Feeding the previous output into the next turn is also insufficient because it directly carries only the latest generation and can propagate errors along the chain.

Long-video editing exposes the same problem: LucyEdit can turn an apple into an orange in a short clip, but independently edited segments may contain visually different oranges. The desired property is preservation of earlier editing choices, not merely smooth motion between adjacent frames; useful memory can come from an older segment or a revisited viewpoint. Existing long-video memory methods mostly organize context around a continuous temporal stream, whereas each editing turn here performs independent denoising, making temporal proximity an inadequate retrieval rule. Concatenating the entire history rapidly increases attention cost and introduces irrelevant views and redundant content.

The paper therefore retains the original input as the current editing reference while retrieving visual evidence genuinely relevant to the new turn. Retrieval selects which historical videos to consult, dynamic tokenization determines their spatial detail, and merging compresses less responsive content. This is neither explicit 3D reconstruction nor a hard constraint guaranteeing identical pixels; the generator learns to use historical visual conditioning. Core Idea: treat previous generations as external memory retrieved by task relevance and represented with relevance-dependent capacity, so independently denoised editing turns share already established visual details.

Method

Overall Architecture

Inputs comprise the current source video, a camera trajectory or editing instruction, and an external cache of previously generated videos. Novel-view synthesis builds on ReCamMaster, while long-video editing builds on LucyEdit; both share retrieval and compression principles but use different retrieval signals. Task-aware retrieval first selects memory, dynamic tokenization allocates resolution, and adaptive token merging inside the DiT reduces redundancy. After the current turn is denoised, the edited video is encoded and written back to memory for subsequent turns.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Current source video<br/>Camera trajectory or instruction"] --> RET["Task-aware retrieval"]
    CACHE["Historical edit cache"] --> RET
    RET --> TOK["Dynamic tokenization"]
    TOK --> MERGE["Adaptive token merging"]
    MERGE --> OUT["Finish current-turn denoising<br/>Output edited video"]
    INPUT --> OUT
    OUT -->|Encode and store| CACHE

Solid arrows describe inference data flow; extending target videos constructs training data and is not an additional generation branch required at inference. The cache contains VAE-encoded video latents, rather than requiring concatenation of complete attention states from every previous turn. The paper denotes the latent shape as \(F\times H\times W\times C\), corresponding to latent temporal, spatial, and channel dimensions. These latent dimensions should not be directly equated with raw video frame counts or pixel dimensions.

Key Designs

1. Task-aware retrieval: locate historical edits relevant to the current turn

For novel-view synthesis, each output latent is stored together with its camera trajectory after generation. The next turn does not prioritize the latest video: it compares target and historical field-of-view coverage and retrieves the top-k most relevant videos. VideoFOV places a unit sphere at the first camera position of the target trajectory and uniformly samples 64,800 directions. For each frame, sampled points are transformed into camera coordinates; only points with positive depth whose projections fall inside the image bounds count as visible. The union of visible sets across frames represents the video's field of view, so the comparison concerns an entire trajectory rather than one keyframe. Two complementary scores consider intersection over union and the intersection relative to target coverage; the latter emphasizes whether history covers the region currently requested. The paper combines them with weighting parameter \(\lambda=0.5\) before ranking; the extracted equations are damaged, so no unverifiable symbol-level expression is reconstructed here.

This retrieval can recover views generated several turns earlier, but field-of-view overlap remains a geometric proxy rather than full occlusion-aware 3D visibility. Long-video editing lacks reliable camera trajectories, and multiple segments can share the same instruction, making text similarity insufficiently discriminative. The model therefore maintains source-to-edited-segment associations, compares current and historical source segments using DINO features, and retrieves the corresponding edited video latents. Retrieval uses visual content before editing, while the reference injected into the generator comes from after editing; this distinction is central. Retrieval is non-parametric; detailed similarity aggregation and algorithms are deferred to the supplement, so no additional frame-pooling rule is assumed here. The main text also does not specify one fixed top-k value for all experiments, and training memory counts should not be mistaken for inference retrieval counts.

2. Dynamic tokenization: give relevant videos finer spatial representations

Video-level retrieval can still introduce many frames: a novel-view video contains 81 frames, so multi-turn memory quickly reaches hundreds of frames. Instead of representing every conditioning video at equal precision, Memory-V2V trains tokenizers at different compression scales and translates retrieval rank into a token budget. The user input uses \(1\times2\times2\) compression, the 3 most relevant retrieved videos use \(1\times4\times4\), and the remaining videos use \(1\times8\times8\). These dimensions correspond to time and the two spatial axes, so this stage primarily changes spatial granularity rather than directly dropping temporal frames at those rates. For equal latent resolution, larger spatial patches produce fewer tokens, retaining coarse structure from less relevant history without assigning it equal attention capacity. The finest allocation remains with the user input, preventing the current source video from being overcompressed relative to abundant historical memory.

Unlike simply truncating history, retrieval decides whether a video enters the context, after which relevance determines its representational precision. This explains why adding previous videos alone is insufficient: whole-video memory requires both selection and control over representation cost. The tokenizers are learned modules, not training-free preprocessing that can be replaced merely by resizing images at inference. Training alternates single-scale and mixed-scale tokenization to prevent one scale from always relying on detail supplied by another.

3. Adaptive token merging: retain low-response information instead of deleting it

Sequences can remain long after dynamic tokenization, leaving the quadratic cost of DiT self-attention as a major bottleneck. The model first spatially averages key features within each frame, obtaining a cheap frame-level representative for estimating its relevance to current target generation. Following Equation (4), if \(\mathcal{I}_t\) denotes the token indices in frame \(t\), the frame-level key is:

\[ \bar{K}_t=\frac{1}{|\mathcal{I}_t|}\sum_{i\in\mathcal{I}_t}K_i. \]

It then evaluates scaled attention responses from target queries to these frame-level keys and uses the maximum response over target queries to measure a frame's importance. The maximum preserves history strongly needed by at least one target location, rather than letting unrelated locations dilute that need through averaging. Highly responsive frames retain fine-grained representations; low-response frames pass through a learnable convolutional operator that fuses their tokens into a compact representation. Larger histories use stronger compression: the method reduces the representational capacity of weakly responsive information rather than declaring it useless. The qualitative ablation in Figure 7 shows that reducing high-response tokens damages consistency in overlapping novel-view regions, while discarding low-response tokens can introduce detail and motion artifacts.

Merging location also matters: early attention is unstable, so a frame judged unimportant may become important in later layers. The authors analyze 30 DiT blocks; Table 1 reports Pearson correlation of \(0.608\pm0.137\) between Block 1 and subsequent blocks, versus \(0.753\pm0.144\) for Block 21 and subsequent blocks. They consequently apply merging at Blocks 10 and 20 rather than aggressively compressing all conditions at the beginning. These correlations support more stable decisions in middle and later layers, but do not theoretically guarantee lossless merging. Equation (5) is damaged in extraction; Equation (6)'s \(N_t/r\) notation also raises a convention question against the training range \(r\in[0.3,0.7]\), so exact retained token counts are not inferred from it.

A Worked Example

Consider the apple-to-orange edit in Figures 1 and 5: a video exceeding 200 frames cannot directly fit into the base editor's roughly 81-frame window. The model divides the source into manageable segments; once the first segment is edited, its output becomes a visual example in historical memory. When the same fruit appears later, DINO features of source segments retrieve relevant history, and the previously generated orange video becomes a condition. Dynamic tokenization preserves more spatial detail for the most relevant segments, while adaptive merging compresses memory frames receiving little attention from current generation. The current segment still undergoes independent denoising conditioned on its own source and instruction; it neither simply copies old frames nor generates all segments as one continuous noisy stream. Edited segments are stitched into a long video, aiming to maintain fruit identity and appearance while preserving each segment's original motion and background. Memory constrains what should remain consistent, while the source specifies what happens in the current segment; this example assumes no unreported overlapping-window or stitching algorithm.

Loss & Training

The novel-view branch uses synthetic multi-view data containing 10 synchronized camera videos per scene, randomly selecting 1–6 videos as memory during training. Fine-tuned components include the base model's self-attention layers, MLP projector, and camera encoder, alongside the new tokenizers and compressors. For 50% of training, only one tokenizer is used; the other 50% mixes scales, allowing each scale to learn useful conditioning. Adaptive token merging is enabled with 50% probability, with a reported random compression parameter in \([0.3,0.7]\); its convention relative to Equation (6) requires implementation verification. The long-video branch uses 56K samples filtered from Señorita-2M, each originally containing an instruction, source video, and target video. An existing video extension model extends only the target video, whose additional segments serve as training memory, avoiding the need for genuine long source-target pairs. This augmentation teaches editing with target-side visual history beyond the current short source segment, although generated extensions can introduce their own biases. Both branches use rectified flow matching loss on 32 A100 GPUs with total batch size 32, reaching stable convergence in approximately 1–2K fine-tuning steps. The contribution concerns conditioning organization and compression: it is not a training-free retrieval plugin, and no additional cross-turn consistency loss should be invented.

Key Experimental Results

Main Results

Novel-view experiments use 40 public videos, each with 3 generated camera trajectories having substantial field-of-view overlap; the following selection comes from Table 2, page 11. MEt3R measures cross-view geometric consistency, with lower values preferred; the average aggregates pairwise comparisons between turns 1–2, 1–3, and 2–3. RotErr and TransErr are also lower-is-better; the table does not specify units, so no angular or distance units are added here.

Method	Average MEt3R	Turns 1–3 MEt3R	RotErr	TransErr
TrajCrafter	0.1818	0.2110	3.66	57.44
ReCam (Ind)	0.1892	0.1982	1.97	24.23
ReCam (AR)	0.1485	0.1985	Not reported	Not reported
Memory-V2V	0.1357	0.1525	1.65	13.47

ReCam (Ind) reuses the same original input each turn, whereas ReCam (AR) uses the previous output as its next input. Memory-V2V lowers average MEt3R by 0.0535 against ReCam (Ind); its turns 1–3 result also improves on ReCam (AR), showing benefits beyond conditioning on adjacent turns. However, its turns 2–3 MEt3R in Table 2 is 0.1379, higher than ReCam (AR)'s 0.1290, so it is not best for every turn pair.

Long-video evaluation uses 50 videos from the Señorita test set; the next selection reports representative methods and metrics from Table 3, page 11, whose columns are all labeled higher-is-better. Subject consistency and imaging quality come from VBench; DINO-F and CLIP-F are cross-frame feature similarities, not direct measures of instruction-following success.

Method	Subject consistency	Imaging quality	DINO-F	CLIP-F
LucyEdit (Ind)	0.8683	0.6429	0.6856	0.8225
LucyEdit (FIFO)	0.8737	0.5598	0.6784	0.8198
FlowEdit	0.9027	0.6433	0.8612	0.8943
DynVFX	0.9119	0.6666	0.8387	0.8682
Memory-V2V	0.9326	0.6759	0.8019	0.8741

Memory-V2V improves subject consistency and imaging quality over these comparisons, but FlowEdit has higher DINO-F and CLIP-F, and DynVFX also has higher DINO-F. The broad claim of outperforming all baselines across metrics on page 13 does not fully match Table 3; individual values take precedence, and the authors also warn that high similarity may reflect insufficient editing.

Ablation Study

The following selection comes from Table 4, page 12; each individual addition uses dynamic tokenization alone as its baseline, while the full model includes both retrieval and merging. The surrounding text does not fully specify timing hardware and memory size, so time comparisons should remain within this ablation, and its scores should not be treated as sharing Table 2's protocol.

Config	Time / seconds	MEt3R, lower is better	Subject consistency, higher is better	Imaging quality, higher is better
Dynamic tokenization only	980.80	0.2234	0.9368	0.7307
Add video retrieval	965.95	0.2169	0.9338	0.7288
Add adaptive token merging	661.31	0.2344	0.9361	0.7287
Full model	648.5	0.2208	0.9351	0.7300

Key Findings

• The full model saves 332.30 seconds against dynamic tokenization alone, approximately 33.9%; MEt3R changes from 0.2234 to 0.2208, making efficiency the main improvement in this comparison.
• Retrieval alone produces the lowest MEt3R, 0.2169, versus 0.2208 for the full model with merging, revealing a small compression-quality trade-off.
• The text accompanying Figure 9 on page 15 reports over 90% lower FLOPs and latency than naive context expansion and approximately 30% additional reduction from merging; these are the authors' reported findings, not independently reproduced measurements.

Highlights & Insights

• Separating cross-turn consistency from within-segment temporal continuity is valuable. History selection follows task relevance rather than generation order, fitting revisited viewpoints and recurring objects.
• Retrieving with source segments but conditioning with target segments avoids the ambiguity of text retrieval under shared instructions. Source-content matching and edited-appearance memory thus serve distinct purposes.
• Low responsiveness does not imply uselessness, making merging better suited to visual generation than deletion. Details, structure, and motion may depend on historical conditions receiving limited current attention.

Limitations & Future Work

• The authors explicitly acknowledge inherited base-model limitations, including large viewpoint changes; external memory cannot supply capabilities the underlying generator has not learned.
• Training covers continuous single-shot videos, so multi-shot videos with abrupt transitions may fail. Shot-boundary detection and scene-specific memory management are plausible extensions, but are not evaluated here.
• Reader analysis: test sets of 40 and 50 videos and primarily three-turn quantitative evaluation do not establish unlimited-turn stability; repeated reuse of erroneous history also deserves separate measurement.
• Extraction and reproducibility boundary: supplementary algorithms and code were not verified, parts of the FOV and responsiveness equations are damaged, and compression conventions are unclear; the main text does not establish every retrieval, merging, or timing detail.

Related Work & Insights

• vs ReCamMaster: the base model performs camera-controlled single-turn generation, while this work adds selectable visual history beyond the original input. An autoregressive chain directly conditions on the latest output, whereas retrieval can cover much earlier revisited regions.
• vs LucyEdit and FIFO-Diffusion: this method still independently denoises short segments and preserves appearance through history conditioning; FIFO-style temporal noise scheduling is not equivalent to explicitly remembering an edited object's identity.
• vs TokenFlow, RAVE, and memory-based long-video generation: the first two rely on cross-frame feature propagation or noise organization, while memory-generation methods mostly serve continuous generation; this paper targets conditioning consistency between independent editing turns.
• Possible extension: attach quality or conflict confidence to historical edits, selectively releasing old constraints when a new instruction requests a changed attribute. This is a reader-proposed research direction, not an established capability of the paper.

Rating

• Novelty: 4/5. Task-aware memory combined with two-level compression addresses a practical multi-turn editing problem.
• Experimental Thoroughness: 4/5. Two tasks, multiple baselines, and component ablations are covered, but long-horizon, scene-transition, and erroneous-memory evaluation remain limited.
• Writing Quality: 4/5. The main mechanism is clear, but broad claims, table rankings, and compression-parameter conventions need careful reconciliation.
• Value: 4/5. A useful approach to cross-turn consistency for existing video editors, rather than a universal training-free plugin.