Consistent Video-to-Video Translation via Explicit Correspondences

Conference: ECCV2026
Paper: ECCV Official Page / Paper PDF
Authors: Gaurav Parmar, Zhengqi Li, Richard Zhang, Jun-Yan Zhu, Srinivasa Narasimhan, Eli Shechtman, Yotam Nitzan
Area: Video Generation / Video-to-Video Translation
Keywords: explicit correspondences, latent retrieval, dynamic memory, long-range consistency, autoregressive diffusion

TL;DR

vid2vid-long uses the current input block to retrieve matching outputs generated much earlier, then conditions video translation through per-token correspondence cross-attention, reducing five-step Rolling Forcing LPIPS on VACE-Bench from 0.427 to 0.240 while also reducing throughput from 10.99 to 8.10 FPS.

Background & Motivation

Streaming video translation must do more than produce an attractive current frame: an object that leaves the view should retain its appearance when it returns. Autoregressive diffusion models generate successive video blocks and use a cache of past keys and values to maintain continuity. To control cost, they generally retain only a fixed-length sliding window. This remembers recent motion but cannot preserve colors, textures, and identity once those details leave the window. An object returning after a few seconds can therefore receive a plausible yet different appearance.

Simply enlarging the window raises attention and cache costs without ensuring that the model locates the relevant history. An attention sink retaining the initial frames only protects early content, sparse keyframes can miss important local regions, and pooling or resolution compression can erase detail. In video-to-video translation, these strategies also overlook an additional source of information: the input video remains available. The system can first ask whether an input region has appeared before, instead of forcing the generator to rediscover its correspondence among all historical outputs.

The paper turns this observation into a searchable input-output memory. Repeated input regions index appearances that were actually generated earlier. The recent sliding window still supports local motion continuity, while the external memory restores content that the model would otherwise forget. Core idea: use explicit local correspondences in the input to select historical output patches, and expand memory only when novel content appears, rather than retaining or compressing history mechanically by time.

Method

Overall Architecture

The input is a stream of video conditions, with depth video used in the main experiments and text conditioning retained; the output is an appearance video generated block by block. vid2vid-long augments an autoregressive VACE student with an external memory whose keys are input latent patches and whose values are fully generated output latent patches at the corresponding locations. This key-value store is distinct from the Transformer layers' internal KV cache: the external store retrieves distant history by content, while the internal cache continues to supply recent temporal context.

Each new block is VAE-encoded and patchified, then undergoes key construction for latent key-value pairing and thresholded neighbor retrieval. The retrieved historical outputs participate in every denoising step through correspondence cross-attention. Only after generation finishes does novelty-based writing add useful input-output pairs to the store. Retrieval runs once per block, and its results are reused across denoising steps to avoid repeating the same matching work.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Current input block<br/>VAE encoding and patchification"] --> Pair["Latent Key-Value Pairing"]
    Pair --> Read["Thresholded Neighbor Retrieval"]
    Memory["Historical Input-Output Store"] --> Read
    Read --> Corr["Correspondence Cross-Attention"]
    Context["Text and depth conditions<br/>Recent KV cache"] --> Corr
    Corr --> Write["Novelty-Based Writing"]
    Write -->|Updated for the next block| Memory
    Write --> Output["Output video block"]

Key Designs

1. Latent Key-Value Pairing: use the input to locate content and the output to preserve appearance

The correspondence unit is not an entire frame but a latent patch aligned with one Transformer token. After VAE compression, VACE divides the latent representation into patches of shape \(1\times2\times2\); the patch does not additionally merge the temporal dimension. At global spatiotemporal location \(g\), the input patch \(z_g\) directly becomes the retrieval key \(k_g=z_g\), while its value \(v_g\) is the fully generated output latent patch at that location. A shared index therefore connects two questions: which earlier input region resembles the current depth region, and how was that earlier region rendered?

This distinction is essential. Using the input patch as the retrieved output condition would merely repeat the structure without recovering previously chosen colors and textures. Searching only in output space would lack a reliable query because the current output has not yet been generated. The input establishes correspondence, and the output carries the committed appearance, exploiting the conditions specific to video translation. Local pairing also allows different regions of a current block to retrieve content from different historical moments, without requiring the whole frame to match one old frame.

The authors evaluated external visual features and state that directly using latent patches worked best, while avoiding latent decoding and an additional feature extractor. However, the feature ablations are deferred to supplementary material that is absent from the local cache. The note therefore does not assign numerical disadvantages to any particular visual encoder. Direct latent matching should be understood as the authors' empirical choice for this task, not a universal claim about optimal descriptors for all video conditions.

2. Thresholded Neighbor Retrieval: provide history only when a match exists

Before generating the next block, each input key searches the existing memory for at most \(k=3\) neighbors by cosine similarity. A candidate must also meet the retrieval threshold \(\tau_m=0.75\). Here, a neighbor is similar in input content, not necessarily close in time. An object that disappeared seconds earlier can therefore recover its historical output even after the corresponding frame has left the KV window, provided that its current input matches a retained key.

The threshold allows the system to reject a match. If no candidate qualifies, a binary validity mask is set to \(m_g=0\), and the history branch contributes no residual; if candidates exist, \(m_g=1\). A region never seen before can thus receive a new appearance from the ordinary depth and text conditions, instead of being forced to follow an old patch that is merely the least dissimilar item in the store. The procedure filters by similarity and then takes top-\(k\); it is not unconditional retrieval of three historical patches.

Retrieval uses the current input and already generated history. It neither accesses future outputs nor precomputes global optical flow over the video to be generated. It runs once for each video block, and the selected neighbors are reused throughout denoising because the input correspondences do not need to be recomputed at every noise step. Avoiding repeated retrieval, however, does not make search complexity independent of memory size. The main text does not specify a search-index implementation or establish a cost bound for an indefinitely long stream.

3. Correspondence Cross-Attention: each token reads only its own historical candidates

Retrieved output latent patches are converted into history tokens using the same patch embedding as the input latents. Within every Transformer block, the new correspondence cross-attention layer sits between self-attention and text cross-attention. The current token's hidden state supplies the query, and only the history tokens retrieved for that token supply keys and values. Rather than letting a whole frame attend to a mixed pool of history, the layer preserves the local constraint that this input region corresponds to these historical outputs.

Let \(\mathcal{H}_g\) denote the candidate history tokens for location \(g\). The central operation from the paper's Equation (4) is:

\[ \operatorname{CorrXA}(g)=m_g W_o\operatorname{Attn}(W_qx_g,W_k\mathcal{H}_g,W_v\mathcal{H}_g), \qquad x_g\leftarrow x_g+\operatorname{CorrXA}(g). \]

History provides a learned residual condition, rather than hard-pasting old pixels onto the current frame. The model can still combine current structure, text, and recent context during generation, with history helping it recover an earlier appearance choice. Zero-initializing the output projection \(W_o\) makes this new residual branch zero at the start of training, preserving the pretrained model's behavior. The validity mask prevents unmatched tokens from being influenced by history.

For efficient implementation, the authors fold the sequence dimension into the batch dimension, turning the operation into independent \(1\times k\) attention problems. The candidate count for this additional attention is therefore bounded by \(k\), rather than expanding directly with the entire history. Two costs must nevertheless be separated: this construction bounds the attention used to inject retrieved history, but searching the external store still has its own cost. This helps explain why a lightweight architectural addition can still slow down measured five-step generation.

4. Novelty-Based Writing: append only input content not already represented

Only after the new block is fully denoised can the current input key and its generated output value become a reliable record for later retrieval. The method computes each new key's maximum cosine similarity to the old store. It appends the pair only if that similarity is below the history threshold \(\tau_h=0.75\); otherwise, the content is treated as sufficiently represented and the duplicate record is skipped. The retrieval threshold asks whether an old patch is usable, whereas the writing threshold asks whether a new patch is worth storing. They serve different purposes even though their defaults coincide.

Using the method section's convention that \(H_b\) is the memory after block \(b\), Equations (1)-(2) can be written as:

\[ s(k_g,H_b)=\max_{(k_{g'},v_{g'})\in H_b}\frac{k_g^\top k_{g'}}{\lVert k_g\rVert\lVert k_{g'}\rVert}, \qquad H_{b+1}=H_b\cup\{(k_g,v_g):g\in G_{b+1},\ s(k_g,H_b)<\tau_h\}. \]

Here, \(G_{b+1}\) is the set of token indices in the new block. Each video starts with an empty store, so its first block has no valid history to read. The main text does not separately define the numerical convention for maximum similarity over an empty set; this note does not invent that implementation detail. The update also shows that the method appends records to the old store, rather than overwriting values, learning a fixed-size hidden state, or repeatedly averaging historical images.

When scenes recur frequently, repeated patches do not accumulate indefinitely, and memory growth is driven primarily by visual novelty. This is not an unconditional fixed-memory guarantee: continually novel content can still enlarge the store, and the main text gives neither a capacity limit nor an eviction policy. Similar old records are not automatically replaced by a new generation, which can preserve the original appearance choice. Conversely, an erroneous initial record might be reused repeatedly; this last point is an inference from the writing mechanism, not a separately quantified author result.

A Worked Example

Figure 5 illustrates a depth-conditioned video in which a dog initially receives a yellow snout. After leaving the view and returning later, the model without correspondences changes the snout to blue. After the first generation, vid2vid-long can retain paired input patches and the output patches carrying the yellow appearance. When the dog returns, its current input patches search for at most 3 historical neighbors above the 0.75 threshold, recovering that earlier appearance choice even if the old frames have left the sliding window.

Correspondence cross-attention then provides the matched outputs as additional conditions for the current tokens. The same retrieval result is reused through denoising, helping preserve the yellow appearance. Similar regions generally need no new records, while newly exposed unmatched regions can be generated from the ordinary conditions and stored when the novelty rule permits. This is a walkthrough of the figure and algorithm, not a patch-level measurement: the cache gives no similarity values for this example, so “at most 3” must not be presented as exactly 3 observed matches.

Loss & Training

The default student is initialized from VACE 1.3B and distilled with five-step Rolling Forcing. The teacher is bidirectional VACE 14B and does not use correspondence memory. All student parameters are finetuned jointly with the new cross-attention layers; training is not restricted to an inserted adapter. Memory is active during training: the student reads history formed from its own previously generated blocks and writes according to the same novelty rule, so the retrieved values reflect the student's actual generation distribution.

The paper explicitly retains the standard Rolling Forcing objective without adding a correspondence loss or any other extra loss term. The contribution is therefore the organization and injection of conditioning information, not explicit optical-flow supervision or a new temporal reconstruction loss. Optimization uses AdamW with batch size 64, a student learning rate of \(2\times10^{-6}\), and a critic learning rate of \(4\times10^{-7}\); the main experiments use 480p. The cache does not contain the supplementary material's full training configuration, training duration, or dataset size.

Key Experimental Results

Main Results

VACE-Bench interleaves two different 5-second clips as A→B→A→B and compares generated content at its first and final occurrences. DL3DV uses calibrated cameras and 3D Gaussian Splatting to render exact pose revisits. Lower LPIPS and DreamSim and higher CLIP-Sim indicate greater consistency between generated occurrences, not framewise reconstruction accuracy against a real target video. Quality averages six V-Bench scores: subject consistency, background consistency, temporal flickering, motion smoothness, aesthetic quality, and imaging quality.

The following table selects results from Tables 1 and 2. Latency means time to the first frame, and throughput is measured on a single NVIDIA H100. Different sampling-step counts must not be treated as equal compute budgets.

Dataset	Method	LPIPS ↓	DreamSim ↓	CLIP-Sim ↑	Quality ↑	First-Frame Latency / s ↓	FPS ↑
VACE-Bench	Rolling Forcing, 5 steps	0.427	0.172	0.905	0.799	2.58	10.99
VACE-Bench	vid2vid-long, 2 steps	0.302	0.079	0.941	0.814	1.09	13.88
VACE-Bench	vid2vid-long, 5 steps	0.240	0.066	0.955	0.828	3.72	8.10
DL3DV	Rolling Forcing, 5 steps	0.445	0.163	0.907	0.806	2.58	10.99
DL3DV	vid2vid-long, 2 steps	0.379	0.083	0.932	0.799	1.09	13.88
DL3DV	vid2vid-long, 5 steps	0.305	0.058	0.952	0.821	3.72	8.10

At five steps, LPIPS falls by approximately 43.8% on VACE-Bench and 31.5% on DL3DV, calculated from the raw table values. First-frame latency simultaneously increases by 1.14 seconds, and throughput decreases by 2.89 FPS. The two-step configuration is faster but less consistent than the five-step version, and its DL3DV Quality is lower than the original Rolling Forcing baseline. “Preserving real-time performance” is therefore better read as retaining low-step streaming capability, not eliminating speed costs or meeting high-frame-rate interaction requirements.

Ablation Study

The following table selects results from Tables 4 and 5. Removing correspondences also disables correspondence cross-attention. Random retrieval preserves memory creation but substitutes random historical patches for similarity-based neighbors. FramePack compares a history-compression strategy; it is not a single-module removal ablation.

Dataset	Configuration	LPIPS ↓	DreamSim ↓	CLIP-Sim ↑	Quality ↑
VACE-Bench	Without correspondences	0.379	0.160	0.910	0.829
VACE-Bench	Random retrieval	0.396	0.144	0.916	0.807
VACE-Bench	FramePack	0.304	0.095	0.938	0.775
VACE-Bench	Full method	0.240	0.066	0.955	0.828
DL3DV	Without correspondences	0.437	0.124	0.921	0.822
DL3DV	Random retrieval	0.451	0.156	0.911	0.801
DL3DV	FramePack	0.344	0.105	0.932	0.802
DL3DV	Full method	0.305	0.058	0.952	0.821

Numerical check: relative to “Without correspondences,” the full method reduces LPIPS by approximately 36.7% and 30.2%. Table 5 labels these as approximately 37% and 30%, whereas Section 4.3 says 37% and 40%. The latter conflicts with 0.437→0.305, so this note follows the raw table values. Table 4's caption refers to Self-Forcing with a window of 21, yet its full-method row matches the five-step Rolling Forcing numbers. Because the text does not fully specify training configurations for every alternative, the table should not be treated as a rigorously verified same-backbone, single-variable comparison.

Key Findings

• Long-range consistency and visual quality are distinct. Removing correspondences actually raises Quality by 0.001 on both datasets while substantially worsening LPIPS. Smooth, clear frames do not necessarily preserve an appearance across revisits.
• Relevant history matters more than simply making history available. Random retrieval gives VACE-Bench LPIPS of 0.396 versus 0.240 for the full method, and DL3DV LPIPS of 0.451 versus 0.305. The benefit cannot be attributed solely to adding an attention layer.
• Table 3 reports consistency improvements with Self-Forcing, LongLive, and Rolling Forcing, but not universal Quality gains. On DL3DV, for example, Self-Forcing Quality falls from 0.790 to 0.775, and LongLive falls from 0.806 to 0.786.
• The main text provides only the defaults \(k=3\) and two thresholds of 0.75. No threshold sweep, feature ablation, or memory-growth curve is available in the cache, so the stable hyperparameter range cannot be established here.

Highlights & Insights

• The method delegates locating relevant history to input space and leaves rendering to output space. This is more task-aware than selecting frames by time: video translation already supplies revisit cues, reducing the need to learn an implicit global memory from scratch.
• Per-token historical candidates connect selection directly to conditioning. Instead of first collecting a large pool and expecting attention to reject irrelevant regions, the system narrows each query's candidates before attention is applied.
• Novelty-based writing retains selected local output values without repeatedly averaging the entire past. The transferable idea is not the particular 0.75 threshold, but the structure of searchable conditions as keys and previously generated content as values.

Limitations & Future Work

• The authors acknowledge failures on fast videos with large motion because cross-frame correspondences become harder to establish. Text prompts inconsistent with the input depth also create competing semantic and structural requirements.
• The main quantitative evidence concerns depth-to-video translation and controlled revisits. Modality independence is an architectural claim; the cache does not establish performance on other modalities, natural open-ended streams, or strongly nonrigid scenes.
• Novelty compression does not guarantee fixed capacity. Continual exploration can still enlarge memory and retrieval cost. Capacity budgets, hierarchical indexes, and eviction mechanisms are possible extensions, not completed components of this method.
• An erroneous first generation may be stored for later reuse, and similar local inputs do not necessarily identify the same object. Confidence, geometric constraints, or updateable values deserve investigation, while ensuring that updates do not themselves introduce appearance drift.
• Five-step performance of 8.10 FPS and 3.72 seconds to the first frame limits strictly real-time applications. Without the supplementary material, long-duration GPU memory use, training cost, and the appendix's retrieval runtime analysis cannot be verified.

Related Work & Insights

• Compared with VACE: VACE supplies the conditioned video generation foundation, but its bidirectional sequence attention does not naturally support unbounded streaming. This work adapts it into an autoregressive student and adds distant history retrieved by input content beyond the sliding context.
• Compared with Self-Forcing, LongLive, and Rolling Forcing: these approaches primarily organize causal generation and few-step distillation; this paper changes access to historical information. They are composable technical dimensions, not mutually exclusive complete model families.
• Compared with attention sinks and FramePack: the former favors initial content, while the latter trades older spatial detail for history coverage. vid2vid-long retains selected local output values and retrieves them according to current input needs, prioritizing relevance over temporal position.
• Compared with explicit 3D memory: inference requires neither camera poses nor a reconstructed 3D world, but the method also lacks the spatial consistency guarantees of a geometric map. DL3DV evaluation uses 3D Gaussian Splatting to construct revisits; this must not be confused with a reconstruction dependency of the model itself.

Rating

• Novelty: 4/5. Turning video conditions into an index for historical generated content is a clear task-specific contribution; nearest-neighbor retrieval and cross-attention are not themselves new operators.
• Experimental Thoroughness: 4/5. Two revisit benchmarks, three distillation schemes, and multiple memory comparisons support the central claim, but supplementary results are absent from the cache and open-ended streaming and cost evidence remain limited.
• Writing Quality: 3/5. The method and equations are understandable, but ablation percentages, some configuration descriptions, and claims of no speed penalty require careful checking.
• Value: 4/5. The work offers a composable local long-term memory mechanism for conditioned video consistency, with deployment still constrained by first-frame latency, matching reliability, and memory growth.