HorizonRelight: Relighting Long-horizon Videos Consistently via Diffusion Transformers¶

Conference: ECCV 2026
Paper: ECCV 2026 official page · Project page
Area: Image Generation
Keywords: video relighting, long-horizon video, diffusion transformer, chunk-consistency, target-domain self-conditioning

TL;DR¶

The paper recasts long-horizon video relighting from "independent short-clip generation per chunk" into "chunk-by-chunk relay of the relit target-domain latent state": training with arbitrary temporal masks teaches the model to continue from partial target-domain context, inference hands the tail latent frames of the previous chunk to the next chunk as anchors, and an externally generated relit starter frame warm-starts the first chunk, which together suppress chunk-boundary seams and cross-chunk appearance drift.

Background & Motivation¶

Relighting must modify scene illumination while preserving content and material appearance, a core capability for content creation, AR/VR, and visual effects. The recent mainstream approach couples inverse decomposition with forward re-synthesis to exploit video diffusion priors from a single input video: DiffusionRenderer places both inverse decomposition and forward re-synthesis inside one diffusion pipeline, while UniRelight performs end-to-end relighting in a single model; both already deliver strong results on short clips on the order of a few seconds. Real takes, however, typically last a few to a few tens of seconds (average shot length across eras is roughly 5–25 s), which already exceeds the fixed clip length that current diffusion backbones can process reliably at high resolution — modern video diffusion backbones rely on spatiotemporal self-attention, whose memory grows rapidly with sequence length and resolution; training is therefore confined to fixed-length clips that fit in GPU memory, and long-video inference degenerates into a sliding-window procedure over time. This is a compute constraint rather than a modeling preference: public Cosmos-based relighting models are explicitly framed around 57-frame clips.

Under this fixed-window regime each chunk is processed largely independently, at best with simple overlap or blending. Every window must re-infer latent causes such as intrinsics and appearance cues, which exposes a train–test mismatch between short-clip training and long sliding-window deployment: as soon as the window shifts, relit results under the same target illumination differ visibly, temporal discontinuities appear near chunk boundaries, and long-horizon relighting becomes unreliable.

This paper's angle is to change how the problem is formulated: long-horizon relighting should not be seen as K unrelated short-clip generations but as temporally conditioned latent domain translation — a chain of state that continues in the relit target domain rather than repeatedly rebuilding itself. If repeated re-inference of the target-domain state is the source of the discontinuity, then that state should be relayed instead. Core idea: use masked target-domain self-conditioning to turn "continue from partial target-domain context" into a task the model has actually been trained on, relay the tail relit latent frames of each chunk to the next one at inference, and warm-start the first chunk with a starter frame produced by an external generative model.

Method¶

Overall Architecture¶

The input is a long video \(I_{1:T}\) together with a per-frame aligned target lighting condition \(\ell_{1:T}\) (represented as equirectangular environment maps, i.e. HDRIs), and the output is a relit video of the same length. The method follows the two-stage structure of DiffusionRenderer: an inverse stage first predicts a set of intrinsics \(\hat{X}_k\) (six domains: basecolor, normal, metallic, roughness, depth, specular), and a forward re-synthesis stage then synthesizes relit RGB conditioned on those intrinsics and the target lighting \(\ell_k\) of chunk \(k\). Both stages use a Diffusion-Transformer-based latent video diffusion model denoised in VAE latent space under the EDM parameterization. The actual change is that both stages are wired to the same cross-chunk propagation mechanism — the inverse stage propagates in the intrinsic latent domains, the forward stage propagates in the relit RGB latent domain — so chunked execution is no longer K independent generations but one conditioned latent-domain translation chain.

The chain runs as follows: in training, masked target-domain self-conditioning feeds the model samples that expose only part of the target-domain evidence, teaching it to complete the remaining frames; at inference, warm-start prompting first obtains a relit starter frame from an external controllable generative model as the target-domain appearance initialization of the first chunk; then cross-chunk context propagation relays the tail latent frames predicted for each chunk to the next chunk until the whole video is covered.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input video + target HDRI"] --> B["Masked target-domain self-conditioning<br/>random masks teach continuation"]
    B --> C["Warm-start prompting<br/>external model gives the starter anchor"]
    C --> D["Cross-chunk context propagation<br/>tail latents relay to the next chunk"]
    D --> E["Long-horizon relit video"]

Key Designs¶

1. Masked target-domain self-conditioning: turning "continue from partial target-domain context" into a trained task

Chained inference requires the model to complete the remaining frames of the current chunk while seeing only a few frames of target-domain evidence (the tail of the previous chunk), yet if training always supplies the full target-domain latent as input, the model has never seen such partial conditions and its inference-time behavior falls outside the training distribution. The authors therefore sample an arbitrary temporal mask over the target-domain latent during training and broadcast it to the latent shape \(M \in \{0,1\}^{F'\times h\times w\times C}\), multiplying it element-wise with the clean target-domain latent to obtain \(\bar{z}^Y_k = M \odot z^Y_k\): masked entries are zeroed out and only the retained frames enter the conditioning as observed anchors. The conditioning term thus grows from "source-domain latent" to "source-domain latent + masked target-domain latent", while the model is still trained to reconstruct the full target-domain latent:

\[\mathcal{L}_{\mathrm{VDM}} = \mathbb{E}_{z^{Y}_k,\epsilon}\Big[\,\big\|\epsilon - f_{\theta}(z_t \mid \langle z^{I}_k,\; \bar{z}^{Y}_k\rangle)\big\|^2\,\Big]\]

(⚠️ the equation is lossy in the extracted full text; it is restored here in the standard form of the EDM conditional denoising objective, with the symbols matching the paper.) The benefit is that the inference-time situation — providing only a few tail frames as anchors — becomes a subcase of the training distribution ("the mask happens to fall on the last frames") rather than a new mode of inference. This is exactly the basis for the authors' response to reviewers that the inference mask is a trained subcase introducing no train–test mismatch: the model learns not to "re-infer the target domain" but to "complete the remaining frames given existing target-domain evidence".

2. Cross-chunk context propagation and chained inference: adjacent chunks sharing one target-domain state

This is the backbone of the method. If each chunk is generated independently under chunked inference, every window shift forces the model to re-establish the target-domain state (intrinsics, appearance, lighting response), producing seams and drift between chunks. HorizonRelight instead turns the previous chunk's prediction into the next chunk's input: when inferring chunk \(k\), it takes the last \(b\) frames of the previously predicted latent along the temporal axis as the propagated context \(s_{k-1}=\mathrm{Tail}(\hat{z}^Y_{k-1})\) and uses them as the target-domain anchors of this chunk, letting the model predict the remaining frames from this partial context; after generation the context is updated as \(s_k\) for the next chunk. The same propagation applies to both stages: the inverse stage propagates in the intrinsic latent domains (its first chunk has no predecessor and is cold-started with DiffusionRenderer), while the forward stage propagates in the relit RGB latent domain and additionally accepts the warm-start anchor. Adjacent chunks may also keep overlapping frames to further smooth the boundary transition. The source of the consistency gain is unambiguous — cross-chunk variation is compressed to what true scene motion explains: the paper initializes the first inverse chunk with DiffusionRenderer, so improvements at later boundaries can only be attributed to chained propagation rather than to a better starting point.

One point deserves clarification because it is easy to misread: HorizonRelight is not autoregressive video generation. Autoregressive schemes treat previous outputs as the content of subsequent generation, so geometry, motion, and texture roll out and drift as the chain grows; here every chunk is still re-controlled by the source video and G-buffers, and the previous relit output only serves as a boundary color cue — it stabilizes appearance, not geometry. Consequently no self-generated history needs to be rolled out and no scheduled sampling is required for it.

3. Warm-start prompting: initializing the first chunk with an externally generated relit anchor

The chain has one inherent gap: the first chunk has no target-domain state to propagate. With \(s_0=\emptyset\) the generation is cold-started and can only rely on the relighting prior learned from training data, which weakens strong lighting cues such as specular detail. Warm-start prompting borrows the prompt intuition of GPT-style models — a prefix defines the task and steers generation without changing model parameters: an external controllable generative model (Nano Banana Pro in the paper, ChatGPT 4o in the ablation) takes the first frame of the input video plus the target HDRI and produces a relit image as the anchor, with a prompt that asks for lighting transfer only while preserving content, resolution, and aspect ratio. This anchor defines the target-domain appearance the first chunk should continue from in the forward stage, after which the whole chain proceeds under the same continuation process; the inverse stage still cold-starts and only supplies the intrinsics and G-buffers needed by the forward stage. The anchor is not restricted to a single image — an aligned short clip works too, and the empty set (cold start) is just another value of the same interface.

One practical case must be handled: images returned by external editors often do not align exactly with the input frame — the framing may shift slightly, or the image comes back stretched to another aspect ratio — in which case the prompt RGB of that frame and the predicted G-buffer intrinsics contradict each other, the model receives incompatible cues, and the result is blur or content mixing. The authors' fix is minimal: disable the warm-start G-buffer for that frame, keeping the appearance prior while removing the conflict. This keeps warm-start prompting effective even when the starter image is imperfect and makes image-based prompting more practical than video-based prompting — the latter demands strong spatial alignment with the target sequence, whereas for the former any mismatch is confined to the starter frame and can be removed in place. Incidentally the same mechanism yields a prompt-based editing interface: as long as the anchor is aligned with the input's structural conditions, the prompted appearance propagates over time, which the paper uses for long-horizon style transfer and content editing within one framework.

A Worked Example¶

Take the long take in Fig. 5 of the paper (frame indices follow the paper's figures; ⚠️ consult the paper/supplement for the exact chunk window and overlap): the input is a long take of several hundred frames, processed as 57-frame chunks, which become about 8 latent frames after 8× temporal compression. First, the first frame plus the target HDRI are fed to Nano Banana Pro to produce a relit starter frame (the prompt in the paper asks for lighting transfer only, preserving content and aspect ratio), which defines the target-domain appearance of the first chunk. Second, the inverse stage cold-starts and estimates the intrinsics and G-buffers of the first chunk. Third, the forward stage generates the relit result for that chunk conditioned on the G-buffers, the target HDRI, and the starter latent. Fourth, the tail latent frames of this chunk's prediction are taken as anchors and relayed to the second chunk, which no longer needs an external anchor: appearance is carried by the anchor while content stays controlled by the source video and G-buffers. Fifth, the relay repeats until the whole video is covered (boundaries compared in the paper include f56→f59, f56→f66, f169→f185, and f170→f171). Changing the anchor source (generated image, generated video, or plain cold start) leaves the flow unchanged and only alters how the first chunk's target domain is initialized; dropping propagation instead sends every chunk back to the DiffusionRenderer/UniRelight behavior of re-inferring on its own, and the boundary jumps immediately.

Loss & Training¶

Both stages use the standard conditional denoising objective (the form of Eqs. 1/2 above); they differ only in the conditioning term. The inverse stage adds a domain embedding \(e_d\) next to the input RGB latent \(z^I_k\) to specify which intrinsic quantity is being predicted, giving \(\langle z^I_k, e_d, \bar{z}^{X_d}_k\rangle\); the forward stage conditions on the set of six intrinsic latents, the target lighting \(\ell_k\), and the masked relit RGB latent \(\bar{z}^{I^\ell}_k\). In terms of scale, both models are initialized from Cosmos-Predict1-7B-Video2World pretrained weights; the forward (relighting) model is trained for about one month on 16 NVIDIA H100 GPUs and the inverse model for about two weeks on the same setup. Training uses BF16 mixed precision with AdamW, and all inputs (video, intrinsics, lighting cues) are encoded into tokens and concatenated with type embeddings. Temporal masks over the target-domain latents are sampled randomly during training so the model sees arbitrary fractions of partial context; at inference only a few anchor frames are kept (in the controlled evaluation, only the first frame) and the remaining frames are predicted.

Key Experimental Results¶

Main Results¶

Qualitative evaluation uses roughly 100 YouTube Creative Commons long takes (collected with broad keywords such as "vlog", "travel long take", "film long take") against DiffusionRenderer and UniRelight. Because chunk-boundary discontinuity is the dominant failure mode of long-horizon chunked inference, the paper compares frame-difference maps to determine whether each method captures true scene dynamics or introduces chunk-specific appearance changes; ideally the difference map responds only in regions of actual motion and stays largely black in static regions. The difference maps of this method are closest to those of the input video, whereas the baselines respond broadly in non-moving regions (blur, flicker, cross-chunk drift).

Quantitatively there are three tables: cross-chunk consistency on the controlled synthetic set (main-paper Table 1), boundary consistency on the 100 in-the-wild clips, and fidelity on paired Multi-Illumination (the latter two in the supplement). The controlled setting concatenates the last 57 frames of each 121-frame test video with its own reversed copy to form two adjacent chunks, so the overlapping frames are identical in content and any cross-boundary prediction difference reflects only the inconsistency introduced by chunked inference.

Table 1: MSE under the controlled repeated-chunk setting (lower is better).

Method	Scope	Normal	BaseColor	Depth	Roughness	Metallic	Specular	Relighting
DiffusionRenderer	Boundary	0.1379	0.1613	0.1576	0.2614	0.2050	0.1205	0.1048
Ours	Boundary	0.0971	0.1173	0.1181	0.1042	0.0542	0.0935	0.0729
DiffusionRenderer	Sequence	0.1425	0.1670	0.2208	0.2664	0.2409	0.1242	0.1063
Ours	Sequence	0.1007	0.1325	0.1383	0.1199	0.1088	0.1137	0.0906

Table 2: boundary metrics of relit RGB on 100 in-the-wild YouTube clips (the window column gives each method's native window length).

Method	Window	Continuous MSE↓	Continuous SSIM↑	Duplicate-reverse MSE↓	Duplicate-reverse SSIM↑
DiffusionRenderer	57	0.0327	0.670	0.0209	0.718
UniRelight	57	0.0131	0.783	0.00912	0.837
TC-Light	57	0.00127	0.937	0.00148	0.911
Light-A-Video	32	0.00623	0.853	0.00149	0.926
RelightVid	16	0.00763	0.760	0.00336	0.861
Ours-no-anc.	57	0.0254	0.679	0.0160	0.734
Ours	57	0.00080	0.934	0.00048	0.954

Table 3: held-out target-light reconstruction on paired Multi-Illumination (fidelity; the warm-start here uses a UniRelight first frame as an appearance cue, not as ground truth).

Method	PSNR↑	SSIM↑	MAE↓
Ours-cold	12.80	0.594	0.197
Ours-warm	14.39	0.679	0.163
DiffusionRenderer	12.39	0.611	0.197
UniRelight	14.72	0.708	0.157
TC-Light	7.58	0.255	0.378
Light-A-Video	9.42	0.469	0.282
RelightVid	14.42	0.666	0.145

Ablation Study¶

The paper's ablations rest mainly on qualitative figures and supplement tables (main-paper Table 1 has DiffusionRenderer as its only comparison; the full-method comparison on in-the-wild clips is in the supplement).

Config	Metric	Note
Full model (warm start + anchor propagation)	YouTube boundary MSE 0.00080 / SSIM 0.934	complete configuration
w/o target-domain anchor (Ours-no-anc.)	MSE 0.0254 / SSIM 0.679	keeps source and G-buffer conditioning, removes only the anchor → boundary MSE up ~31.8×
Cold start (Ours-cold)	Multi-Illumination PSNR 12.80 / SSIM 0.594 / MAE 0.197	no external anchor for the first chunk, relies on the learned relighting prior → 1.59 dB below warm start
Disable the G-buffer of a misaligned starter frame	qualitative (Figs. 6, 7)	removes the conflict between prompt RGB and predicted intrinsics that causes blur and content mixing
Different anchor sources (image / video / cold start / ChatGPT 4o)	qualitative (Fig. 7)	the framework is not tied to one initialization source; image prompting is more robust than video prompting

Key Findings¶

The target-domain anchor contributes the most. Removing only the target-domain anchor on the in-the-wild clips (all other conditioning unchanged) raises boundary MSE from 0.00080 to 0.0254 and drops SSIM from 0.934 to 0.679. The supplement reports relative gains of 94.0%/94.7% boundary MSE reduction (continuous / duplicate-reverse) versus UniRelight, 97.6%/97.7% versus DiffusionRenderer, and 37.0% / 87.2% / 89.5% (continuous) and 67.6% / 67.8% / 85.7% (duplicate-reverse) versus TC-Light / Light-A-Video / RelightVid.
Under the controlled setting, both the inverse and forward outputs improve consistently. In Table 1 all six intrinsic domains and the final relighting output beat DiffusionRenderer on both boundary and sequence MSE; Metallic drops from 0.2050 to 0.0542 and Roughness from 0.2614 to 0.1042 at the boundary, indicating the propagation mechanism is especially effective against the reinference of material-type intrinsics across chunks.
Consistency has a price, and the price is small. On Multi-Illumination the warm-start variant reaches PSNR 14.39, i.e. 0.33 dB and 0.029 SSIM below UniRelight (14.72), and only 0.03 dB below RelightVid (14.42 dB) while exceeding its SSIM. In other words, buying an order-of-magnitude boundary consistency gain on real clips costs only a small amount of per-frame fidelity. Conversely, TC-Light's boundary numbers are close to this method's (its continuous SSIM of 0.937 is even marginally higher than 0.934), yet it reaches only 7.58 dB on Multi-Illumination — its high consistency comes from conservative output rather than from better translation.
The longer the chain, the larger the benefit. Frame-wise statistics in the supplement show that boundary MSE stays lowest over the 57→285 frame span and that source-structure correlation decays gradually rather than jumping, indicating drift is a gradual quality decay rather than a break at chunk boundaries.
Anchor quality sets the ceiling but not the feasibility. When the anchor is imperfect (stretched, slightly reframed) the framework still works, provided the conflicting G-buffer of that frame is removed; cold start is feasible but weakens strong lighting cues such as specular highlights.

Highlights & Insights¶

Rewriting consistency from "generation quality" into "state relay". Much long-video work treats chunk-boundary discontinuity as a problem to be solved with stronger generation, whereas this paper traces it to the target-domain state being re-inferred per chunk and turns "continuation" into an explicit training task. The view transfers directly to any chunked generation task: long-video super-resolution, long-video editing, world-model rollout.
Masked self-conditioning has an excellent cost/benefit ratio. The change is a single random temporal mask on the target-domain latent during training, yet it makes "provide only a few tail frames" a subcase of the training distribution, removing the mismatch between short-clip training and sliding-window deployment. It is the lightest of the available options compared with enlarging the window, changing the attention structure, or scaling context across GPUs.
The "not autoregressive" argument matters. Previous outputs act only as boundary color cues while content remains controlled by the source video and G-buffers, so geometry and motion errors do not snowball along the chain the way rolling-forcing-style autoregressive schemes do; the paper's comparison table separating content source, consistency cue, and autoregressiveness is a clean way to make that argument.
Warm start uses an external model as a prefix. Without changing parameters or fine-tuning, a generated anchor frame defines the target-domain distribution and incidentally provides a prompt-based editing interface (used for Disney-animation, cartoon, and science-fiction long-horizon style edits), showing the continuation mechanism is not specific to relighting.
The misaligned-anchor fix is worth reusing. When multiple conditioning sources conflict, dropping the conflicting one (that frame's G-buffer) is more stable than forcing a blend — a useful engineering instinct in any generative system with several conditioning paths.

Limitations & Future Work¶

Detail degradation at long chain lengths. The authors acknowledge that beyond roughly five chunks (about 300 frames) relighting stays stable but scene content gradually loses high-frequency details, stemming from the gap between the learned appearance prior and the long-horizon appearance space encountered in practice. They position the method as practical long-take relighting rather than unrestricted 800-frame generation, noting that in editing practice the chain resets at shot changes and periodic re-anchoring handles unusually long continuous takes.
Uneven quantitative coverage. The main table (controlled repeated chunks) compares against DiffusionRenderer only; the in-the-wild and paired Multi-Illumination comparisons live in the supplement. The duplicate-reverse construction decouples boundary differences from real motion, but it is an artificial two-chunk setting, still some distance from the dozens of chunks in a real long take.
Key hyper-parameters are not systematically reported. The tail frame count \(b\) used for propagation, the chunk overlap, and the window length have no ablation (the authors state in their response that they will report \(b=1\), the mask schedule, and window sizes), leaving the central "propagation strength" degree of freedom without guidance.
High training and inference cost. The forward model trains for about a month on 16 H100s and the inverse model for about two weeks; inference runs two stages serially on a 7B-scale model, far from real-time or on-device use.
Dependence on external models. The warm-start anchor comes from closed services such as Nano Banana Pro / ChatGPT 4o, so anchor quality directly determines the first chunk's appearance and introduces external coupling for reproducibility and deployment compliance; the paper shows the framework survives a change of anchor source but does not offer an equivalent without external models.
Possible improvements. Make \(b\) and the overlap adaptive to content (for example by motion magnitude); weight or decay anchor influence over time to slow detail degradation along long chains; or let the inverse and forward stages share one propagated state to further reduce error accumulation between them.

vs DiffusionRenderer: Both are two-stage diffusion pipelines of inverse decomposition plus forward re-synthesis, and this paper even reuses its architecture and cold-starts its first inverse chunk with it. The difference is that both stages here are wired to cross-chunk propagation (one in the intrinsic domains, one in the relit RGB domain), whereas DiffusionRenderer processes chunks independently with models framed around 57-frame clips. As a result this method is lower on boundary and sequence MSE for all six intrinsics and the relighting output in the controlled setting, and about 97% lower on boundary MSE on the in-the-wild clips.
vs UniRelight: UniRelight jointly models relit appearance and albedo in one diffusion model, avoiding the explicit G-buffer bottleneck and reducing error accumulation between stages; its short-clip fidelity is the strongest among the compared methods (Multi-Illumination PSNR 14.72). This paper leaves the backbone alone and changes how chunks are coupled, leading by a wide margin on boundary consistency for real long videos (boundary MSE 0.00080 vs 0.0131) at the cost of 0.33 dB per-frame fidelity. The trade-off is clear: UniRelight for per-frame quality, this method for long-horizon stability.
vs long-video diffusion generation (Rolling Forcing / Resampling Forcing / Diffusion Forcing / StreamingT2V / FreeNoise, etc.): These also target long-horizon consistency, but their content source is generated or self-resampled history, i.e. autoregressive rollout; here every chunk is re-controlled by the source video and G-buffers while the previous relit output is only a boundary color cue, so the method is not autoregressive and needs no training for history rollout or scheduled sampling.
vs image/short-clip relighting methods (TC-Light, Light-A-Video, RelightVid, IC-Light, DiLightNet, etc.): They target isolated objects, portraits, or short clips, and when applied frame-by-frame or in short windows their boundary consistency is far behind this method (continuous MSE higher by 37%–89%). TC-Light's low paired-reconstruction fidelity (7.58 dB) is a reminder that methods whose boundary consistency metric is inherently conservative get an advantage for free — fidelity and consistency must be read together.

Rating¶

Novelty: ⭐⭐⭐⭐ At the component level this is a combination of known mechanisms (masked self-conditioning, anchor propagation, prompt prefix), but reformulating long-horizon relighting as temporally conditioned latent domain translation and designing training and inference around it changes the way the problem is viewed.
Experimental Thoroughness: ⭐⭐⭐ Controlled synthetic, 100 in-the-wild clips, and paired reconstruction complement each other and the qualitative figures are convincing; but the main table has a single baseline, ablations on \(b\)/overlap/window are missing, and most quantitative comparisons sit in the supplement.
Writing Quality: ⭐⭐⭐⭐ The motivation chain (compute constraint → chunked deployment → train–test mismatch) is clearly told and the comparison table against autoregressive work is crisp; the equations are lossy in the extracted text and need checking against the original.
Value: ⭐⭐⭐⭐ The tail-latent relay plus masked self-conditioning mechanism does not depend on a specific backbone and transfers directly to other long-horizon chunked generation tasks, which matters more in practice than the size of the metric lead.