Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning

Conference: ECCV2026
Paper: ECCV Paper
Area: Robotics & Embodied AI
Keywords: world models, autoregressive rollouts, exposure bias, multi-view rewards, contrastive denoising

TL;DR

PersistWorld branches multiple future videos from its own increasingly imperfect histories and uses multi-view visual rewards for contrastive denoising post-training, reducing external-view LPIPS from 0.081 to 0.070 against reproduced Ctrl-World on 14-step DROID rollouts and receiving approximately 80% preference in a blind study.

Background & Motivation

Action-conditioned world models predict how a scene changes in response to a sequence of robot actions. Methods such as Ctrl-World can generate convincing short videos from real histories, but operating as a simulator requires feeding generated videos back into the next prediction. A slight deformation of a bowl becomes part of the scene the model assumes to be real; after repeated rollouts, the object may lose its identity and the robot arm may deviate from the commanded trajectory. The issue is not simply single-frame sharpness, but errors repeatedly entering the context.

Conventional teacher forcing always supplies clean ground-truth histories and supervises predictions of real futures. More training of this kind does not directly teach the model to handle its own mistakes, so the context distribution still differs between training and deployment. Regressing real futures from self-generated histories reduces this input-distribution gap, but does not directly exploit which generation choices are better from the same corrupted state. Meanwhile, reinforcement learning through an entire diffusion sampling chain faces likelihood-computation or back-propagation costs, preventing a straightforward transfer of token-level policy optimization from language models.

The paper treats the history buffer as a shared condition that can be frozen: roll out to some depth, then start independent candidates from exactly the same state and compare their fidelity to real video. This exposes training to deployment-time errors while providing meaningful within-group quality signals. Core Idea: construct comparable future branches from self-generated histories at randomized depths, then turn multi-view visual rewards into positive/negative contrastive denoising updates so the world model learns to preserve scene structure despite existing errors.

Method

Overall Architecture

PersistWorld post-trains the Ctrl-World world model, not the robot action policy. The base model uses a Stable Video Diffusion (SVD) backbone. Each prediction reads the latest 6 latent frames, their corresponding end-effector poses, and 5 future targets expressed as 7-dimensional end-effector poses. It jointly generates 5 future frames for two external cameras and one wrist camera. Predictions are encoded and written back into the history to condition the next generation.

Training follows shared-history branching, multi-view group-relative rewards, and positive/negative contrastive denoising. Real video is used for initialization and reward comparison; after branching, each candidate advances its own history using its own predictions. Deployment only generates the next segment under the supplied actions and updates the history. It neither accesses real future frames nor needs the candidate-scoring procedure.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Initial observation and actions"] --> B["Shared-history branching"]
    B --> C["Multi-view group-relative rewards"]
    G["Real future frames<br/>Training only"] -.-> C
    C --> D["Positive/negative contrastive denoising"]
    D -.->|Update parameters| E["Deployment: predict next segment<br/>and update history"]
    A --> E
    E -->|Continue supplied actions| E

Key Designs

1. Shared-history branching: compare candidates from the same corrupted state

Training starts with one real observation, whose encoded latent is replicated to fill the history buffer. The model autoregressively generates a prefix, writing its outputs back into the buffer after each segment; the prefix length is sampled uniformly from 0 to 9 segments. This shared state is then frozen, and 16 independent candidate continuations are sampled. Each candidate contains a sequence of generation chunks and uses a private buffer copy. Errors can therefore accumulate within each candidate without candidates contaminating one another's histories. The randomized prefix length is an explicit setting; the continuation length is denoted F in the method, but the main text does not specify a unique default. These two lengths should not be conflated.

The essential detail is not merely sampling more videos. If candidates start from different states, reward differences may reflect easier histories rather than better predictions. A shared buffer removes this conditioning difference, while randomized branching depth prevents learning only to handle mild errors. A prefix of 0 provides nearly clean context, whereas a longer prefix contains substantial drift, exposing training to error regimes from early to late rollout stages. The recorded action sequence remains the condition: the model must improve its predictions of those actions' consequences, rather than choose easier actions to obtain higher rewards.

2. Multi-view group-relative rewards: compare local structure, perceptual appearance, and global drift

Each candidate clip is compared with the real future at the same time. Metrics are first averaged over the clip's 5 frames, then equally across the three cameras, avoiding a single terminal reward for a long trajectory that would make credit assignment difficult. LPIPS measures perceptual differences in deep features and is lower-is-better; SSIM measures local luminance, contrast, and structural similarity; PSNR measures the signal-to-noise ratio associated with pixel errors. The latter two are higher-is-better. The reward adds negative LPIPS, positive SSIM, and PSNR scaled by 1/32 to bring the components to similar numerical scales. These are full-frame visual rewards. Object and robot masks are only used in evaluation, not as additional training supervision.

Early clips are generally closer to real video than late clips, so absolute rewards should not directly determine update strength. Within each group of 16 candidates, the authors subtract the mean and divide by the standard deviation plus a stabilizer to obtain relative advantages. These are clipped to [-1, 1] and linearly mapped to [0, 1]. Values near 1 favor the positive branch, while values near 0 favor the negative branch; candidates near the group mean map to approximately 0.5 instead of uniformly becoming failures. The model consequently learns which future is better from the current state, retaining a useful direction even when all absolute rewards are low late in a rollout.

3. Positive/negative contrastive denoising: encode preferences in a forward-noised training objective

DiffusionNFT originally develops contrastive updates for velocity-prediction flow matching, whereas Ctrl-World uses EDM clean-sample prediction. This work exploits the affine relationship between the network output and clean-data prediction to transfer the positive/negative branch construction. A frozen exponential moving average (EMA) model acts as the reference. The difference between the current and reference predictions defines a direction: the positive branch moves along it, the negative branch moves against it, and group-relative rewards control the weights assigned to fitting these branches. The authors state that the policy-improvement guarantees transfer under this transformation; this is not a guarantee of physical stability for a real robot.

The objective neither differentiates directly through sampling nor uses the real future latent as the regression target for every candidate. The model first generates a clean candidate and then applies forward noise to it. The current model and frozen reference predict from the same noisy input, and the reward determines whether training reinforces or repels the generated sample. The following uses notation introduced in this note to summarize the mechanism, rather than reconstructing the damaged cached equations as exact author formulas: d denotes the change from the reference prediction, c is a mixing coefficient, and y is the generated clean candidate.

\[ d = \hat{y}_{\theta}-\hat{y}_{\mathrm{ref}},\qquad \hat{y}_{+}=\hat{y}_{\mathrm{ref}}+c d,\qquad \hat{y}_{-}=\hat{y}_{\mathrm{ref}}-c d. \]

\[ \mathcal{L}=\mathbb{E}\left[r\lVert\hat{y}_{+}-y\rVert_2^2+(1-r)\lVert\hat{y}_{-}-y\rVert_2^2\right]. \]

High-reward samples primarily pull the positive branch toward the candidate. Low-reward samples primarily pull the opposite-direction branch toward the candidate, pushing the current model away from poor generations. Reference predictions and candidates are treated as fixed during the update. There is no need to retain gradients through the entire reverse denoising chain, and rewards need not be differentiable. This accommodates different samplers and visual metrics, but candidates must still be generated: the method saves back-propagation overhead, not rollout-sampling cost.

A Worked Example

Consider a three-camera sequence in which a robot moves a bowl. Suppose the sampled prefix contains 4 segments, and use candidates extending only 1 segment for illustration. After the initial image fills the 6-frame history, the model generates 4 groups of 5 frames, advancing 20 frames. Even if the bowl's rim is now slightly distorted, the history is not replaced with real observations. All 16 candidates start from that same buffer and generate frames 21 through 25 under the next actions.

If one candidate preserves the bowl's shape and arm position while another turns the bowl into blurred texture, the combined metrics compare their deviations from the real future. After group normalization, the former favors the positive-branch update and the latter favors the negative-branch update; the updated parameters belong to the world model. This example illustrates data flow, not a reported experimental instance, and does not claim that a 1-segment continuation is the paper's default hyperparameter.

Loss & Training

The authors train only rank=64, alpha=64 LoRA adapters in the UNet and the action encoder; other UNet parameters and the VAE remain frozen. Training uses Muon with learning rate 1e-4, batch size 64, and 6,000 steps. Each group generates 16 candidates, but an update uses the 5 highest-reward and 5 lowest-reward samples, totaling 10. Selecting both extremes emphasizes distinguishable quality differences within a limited update budget; it does not mean the middle 6 candidates were never generated.

The reported budget is 8 NVIDIA H200 GPUs for 2 days, with autoregressive rollout generation dominating the cost. Three computations should be distinguished: prefixes construct training states, candidates produce reward differences, and denoising predictions after forward noising provide parameter updates. The first two do not back-propagate through the full sampling chain; gradients enter the trainable parameters through the third. The main text does not fully specify reproduction details such as continuation length and mixing coefficient. The supplied cache also omits the referenced Appendices A through E, so their ablation numbers and proof steps are not invented here.

Key Experimental Results

Main Results

The table below selects results from the paper's Table 1. Evaluation uses DROID, the Franka Emika Panda robot, and Ctrl-World's held-out validation split, rolling out recorded actions for 14 segments of 5 frames and averaging metrics over the full prediction sequence. External and wrist views are reported separately. Higher SSIM and PSNR and lower LPIPS are better; PSNR is in dB. Rows labeled cited inherit results from the Ctrl-World paper, whereas reproduced rows are this paper's experiments. These sources should not be mixed when calculating improvements.

View	Model and source	SSIM	PSNR	LPIPS
External	WPE, cited	0.77	20.33	0.131
External	IRASim, cited	0.77	21.36	0.117
External	Ctrl-World, cited	0.83	23.56	0.091
External	Ctrl-World, reproduced	0.84	23.02	0.081
External	PersistWorld	0.86	24.42	0.070
Wrist	Ctrl-World, reproduced	0.62	17.80	0.310
Wrist	PersistWorld	0.67	19.39	0.277

Against the reproduced baseline, external and wrist PSNR improve by 1.40 dB and 1.59 dB, respectively. The authors report a 14.0% external LPIPS reduction and a 9.1% wrist SSIM increase, whereas the rounded table entries yield approximately 13.6% and 8.1%; these should not be presented as exactly consistent. The paper also states 70 frames, 5 Hz, and approximately 11 seconds. The first two imply approximately 14 seconds; this discrepancy is preserved rather than silently changing the source data.

Ablation Study

To distinguish exposure to corrupted histories from the contrastive reward objective, the authors construct a Self-Forcing-style baseline that also consumes its own rollout history but regresses real latents. This is not a full Self-Forcing reproduction with a large bidirectional teacher. The following results come from Section 4 and use 10 interactions, unlike the 14-step setting above; values should not be compared directly across tables.

View	Post-training method	SSIM	PSNR	LPIPS
External	Self-generated history + real-latent regression	0.868	25.35	0.119
External	PersistWorld contrastive rewards	0.872	25.52	0.116
Wrist	Self-generated history + real-latent regression	0.710	20.17	0.338
Wrist	PersistWorld contrastive rewards	0.721	20.40	0.338

The direction of these differences supports the contrastive objective, but wrist LPIPS is tied and the other gains are modest. This comparison cannot establish that reinforcement learning accounts for every improvement. More detailed reward-combination and hyperparameter ablations are referred to the appendix, which is absent from the supplied cache.

Key Findings

• The 14-step masked evaluation in Table 2 shows external object-region LPIPS changing from 0.025 to 0.021 and wrist object-region SSIM from 0.73 to 0.76. Robot-region PSNR changes from 17.62 to 19.25 externally and from 25.50 to 27.24 at the wrist. Improvements are not confined to static backgrounds, although masked image quality is still not a direct measurement of physical error.
• The blind study supplies real video as a reference and compares realism and temporal consistency between PersistWorld and the baseline, recording 174 wins and 43 losses, approximately 80%. The approximately 98% paired validation-sample win rate comes from a separate automated-metric analysis, not a 98% robot-task success rate.
• Cross-embodiment experiments first fine-tune SVD on Bridge and RT-1 for 100k steps and then apply the proposed post-training. LPIPS changes from 0.1061 to 0.0957 and from 0.1683 to 0.1305, respectively. These results concern transfer after adaptation, not zero-shot generalization.
• A small real-robot policy-improvement experiment fine-tunes pi0 with 50 rollouts from each world model. Across 4 spatial picking targets with 5 trials per block, average task progression is 0.350 for the base policy, 0.558 with Ctrl-World, and 0.667 with PersistWorld. This is a task-progression score, not binary success rate.

Highlights & Insights

• Treating shared history as the conditioning prompt brings group-relative optimization into a continually changing video-generation environment. The reusable principle is fixing the comparison's starting state, not merely increasing candidate count.
• Post-training incorporates the model's own error distribution into training data. Randomized prefixes optimize not only clean starts but also prediction from states that have already drifted from the real trajectory.
• Visual rewards need no additional human labels, and positive/negative contrastive updates do not require differentiable rewards. This convenience depends on time-aligned real trajectories; it does not provide unsupervised scoring for arbitrary unknown actions.

Limitations & Future Work

• The authors explicitly identify the cost of generating 16 candidates and the lack of explicit physical or geometric constraints in visual rewards. Contact, geometric consistency, or task-specific signals could be added, but the paper does not establish their effectiveness.
• This note's assessment: visual matching on recorded trajectories does not guarantee correct simulation of out-of-distribution contacts caused by arbitrary new policies. Wrist and foreground metrics are more targeted, but cannot rule out plausible appearance with incorrect dynamics.
• This note's assessment: real-robot policy improvement is small-scale, and the 217 human choices do not imply 217 independent tasks or participants. More tasks, repeated training runs, and explicit uncertainty estimates are needed to assess robustness.
• The duration discrepancy, rounded percentages, and missing appendices limit exact verification. The evidence demonstrates slower error accumulation, not indefinite non-degradation or global physical consistency.

Related Work & Insights

• vs Ctrl-World: The action conditioning, multi-view video setup, and SVD backbone are retained; post-training changes the context distribution and optimization signal. The contribution is not a redesigned robot policy.
• vs DiffusionNFT: The method inherits forward-process contrastive reinforcement and extends it to clean-sample prediction and autoregressive candidates with shared histories. Transferred theoretical guarantees concern objective construction, not closed-loop stability of a real system.
• vs Self-Forcing: Both address training-deployment distribution mismatch, but this work uses reward comparisons instead of only real-latent regression. The authors' style-inspired baseline should be distinguished from the complete teacher-distillation method.
• vs WorldCompass / RLVR-World: The former also studies contrastive world-model post-training with prefix context; this paper emphasizes randomized prefixes, multi-view dynamic manipulation, and efficient visual rewards. The latter targets token-based world models, whereas this work targets video diffusion models.

Rating

• Novelty: 4/5. Shared-history grouping and parameterization adaptation are concrete and targeted, but build on existing contrastive diffusion post-training.
• Experimental Thoroughness: 4/5. Full-frame, foreground, human-preference, cross-embodiment, and small real-robot evaluations are covered; complete ablations and uncertainty assessment still require appendix support.
• Writing Quality: 4/5. The problem and training protocol are clear, but duration and rounded numerical claims need clarification.
• Value: 4/5. The work offers a reusable deployment-distribution post-training recipe for robot video simulators, with physical reliability and computational efficiency remaining barriers.