MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

Conference: ECCV2026
Authors: Haoyu Chen, Kaichen Zhou, Hang Hua, Kaile Zhang, Jingwen Qian, Wufei Ma, Haonan Chen, Chunjiang Liu, Yizhou Zhao, Xiaoyuan Wang, Weiyue Li, Alan Yuille, Paul Pu Liang, Yilun Du
Paper: ECCV Paper
Code: https://github.com/MemoBench-Team
Area: Video Generation
Keywords: world models, object permanence, dynamic memory, camera control, video evaluation

TL;DR

MemoBench tests whether video generators remember and update objects outside the field of view using 360 visible-disappear-reappear clips; evaluation of 10 models yields a best Object Reappearance Score (ORS) of only 0.582 and shows that avoiding camera motion can inflate consistency scores.

Background & Motivation

A video generator acting as a world simulator must do more than keep adjacent frames visually smooth. When the camera turns away, pedestrians keep walking, powder keeps pouring, and materials continue changing state. A correct return view should depict that evolving world, rather than regenerate a similar object or simply restore its previous appearance.

Existing evaluations cover visual quality, physical plausibility, multiview consistency, and embodied task success, but these do not establish object permanence in dynamic environments. A continuously visible target can be copied from recent observations; even an out-of-view test on a static scene does not require updating the hidden state. MemoBench therefore combines camera movement with scene dynamics and supplies ground-truth videos, making a correct return and an updated state observable evaluation targets.

This setup also reveals an evaluation loophole: if a model never turns the camera, the target never leaves, and optical flow, appearance similarity, and even semantic consistency can look strong. Core Idea: jointly evaluate camera execution, object recovery, and dynamic state correctness under a disappear-and-reappear protocol, instead of treating visual stability as evidence of memory.

Method

Overall Architecture

MemoBench is a diagnostic benchmark, not a new generator architecture. A model receives the reference first frame, a text prompt, and a camera-control signal when supported; its generated video is then evaluated alongside the temporally annotated ground-truth video.

The evaluation comprises dynamic data construction, three-phase alignment, automated metric diagnosis, and semantic question validation. The first two establish when disappearance and reappearance should occur, while the latter branches inspect observable image signals and higher-level state semantics before their scores are interpreted jointly.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Data["Dynamic Data<br/>Construction"] --> Align["Three-Phase<br/>Alignment"]
    Align --> Auto["Automated Metric<br/>Diagnosis"]
    Align --> VQA["Semantic Question<br/>Validation"]
    Auto --> Result["Joint interpretation of<br/>camera execution and memory"]
    VQA --> Result

Key Designs

1. Dynamic Data Construction: cover spatial and material changes separately

The synthetic branch builds scenes in Unreal Engine 5, animates targets along predefined paths, and attaches the camera to a first-person observer following a scripted trajectory. The observer initially faces the target, turns their head or makes a U-turn to remove it from view, and subsequently brings it back. Partial observability comes from viewpoint changes rather than a black rectangle covering the target, so the test is not limited to occlusion within a fixed composition.

This branch contains 196 clips across 5 environment categories and 14 scene subdomains, rendered at 1920×1080 and 60 FPS. RGB, metric depth, camera intrinsics, and camera-to-world poses are exported per frame. Geometry is therefore tightly paired with video, but a game engine cannot accurately simulate every material property; the authors add a real-world branch rather than equate synthetic diversity with physical coverage.

The real-world branch records 164 clips in controlled indoor settings, covering 30 physical-state-change processes in 7 categories. These supplement material dynamics involving viscosity, elasticity, and thermal conductivity. A fixed-position camera pans away and back; intrinsics come from manufacturer calibration, while MapAnything estimates extrinsics from RGB before trajectory smoothing. Together the branches provide 360 high-definition reference clips, but estimated real-world poses should not be treated as equally exact as synthetic poses.

2. Three-Phase Alignment: use reference event times to define evaluation regions

Each reference clip is annotated with \(d_{\mathrm{start}}\) and \(r_{\mathrm{start}}\), marking complete disappearance and full reappearance. These indices are linearly mapped to the generated-video length to define Visible (V), Disappeared (D), and Reappear (R). The paper uses intervals \([0,d_{\mathrm{start}})\), \([d_{\mathrm{start}},r_{\mathrm{start}})\), and \([r_{\mathrm{start}},N-1]\), where \(N\) is the frame count used for phase partitioning.

Generated and reference videos may have different frame counts and frame rates. The reference is uniformly downsampled through linear interpolation of frame indices before per-frame comparison. Motion and geometry metrics exclude D by default, whereas pixel fidelity is reported for V, D, R, and the whole video; the main table uses whole-video averages. Consequently, its PSNR is not a reappearance-only PSNR.

These phases follow the reference timeline and do not guarantee that a generated clip actually executes disappearance. If the camera remains static, the scheduled R interval is still scored. Object detection must therefore be interpreted with camera controllability, rather than serving as a standalone declaration of successful memory.

3. Automated Metric Diagnosis: recovering an object is not recovering its correct state

General video quality comprises visual quality, motion smoothness, object identity consistency, and 3D geometry consistency. Visual quality combines the LAION aesthetic predictor with CLIP-IQA+. Equation (1) applies clipped min-max normalization with fixed ranges, using \([1,10]\) for aesthetics and \([0,1]\) for CLIP-IQA+, before averaging the two normalized scores:

\[ \mathcal{N}(m;a,b)=100\,\operatorname{clip}_{[0,1]}\left(\frac{m-a}{b-a}\right). \]

Motion smoothness uses RAFT-Large optical flow between consecutive sampled frames in V and R, bilinear warping, and mean L1 photometric error, followed by a negative exponential with temperature \(\tau=0.15\). Identity consistency compares DINOv2 ViT-B/14 patch tokens with the generated first frame and selects the most similar 40% of patches. Geometry consistency compares consecutive Depth Anything V2 depth maps by cosine similarity after min-max normalization, flattening, and L2 normalization.

Identity and geometry aggregation both consider average and worst-case behavior, with \(\alpha=0.7\) stated in the text. However, operators in cached Equations (4) and (5) are damaged by extraction, so this note does not reconstruct unverified weighted formulas. These metrics measure stability, not the correctness of hidden-state evolution by themselves.

ORS more directly checks whether the target can be found during R. SAM-3 is queried with the target description, retaining masks covering 0.05%–50% of image area, with a 0.05%–70% fallback range to reject large spurious masks and noise. A frame counts as detected if any valid mask exists, using the highest valid confidence. Detection replaces cross-phase mask IoU because different viewpoints make direct spatial overlap unreliable.

Equation (6) multiplies the fraction of detected frames by their mean detection confidence:

\[ S_{\mathrm{ors}}=\frac{n_d}{n_R}\cdot\frac{1}{n_d}\sum_{i=1}^{n_d}p_i. \]

Here \(n_R\) is the R-phase frame count, \(n_d\) counts frames with valid detections, and \(p_i\) is the corresponding highest confidence. Table 2 retains ORS as a 0–1 decimal, not a 0–100 score or an unweighted detection success rate. The displayed equation does not specify the \(n_d=0\) implementation branch, which requires checking the code for reproduction. Even a high ORS only establishes recognizable target presence, not instance identity or the correct physical state.

Pixel fidelity compares generated and reference frames using PSNR, SSIM, and VGG-based LPIPS; PSNR is measured in dB and lower LPIPS is better. Camera controllability estimates generated camera poses with MapAnything, aligns them by the first frame, and evaluates only rotational ATE RMSE in degrees, with scaling based on the reference end-to-end net rotation and a \(10^{\circ}\) floor. Translation is excluded because monocular scale is ambiguous. Cached Equation (7) is also damaged, so this note retains the verifiable definition without guessing its complete formula. Prompt fidelity additionally uses ImageReward followed by sigmoid and normalization.

4. Semantic Question Validation: establish error sensitivity before scoring

Gemini-3.1-Pro drives the question-based evaluation. The prompt and first frame yield an initial bank of 24 Yes/No questions, with 6 for each of 4 dimensions. Yes means pass for positive questions and fail for negative questions; mixed polarity mitigates acquiescence bias. Questions must address observable changes in that particular video rather than merely increase the question count.

The initial bank passes through ground-truth filtering, failure filtering, and human cross-validation. The first stage removes questions the evaluator answers incorrectly on ground-truth video. The second uses known failure clips from the same scene to remove questions that do not penalize those errors. Ph.D.-level researchers and experienced AI engineers then check ambiguity, polarity, and answerability. The final bank scores generated clips by the pass rate in each dimension, so 24 is the initial count, not a guaranteed post-filtering count for every clip.

The four dimensions are instruction following, object and background consistency, continuity of memory, and physics adherence. Continuity of memory most directly targets identity, trajectory, and state across disappearance; physics adherence examines locomotion, gravity, lighting, and shadows. Semantic and automated scores are complementary: high ORS can mean recognizable but incorrect state, while high semantic consistency can still come from never moving away from the target.

Loss & Training

The paper introduces no training loss and does not train a MemoBench generator. It evaluates 5 camera-conditioned I2V models, 2 models grouped by the paper as 3D-based, and 3 I2V models without explicit camera conditioning. This note follows those groups without treating the labels as independent verification of every model's internal representation.

The main text delegates per-model generation configurations to supplementary material, but the available cache contains only the main paper and references. It therefore does not establish each model's output resolution, duration, sampling steps, random seeds, or compute budget. The synthetic reference videos' 60 FPS must not be presented as the output frame rate of every model.

Key Experimental Results

Main Results

The following representative rows come from Table 2. Camera control and visual quality use 0–100 scores; ORS uses 0–1, PSNR is in dB, and SSIM and LPIPS are dimensionless. Pixel metrics average the whole video rather than R alone.

Model	ORS ↑	Camera Control ↑	PSNR ↑	SSIM ↑	LPIPS ↓	Visual Quality ↑
LingBot-World	0.381	37.4	14.41	0.490	0.482	47.4
Wan2.2	0.328	29.8	13.76	0.469	0.529	40.0
HunyuanWorldPlay	0.582	69.9	14.35	0.471	0.505	43.5
Matrix-Game 2.0	0.157	17.3	13.49	0.376	0.550	61.2
Stable Virtual Camera	0.294	65.2	15.36	0.523	0.455	43.3
LTX-Video	0.330	17.1	13.42	0.455	0.518	44.9

HunyuanWorldPlay leads both ORS and camera control, yet its ORS remains below 0.6. Stable Virtual Camera leads pixel fidelity without leading visual quality or semantic memory; these metrics are not interchangeable.

Ablation Study

This benchmark paper provides no module-removal ablation. The following cross-metric diagnostic analysis comes from Table 3 and should not be interpreted as causal attribution to training components. All four columns are question pass scores on a 0–100 scale.

Model	Instruction Following ↑	Object & Background ↑	Continuity of Memory ↑	Physics Adherence ↑
LingBot-World	64.2	44.4	42.1	53.6
Wan2.2	50.6	30.2	36.8	38.9
HunyuanWorldPlay	61.6	66.4	55.6	63.6
Matrix-Game 2.0	37.5	12.7	36.5	21.8
Stable Virtual Camera	49.7	23.8	29.6	33.3
LTX-Video	41.0	76.6	57.0	63.5

Figure 4 separately validates evaluator reliability: 8 questions from each of 12 scenes produce 96 questions, distributed across 4 interleaved survey versions with 30 responses. Human majority answers agree with VLM answers on ground-truth videos at 92.9%, with Cohen's \(\kappa=0.85\). This is not an estimate of evaluator accuracy over all generated failure videos.

Key Findings

• LTX-Video scores 57.0 on continuity of memory versus HunyuanWorldPlay's 55.6, but camera control is only 17.1 versus 69.9. Together with the nearly static outputs in Figure 6, this indicates avoidance of difficult camera motion rather than stronger dynamic memory.
• Matrix-Game 2.0 has the highest visual quality at 61.2, yet object and background consistency is only 12.7 and physics adherence only 21.8. Sharp, attractive frames do not establish semantic or geometric correctness.
• The conclusion claims that no model exceeds ORS 0.4, contradicting the 0.582 result in Table 2 and the below-0.6 statement in Section 4.1. This note follows the detailed result table rather than repeating the conclusion's threshold.

Highlights & Insights

• Object persistence and ongoing state evolution are tested together. This makes the hidden state required of a world model concrete, rather than asking only whether a video looks continuous.
• Validating questions against failure clips adds diagnostic value. A question answered correctly on ground truth may still overlook generated errors, and failure filtering removes such permissive questions.
• Camera execution is a prerequisite for interpreting memory, not just another score to weight arbitrarily. A central finding is that high stability can result from avoiding the task itself.

Limitations & Future Work

• The paper identifies camera inactivity as a source of inflated automated and VQA scores, but the protocol does not fully eliminate it. A possible extension is to verify actual disappearance and reappearance first, then report conditional memory scores on qualifying clips; this is a proposal from this note.
• ORS depends on SAM-3 recognition, confidence, and area filtering, without directly validating instance identity or state evolution. Missed detections reduce scores, while a category-correct but instance-wrong object may still earn credit.
• Real videos are restricted to controlled indoor settings and mostly fixed-position camera rotation. Richer synthetic spatial motion does not fully replace real material behavior and open-world complexity.
• Human agreement is measured only on reference videos using 96 questions and 30 responses. It does not establish equal evaluator reliability for every model-specific artifact.
• Reference-timeline alignment can confound content errors with event-speed differences, and real camera poses contain estimation error. Future evaluation should separate temporal misalignment, camera execution failure, and genuine state-memory failure.

Related Work & Insights

• vs WorldScore: It emphasizes trajectory-constrained traversal and scene consistency. MemoBench additionally requires an out-of-view target to return in an updated state, rather than merely reveal new regions.
• vs World-in-World: It measures task success in closed-loop embodied settings, whereas this paper targets fine-grained visual memory of individual objects. Task success and correct object state are complementary levels of observation.
• vs VBench / WorldModelBench: Quality decomposition and physical plausibility remain necessary, but disappearance and camera-execution checks are needed before inferring persistent world-state capability from ordinary coherent videos.
• Implications for model design: Persistent state representations and memory-aware training objectives merit investigation, but this paper does not ablate such training methods and does not establish the effectiveness of a particular memory module.

Rating

• Novelty: 4/5. Disappearance and reappearance in dynamic scenes provide a clear, targeted diagnostic objective.
• Experimental Thoroughness: 4/5. The study covers 10 models and complementary metrics, but lacks comprehensive human validation on generated failures and strict motion gating.
• Writing Quality: 3/5. The protocol is clear, but the conclusion's ORS threshold conflicts with the main table, and cached equations suffer extraction damage.
• Value: 4/5. Especially useful for avoiding the mistaken identification of static stability as world memory.