IC-World: In-Context Generation for Shared World Modeling

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/wufan-cse/IC-World
Area: Video Generation / World Models
Keywords: shared world modeling, in-context generation, geometry consistency, motion consistency, GRPO

TL;DR

IC-World combines multi-view images of the same world into a grid for joint video generation, then applies GRPO fine-tuning with 3D geometry and motion consistency rewards, reducing the Wan2.1 baseline's M-FID from 185.2354 to 157.6099 on the two-view dynamic-scene benchmark.

Background & Motivation

A single-view image-to-video model can animate an image without answering what another camera should observe at the same moment. When two photographs of the same room are processed separately, both videos may look plausible while disagreeing about furniture positions, occlusions, or human actions. Shared world modeling requires the outputs to represent different observations of one environment, rather than independently generated videos with similar themes. This matters for multiplayer games and robots sharing a workspace: observers may differ, but the world's events must not unfold independently for each observer. The paper investigates this visual consistency, not a complete interactive game engine or robot control policy.

Existing multi-view generation for autonomous driving often assumes a predefined camera system, while object-centric 4D generation organizes views around a particular object. This paper targets general scenes without predefined constraints on each camera trajectory, making those task-specific approaches difficult to apply directly. Generating views one at a time also introduces ordering dependencies: later videos must agree with earlier histories, but independent sampling provides no mechanism that guarantees this agreement. The problem is therefore not merely the quality of an individual video, but how multiple outputs obtain shared context during generation. However, unconstrained trajectories do not imply demonstrated precise camera-trajectory control; the main inputs remain image sets and world descriptions.

Large video diffusion transformers can already interpret visual layouts containing multiple regions, allowing the authors to recast information exchange as input organization. Placing views in one image and explicitly describing them as a synchronized world lets one generation process handle the regions together. This requires no additional cross-view network module, but pretraining alone cannot guarantee correct geometry and actions, motivating targeted feedback on generated outputs. The authors turn 3D reconstruction and point trajectories into computable rewards instead of relying entirely on scarce paired target-video supervision. Core Idea: first activate joint generation through grid inputs and synchronized-world instructions, then use cross-view 3D geometry and motion rewards to move from semantic resemblance toward agreement in spatial structure and events.

Method

Overall Architecture

The inputs are \(N\) images of the same world at the same time from different viewpoints, together with shared text describing the scene and events. The outputs are \(N\) videos corresponding to those initial images; the main experiments use \(N=2\). The model first performs Grid In-Context Generation, downscaling and arranging images on one canvas before generating a video containing the different view regions. It then crops the video by pixel region into sub-videos that can be viewed separately and evaluated for cross-view consistency. During training, the Geometry Consistency Reward and Motion Consistency Reward evaluate these sub-videos in parallel, and Group Relative Policy Optimization updates the generator. At inference time, the trained generator performs coupling, generation, and decoupling; the main text does not require running the reward models online.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Synchronized multi-view images<br/>Shared world description"] --> Grid["Grid In-Context Generation"]
    Grid --> Split["Crop corresponding sub-videos"]
    Split -->|Training evaluation| Geometry["Geometry Consistency Reward"]
    Split -->|Training evaluation| Motion["Motion Consistency Reward"]
    Geometry --> Optimize["Group Relative Policy Optimization"]
    Motion --> Optimize
    Optimize -.->|Training update| Grid
    Split -->|Inference output| Output["Shared-world video set"]

Key Designs

1. Grid In-Context Generation: introduce view dependencies within one generation process

Conventional baselines run image-to-video generation separately for each input, requiring \(N\) independent generations. IC-World instead downscales the inputs and combines them pixel-wise into fixed regions, producing one conditioning image accepted by an ordinary video model. The accompanying instruction requests a grid video and specifies that its regions show the same synchronized world from different camera positions, followed by the world description. The instruction supplies a common semantic condition rather than assigning a separate event sequence to each view. After generation, each frame is cropped according to the input layout to recover the corresponding videos; no new video decoder is trained to separate them. The views therefore share the spatiotemporal context of one video generation process, making agreement easier than with independent continuation.

Parallel generation here means joint generation inside the grid, not simply running independent models on separate GPUs. The original video model carries cross-region dependencies; the authors add neither an explicit shared 3D scene representation nor an extra cross-view module. A zero-shot version can therefore run immediately, while GRPO provides subsequent consistency-oriented post-training rather than being a prerequisite for the grid method. The cost also follows from sharing a canvas: with fixed total output dimensions, each view receives only part of the pixel budget. Adding views increases the burden of maintaining multiple scene regions simultaneously, so two-view success does not establish arbitrary scalability.

2. Geometry Consistency Reward: separate camera differences from scene-shape differences

Direct pixel comparison would penalize valid perspective changes, so the method first feeds each sub-video into Pi3 to reconstruct a dense 3D point cloud. The reconstructions occupy separate coordinate systems and must be aligned before their spatial structures can be compared. Lepard registers the point clouds using learned local features, handling partial overlap and reconstruction noise to place them in comparable coordinates. After registration, the symmetric Chamfer distance \(D_g\) averages the Euclidean distance from each point to its nearest neighbor in the other cloud, then averages the two directions. The bidirectional calculation avoids checking only whether one cloud is contained in the other while overlooking additional or missing structures on the opposite side. A smaller distance indicates more consistent reconstructed geometry; an exponential transformation converts distance into a higher-is-better reward.

\[ r_g=\exp(-D_g),\qquad r_m=\exp(-D_m). \]

The equation also includes the motion reward introduced below; the transformation is shared, but the underlying comparison differs. The geometry reward primarily constrains scene-level layout, such as whether a table occupies the same position, rather than forcing different cameras to produce identical images. Evaluation additionally filters Pi3 points at confidence thresholds 0.1, 0.5, and 0.7 to produce three Geometry scores. These subscripts specify point-confidence filtering, not distance thresholds or geometric accuracy rates. The reward still depends on reconstruction and registration reliability and should not be equated with physical correctness verified against ground-truth 3D annotations.

3. Motion Consistency Reward: compare aligned 3D point trajectories

Even with similar room layouts, a person may start pouring in one view after the corresponding action has already finished in the other. To detect temporal disagreement, the authors use a 3D point tracker to predict camera extrinsics and point trajectories from sampled frames. The method section abbreviates the model as SpatialTracker, while the evaluation section and bibliography identify SpatialTrackerV2. Rotation and translation using the estimated extrinsics place the trajectories in common coordinates, preventing viewpoint offsets from being mistaken for object-motion differences. Correspondences do not rely directly on annotated semantic identities: the method computes each trajectory's temporal-average position and matches it to the nearest average position in the other view. After matching, it computes Euclidean distances between corresponding points at sampled times and averages over trajectories and time to obtain motion distance \(D_m\).

The motion reward \(r_m\) uses the exponential transformation above, encouraging corresponding motions to agree in both space and time. Evaluation samples every 5 frames and reports Motion scores at tracking-density settings 10, 20, and 30. These subscripts indicate density settings, not video lengths, action-class counts, or accuracy percentages. Compared with the geometry reward, this signal more directly penalizes a hand appearing at incompatible positions across views. However, nearest matching by average position is not explicit semantic correspondence and can fail when motions cross or objects approach one another; this is a reader's analysis of the design limitation.

4. Group Relative Policy Optimization: learn from relative consistency among candidates

For the same image set and description, training samples \(M\) complete grid videos, each containing all views. After cropping each candidate, the algorithm computes both rewards and derives a relative advantage from the candidate group's mean and standard deviation. The experiments weight the rewards equally; the following uses clear subscripts to restate the paper's weighted reward and within-group normalization.

\[ r_i=0.5r_{g,i}+0.5r_{m,i},\qquad A_i=\frac{r_i-\operatorname{mean}(r_1,\ldots,r_M)}{\operatorname{std}(r_1,\ldots,r_M)}. \]

The model thus learns which candidates are more consistent than others under the same conditions without requiring a universal absolute reward threshold across scenes. Training follows DanceGRPO, interpreting denoising as a Markov decision process and updating parameters with a clipped objective based on the new-to-old policy probability ratio. The algorithm also subsamples denoising steps for gradient computation to control training cost; the main text does not specify the numerical subsampling ratio. Because the algorithm computes an advantage for every group sample, it should not be summarized simply as learning only from the best video. The Figure 2 caption nevertheless describes finding the best candidate for gradient calculation, which differs from the algorithm; this note follows Algorithm 1's group-advantage procedure while retaining that uncertainty. Group size also affects reward noise: the authors show smoother reward curves when increasing it from 8 to 32, but the full quantitative sensitivity analysis is assigned to an unavailable appendix.

A Worked Example

Consider the drink-preparation scene in Figure 4, where two input images show the same person in a kitchen from different directions. They are arranged in a grid and share a description of the person and scene; one generation then develops the action in both views simultaneously. If a hand is already raised in one view but remains lower in the corresponding action in the other, semantic agreement that both show drink preparation is insufficient for the motion check. After cropping, point tracking and extrinsic alignment measure the motion discrepancy, while point-cloud registration checks the geometry of the kitchen and work surface. Each of the \(M=16\) training candidates receives a reward, and relative advantages update the same generator. This example explains the paper's mechanism without assigning this particular video any unreported per-sample score.

Loss & Training

The main model is initialized from a step-distilled Wan2.1-14B version supporting 4-step inference. The authors choose LoRA with rank 64 instead of full-parameter fine-tuning, reporting more stable training and more coherent results. Training uses AdamW with learning rate \(1\times10^{-5}\), group size \(M=16\), and up to 200 training steps. The experimental platform contains 8 H20 GPUs with 95 GB memory each; the reported fine-tuning duration is 40 hours. The static-scene task samples 1K training and 1K test examples from DL3DV-10K, and the dynamic-scene task samples the same amounts from MultiCamVideo. MultiCamVideo is rendered with Unreal Engine 5 and contains 13.6K dynamic scenes with 10 camera viewpoints per scene, so its dynamic-scene results are not direct validation on real videos. Qwen2.5-VL-32B generates world descriptions covering static scene attributes and dynamic object behaviors. The core method uses GRPO without requiring SFT warm-up; Table 5 separately compares SFT, SFT+GRPO, and GRPO to examine that choice.

Key Experimental Results

Main Results

The selection below comes from Tables 1 and 2 on page 8, both using two views; static scenes have moving cameras, while dynamic scenes have static cameras. M-FID measures perceptual discrepancy between generated views and is lower-is-better, rather than conventional FID against ground-truth target videos. Geometry and Motion are exponentiated distance scores and are higher-is-better; CLIP measures alignment between the grid video and text. The VLM score uses Qwen2.5-VL-32B to judge whether videos represent the same world; the text specifies a scale from 0 to 10, but the tables report values below 1 without explaining the scaling in the available main text. The table preserves the reported VLM values rather than recasting them as accuracy.

Scene / Source	Method	M-FID ↓	CLIP ↑	VLM ↑	Geometry0.5 ↑	Motion20 ↑
Static / Table 1	CogVideoX-I2V-5B	96.1624	0.6208	0.7684	0.7150	N/A
Static / Table 1	Wan2.1-14B	103.5042	0.6204	0.7193	0.7171	N/A
Static / Table 1	IC-World	98.8324	0.6170	0.7822	0.7217	N/A
Dynamic / Table 2	CogVideoX-I2V-5B	202.9558	0.6412	0.7915	N/A	0.7963
Dynamic / Table 2	Wan2.1-14B	185.2354	0.6405	0.7949	N/A	0.8344
Dynamic / Table 2	IC-World	157.6099	0.6400	0.7958	N/A	0.8466

Relative to Wan2.1, dynamic-scene M-FID decreases by 27.6255 and Motion20 increases by 0.0122; these are differences between reported scores, not accuracy gains. IC-World does not beat CogVideoX on static-scene M-FID, and its CLIP score is not the best, so the results do not establish superiority on every quality metric. Table 3 on page 9 additionally reports a VBench weighted average of 81.15 versus 80.89 for the listed Gen-4, but individual dimensions do not all improve, and baseline numbers come from official VBench reports.

Ablation Study

First, the Wan2.1-14B-distill zero-shot ablation in Table 4 on page 9 isolates grid in-context generation without GRPO gains.

Scene / Source	Zero-Shot Setting	M-FID ↓	VLM ↑	Geometry0.5 ↑	Motion20 ↑
Static / Table 4a	Without in-context generation	103.5042	0.7193	0.7171	N/A
Static / Table 4a	With in-context generation	99.1513	0.7818	0.7199	N/A
Dynamic / Table 4b	Without in-context generation	185.2354	0.7949	N/A	0.8344
Dynamic / Table 4b	With in-context generation	160.0052	0.7981	N/A	0.8441

Next, Table 6 on page 11 ablates the reward models; every configuration uses Wan2.1-14B and in-context generation. The configuration without geometry and motion rewards still uses DanceGRPO's video-text alignment reward and is not an untrained or reward-free model.

Reward Setting	Static M-FID ↓	Static Geometry0.5 ↑	Dynamic M-FID ↓	Dynamic Motion20 ↑	Dynamic Motion30 ↑
Without geometry and motion rewards	101.2243	0.7180	165.4712	0.8364	0.8411
Geometry only	97.7391	0.7270	159.5929	0.8380	0.8412
Motion only	99.1874	0.7253	159.1588	0.8445	0.8479
Geometry and motion	98.8324	0.7217	157.6099	0.8466	0.8500

Key Findings

• Grid generation already contributes most of the dynamic-scene M-FID improvement: the zero-shot result is 160.0052 versus 157.6099 for the final model, so the full change cannot be attributed to reinforcement learning.
• Geometry-only training achieves static Geometry0.5 of 0.7270 versus 0.7217 with both rewards; the combined objective is not monotonically best on every metric.
• On the motion task, both rewards achieve Motion20 of 0.8466 versus 0.8445 for motion alone, supporting complementary effects while keeping the magnitude in perspective.
• Table 5 on page 10 reports dynamic-scene M-FID of 166.4103 for SFT with 2K examples versus 157.6099 for GRPO with 1K; this compares specific training settings rather than establishing a universal data-efficiency law.

Highlights & Insights

• The most reusable element is condition organization: placing related outputs in one generative context may exploit pretrained capabilities more directly than adding complex interaction modules. The zero-shot ablation supports this, although deterioration on some LTX-Video metrics shows that the prompting strategy is not universally beneficial.
• The two rewards separate shared-world consistency into scene structure and time-varying point motion instead of measuring agreement through text similarity alone. This decomposition is interpretable, and task-specific ablations reveal trade-offs between geometric and motion objectives.
• A useful methodological lesson is to change joint generation first and learn specialized constraints afterward, checking zero-shot gains before deciding whether expensive post-training is warranted. This is a reader's takeaway, not a transfer experiment already demonstrated on other tasks.

Limitations & Future Work

• The authors acknowledge that pixel-wise coupling reduces effective resolution per view and suggest video super-resolution as a possible remedy, but the main text provides no quantitative result after that remedy.
• The main evidence covers two views and separates static scenes with moving cameras from dynamic scenes with static cameras; it does not establish performance when complex scene and camera motion occur together.
• Training rewards and central evaluation metrics both depend on Pi3, Lepard, and point tracking, creating a risk of optimizing the same proxies; ground-truth 3D supervision or independent evaluators would provide stronger checks.
• The main text refers to Appendices 6 through 10 for additional views, user studies, reward fluctuations, and failure cases, but the supplied cache ends at the references, leaving those details unverified.
• Mutual memory and robot or gaming applications are primarily supported by qualitative examples; the main text does not establish long-horizon memory, real-time interaction, or practical policy-training gains.
• Some extracted equations contain damaged symbols; this note restates only reward definitions supported by the prose and context rather than reconstructing an exact extrinsic-matrix product with uncertain coordinate conventions.

Related Work & Insights

• Compared with Wan2.1 / CogVideoX: these baselines continue input images independently, whereas IC-World reorganizes multi-view conditioning and applies consistency-oriented post-training rather than uniformly improving single-view quality.
• Compared with IC-LoRA / IC-Edit: all exploit diffusion transformers' in-context capabilities; this paper applies them to synchronized cross-view video and adds 3D consistency feedback.
• Compared with DanceGRPO: the optimization framework is inherited, while the candidate unit becomes a complete video set and the rewards target shared-world geometry and motion.
• Compared with multi-view driving / 4D generation: the paper emphasizes general scenes and views without predefined trajectory constraints, but it excludes these task-specific approaches from direct comparison and does not establish superiority in their original settings.
• Research direction: rewards could incorporate occlusion reliability and correspondence confidence, with validation by 3D tools excluded from training; this is a proposed extension rather than a completed result.

Rating

• Novelty: 4/5. Extending grid in-context generation to shared-world video with geometry and motion feedback has clear task-specific value.
• Experimental Thoroughness: 4/5. Main comparisons and zero-shot, training-strategy, and reward ablations are provided, but dynamic scenes are synthetic and appendix evidence is absent from the cache.
• Writing Quality: 3/5. The main mechanism is clear, but the figure caption and algorithm differ, the VLM scale is unclear, and some extracted equations are damaged.
• Value: 4/5. A useful reference for joint video generation and reward design, with substantial gaps remaining before scalable interactive world simulation.