ActionParty: Multi-Subject Action Binding in Generative Video Games

Conference: ECCV 2026
Paper: Official paper page · PDF
Project: https://action-party.github.io/
Area: Video Generation / World Models
Keywords: Multi-subject action binding, subject state tokens, attention masks, spatial RoPE, autoregressive generation

TL;DR

ActionParty jointly predicts a shared game video and each character's coordinates, separating action ownership through attention masks and grounding subjects through spatial RoPE; across 46 Melting Pot games with up to 7 players, it raises movement accuracy from Text-Action's 0.158 to 0.779.

Background & Motivation

A model that generates convincing game footage does not necessarily let different players control their own characters. Single-subject world models usually receive one action stream, so the recipient of an action is largely implicit. In a shared scene, however, "character A moves forward while character B turns" requires action understanding, persistent identity, and correct spatial localization at the same time. Describing everything in a single prompt pushes these different responsibilities through the same conditioning channel. Once characters leave their initial locations, or look identical, the language-based identity can lose its visual grounding.

The desired control is also more demanding than following an absolute trajectory. In Melting Pot, moving forward depends on the character's orientation, and an interaction button can have different effects in different games. The model must inspect the video for orientation, obstacles, and environmental state before determining the outcome. Generating a separate view per player makes identity easier to establish, but increases video-token costs with the number of players and introduces cross-view consistency requirements. ActionParty instead generates one shared view and puts the additional representation budget into lightweight subject states.

The paper borrows a division of responsibilities from game engines: establish which action updates which subject, then express the resulting states in the image. This is not a symbolic game engine implemented alongside a renderer; it is an information-flow constraint inside a pretrained video diffusion transformer. Core idea: maintain a predicted coordinate state for each subject, route actions only to their corresponding subject states, and jointly generate states and video through spatially grounded attention.

Method

Overall Architecture

The inputs are a game description, an initial frame, the initial coordinates of every subject, and discrete actions supplied over time. The outputs are both the next frame and the next coordinates. These generated outputs become the context for the following step. Flattened video latents and subject state tokens are concatenated along the sequence dimension and jointly denoised by one DiT; the coordinates are therefore not annotations produced by an external tracker after video generation.

Within each DiT block, self-attention first combines subject isolation and spatial RoPE to establish which visual entity a state belongs to. Cross-attention then uses an action mask to establish which action stream that state receives. This structure is repeated across blocks and denoising steps. The update-and-render analogy describes responsibilities, not two independent networks invoked exactly once per frame in a strict update-then-render order.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Frames, coordinates<br/>and historical context"] --> B["Joint Subject-State Denoising"]
    B --> C["Spatially Bound Rendering<br/>Self-attention and RoPE"]
    C --> D["Subject-Specific Action Updates<br/>Cross-attention mask"]
    E["Per-subject actions<br/>and game text"] --> D
    D -->|Repeat blocks and denoising| F["Next frame and coordinates"]
    F -->|Sliding-window feedback| A

Key Designs

1. Joint Subject-State Denoising: give identity a persistent spatial representation

A conventional video model represents characters implicitly in pixel features, leaving external actions without a stable attachment point. ActionParty adds a state token for every subject at every timestep and defines the concrete state as a two-dimensional coordinate. This is neither a complete engine record containing orientation, health, and inventory nor an arbitrary learned identity vector. In most of the grid environments considered, two characters cannot occupy the same location, so coordinates provide a sufficient spatial basis for distinguishing them. A subject index preserves the control channel over time, while its coordinates change with movement.

Joint denoising requires the model to predict both where the character appears in the frame and what coordinates its state should contain. Coordinates are explicit and inspectable, but properties such as orientation must still be read from the video. This avoids replacing visual world modeling with prediction from a complete engine state, and explains why subject states cannot evolve entirely independently of visual tokens. Coordinate precision is useful insofar as it disambiguates identity: a small localization error need not immediately break action binding, a tolerance also illustrated in the paper's qualitative examples.

2. Spatially Bound Rendering: read the shared scene without directly mixing subjects

The self-attention mask \(M_{SA}\) lets subjects attend to video tokens and video tokens attend to all subjects, while blocking direct attention between different subjects. Video remains the shared environmental representation: each character can read its orientation and surroundings, and the image can integrate all subject states for rendering. The restriction prevents indiscriminate state mixing, not environmental interaction. Isolation alone is insufficient, however, because a subject token still needs to know which visual location it represents.

Spatial RoPE supplies that connection. A video token at position \((h,w)\) and time \(t\) receives the rotation \(R(t,h,w)\). A subject token uses a rotation based on that subject's coordinates in the previous timestep:

\[ R\bigl(t,h_{t-1}^{i},w_{t-1}^{i}\bigr). \]

The previous position is used because the current one is still being denoised and is not yet available. Provided the subject moves only a limited distance, identifying it across the entire image becomes local refinement within the reach of one action. This is an attention bias, not a hard spatial crop or a supplied ground-truth future trajectory. Its rationale depends on nearby consecutive positions and should not be assumed to extend unchanged to teleportation or reappearance after prolonged absence.

3. Subject-Specific Action Updates: route actions to a subject rather than the entire image

Each discrete action is embedded into the model's latent dimension and combined with game-text embeddings to form the cross-attention conditions. The mask \(M_{CA}\) allows subject \(i\) to attend only to its own action embeddings, preventing other subjects' commands from entering its state pathway. The game-text conditioning pathway connects only to video tokens. Text can therefore preserve its pretrained role in describing the environment, while individual control no longer relies on a long prompt resolving identity by itself.

The mask separates subjects rather than additionally restricting each state to its own frame's action. The paper allows access to a subject's action sequence and discusses current and past actions as context. Replacing this with frame-wise access reduces MA from 0.872 to 0.052 in the Coins ablation. The lesson is not that more sparsity is always better: unwanted cross-subject connections should be removed while useful within-subject temporal connections remain. The cache does not specify the complete implementation-level temporal mask, so this result does not establish that inference requires unknown future actions.

A Worked Example

Consider two Paintball characters: A remains still, while B moves forward, turns, and fires a beam. This is an explanatory example, not an additional measured result. The model receives the initial frame and both coordinates, establishing where the two action channels begin even if the characters look similar.

When predicting B's forward movement, B's state token reads its own action through cross-attention and its orientation through video features. Spatial RoPE uses B's previous location to connect the token to nearby pixels. A cannot directly read B's action or mix directly with B's subject state. After denoising, the new frame and both predicted coordinates become the next step's context.

When B subsequently turns and interacts, "interact" cannot be implemented as a fixed displacement: the generated beam must depend on the orientation in the updated scene. This illustrates why position changes alone are insufficient for evaluating the task and why the paper also measures local visual effects around the character.

Loss & Training

Training uses clean ground-truth context with noisy prediction targets. Historical frames and coordinates remain clean, while the next frame and next subject states are denoised. Unused later sequence positions are padded with fully noisy frames to support variable context lengths. Adam optimizes a flow matching loss over video and subject states. Equations (1) and (2) are corrupted in the text cache, and the body does not specify a complete loss-weighting expression; neither those equations nor an assumed weighting between the two targets is reconstructed here.

The backbone is Wan2.1-1.3B. It first undergoes 22.5k steps of autoregressive adaptation using only game text and video, without action control or subject-state modifications. The complete ActionParty architecture is then trained for 65k steps; both stages use batch size 64. The training window is \(T=5\), corresponding to 4 actions. Scenes contain at most 7 subjects, with padded subject states set to zero. Inference uses 20 sampling steps and a timestep shift of 5.0.

Initial coordinates are required at inference, especially when visually identical entities make the correspondence between action channels and characters ambiguous. Subsequent predictions use generated frames and coordinates, dropping the oldest context when necessary and keeping at most \(T-1\) historical frames. Clean-context training and generated-context inference still create a distribution shift; the architecture does not by itself eliminate accumulating drift. The authors report 6% additional overhead for 7 players at \(T=5\), but the cache's extra-token accounting is not sufficiently consistent to derive an exact complexity expression or real-time frame rate.

Key Experimental Results

Main Results

Training covers 46 Melting Pot games, with 2,000 videos per game at \(512\times512\) resolution and a unified space of 25 actions. Evaluation uses 230 rollouts, or 5 per game. The first two timesteps are restricted to movement or turning, and later steps include at least one interacting subject, reducing the tendency of purely random controls to leave characters near their starting locations. These are evaluations within the set of training games, not zero-shot generalization to 46 unseen games.

MA measures whether movement directions extracted from consecutive generated frames match the input actions. SP measures how many subjects remain at the end of generation. DR measures the fraction of steps where subject positions agree with the ground-truth trajectory, not merely whether a detector sees a character. The \(z_t\) Error is the L2 distance between predicted and ground-truth coordinates; the cache does not establish its normalization or physical units, so it should not be interpreted as pixels or grid cells.

Table 1 reproduces the main comparison from the paper's Table 1. Higher is better for MA, SP, DR, and PSNR; lower is better for LPIPS and FVD.

Method	MA	SP	DR	LPIPS	PSNR	FVD
Pretrained AR	0.065	0.571	0.330	0.0494	26.98	59.30
Zero-shot I2V	0.027	0.422	0.440	0.0906	23.17	281.55
View-based	0.013	0.237	0.588	0.0503	27.38	75.1
Text-Action	0.158	0.668	0.433	0.0353	29.14	56.74
ActionParty	0.779	0.903	0.886	0.0102	36.35	17.16

Against Text-Action with the same backbone, MA improves by 0.621, or 62.1 percentage points; SP improves by 23.5 percentage points and PSNR by 7.21 dB. Zero-shot I2V uses the larger Wan2.1-14B model but lacks equivalent training adaptation, so its failure alone does not show that parameter scale is irrelevant. The stronger evidence is that text-only control with the same backbone still cannot replace explicit action binding.

EA examines changes in a \(3\times3\) tile patch around the ground-truth player location relative to the preceding frame, using SSIM to assess local action effects. The cached text states "SSIM ≥ 0.85" for detecting executed actions while also describing inter-frame dissimilarity. The direction of that decision requires clarification from supplementary material or code. The reported results are retained without silently changing the inequality, and EA should not be read as complete game-rule correctness.

Table 2 selects results from the paper's Table 2; higher EA is better.

Method	Idle EA	Move EA	Turn EA	Interact EA	Overall EA
Pretrained AR	0.399	0.414	0.462	0.346	0.412
Text-Action	0.357	0.420	0.553	0.326	0.433
ActionParty	0.899	0.867	0.914	0.774	0.861

Interact EA rises from Text-Action's 0.326 to 0.774, a gain of 44.8 percentage points, but remains below ActionParty's own Turn EA of 0.914. Context-dependent interactions remain harder than simpler character changes; better binding does not imply that every game rule has been learned.

Ablation Study

Table 3 comes from the paper's Table 3. These experiments use only the two-player, 4-action Coins game, with 2.5k samples at \(256\times256\) resolution. All variants start from the same autoregressive pretrained checkpoint and receive 45k fine-tuning steps. The full model's 0.872 here is therefore not directly comparable to the 0.779 in the all-game evaluation.

Configuration	MA	\(z_t\) Error	SP	DR
Without \(M_{SA}\)	0.580	0.108	1.00	0.716
Without \(M_{CA}\)	0.052	0.215	0.88	0.307
Frame-wise \(M_{CA}\)	0.052	0.240	0.80	0.293
No RoPE in self-attention	0.032	0.291	1.00	0.278
ActionParty	0.872	0.072	1.00	0.913

Key Findings

• Removing RoPE reduces MA by 84.0 percentage points while SP remains 1.00. Keeping every character visible does not mean that actions reach the correct character; preservation cannot replace binding evaluation.
• Removing \(M_{CA}\) or restricting actions to the same frame both reduce MA to 0.052. Subject-specific routing and temporal context are both important. Removing \(M_{SA}\) has a milder effect, but the fall from 0.872 to 0.580 still shows the harm of direct state mixing.
• Figure 5 shows stability across 4 action steps, and Figure 6 illustrates 2-, 4-, and 7-player scenes. These support short-horizon multi-subject control, not stable gameplay over minutes.

Highlights & Insights

• The control interface matters more than a longer prompt. Independent action inputs are attached to subjects and spatially grounded in the image, avoiding the requirement that text embeddings simultaneously perform identity tracking and dynamics control.
• A small state can support a large shared image. Coordinates handle disambiguation, while video features provide orientation and environmental context. This is cheaper than duplicating a full video stream per player, provided coordinates are sufficient to distinguish subjects.
• Different metrics expose different failures. SP, MA, DR, and EA test presence, action assignment, trajectory alignment, and local effects. The ablation with SP=1.00 and MA=0.032 is a particularly clear counterexample to judging control from appearance alone.

Limitations & Future Work

• The authors leave three-dimensional and partially observed settings to future work. Current evidence concerns two-dimensional grids, discrete actions, and at most 7 players, not realistic open worlds.
• The method requires initial subject positions and coordinate supervision during training. Long occlusions, teleportation, overlapping positions, or newly joining subjects may exceed what two-dimensional coordinates and previous-position RoPE can resolve. Identity memory and uncertainty-aware states are plausible extensions to test.
• Training costs restrict architectural ablations to Coins. Cross-game repetitions, uncertainty across random seeds, and longer rollouts are missing, leaving the generality of component-level gains uncertain.
• EA depends on local patches and an empirical threshold, while detector accuracy is primarily validated on real game frames. Measurement reliability under generated artifacts and event-level rule checks need further study; the cached EA decision direction specifically requires source verification.
• Twenty sampling steps and 6% additional overhead do not establish real-time interaction. Hardware, end-to-end latency, and throughput and drift across player counts and sequence lengths should be reported.

Related Work & Insights

• Versus Genie / GameNGen: The focus is ownership of multiple action streams in a shared image, rather than single-player control or first-person generation. Correctly controlling who acts is a distinct objective from visual fidelity.
• Versus Solaris / Multiverse / MultiGen: Multi-view and explicit-game-state approaches address alignment between player views; ActionParty binds actions using lightweight coordinates in one global view. The paper's custom View-based baseline is not a full reproduction of those systems, so the table is not a direct ranking against their complete implementations.
• Versus trajectory control and attribute binding: An external trajectory specifies where an object should appear, whereas this method predicts the next position from actions and historical state. Spatial grounding becomes a generated state fed back over time. That suggests an interface worth studying for shared observations in multi-robot settings, but the paper does not experimentally establish that transfer.

Rating

• Novelty: 4/5. Joint subject-state generation, subject-specific action routing, and spatial RoPE form a focused and testable mechanism for multi-subject control.
• Experimental Thoroughness: 3/5. Coverage across 46 games and several metrics is useful, but architectural ablations are limited to one game and long-horizon and uncertainty reporting remain insufficient.
• Writing Quality: 4/5. The game-engine analogy and attention diagrams communicate the design well; the EA decision direction and extra-token accounting require clarification, while corrupted cached equations limit exact verification.
• Value: 4/5. An effective design for individual control within shared video at low reported additional overhead, with substantial remaining gaps to general real-time multiplayer world models.