World Models for Policy Refinement in StarCraft II¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/yxzzhang/StarWM
Area: LLM Agent
Keywords: Large Language Models, World Models, StarCraft II, Policy Refinement, Hybrid Dynamics
TL;DR¶
StarWM introduces the first player-view, action-conditioned textual world model for StarCraft II and integrates it into a lightweight Generate–Simulate–Refine decision loop (StarWM-Agent), achieving consistent win-rate gains (+30%, +15%, and +30% against Hard, Harder, and VeryHard built-in AI) and dramatically reducing supply block rates through foresight-driven macro refinement.
Background & Motivation¶
Large Language Models (LLMs) have demonstrated impressive multi-step reasoning capabilities, inspiring extensive exploration of their potential in complex decision-making benchmarks. StarCraft II (SC2), recognized for its massive combinatorial state-action space, fine-grained micro-tactical execution, and strict partial observability under the Fog of War, serves as one of the most demanding testbeds for sequential decision-making. Recent LLM-based SC2 agent architectures (such as TextStarCraft II, LLM-PySC2, and StarEvolve) have investigated input summarization, multimodal perception, and hierarchical decoupling between high-level strategy and low-level micromanagement. Nevertheless, virtually all existing methodologies exclusively focus on improving the policy itself, leaving the integration of a learnable, action-conditioned dynamics model into the decision loop largely unexplored.
In complex, dynamic environments, purely reactive or heuristic LLM policies frequently suffer from shortsighted decisions, such as prematurely exhausting minerals on surplus supply structures or marching units blindly into fatal engagements. Cognitive science research demonstrates that human experts heavily rely on internal mental world models to conduct short-term simulation, evaluating consequences and avoiding adverse traps before committing to physical execution. However, establishing a player-view textual world model for SC2 encounters two primary barriers. First, SC2 dynamics are inherently hybrid and multi-faceted: resource flows follow continuous accumulation and consumption, task progress adheres to deterministic game clocks, unit trajectories follow spatial kinematics, and combat outcomes are dictated by non-linear damage attrition—all under the shroud of partial observability. Second, given the vast action space, turning predicted future observations into tangible policy improvements without falling into computationally prohibitive Monte Carlo Tree Search (MCTS) or Model Predictive Control (MPC) presents a severe practical trade-off.
To resolve these challenges, this paper conducts the first systematic investigation into the learnability and decision utility of player-view, action-conditioned textual world models for SC2. The authors devise a structured textual representation that decomposes the hybrid dynamics into five distinct semantic modules, assemble SC2-Dynamics-50K as the first instruction-tuning dataset for SC2 dynamics prediction, and propose StarWM-Agent to operationalize foresight-driven policy refinement. Core idea: factorize SC2 hybrid dynamics into a five-module textual observation representation to train an action-conditioned world model, and couple it with a lightweight Generate–Simulate–Refine inference loop that leverages short-horizon forward simulation to refine suboptimal action proposals without costly tree search.
Method¶
The overall methodology centers on action-conditioned dynamics modeling under partial observability and lightweight forward simulation for policy refinement. StarWM acts as an autoregressive textual simulator operating strictly from a single player's restricted perspective. In online gameplay, StarWM-Agent adopts a two-stage decision pipeline: the policy model first proposes candidate actions, StarWM projects their short-term consequences \(\tau\) seconds into the future, and the policy model conducts a second pass over the augmented context to confirm or amend its tactical and macro decisions.
Overall Architecture¶
The agent interacts with the StarCraft II engine through textual interfaces: an observation-to-text operator \(\Phi(x_t)\) serializes raw structured engine observations into categorized text blocks, while an action-to-text operator \(\Psi(u_t)\) maps textual commands into executable engine commands. The decision flow executes via the Generate–Simulate–Refine pipeline: given the current observation \(o_t\), the policy \(\pi_\theta\) first generates an initial action proposal \(a_{\text{init}} = \pi_\theta(o_t)\). Next, the learned world model \(M_\phi\) simulates the expected future state \(\hat{o}_{t+\tau} = M_\phi(o_t, a_{\text{init}})\) over a short horizon (default \(\tau = 5\) seconds). Finally, an augmented context \(c_t = [o_t; a_{\text{init}}; \hat{o}_{t+\tau}]\) is constructed, prompting the policy model to perform foresight-driven refinement to output the final executed action \(a_{\text{final}} = \pi_\theta(c_t)\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Engine Observation xt"] --> B["Five-Module Textual Representation Φ(xt)"]
B --> C["Initial Action Proposal a_init = π(ot)"]
C --> D["Action-Conditioned Forward Simulation M_ϕ(ot, a_init)"]
D --> E["Generate-Simulate-Refine Decision Loop π([ot; a_init; o_pred])"]
E --> F["Final Executed Actions a_final to Engine"]
Key Designs¶
1. Five-Module Textual Representation: Decoupling SC2 Heterogeneous Hybrid Dynamics
To convert raw engine variables (continuous resource counts, floating-point coordinates, categorical statuses, and percentage bars) into an LLM-digestible format without losing critical physics, the observation-to-text operator \(\Phi(x_t)\) factorizes the environment state into five specialized semantic modules. The [Info] module captures global status, race matchup, mineral/gas counts, gathering rates, supply usage/caps, active alerts, and upgrades, isolating continuous resource flows. The [Queue] module tracks active production, construction, and research progress percentages, capturing deterministic temporal evolution governed by the engine clock. The [My Units] module records friendly worker and army IDs, positions, health percentages (HP), energy, and status tags, governing spatial kinematics and combat degradation. The [My Structures] module tracks friendly base facilities and tech add-ons. Finally, the [Visible Hostiles] module logs spotted enemy units, structures, and fog-of-war snapshot buildings, encapsulating epistemic uncertainty under partial observability. This semantic decomposition prevents interference across diverse dynamic mechanisms and enables stable autoregressive generation.
2. Action-Conditioned Forward Simulation: Capturing Causal Transitions under Partial Observability Unlike god-view simulators with perfect information, a player-view world model \(M_\phi\) must predict future observations \(\hat{o}_{t+\tau}\) conditioned on local observation \(o_t\) and intended action sequences \(a_{t:t+\tau}\) while facing incomplete knowledge of opponent behavior. The authors collect 100 game trajectories played by a scripted agent against SC2 built-in AI at Harder (LV6) and VeryHard (LV7) with randomized opening and tactical styles, generating 50,407 instruction-tuning triples \((o_t, a_{t:t+\tau}, o_{t+\tau})\) with a sliding step of 1 second and horizon \(\tau = 5\) seconds (SC2-Dynamics-50K). StarWM is fine-tuned on Qwen3-8B using standard supervised negative log-likelihood: $\(L_{\text{SFT}} = - \sum_{\mathcal{D}} \log P_{M_\phi}(o_{t+\tau} \mid o_t, a_{t:t+\tau})\)$ Through this training, the world model masters deterministic state transitions (e.g., deducting resource costs, appending active items to the queue, updating building percentages) as well as probabilistic interactions (e.g., inferring that marching towards hostile territory frequently encounters enemy defenders, or that engaged units suffer health attrition according to estimated DPS).
3. Generate–Simulate–Refine Decision Loop: Lightweight Foresight-Driven Policy Correction In real-time strategy games, unconstrained combinatorial action spaces make exhaustive lookahead algorithms such as MCTS impractical for inference-time LLM agents. StarWM-Agent resolves this dilemma through a lightweight, non-search Generate–Simulate–Refine pipeline. When the policy proposes an initial action sequence \(a_{\text{init}}\), StarWM executes a single forward rollout over the next 5 seconds to synthesize \(\hat{o}_{t+\tau}\). The policy model is then prompted to conduct a structured tactical look-back comparison between \(o_t\) and \(\hat{o}_{t+\tau}\). If the simulation reveals impending resource depletion, supply bottlenecks, or unfavorable combat trades, the policy revises \(a_{\text{init}}\) to cancel wasteful expenditures or reposition vulnerable forces. Across online evaluations, this mechanism produces an Action Revision Rate of 20% to 33%, achieving human-like "think before you leap" behavior at the nominal cost of just one additional forward pass.
A Worked Example¶
Consider an early-game macro-management scenario at game time 01:02:
1. Initial Game State: The player possesses 145 minerals, 12 vespene gas, an idle Command Center, and an unused supply margin of 18 (Supply: 13/31, unused: 18), while the Barracks is currently at 29% construction.
2. Initial Action Proposal: Prompted by generic heuristic reflexes, the zero-shot policy proposes constructing another supply depot: TERRANBUILD_SUPPLYDEPOT(target: [30, 61]).
3. World Model Forward Simulation: Conditioned on this proposed action, StarWM simulates the state 5 seconds later (\(\tau = 5\)s) and reports: minerals drop sharply to 50, gas increases to 24, a new supply depot enters the queue at 23% completion, and unused supply remains exceedingly high at 18.
4. Policy Refinement: Inspecting the simulated future report, the policy recognizes that supply is already fully sufficient (margin 18) while minerals have become critically constrained (dropping to 50), which will starve subsequent worker training. Consequently, the policy amends its command: it removes the redundant supply depot construction and issues COMMANDCENTERTRAIN_SCV(units: [137]) instead, preserving smooth economic growth.
Loss & Training¶
StarWM adopts Qwen3-8B as its base foundation and is trained via supervised fine-tuning (SFT) using LLaMA-Factory. Training uses LoRA with rank 8, an initial learning rate of \(5 \times 10^{-5}\), and runs for 10 epochs on 8 \(\times\) NVIDIA H100 GPUs. Prompts enforce strict raw-text output conforming to the five-module schema without markdown code blocks or conversational chatter, securing a 99.44% parsing success rate on downstream evaluation frames.
Key Experimental Results¶
Main Results¶
StarWM was evaluated across offline dynamics accuracy (Table 1) and full online matches against the SC2 built-in AI across three difficulty levels (Table 2). Online evaluations comprise 20 games per setting in Terran-vs-Terran matches on Flat64:
| Setting / Opponent Level | Win Rate (%)↑ | Supply Block Rate (%)↓ | Resource Conversion Rate (%)↑ | Kill-Loss Ratio (%)↑ | Valid Action Rate (%)↑ |
|---|---|---|---|---|---|
| LV5 (Hard - OOD Opponent) | |||||
| Qwen3-8B Baseline Policy | 0.0% | 63.58 ± 13.54 | 29.09 ± 19.14 | 15.22% | 17.89 ± 12.25 |
| StarWM-Agent (8B) | 10.0% (+10.0) | 5.42 ± 4.97 (-58.16) | 84.20 ± 14.21 (+55.11) | 50.44% (+35.22) | 86.29 ± 12.88 (+68.40) |
| Qwen3-32B Baseline Policy | 20.0% | 25.45 ± 17.47 | 52.32 ± 23.03 | 62.42% | 41.31 ± 20.81 |
| StarWM-Agent (32B) | 50.0% (+30.0) | 6.09 ± 4.61 (-19.36) | 81.11 ± 7.56 (+28.79) | 89.87% (+27.45) | 85.57 ± 11.01 (+44.26) |
| LV6 (Harder) | |||||
| Qwen3-8B Baseline Policy | 5.0% | 58.31 ± 21.15 | 36.12 ± 21.36 | 6.25% | 28.00 ± 21.64 |
| StarWM-Agent (8B) | 10.0% (+5.0) | 10.57 ± 9.28 (-47.74) | 78.77 ± 8.82 (+42.65) | 28.12% (+21.87) | 82.37 ± 10.23 (+54.37) |
| Qwen3-32B Baseline Policy | 25.0% | 21.41 ± 21.29 | 58.81 ± 21.34 | 27.42% | 55.49 ± 25.58 |
| StarWM-Agent (32B) | 40.0% (+15.0) | 5.93 ± 5.20 (-15.48) | 78.67 ± 8.66 (+19.86) | 41.30% (+13.88) | 84.00 ± 11.22 (+28.51) |
| LV7 (VeryHard) | |||||
| Qwen3-8B Baseline Policy | 0.0% | 58.74 ± 19.99 | 32.78 ± 21.61 | 12.30% | 25.29 ± 20.86 |
| StarWM-Agent (8B) | 20.0% (+20.0) | 5.51 ± 5.21 (-53.23) | 82.89 ± 8.96 (+50.11) | 20.00% (+7.70) | 82.13 ± 11.91 (+56.84) |
| Qwen3-32B Baseline Policy | 20.0% | 16.39 ± 21.25 | 55.74 ± 20.72 | 29.26% | 61.64 ± 26.31 |
| StarWM-Agent (32B) | 50.0% (+30.0) | 5.39 ± 4.90 (-11.00) | 76.27 ± 16.14 (+20.53) | 50.51% (+21.25) | 81.99 ± 22.50 (+20.35) |
Ablation Study¶
The authors conducted component-wise ablations against LV7 (VeryHard) using zero-shot Qwen3-8B as the policy model (Table 4), and assessed complementarity with the Chain-of-Summarization (CoS) history-compression baseline adapted from TextStarCraft II (Table 3):
| Configuration / Pipeline | Win Rate (%)↑ | SBR (%)↓ | RCR (%)↑ | KLR (%)↑ | VAR (%)↑ | Action Revision Rate (%) | Decision Latency (s)↓ |
|---|---|---|---|---|---|---|---|
| Ablation Component Hierarchy (LV7) | |||||||
| Direct Generate (Policy-only) | 0.0 | 58.74 | 32.78 | 12.30 | 25.29 | — | 1.96 ± 0.62 |
| + Self-Reflection (+ Refine) | 5.0 | 9.19 | 76.44 | 13.97 | 78.04 | 34.64 | — |
| + Zero-Shot WM Simulate | 10.0 | 9.98 | 72.80 | 14.80 | 78.44 | 26.66 | — |
| + StarWM Simulate (Full StarWM-Agent) | 20.0 | 5.51 | 82.89 | 20.00 | 82.13 | 32.33 | 11.61 ± 3.78 |
| Complementarity Analysis (LV7) | |||||||
| CoS History Summarization Baseline | 10.0 | 31.00 ± 21.51 | 56.07 ± 17.03 | 11.34 | 54.37 ± 16.16 | — | 6.34 ± 0.81 |
| CoS + StarWM-Agent Combined | 35.0 | 6.13 ± 7.48 | 86.92 ± 10.00 | 29.40 | 83.16 ± 15.02 | — | 14.92 ± 3.25 |
Key Findings¶
- Accurate Dynamics Modeling Outperforms Mere Self-Reflection: The ablation progression reveals that while policy self-reflection (+ Refine) curtails supply block rates, it yields only a modest win-rate bump (0% \(\to\) 5%). Incorporating zero-shot Qwen3-8B simulation pushes win rate to 10%, while the specialized StarWM simulator drives win rate to 20% and achieves superior combat ratios (KLR 20.00%), confirming that accurate physical foresight provides value well beyond reflective prompting.
- Orthogonal Synergy with History Summarization: While CoS summarizes past observation history (\(o_{\le t}\)), StarWM forward-projects candidate action outcomes (\(o_{t+\tau}\)). Unifying CoS with StarWM-Agent achieves the highest performance on LV7: 35.0% win rate, 86.92% resource conversion, and 29.40% kill-loss ratio.
- 100-Match Statistical Significance Validation: In an expanded 100-match evaluation against LV7 (Table 11), StarWM-Agent (8B) maintained a 23.0% win rate (95% Wilson CI [15.8%, 32.2%]) against the Qwen3-8B baseline's 5.0% ([2.2%, 11.2%]), demonstrating statistical significance under a two-sided Fisher exact test (\(p = 0.00037\)).
Highlights & Insights¶
- Semantic Factorization for Complex Physical Systems: Decomposing raw game telemetry into five semantically decoupled text modules (economic flow, deterministic task progress, kinematic entity states, structural assets, and fog-covered hostiles) provides an effective recipe for training LLMs on hybrid discrete-continuous simulation tasks.
- Pragmatic Non-Search Foresight Architecture: Rather than attempting intractable tree searches over combinatorial action spaces, the single-step Generate–Simulate–Refine paradigm demonstrates that a 5-second hypothetical lookahead suffices to eliminate catastrophic macro mistakes (reducing supply block rates from ~60% to ~5%).
- Empirical Characterization of POMDP Boundaries: The offline evaluation candidly reveals the epistemic boundary of single-frame prediction under the Fog of War (where enemy entity distribution prediction AWD does not surpass static copying), while showing that incorporating temporal history (\(h=3\)) effectively tightens prediction consistency.
Limitations & Future Work¶
- Epistemic Boundaries under the Fog of War: Under partial observability, future enemy movements cannot be deterministically inferred from a single observation. Developing explicit belief-state representations or latent particle filtering is an essential next step.
- Evaluation Restricted to Mirror Matchups: The training corpus SC2-Dynamics-50K is currently restricted to Terran-vs-Terran matches on Flat64. Extending the framework to asymmetric matchups (TvZ, TvP) and non-flat competitive ladder maps remains future work.
- Inference Latency in Real-Time Regimes: The agent requires approximately 11 to 14 seconds per decision frame, relying on game pausing protocols. Deploying world-model agents under strict real-time APM constraints will require state-delta prediction, model distillation, and high-throughput speculative decoding.
Related Work & Insights¶
- vs AlphaStar (Vinyals et al., 2019): AlphaStar relies on massive self-play reinforcement learning and deep neural networks, demanding millions of compute hours while lacking natural language interpretability and generalizability; StarWM leverages an instruction-tuned LLM as a modular, interpretable world model that operates with substantially lower training overhead.
- vs TextStarCraft II / CoS (Ma et al., 2024): TextStarCraft II utilizes Chain of Summarization to compress past observation streams; StarWM-Agent projects candidate actions forward in time. They are conceptually complementary, and their combination yields the highest win rates in online testing.
- vs RAP (Hao et al., 2023) / WebDreamer (Gu et al., 2025): Prior world-model LLM frameworks rely on expensive MCTS or MPC search trees, which suffer combinatorial explosion in RTS games; StarWM establishes a lightweight, non-search Generate–Simulate–Refine loop tailored for real-time competitive domains.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering study formulating the first player-view, action-conditioned textual world model and dataset for StarCraft II.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous multi-dimensional offline metrics, online match benchmarks, out-of-domain probes, progressive ablations, and 100-game statistical significance tests.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical modeling, and illustrative real-game case studies.
- Value: ⭐⭐⭐⭐⭐ Provides a foundational blueprint for integrating forward predictive world models into embodied and sequential decision-making LLM agents.