Skip to content

Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/loyiv/ITP
Area: LLM Reasoning
Keywords: Large Language Model Agents, Textual World Models, Adaptive Lookahead, Task Planning, Reinforcement Learning

TL;DR

Addressing the "shallow grounding" and trial-and-error breakdowns caused by purely reactive perception in LLM agents, this paper introduces Imagine-then-Plan (ITP) within a newly conceptualized POIMDP formulation, adaptively balancing task progress against rollout costs to guide both training-free reflection (ITPI) and reinforced online actor-critic optimization (ITPR), achieving new state-of-the-art results across ALFWorld, ScienceWorld, and tool-use benchmarks.

Background & Motivation

Autonomous agents powered by Large Language Models (LLMs) have demonstrated remarkable decision-making capabilities across digital interfaces, interactive environments, and embodied household tasks. However, mainstream interactive paradigms such as ReAct and Reflexion remain strictly reactive, conditioning each subsequent action purely on immediate observations and past execution history. This passive, backward-looking feedback loop traps agents in a state of "shallow grounding": while models can linguistically interpret their current sensory feedback, they lack an active, causal understanding of how prospective actions will reshape environment dynamics over extended horizons. Without internal predictive foresight, agents cannot anticipate downstream bottlenecks or irreversible state conflicts, repeatedly stumbling into catastrophic errors that are recognized only after irreversible environment execution.

To bridge this fundamental grounding gap, recent efforts have increasingly explored utilizing LLMs as textual world models to simulate environment transitions within a "mental sandbox." Nevertheless, current world-model-based agent planners remain constrained by rigid rollout schemes: they either restrict imagination to a fragile single-step transition verification, or impose fixed-horizon multi-step rollouts and costly Monte Carlo Tree Search (MCTS). Such static strategies introduce severe structural dilemmas across variable task distributions. On trivial or routine actions (e.g., navigating a corridor or grabbing an obvious item), deep rollouts induce massive computational overhead and latency without adding informative signals; conversely, at pivotal, high-stakes decision points, shallow rollouts fail to expose long-range dependencies and latent conflicts. Furthermore, autoregressive world models inevitably suffer from compounding prediction errors over long horizons, meaning excessive rollouts frequently poison policy decisions with hallucinated states.

Consequently, an intelligent agent must not imagine blindly at a fixed horizon across all time steps, but must instead deliberate adaptively—dynamically modulating its lookahead depth by trading off expected task progress against the predictive degradation and computational expense of deeper simulation. The core idea of this paper is to generalize the standard POMDP into a Partially Observable and Imaginable Markov Decision Process (POIMDP), establishing the Imagine-then-Plan (ITP) framework where an adaptive lookahead mechanism dynamically governs mental sandbox rollouts, leveraging the resulting imagined future trajectories as rich dual-stream signals to steer both plug-and-play inference-time reflection (ITPI) and joint policy-horizon actor-critic reinforcement learning (ITPR).

Method

Overall Architecture

ITP reformulates agent decision-making from a conventional POMDP into a Partially Observable and Imaginable Markov Decision Process (POIMDP). At each environment interaction step \(t\), the agent receives the current partial observation \(s_t\). Instead of immediately emitting an action, an adaptive lookahead mechanism evaluates the current state and task instruction to dynamically assign an imagination horizon \(K_t \in [0, K_{max}]\). When \(K_t = 0\), the framework smoothly collapses to standard reactive POMDP decision-making. When \(K_t > 0\), the agent policy \(\pi_\theta\) and a learned autoregressive textual world model \(M_\phi\) interact in an internal mental sandbox for \(K_t\) continuous steps, generating an imagined future trajectory \(\hat{\tau}_t^{(K_t)}\). This foresighted trajectory provides crucial implicit signals regarding goal progress, dead ends, or precondition violations, and is fused with the concrete present observation into a dual-stream representation that guides the final environmental action \(a_t \sim \pi_\theta(\cdot | s_t, \hat{\tau}_t^{(K_t)})\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Task Instruction & Environment State<br/>Current observation s_t and task goal"] --> B["Adaptive Lookahead Horizon Selection<br/>Dynamic trade-off between progress and budget K_t"]
    B --> C["POIMDP Mental Sandbox Imagination<br/>Multi-step interaction with world model yielding tau_t"]
    C --> D["Dual-Stream Policy Planning<br/>Fusing present observation and imagined trajectory for action a_t"]
    D --> E["Environment Execution & State Transition<br/>Emit reward r_env and transition to next state s_t+1"]

Key Designs

1. POIMDP Formalization and World Model Sandbox Rollouts: Overcoming shallow grounding via foresight projection in an internal predictive environment
Traditional reactive agents map observation histories directly to actions, blinding them to the future causal implications of their choices. To resolve this, ITP introduces the POIMDP framework, augmenting the agent-side information state with an internal lookahead operator without altering external environment dynamics. The textual world model \(M_\phi\) is parameterized by an autoregressive language model trained to capture transition regularities \(p_\phi(s_{t+1}|s_t, a_t)\). To ensure grounding and out-of-distribution robustness against sub-optimal exploratory actions, \(M_\phi\) is supervised on a joint corpus \(\mathcal{D}_{WM} = \mathcal{D}_{exp} \cup \mathcal{D}_{roll}\), combining expert demonstrations with exploratory rollout traces via negative log-likelihood. During imagination, the agent policy \(\pi_\theta\) and the world model \(M_\phi\) recursively interact: $\(\hat{a}_{t+i} \sim \pi_\theta(\cdot | s_t, \hat{\tau}_t^{(i)}), \quad \hat{s}_{t+i+1} \sim M_\phi(\cdot | \hat{s}_{t+i}, \hat{a}_{t+i})\)$ producing a multi-step imagined rollout \(\hat{\tau}_t^{(K_t)} = \{(\hat{a}_{t+i}, \hat{s}_{t+i+1})\}_{i=0}^{K_t-1}\). This mental rehearsal allows the agent to anticipate critical task progress or uncover latent physical constraints before taking any irreversible step in the physical environment.

2. Adaptive Lookahead Horizon Selection: Reconciling deep foresight requirements with compounding simulation errors and computational cost
Static-horizon rollouts either terminate prematurely before uncovering long-term dependencies or expand excessively, incurring prohibitive token budgets and compounding simulation errors (as Fact-F1 rapidly deteriorates at deeper steps). ITP dynamically selects the imagination horizon \(K_t\) conditioned on state progress. For training-free inference-time deployment (ITPI), the frozen agent explicitly evaluates instruction complexity and current progress to output a discrete horizon \(K_t \in [0, K_{max}]\), executes sandbox rollouts, and refines its action through a structured reflection operator: $\(a_t \sim \pi_\theta(\cdot | s_t, \text{Reflect}(\hat{\tau}_t^{(K_t)}))\)$ For the reinforcement-trained variant (ITPR), a lightweight linear classification layer (\(K\)-head predictor) \(P_\theta(K_t | s_t)\) is appended to the policy LLM backbone. To bypass the fundamental cold-start challenge that expert demonstrations lack lookahead annotations, ITP introduces an offline pseudo-labeling algorithm: teacher-forced candidate rollouts \(\{ \hat{\tau}_t^{(k)} \}_{k=0}^{K_{max}}\) are rolled out via the world model, and each horizon \(k\) is scored by the log-likelihood gain of the ground-truth expert action penalized by lookahead length: $\(\tilde{K}_t = \arg\max_{0 \le k \le K_{max}} \left[ \log p_{\theta 0}(a_t^* | s_t, \hat{\tau}_t^{(k)}) - \lambda_K k \right]\)$ This guarantees that deep imagination is invoked only when it provides decisive log-likelihood gains that outweigh rollout penalties.

3. Two-Stage Warm-Up and Online Actor-Critic Co-Optimization: Harmonizing dual-stream action execution with horizon meta-decisions
Jointly learning when to look ahead and how to exploit lookahead creates a non-stationary policy optimization landscape. ITPR resolves this through a structured curriculum. In Stage 2, supervised warm-up training optimizes the policy on tuples \((s_t, a_t^*, \tilde{K}_t, \hat{\tau}_t^{(\tilde{K}_t)})\) under a joint objective: $\(\mathcal{L}_{WT}(\theta) = \mathcal{L}_\pi(\theta) + \eta \mathcal{L}_K(\theta)\)$ which aligns action generation under imagined rollouts while calibrating the \(K\)-head predictor. In Stage 3, the agent undergoes online reinforcement learning via Advantage Actor-Critic (A2C). To prevent horizon collapse or unchecked rollout inflation, environment rewards are augmented with explicit penalty terms: $\(r_{t+1} = r_{env} - \lambda_K K_t - \lambda_{step}\)$ The A2C objective simultaneously updates the Actor (jointly maximizing the probabilities of the sampled horizon and the subsequent action weighted by advantage \(A_t\)), the Critic value head, and an entropy regularization term over the horizon distribution, enabling stable convergence and budget self-regulation across long-horizon interactive tasks.

Loss & Training

The complete training pipeline of ITPR is structured into three coordinated stages:

  1. World Model Pre-training: Optimized over the aggregated demonstration-rollout dataset \(\mathcal{D}_{WM}\) by minimizing cross-entropy transition loss: $\(\mathcal{L}_{WM}(\phi) = - \mathbb{E}_{(s', s, a) \sim \mathcal{D}_{WM}} \left[ \log p_\phi(s' | s, a) \right]\)$

  2. Warm-Up Supervised Training: On the pseudo-labeled dataset \(\mathcal{D}_K\), jointly updates the policy parameters and the \(K\)-head predictor with weighting hyperparameter \(\eta = 0.5\): $\(\mathcal{L}_{WT}(\theta) = - \mathbb{E} \left[ \log \pi_\theta(a_t^* | s_t, \hat{\tau}_t^{(\tilde{K}_t)}) \right] - \eta \mathbb{E} \left[ \log P_\theta(\tilde{K}_t | s_t) \right]\)$

  3. Online Actor-Critic Optimization: Decomposed into policy-horizon actor loss, critic value regression loss, and exploration entropy regularization: $\(\mathcal{L}_{A2C}(\theta) = \mathcal{L}_{act}(\theta) + \alpha \mathcal{L}_{value}(\theta) + \beta \mathcal{L}_{ent}(\theta)\)$ where: $\(\mathcal{L}_{act}(\theta) = - \mathbb{E} \left[ A_t \left( \log P_\theta(K_t | s_t) + \log \pi_\theta(a_t | s_t, \hat{\tau}_t^{(K_t)}) \right) \right]\)$ $\(\mathcal{L}_{value}(\theta) = \mathbb{E} \left[ (V_\theta(s_t) - \hat{V}_t)^2 \right], \quad \mathcal{L}_{ent}(\theta) = - \mathbb{E} \left[ \mathcal{H}(P_\theta(K_t | s_t)) \right]\)$ Standard hyperparameter settings include discount factor \(\gamma=0.99\), lookahead penalty \(\lambda_K=0.2\), step penalty \(\lambda_{step}=0.01\), value coefficient \(\alpha=1.0\), and entropy coefficient \(\beta=0.01\), using AdamW with cosine learning rate scheduling (\(2 \times 10^{-5}\) for warm-up, \(5 \times 10^{-6}\) for RL).

Key Experimental Results

Main Results

Evaluation spans four prominent agent benchmarks: embodied household navigation in ALFWorld, interactive procedural experiments in ScienceWorld, multi-step web browsing in WebShop, and executable tool calling in StableToolBench, evaluated across Qwen2.5-7B, Qwen3-8B, and Llama-3.1-8B-Instruct.

Backbone Suite Method Paradigm Method ALFWorld Overall SR (%) ScienceWorld Seen SR (%) ScienceWorld Unseen SR (%) WebShop SR (%)
Qwen2.5-7B Prompting CoT 14.29 3.09 4.63 5.10
Prompting ReAct 17.14 8.24 9.93 15.28
Prompting RAP (Tree Search) 27.86 10.30 16.55 11.28
Prompting (Ours) ITPI 35.71 16.49 17.88 20.10
Training SFT 67.86 55.67 49.00 51.60
Training WKM 76.43 54.12 56.29 58.80
Training IWM 82.80 60.82 57.61 56.20
Training (Ours) ITPR 85.07 62.58 58.94 60.20
Qwen3-8B Prompting CoT 13.57 2.44 1.99 6.32
Prompting ReAct 19.29 9.79 8.61 18.62
Prompting RAP 28.57 15.46 27.14 12.40
Prompting (Ours) ITPI 41.43 20.61 19.86 25.25
Training SFT 70.71 56.70 49.67 52.30
Training WKM 79.29 60.31 47.68 61.15
Training IWM 82.14 59.27 54.30 57.30
Training (Ours) ITPR 88.57 61.85 56.95 68.10
Llama-3.1-8B Prompting CoT 15.00 3.09 3.31 8.20
Prompting ReAct 21.43 9.27 13.24 19.32
Prompting RAP 22.86 11.34 17.21 16.05
Prompting (Ours) ITPI 37.86 19.58 19.20 24.30
Training SFT 79.28 57.21 50.33 47.30
Training WKM 77.86 61.34 54.96 65.58
Training IWM 85.90 57.56 56.29 58.60
Training (Ours) ITPR 87.14 63.91 57.61 67.50

On the multi-turn tool-calling benchmark StableToolBench with Qwen3-8B, ITPI achieves a Solvable Pass Rate (SoPR) of 35% (+7% over RAP) and Solvable Win Rate (SoWR) of 28% (+2%), while ITPR hits 68% SoPR (+24% over IWM) and 54% SoWR (+18%), validating its efficacy in high-stakes API function calling.

Ablation & In-Depth Analysis

To verify the role of online reinforced training, budget efficiency, and planning mechanisms, extensive ablation studies were conducted on ALFWorld and ScienceWorld using Qwen3-8B:

Study Focus Method Variant Success Rate SR (%) Normalized Budget NB ↓ Empirical Insight / Comparison
Online RL Ablation ITPR (Full Model) 88.57 (ALFWorld) / 59.70 (ScienceWorld) 0.21 (ALFWorld) Full three-stage curriculum with online A2C optimization
ITPR w/o RT (No Online RL) 71.42 (ALFWorld) / 46.00 (ScienceWorld) — Severe performance drop (-17.15% / -13.70%) without online RL
Tree Search Planning RAP 28.57 0.42 MCTS planner prone to branching explosion and high latency
ToT (Tree of Thoughts) 16.29 0.58 Breadth-first thought generation induces heavy budget bloat
MCTS 25.00 0.39 Classic Monte Carlo search without adaptive horizon bounds
ITPI (Ours) 41.43 0.25 Surpasses all search baselines with nearly 40% less token budget
World Model RL SFT (Base Imitation) 70.71 — Pure supervised cloning lacking predictive sandbox signals
D2PO 76.28 0.29 Dual preference optimization baseline
VaGen 78.89 0.18 Embodied reinforcement learning baseline
ITPR (Ours) 88.57 0.21 Highest task success rate through adaptive K-head supervision
Compute-Matched Controls Self-Refine (No WM) 22.59 (Prompt) / 68.89 (Train) Matched Budget Iterative verbal rewriting lacks causal environment grounding
Multi-Attempt (No WM) 20.48 (Prompt) / 59.68 (Train) Matched Budget Repeated sampling cannot compensate for blind trial-and-error
K-candidate Selection 36.42 (Prompt) / 78.14 (Train) Matched Budget Direct action re-ranking falls short of multi-step lookahead

Analysis of world-model fidelity across lookahead horizons demonstrates that Fact-F1 degrades systematically as \(k\) deepens (\(k=1\): 0.972, \(k=2\): 0.960, \(k=3\): 0.890, \(k=4\): 0.423, \(k=5\): 0.018). Under fixed-horizon rollouts, success rates peak at \(k=3\) (85.0%) before crashing to 58.4% at \(k=5\) due to severe compounding hallucinations. ITP dynamically anchors rollouts within the world model's reliable predictive range (averaging 3 steps on ALFWorld, 8 on ScienceWorld, and 5 on WebShop), achieving superior performance and budget stability.

Key Findings

  • Online reinforcement training is essential for adaptive foresight: Discarding online RL (w/o RT) causes a dramatic collapse of 17.15% on ALFWorld and 13.70% on ScienceWorld. Supervised warm-up alone cannot instill the meta-cognitive capability to judge when to imagine versus act; policy learning must interact online with penalized rollout costs to calibrate horizon allocation.
  • Gains derive from causal foresight rather than compute expansion: In compute-matched comparisons (Table 5), no-world-model alternatives (Self-Refine, Multi-Attempt, K-candidate Selection) consistently trail ITP by 5.01 to 10.43 points under identical token budgets. Performance breakthroughs stem not from spending more tokens, but from directing computation toward future state causal dynamics.
  • Model-agnostic distillation across backbones: While zero-shot open-weights models like DeepSeek-V3.2 show domain transfer gaps on specialized embodied environments in the training-free regime, ITPR's joint optimization successfully bridges this gap, verifying that ITP can effectively distill lookahead planning capabilities across diverse model architectures.

Highlights & Insights

  • POIMDP formalization unifies reactive observation and proactive imagination: By generalizing the classic POMDP to incorporate an internal lookahead imagination operator, the framework provides a mathematically grounded paradigm for integrating imagined future trajectories into agent decision-making.
  • Teacher-forced log-likelihood pseudo-labeling: The method elegantly addresses the missing supervision problem for lookahead depth in offline expert demonstrations by evaluating expert action likelihood gains against lookahead penalties, offering a clean, automated labeling pipeline.
  • Adaptive horizon control prevents compounding hallucination: By linking rollout horizon decisions to task progress and imposing an explicit penalty in the RL reward function, ITP prevents the agent from falling victim to the compounding simulation errors inherent in autoregressive world models.

Limitations & Future Work

  • Evaluation restricted to discrete textual environments: The experiments are grounded in text-based environments (ALFWorld, ScienceWorld, WebShop, StableToolBench). Transitioning to continuous real-world robotics with raw visual and sensorimotor observations may introduce spatial-temporal noise that could challenge the stability of autoregressive lookahead.
  • Inference latency of multi-step language rollouts: Generating multi-step textual rollouts sequentially still incurs multi-second latency per environmental step. Future work should investigate speculative decoding, latent-space world modeling, or parallel simulation engines to accelerate interactive inference.
  • vs RAP and MCTS Search-Based Planners: RAP and MCTS rely on tree search over discrete branching options, suffering from exponential token cost and sensitivity to single-step evaluation errors. ITP replaces exhaustive search with an adaptive-horizon linear rollout, cutting normalized budgets from 0.42 to 0.25 while outperforming RAP by +12.86% on ALFWorld.
  • vs Implicit World Model Tuning (IWM): IWM internalizes environment transitions via auxiliary representation loss during training. ITP shows that explicit, adaptively scaled rollouts provide indispensable structural feedback that substantially outperforms implicit internal representations on complex multi-turn tasks (+6.43% on ALFWorld).

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the POIMDP framework and pioneers adaptive lookahead horizon selection balancing progress against rollout cost.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4 diverse benchmarks, 3 open-source LLM families, both training-free and RL regimes, with rigorous budget-matched controls.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical exposition, compelling narrative flow, and informative empirical diagrams.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly practical and theoretically sound blueprint for liberating autonomous agents from myopic trial-and-error behaviors.