Skip to content

ProAct: Agentic Lookahead in Interactive Environments

Conference: ECCV2026
Paper: ECCV
Code: https://github.com/GreatX3/ProAct
Area: LLM Reasoning
Keywords: LLM agent, lookahead reasoning, Monte-Carlo tree search, multi-turn reinforcement learning, value estimation

TL;DR

ProAct moves lookahead out of inference time and into data construction: it first probes the real environment with MCTS and compresses the resulting futures into observation-analysis-conclusion causal chains for SFT (GLAD), then stabilizes PPO/GRPO training with a parameter-free Monte-Carlo Critic (MC-Critic) that estimates values from cheap random-policy rollouts, letting a 4B model substantially outperform open-source baselines of the same and larger scale on 2048 and Sokoban.

Background & Motivation

Long-horizon interactive tasks — games such as 2048 and Sokoban that demand hundreds of consecutive decisions — require human-like System 2 processing: mentally simulating several futures and comparing them before committing to a move. Chain-of-Thought and ReAct do make LLM reasoning longer, but their simulation ground is the model's own world model: the model describes in language what the board will look like after a move. The trouble is that this internal world model does not agree with the true environment transition function \(P(s'|s,a)\). In short-horizon tasks the discrepancy can be absorbed by later steps; in long-horizon tasks it surfaces, because each step's small prediction error is treated as fact by the next step, errors compound exponentially along the lookahead depth, and the whole plan drifts away from reality. The paper names this phenomenon simulation drift. Its consequences cut both ways: making the reasoning chain longer (a "deeper" lookahead) merely amplifies hallucination and context drift, while retreating to single-step reasoning loses sight of long-term consequences altogether.

The core tension follows. Lookahead requires simulating the future, and the only faithful future simulator is the environment itself; yet querying the environment repeatedly at inference time — running BFS/MCTS at every decision point, as ToT and RAP do — is precisely the cost one cannot afford, since a single trajectory spans hundreds of turns. There is a second difficulty as well: even with good lookahead data, training an LLM agent with online RL is unstable. Policy gradients need a value function to judge how good a state really is, and traditional deep RL can train an accurate critic only because an MLP policy interacts with the environment thousands of times per second and accumulates millions of steps. An LLM, by contrast, must autoregressively generate hundreds or thousands of tokens for a single step: generating one reasoning chain plus action with a 4B model takes 3-6 seconds. With interaction throughput several orders of magnitude lower, critic value estimates stay high-variance and training repeatedly collapses.

ProAct's angle is to relocate the whole act of looking ahead: do it not at inference time but during data construction. The real environment serves as the oracle that reveals the futures, the model reads the true outcomes while writing its analysis, and the entire search tree is then compressed into a single token-efficient natural-language causal chain so that lookahead becomes one-step policy intuition. Value estimation is anchored the same way — instead of forcing a data-starved critic network to converge, a random policy runs a large batch of inexpensive Monte-Carlo rollouts in the environment to estimate state values. Core idea: lookahead happens in the real environment, is compressed in language, and is internalized in the policy, yielding accurate and stable long-horizon decision-making without paying search cost at inference time.

Method

Overall Architecture

The paper models LLM agent-environment interaction as a standard MDP \(\mathcal{M}=\langle \mathcal{S},\mathcal{A},P,R,\gamma\rangle\), but unlike conventional RL agents an LLM agent emits a reasoning chain \(z_t\) before its action, so the policy is written as a joint distribution over reasoning and action, \(\pi_\theta(z_t,a_t\mid s_t)=\pi_\theta(z_t\mid s_t)\cdot\pi_\theta(a_t\mid s_t,z_t)\) — i.e. deliberation followed by execution. ProAct's inputs are a policy model to be trained (Qwen3-4B-Instruct-2507) and a set of interactive environments that can be replayed and reset; its output is a lookahead-capable policy that needs no search at inference time. The framework runs in two serial stages.

Stage one, GLAD (Grounded LookAhead Distillation), both manufactures the lookahead and compresses it in. It runs MCTS in the real environment to probe several futures from the current state (retaining optimal paths as well as dead ends), reads those raw futures directly into the LLM's context for analysis and decision-making, and lets the model emit <BACKTRACK> to revert and re-probe when the analysis shows the current branch is worse than previously explored alternatives. Once an episode ends, a teacher model (or the model itself) rewrites each step's verbose search record into a compact causal reasoning chain, and standard SFT is performed on (state, compressed chain, action) triples. Stage two, MC-Critic, calibrates the lookahead against real returns: a parameter-free Monte-Carlo critic driven by random-policy rollouts supplies low-variance state/action values that replace or blend into the advantage estimates of PPO/GRPO for online RL fine-tuning. The flow below uses the same names, in the same order, as the Key Designs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Interactive environment<br/>2048 / Sokoban"] --> B["Environment-probed lookahead<br/>MCTS samples real futures"]
    B -->|current branch worse than probed ones| C["Backtrack and re-probe<br/>self-correction via BACKTRACK"]
    C --> B
    B --> D["Cognitive compression<br/>search tree to causal chain"]
    D --> E["SFT internalizes lookahead<br/>no search at inference"]
    E --> F["Monte-Carlo Critic<br/>cheap batched random rollouts"]
    F --> G["MC-GRPO / MC-PPO online RL<br/>low-variance advantage estimation"]
    G --> A

Key Designs

1. Environment-probed lookahead construction: let the real environment, not the model's world model, answer "what happens if I take this move"

Since the root cause of drift is that the model simulates the future internally, the most direct fix is to stop simulating — and externalize the lookahead into the environment. At each decision step \(t\), GLAD runs MCTS from the current state \(s_t\) in the real environment and samples \(N\) trajectories of length \(T\), \(\{\tau_1,\dots,\tau_N\}\). One detail matters a great deal: the probe keeps not only the optimal paths but also suboptimal paths and dead ends, because the dead ends are exactly where the supervision for "why this move is wrong" comes from. The raw trajectories are placed straight into the LLM's context as a "ground-truth future map." The model then produces two things: an Analysis comparing and simulating these futures (for instance, "Trajectory A leads to a merge, while Trajectory B leads to a gridlock"), and a Decision — the next action, or a special <BACKTRACK> token. When the analysis reveals that the current branch is worse than an alternative probed earlier, the model emits <BACKTRACK> and reverts the state to \(s_{t-1}\) to re-probe. This Probing-Decision-Reflection loop gives the data-collection phase its own self-correction ability; note also that the recovery process after backtracking is recorded into the dataset just like everything else, so what the model learns is how to climb out of a bad position, not merely to avoid bad moves.

What actually prevents hallucination and self-deception here is the provenance of the supervision signal. Every "future" the model writes in its analysis corresponds to a real transition it has just read in its context; it neither needs nor is permitted to guess \(P\) from an internal world model. Lookahead depth is set by the probe's \(T\) steps and branching width by the number of sampled trajectories \(N\), and both are compute knobs of data construction rather than costs at inference time — the main text does not report the specific \(T\) and \(N\) used for 2048/Sokoban (see Appendix A-C; ⚠️ refer to the original paper), though mechanically they determine how far and how wide the futures visible to supervision are. This is also exactly where it differs from plain CoT/ReAct: a ReAct lookahead is a monologue with no external verification, whereas a ProAct lookahead is a query that comes back with a receipt from the environment.

2. Cognitive compression: collapse a structurally tagged search tree into an observation-analysis-conclusion causal chain

The raw contexts produced by probing are long and dirty — full of search traces, structural tags, and backtracking steps. Fine-tuning directly on them is both expensive (very long tokens) and prone to overfitting to the format: the model learns to recite <search> tags instead of reading the position. The paper therefore adds a compression step, in which a teacher model (or the model itself) synthesizes the raw context into the final reasoning path \(z\) under four enforced principles. Format simplification strips all structural artifacts and rewrites everything in natural language ("Let's analyze the board...", "If I move up, the tiles will merge...") to align with the pretrained distribution. Explicit reasoning chains require every step to follow a strict observation → analysis → conclusion logic, with the analysis explicitly linking the current action to future states based on the environment rules actually observed during probing. Future trend estimation requires the compressed reasoning to explain not only why the chosen action was chosen but why the others were rejected — "moving left is safe now but blocks a critical merge in the future." Preserve diversity requires the reasoning to retain the trade-offs present in the search, in the voice of "Option A is good for score, but Option B provides better safety; considering the long term, I choose B," rather than dogmatically announcing an answer. SFT then minimizes the standard negative log-likelihood over \(\mathcal{D}=\{(s,z_{\text{compressed}},a)\}\).

The third and fourth principles are the load-bearing ones. Forcing the model to explain why the rejected actions were rejected is in effect forcing counterfactual reasoning: it has to distill transferable environment dynamics ("this cell is a dead corner; you pay for filling it sooner or later") out of genuinely observed trajectories instead of memorizing "on a board like this, move up." Preserving the trade-offs keeps the policy from collapsing into a slogan, which is another solution to the same worry that RAGEN captures with its "Echo Trap." Compared with approaches such as VAGEN and WALL-E that explicitly force the generation of world-model states, ProAct distills neither the search tree itself nor labeled state estimates, but the conclusion of the search — a tree becomes one chain, inference-time token cost matches ordinary CoT, and the content is calibrated on ground-truth transitions.

3. Monte-Carlo Critic: replace an untrainable value network with cheap random-policy rollouts

The second stage addresses value estimation. The paper's diagnosis is blunt: bad critics in LLM agent RL are not an algorithmic problem but a sample-throughput problem — an MLP policy in traditional deep RL grinds out millions of interaction steps, so its critic is accurate, whereas an LLM spends hundreds or thousands of tokens per step, so the sampling rate is orders of magnitude lower, the trained critic is high-variance, and that in turn destabilizes training. MC-Critic simply refuses to train a parametric critic: the value of a state is defined directly as the average discounted return of \(M\) trajectories launched from it.

\[V^{\text{MC}}_{\pi_{\text{random}}}(s_t)=\frac{1}{M}\sum_{i=1}^{M}\sum_{k=0}^{T-1}\gamma^{k}r^{i}_{k+t}\]

That estimator is unbiased and its variance shrinks with \(M\) — provided the rollouts use the current policy \(\pi_\theta\), but generating a single step's reasoning plus action with a 4B model takes 3-6 seconds, so rolling out \(M\) trajectories per state is simply infeasible in wall-clock terms. The paper therefore makes a deliberate trade: it uses a random policy \(\pi_{\text{random}}\) as a surrogate for \(\pi_\theta\) to produce those \(M\) rollouts. The resulting \(V^{\text{MC}}_{\pi_{\text{random}}}\) is theoretically inferior to \(V^{\text{MC}}_{\pi_\theta}\), but it is extremely cheap — in 2048 the random policy rolls out more than 1,000 trajectories in under 3 seconds, faster than the LLM generates a single step. This is a distinctly systems-level observation: an LLM step and an environment step differ in cost by several orders of magnitude, so use cheap repetition on the environment side to substitute for expensive estimation on the model side. Crucially, although the random policy's returns understate what an optimal policy could achieve, they are determined entirely by the true environment dynamics, so its verdict on "how objectively good this state is" cannot deceive itself the way a learned critic can.

There are two ways to plug it into policy gradients. For GRPO (MC-GRPO), the skeleton is Step-GRPO: roll out one trajectory, store every visited state in a state pool, then at each training step draw a batch of states from the pool and sample \(G\) independent single-step samples \(s_{t_u},c^i_{t_u},r^i_{t_u}\) per state — but the advantage no longer uses the immediate step reward; it uses the action value estimated by MC-Critic,

\[Q^{\text{MC}}_{\pi_{\text{random}}}(s_{t_u},a^i_{t_u})=r^i_{t_u}+\gamma V^{\text{MC}}_{\pi_{\text{random}}}(s_{t_u+1})\]

followed by group normalization. A pitfall the paper hit itself is worth recording: if the \(G\) sampled actions within a group happen to be all identical, their \(Q\) values are identical too, group normalization yields zero advantage for the whole group, and that state never receives gradient again. The common remedy (DAPO-style dynamic sampling) discards such samples; ProAct instead keeps them — when all actions coincide, the baseline switches from the group mean to the mean of \(Q\) over every action in the whole action space \(\mathcal{A}\) (an absolute baseline), so even with identical samples the state still gets positive advantage whenever that action beats the action-space average. For PPO (MC-PPO), the base is Step-PPO: to avoid concatenating the full history in multi-turn settings and blowing the context limit (e.g. 32,768 tokens), Step-PPO lets the agent see only the current state, discards history, and computes GAE with a turn-level critic \(V_\phi\). MC-PPO then forms a convex combination of the MC value and the critic value, \(V^{\text{MC-PPO}}(s_t)=(1-\omega)V_\phi(s_t,c_t)+\omega V^{\text{MC}}_{\pi_{\text{random}}}(s_t)\) with \(\omega\in[0,1]\) the weight on the MC value (restated from Eq. (18); ⚠️ refer to the original paper), letting a low-variance external signal backstop the higher-variance learned critic. MC-Critic is therefore fully plug-and-play: it adds no trainable parameters, and any RL algorithm that needs state-value estimation can mount it.

A Worked Example: how one 2048 decision is rewritten by lookahead

Take 2048, the example the main text itself uses. Suppose the board is in state \(s_t\) and the model is about to move. Step one, probing: MCTS runs in the real game from \(s_t\) and samples several futures of length \(T\) (exact counts and depth are in the appendix; ⚠️ refer to the original paper), among them a Trajectory A that slams two equal tiles together for a merge and a Trajectory B that drives the board into a deadlock where nothing can merge again. Step two, deciding: these real futures are written into the context, the model produces the analysis — "Trajectory A leads to a merge, while Trajectory B leads to a gridlock" — and picks an action; if what it reads is instead "this branch is worse than the one discarded last step," it emits <BACKTRACK> and reverts to \(s_{t-1}\). Step three, compressing: that long context, full of search tags and backtrack records, is rewritten into a human-sounding chain — "let's analyze the board; if I move up the tiles merge, but then this corner gets locked; moving left is safe now but blocks a critical merge later; so up is better here." Step four, internalizing and calibrating: the model is SFT-trained on (that chain, the chosen action) to acquire the intuition of saying all this without searching; in the RL phase, when \(s_{t_u}\) is drawn from the state pool, \(G\) single-step samples are taken and the batch of Monte-Carlo returns from the random policy prices each candidate action — if the \(G\) actions differ, the group-relative baseline applies; if they are all identical, the absolute action-space baseline takes over, and the gradient still has something to work with.

Loss & Training

Stage one is pure supervised learning: minimize the negative log-likelihood of the reasoning chain and action on the GLAD dataset — 25K trajectories for 2048 and 8K for Sokoban, all on the Qwen3-4B-Instruct-2507 backbone. Stage two is online RL, and both configurations were tried: RL fine-tuning from the GLAD checkpoint, and training from scratch directly from the base instruction-tuned model without any GLAD supervision. MC-GRPO's loss is item-for-item identical to Step-GRPO's; the only difference is the advantage computation, which swaps the immediate step reward for the MC-Critic action value before group/action-space normalization, thereby shifting the objective from this step's immediate gain toward the long-term return this step leads to. MC-PPO's loss is Step-PPO's clipped policy loss plus a value loss, with the value target replaced by the convex combination above. For evaluation, 2048 uses the cumulative tile-merge score and Sokoban the average number of successful box pushes.

Key Experimental Results

Main Results

Evaluation covers two long-horizon decision-making benchmarks: 2048 is stochastic, with hundreds of turns per trajectory, requiring planning under uncertainty; Sokoban is a deterministic planning task with shorter trajectories but sparse rewards. Together they stress simulation drift from complementary directions. Columns marked * denote environment variants strictly unseen during both SFT and RL training.

Model 2048 4×4 2048 3×3* 2048 3072* Sokoban Unseen Sokoban Action* Sokoban Symbol*
GPT-5 4040.0 1184.0 6962.0 1.89 1.83 1.94
Claude-4.5-Sonnet 166.7 109.3 120.0 1.06 0.78 1.33
Doubao-Seed-1.6 1877.3 300.0 2054.0 1.22 0.61 1.56
Doubao-Seed-1.8 4662.7 545.3 4210.0 1.80 1.44 2.00
UI-TARS-1.5 2616.0 466.7 3920.0 0.44 0.44 0.39
Qwen3-235B-A22B 634.7 274.7 2688.0 0.61 0.33 0.56
Qwen3-30B-A3B 838.7 230.7 1740.0 0.44 0.39 0.50
Qwen3-4B (base) 721.3 187.3 1603.0 0.39 0.44 0.56
Base + GLAD (ours) 3335.3 429.2 4565.7 0.72 0.52 0.67
Base + GLAD + MC-Critic (ours) 4503.8 464.4 6013.7 0.94 0.60 0.70

Ablation Study

MC-Critic is ablated in two complementary settings: (i) RL fine-tuning starting from the GLAD checkpoint, and (ii) training from scratch from the base model with no GLAD supervision. The table reports both settings on environment variants; results on the standard training distribution are given only as curves in the main text (Fig. 3) without numbers.

Method 2048 3×3 2048 3072 Sokoban Unseen-RL Sokoban Action Sokoban Symbol
Trained from GLAD SFT checkpoint
Traj-GRPO 459.0 5457.0 1.10 0.60 0.64
Step-GRPO 457.0 5820.3 1.00 0.56 0.63
Step-PPO 487.3 6248.7 0.90 0.62 0.65
MC-GRPO 464.4 6013.7 1.05 0.60 0.70
MC-PPO 465.0 5818.1 1.18 0.62 0.68
Trained from scratch
Traj-GRPO 202.0 1760.7 0.50 0.73 0.90
Step-GRPO 188.0 1792.9 0.25 0.53 0.86
Step-PPO 202.7 1912.9 0.45 0.55 0.88
MC-GRPO 194.2 1754.7 0.55 0.80 1.03
MC-PPO 239.8 2229.1 0.53 0.96 0.98

Key Findings

  • GLAD's gain is implausibly large for a single SFT step: on the standard 2048 board it lifts Qwen3-4B from 721.3 to 3335.3 (roughly 4.6×) and surpasses Qwen3-235B-A22B (634.7) and Qwen3-30B-A3B (838.7), two models two orders of magnitude larger. More striking still, scale barely helps in these two games — both larger Qwen3 models score below the 4B base on 2048. The bottleneck is not parameter capacity but whether the lookahead supervision has been calibrated by the environment, which is precisely the paper's thesis.
  • MC-Critic helps under both initializations, but the gain is largest when training from scratch: from the base model, MC-PPO reaches 239.8/2229.1 on 2048 3×3/3072 against Step-PPO's 202.7/1912.9, and on Sokoban Action 0.96 versus 0.55 — nearly double. On top of the GLAD checkpoint, however, the ablation is not one-sided: Step-PPO (487.3/6248.7) still leads on 3×3 and 3072, while MC-PPO leads only on Sokoban Unseen-RL (1.18) and MC-GRPO only on Symbol (0.70). A plausible reading (not argued by the paper) is that GLAD already supplies a strong long-horizon prior, compressing the marginal contribution of the MC value; the two mechanisms overlap functionally to some degree.
  • Trajectory-level GRPO destabilizes on long-horizon tasks, and MC-GRPO holds: Sokoban training levels are mostly solved in a few steps, so treating a whole trajectory as one training sample incurs little variance, and Traj-GRPO performs comparably to MC-GRPO there. Once trajectories get long (2048 spans hundreds of turns), return variance accumulates, Traj-GRPO visibly degrades, and MC-GRPO stays stable and keeps improving. This is exactly the value of refining the advantage from "one shared value per trajectory" to "one MC estimate per state."
  • Hyper-parameter recipe (\(M\) and \(T\)): \(M\) is the number of rollouts per state and \(T\) the maximum steps per rollout. The paper's conclusion is qualitative — for dense-reward environments (2048) \(M\) should be as large as interaction efficiency allows; for sparse-reward environments (Sokoban) \(M\) should not be excessive, and a smaller \(M\) can work better; \(T\) should be set to the average number of steps of a successful trajectory and should not be too large. Concrete values and curves live in Appendix D and are absent from the main text (⚠️ refer to the original paper).
  • The headline claim needs a discount: the abstract says the 4B model "rivals state-of-the-art closed-source models," and that holds on 2048 — 4503.8 against GPT-5's 4040.0, and 6013.7 against Doubao-Seed-1.8's 4210.0. On Sokoban the gap remains wide: on unseen levels 0.94 against GPT-5's 1.89 and Doubao-Seed-1.8's 1.80, roughly half. A more accurate statement is that it matches the closed-source first tier on the stochastic, long-horizon, lookahead-hungry 2048, while clearly trailing on the deterministic, short-horizon, sparse-reward Sokoban.
  • ⚠️ A small consistency issue: Tab. 1 gives 0.94 for "Base + GLAD + MC-Critic" on Sokoban Unseen, while Tab. 2 gives 1.05 for MC-GRPO on Sokoban Unseen-RL, which appear to be the same configuration. The likely explanation is different held-out splits (Tab. 1's Unseen is unseen in SFT; Tab. 2's Unseen-RL is unseen in both SFT and RL), but the paper does not say so explicitly — refer to the original paper when citing.

Highlights & Insights

  • Externalize the lookahead instead of lengthening the reasoning: the ToT/RAP route searches harder at inference time, whereas ProAct keeps search only in data construction and distills merely the conclusion of the search rather than the search tree itself. This search-then-compress inversion keeps inference-time token cost at ordinary-CoT level while the content is calibrated on ground-truth transitions — the most transferable idea in the paper.
  • Use the rejected actions as supervision: the compression principle demanding an explanation of why other actions were rejected looks like a style guideline, but it is what actually forces the model to internalize environment dynamics. Any process-supervision or reasoning-distillation work can borrow it: teaching only "what to choose" degrades into corpus memorization, whereas teaching "why not the others" forces out transferable decision rules.
  • Swap the critic for many rollouts of a cheap policy: MC-Critic's insight is not mathematical (Monte-Carlo value estimation is old) but systemic — an LLM step (3-6 s) and an environment step (>1,000 trajectories in under 3 s) differ by orders of magnitude, so use cheap repetition on one side to replace expensive estimation on the other. Any setting that needs a value function for an LLM agent (GUI agents, tool use, multi-turn dialogue RL) can lift this "estimate with a cheap surrogate policy" pattern directly.
  • The absolute baseline for degenerate groups: when all \(G\) sampled actions coincide, the advantage vanishes — an easy-to-miss detail that most methods handle by discarding the samples; this paper instead uses the mean \(Q\) over the whole action space as the baseline, salvaging the learning signal. It is a small but plug-and-play trick usable by any GRPO-family variant.

Limitations & Future Work

  • Only two grid games are evaluated: the paper tests 2048 and Sokoban exclusively, both cheap, infinitely resettable, and fast to simulate — precisely the regime where GLAD's probing is cheapest and MC-Critic's random rollouts are fastest. The motivation repeatedly mentions web, embodied, and long-horizon interaction, yet no GUI, embodied, or tool-use environment is validated. Whether GLAD's and MC-Critic's core premises survive when probing is expensive or the environment cannot be reset (real web pages, physical robots) is the biggest open question.
  • The random-policy surrogate is an acknowledged but unquantified bias: the paper admits \(V^{\text{MC}}_{\pi_{\text{random}}}\) is theoretically inferior to \(V^{\text{MC}}_{\pi_\theta}\) but never measures how large the bias is, nor compares against spending more time rolling out with \(\pi_\theta\), training a critic longer, or using an intermediate surrogate such as an \(\varepsilon\)-greedy or temperature-sampled policy. Since surrogate quality directly determines value-signal quality, this design space deserves a systematic sweep.
  • The Step variants discard history: Step-PPO/Step-GRPO let the agent see only the current state to avoid context overflow, which directly loses information in partially observable tasks. The paper notes this as an engineering compromise without ablating "sliding-window history versus no history."
  • The faithfulness of the compressed chains is never tested: after SFT the model writes elegant trade-offs and counterfactuals, but the paper never measures whether the stated reasons actually correspond to the action taken. There is a risk it learned a confident tone of weighing options rather than genuinely internalized dynamics.
  • The hyper-parameter recipe is too qualitative: "\(M\) as large as possible" and "\(T\) at the average successful-trajectory length" are directional suggestions, with all quantitative results deferred to Appendix D, leaving the main text unable to stand alone for reproduction. A cross-environment \(M\)–\(T\) sensitivity heatmap would make the recipe far more usable.
  • Concrete improvement directions: upgrading the surrogate from pure random to a temperature-sampled version of the current policy with importance-weight correction (\(\pi_\theta/\pi_{\text{random}}\) weighting) could remove part of the bias at little extra cost; replacing GLAD's MCTS probe with a cheaper model-guided search could push the method into domains where probing is expensive.
  • vs CoT / ReAct: they lengthen the reasoning chain to approximate System 2, but the future is always simulated internally, so simulation drift accumulates over long horizons and more deliberation means more hallucination. ProAct has the environment supply the future and lets the model merely read and compare, cutting drift off at the data stage.
  • vs ToT / RAP: these attach explicit search (BFS/MCTS) at inference time; effective, but test-time cost explodes with trajectory length and cannot sustain hundreds of interaction steps. ProAct relocates search to data construction, compresses it into one chain internalized in the parameters, and pays nothing at inference.
  • vs VAGEN / WALL-E: they explicitly force the model to generate world-model states (state estimation, dynamics simulation) or align via neurosymbolic learning. ProAct argues there is no need to clone verbose search traces or explicit state labels, and instead compresses into a single natural-language future trend estimate while preserving the diversity of deliberation — what is distilled is judgment, not intermediate artifacts.
  • vs ArCHer / SWEET-RL / Turn-PPO: all three use parametric critics (hierarchical utterance-level value, asymmetric critic with privileged information, turn-level critic + GAE). ProAct keeps Step-PPO's turn-level critic but mixes in a parameter-free MC value; on the GRPO route it replaces the immediate step reward with an MC action value outright, refining credit assignment from "shared across the whole trajectory" to "per state/action."
  • vs RAGEN / AgentGym-RL: those address the infrastructure and stability of multi-turn RL (Echo Trap, StarPO, curriculum-based horizon scaling). ProAct targets the quality of the agent's internal reasoning process and the provenance of the value signal; the two are orthogonal and can be combined.

Rating

  • Novelty: ⭐⭐⭐⭐ Moving lookahead from inference time to data construction, and replacing a parametric critic with many random-policy rollouts, both hit real pain points of LLM agent RL; the combination is new.
  • Experimental Thoroughness: ⭐⭐⭐ Only two environments, with the main results leaning on black-box comparisons against closed APIs such as GPT-5 and Doubao, the \(M\)/\(T\) analysis deferred to the appendix, and one unclear discrepancy between the two tables; still, the GLAD/MC-Critic ablation covers both "from scratch" and "from the GLAD checkpoint" settings, so the structure is complete.
  • Writing Quality: ⭐⭐⭐ The two-stage storyline and the method motivation are clear and the derivations are complete (every GRPO/PPO variant is spelled out), at the cost of a main text crowded with LaTeX and variant naming, plus key hyper-parameters always pointing to the appendix.
  • Value: ⭐⭐⭐⭐ The combination of "environment as oracle + compress into a chain + cheap MC values" is directly transferable to any effort at long-horizon LLM agent training, and a 4B model beating a 235B model on 2048 shows how cost-effective this route can be.