Agent-G^2: Gaussian Guidance for Agentic Reinforcement Learning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ZJU-REAL/Agent-G2
Area: LLM Agent
Keywords: Agentic Reinforcement Learning, Hint Guidance, Gaussian Schedule, Sparse Rewards, Group Relative Policy Optimization
TL;DR¶
To resolve the structural mismatch of scalar hint depths in long-horizon agent RL with sparse rewards, Agent-G^2 reveals that informative guidance depths form a Gaussian band around the optimum and introduces an online Gaussian schedule that dynamically updates depth distributions using regular policy rollouts with zero probe overhead.
Background & Motivation¶
Reinforcement learning applied to large language model agents has shown notable promise across complex, long-horizon decision-making tasks such as web navigation, embodied control, and scientific exploration. However, these environments typically require dozens of sequential actions while issuing only a binary success signal at termination. Consequently, on-policy exploration from the initial state rarely hits a successful terminal state during early training, causing severe advantage collapse in GRPO-style algorithms. Hint-based reinforcement learning mitigates this reward sparsity by retaining a prefix of an expert demonstration before each rollout, allowing the policy to explore from an intermediate state closer to completion.
Nevertheless, the effectiveness of hint-based RL hinges critically on the guidance depthโspecifically, the fraction of the expert trajectory retained. Too little guidance fails to yield successful exploration rollouts, whereas excessive guidance saturates the reward signal and eliminates the advantage contrast required for gradient updates. Existing approaches uniformly treat this guidance depth as a deterministic scalar: scheduled methods decay or adapt a single depth globally across a batch, ignoring high per-task difficulty heterogeneity; per-sample probing methods search for individual depths via binary search or grid enumeration, incurring prohibitive \(O(\log n)\) probe rollout overheads that remain noisy under constrained budgets. Crucially, both paradigms rely on the flawed assumption that an optimal guidance depth is a single, isolated scalar point. Diagnostic analysis demonstrates that informative guidance depths actually occupy a broad band around the optimal point, exhibiting an informativeness profile well fit by a Gaussian distribution (\(\sigma=0.22, R^2=0.92\)).
Recognizing that useful guidance is a neighborhood to be covered rather than a point to be localized, this paper develops Agent-G^2. Core idea: Model guidance depth as a per-task Gaussian distribution whose center and spread are adaptively updated online from regular policy rollouts via global-local decomposition, achieving robust coverage of the informative depth band with zero additional probe rollouts.
Method¶
Overall Architecture¶
Agent-G^2 operates within a five-stage closed loop designed for long-horizon agent RL with sparse terminal rewards: offline difficulty clustering, adaptive Gaussian scheduling, per-task depth sampling and execution, joint policy optimization, and schedule feedback. First, tasks are partitioned offline into \(K\) difficulty clusters based on expert trajectory length. During each training batch, the scheduler combines a global baseline \(\mu_{\text{global}}\) with cluster-level empirical success rates \(A_k\) and variances \(V_k\) to construct per-task Gaussian parameters \((\mu_i, \sigma_i)\). Next, a guidance ratio is sampled independently for each task and converted into an expert prefix length. The environment executes these prefix actions, after which the policy generates multiple parallel rollouts from the resulting post-prefix state. Finally, the terminal rewards are used both to update the policy via joint GRPO and prefix SFT losses and to refresh the Gaussian scheduler parameters for subsequent batches without extra probing.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Task Instruction & Expert Trajectories"] --> B["Offline Difficulty Clustering<br/>Partition into K clusters by expert length"]
B --> C["Adaptive Gaussian Schedule<br/>Global baseline + cluster success & variance"]
C --> D["Per-Task Gaussian Depth Sampling<br/>Sample prefix length and execute to intermediate state"]
D --> E["Policy Rollouts & Multi-Branch Exploration<br/>Perform R rollouts from post-prefix state for binary rewards"]
E --> F["Joint Policy Update<br/>Prefix SFT loss + trajectory GRPO loss"]
E -->|Reuse regular exploration rollout rewards| C
Key Designs¶
1. Offline Difficulty Clustering: Decoupling Prior Task Horizons Because shared-depth schedulers collapse heterogeneous tasks within a batch into a single scalar depth (such as a 2-step "Pick" vs. a 20-step "Pick Two" in ALFWorld), a single guidance ratio inevitably leaves easy tasks over-guided and hard tasks under-guided. Agent-G^2 uses the expert trajectory length \(L_i\) as an unobtrusive proxy for task difficulty, partitioning the training dataset offline into \(K\) difficulty clusters \(\{C_k\}_{k=1}^K\) via length quantiles (default \(K=3\), representing Short, Medium, and Long). This strategy requires no manual task-type labels or semantic embeddings, ensuring manageable within-cluster difficulty variation while preserving sufficient batch samples to compute reliable online statistics.
2. Adaptive Gaussian Schedule: Online Global-Local Decomposition To dynamically track informative guidance boundaries without exploratory overhead, the scheduler maintains a global baseline \(\mu_{\text{global}}\) alongside exponential moving average (EMA) estimates of success rate \(A_k\) and variance \(V_k\) for each cluster. The global baseline adjusts by a fixed step \(\Delta\) according to the divergence between batch success \(\text{acc}_B\) and the midpoint target \(p_{\text{target}}=0.5\): $\(\mu_{\text{global}} \leftarrow \text{clip}\left(\mu_{\text{global}} + \text{sign}(p_{\text{target}} - \text{acc}_B) \Delta, 0, 1\right)\)$ For task \(i\) in cluster \(k=k(i)\), batch empirical success \(\bar{p}_{B_k}\) and variance \(v_{B_k}\) update \(A_k\) and \(V_k\). The task-specific Gaussian mean \(\mu_i\) and spread \(\sigma_i\) are then calculated as: $\(\mu_i = \text{clip}\left(\mu_{\text{global}} + \lambda(p_{\text{target}} - A_k), 0, 1\right), \quad \sigma_i = \max(\gamma V_k, \sigma_{\min})\)$ When a cluster struggles, its mean shifts toward deeper prefixes; when task responses within a cluster disperse widely, the spread expands, mathematically guaranteeing broad coverage across the informative learning band.
3. Stochastic Depth Sampling & Zero-Cost Feedback: Replacing Probing with Distributional Coverage Instead of spending dozens of candidate probe rollouts to pinpoint an empirical optimum, Agent-G^2 draws a single guidance ratio \(z_i \sim \mathcal{N}(\mu_i, \sigma_i^2)\) clipped to \(r_i \in [0, 1]\) for each task. The prefix length \(n_i = \min(\lceil r_i L_i \rceil, L_i - 1)\) is executed once to position the agent at post-prefix state \(s_i\), from which the policy \(\pi_\theta\) conducts \(R\) parallel rollouts. Crucially, the empirical success rate \(\hat{p}_i\) obtained from these exact policy rollouts is recycled to refresh \(\mu_{\text{global}}\), \(A_k\), and \(V_k\). This closed-loop design eliminates separate probing phases, providing continuous adaptive coverage of the optimal neighborhood at the computational cost of standard schedule-based baselines.
Loss & Training¶
The overall training loss combines the standard Group Relative Policy Optimization (GRPO) objective with a teacher-forced auxiliary imitation loss evaluated over the sampled prefix actions: $\(\mathcal{L}(B) = \mathcal{L}_{\text{GRPO}}(B) + \eta \mathcal{L}_{\text{aux}}(B)\)$ where the auxiliary imitation term is formulated as: $\(\mathcal{L}_{\text{aux}}(B) = - \sum_{i \in B} \sum_{t=1}^{n_i} \log \pi_\theta(a_{i,t}^\star \mid s_{i,t}, q_i)\)$ The hyperparameter weight \(\eta=0.5\) preserves imitation supervision on prefix tokens while GRPO updates policy advantages from post-prefix rollouts. Across benchmarks, the global baseline step is \(\Delta=0.1\), EMA factor \(\alpha=0.2\), scaling coefficients \(\lambda=1.0\) and \(\gamma=1.0\), spread floor \(\sigma_{\min}=0.1\), and initial baseline \(\mu_{\text{global}}^{(0)}=0.8\). Training is executed in the verl framework across 8 NVIDIA H800 GPUs using the AdamW optimizer (peak learning rate \(1\times 10^{-5}\), cosine decay with 10% warmup).
Key Experimental Results¶
Main Results¶
Evaluation is conducted on the embodied environment ALFWorld (comprising Short, Medium, and Long tasks across 6 categories) and the web interaction environment WebShop, benchmarked on Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct backbones.
| Base Model | Paradigm | Method | ALFWorld All (Succ %) | WebShop Score | WebShop Succ (%) |
|---|---|---|---|---|---|
| Qwen2.5-1.5B | Imitation | Full SFT | 56.3 | 86.6 | 69.5 |
| Qwen2.5-1.5B | RL w/o Hints | GRPO | 72.8 | 75.8 | 56.8 |
| Qwen2.5-1.5B | RL w/o Hints | GiGPO | 86.1 | 83.1 | 65.0 |
| Qwen2.5-1.5B | RL w/o Hints | BEACON | 91.4 | 86.1 | 75.6 |
| Qwen2.5-1.5B | Aux-RL | RLVMR | 87.9 | โ | โ |
| Qwen2.5-1.5B | Hint-RL (Schedule) | Step decay | 89.8 | 89.1 | 74.2 |
| Qwen2.5-1.5B | Hint-RL (Schedule) | Target acc | 93.8 | 86.4 | 75.0 |
| Qwen2.5-1.5B | Hint-RL (Search) | Binary Search | 85.2 | 89.7 | 77.3 |
| Qwen2.5-1.5B | Hint-RL (Search) | Enumeration | 86.0 | 90.1 | 78.1 |
| Qwen2.5-1.5B | Hint-RL (End-to-End) | StepHint | 77.3 | 87.6 | 71.9 |
| Qwen2.5-1.5B | Hint-RL (End-to-End) | TRAPO | 59.4 | 89.4 | 75.8 |
| Qwen2.5-1.5B | Ours | Agent-G^2 (Ours) | 95.3 | 92.3 | 78.9 |
| Qwen2.5-7B | RL w/o Hints | BEACON | 94.5 | 87.7 | 79.7 |
| Qwen2.5-7B | Hint-RL (Schedule) | Target acc | 92.9 | 89.3 | 79.7 |
| Qwen2.5-7B | Hint-RL (Search) | Enumeration | 96.1 | 96.0 | 89.8 |
| Qwen2.5-7B | Ours | Agent-G^2 (Ours) | 98.4 | 92.3 | 84.4 |
Ablation Study¶
Ablations on Qwen2.5-1.5B-Instruct / ALFWorld isolate the contributions of sampling mechanisms, adaptive moments, clustering, and loss components:
| Ablation Dimension | Configuration | Long Task Succ (%) | All Tasks Succ (%) | Relative Change \(\Delta\) |
|---|---|---|---|---|
| Full Model | Agent-G^2 (Full) | 94.7 | 95.3 | โ |
| Sampling Form | Deterministic mean (\(d_i=\mu_i, \sigma_i=0\)) | 84.6 | 89.8 | -5.5 |
| Sampling Form | Variance-matched uniform (\(\mathcal{U}[\mu_i \pm \sigma_i]\)) | 80.8 | 88.3 | -7.0 |
| Adaptive Moments | Fixed global center (\(\mu_i=\mu_{\text{global}}\)) | 79.2 | 91.4 | -3.9 |
| Adaptive Moments | Fixed spread floor (\(\sigma_i=\sigma_{\min}\)) | 78.3 | 93.8 | -1.5 |
| Clustering Design | Single cluster (\(K=1\)) | 65.4 | 89.1 | -6.2 |
| Loss Components | w/o auxiliary loss (\(\text{w/o } \mathcal{L}_{\text{aux}}\)) | 76.7 | 86.7 | -8.6 |
| Loss Components | w/o RL exploration (\(\text{w/o } \mathcal{L}_{\text{GRPO}}\)) | 10.0 | 26.6 | -68.7 |
Regarding wall-clock training efficiency on Qwen2.5-1.5B / ALFWorld, Agent-G^2 takes 88s per gradient step (1.00ร ratio); in contrast, probe-based methods like Binary Search require 285s (3.24ร) and Enumeration takes 425s (4.83ร).
Key Findings¶
- Stochastic Coverage Beats Point Optimization: Collapsing Gaussian sampling to the deterministic mean reduces overall success by 5.5 points, while variance-matched uniform sampling drops performance by 7.0 points, confirming that Gaussian center-density matching is essential for staying within the just-learnable \([0.4, 0.6]\) success band.
- Long-Horizon Tasks Demand Cluster Adaptivity: Disabling clustering (\(K=1\)) drops overall performance by 6.2 points, but severely degrades Long tasks (Pick2) by 29.3 points (94.7% \(\to\) 65.4%); removing adaptive spread (\(\sigma_i=\sigma_{\min}\)) also triggers a steep drop on Long tasks (94.7% \(\to\) 78.3%).
- Small Backbone Outperforms Larger Baselines: The 1.5B Agent-G^2 model (95.3% on ALFWorld, 92.3 reward score on WebShop) consistently outperforms all 7B non-probing baselines (e.g., 7B BEACON at 94.5% / 87.7 score), demonstrating that principled guidance scheduling can substitute for model scale expansion.
Highlights & Insights¶
- Reconceptualizing Guidance as Neighborhood Coverage: The paper challenges the dogma of pinpointing a single optimal depth via noisy probes, demonstrating that training utility forms a Gaussian informativeness curve and that stochastic neighborhood sampling effectively resolves rollout selection variance.
- Zero-Rollout-Overhead Adaptive Feedback: By updating Gaussian centers and widths directly from regular policy rollouts, the framework provides instance-specific adaptivity without paying the steep \(O(\log n)\) probe rollout penalties of search methods.
- Lightweight Horizon-Based Scaffolding: Using demonstration trajectory lengths as an offline difficulty proxy establishes a simple, label-free clustering baseline that generalizes cleanly across different agent domains.
Limitations & Future Work¶
- Dependency on High-Quality Expert Demonstrations: The framework requires at least one successful demonstration trajectory per training task; extending it to suboptimal rollouts or natural language hints remains open.
- Static Offline Clustering: Task clusters are fixed by initial trajectory lengths and do not dynamically adapt as the policy masters specific tasks during training.
- Parametric Gaussian Assumption: Although the Gaussian parameterization fits the observed empirical profile well on ALFWorld and WebShop, highly skewed or multimodal environments may require richer distribution families.
Related Work & Insights¶
- vs Schedule-based Hint RL (Linear / Cosine / Target acc): Traditional schedulers broadcast a single depth across all batch tasks, inducing severe under- or over-guidance on heterogeneous tasks; Agent-G^2 resolves this via cluster-aware Gaussian decomposition.
- vs Search-based Hint RL (Binary Search / Enumeration): Search-based methods isolate depths via expensive, noisy probe rollouts (consuming 3รโ5ร wall-clock training time); Agent-G^2 eliminates extra probing by sampling from an online-updated parametric distribution.
- vs Auxiliary Supervision RL (RLVMR / ETO / PRM): Prior methods train secondary value networks, PRMs, or verifier heads; Agent-G^2 utilizes raw expert trajectories directly within a unified GRPO objective, avoiding auxiliary model training complexities.
Rating¶
- Novelty: โญโญโญโญโญ Formulates hint depth allocation as Gaussian neighborhood coverage rather than deterministic scalar search.
- Experimental Thoroughness: โญโญโญโญโญ Thorough evaluations across two benchmarks, two parameter scales, in-depth diagnostic profiles, and complete efficiency metrics.
- Writing Quality: โญโญโญโญโญ Rigorous motivation, coherent mathematical derivations, and transparent empirical analyses.
- Value: โญโญโญโญโญ Delivers an efficient, plug-and-play guidance framework for long-horizon agent RL with sparse rewards.