Dynamic Dual-Granularity Skill Bank for Agentic RL¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/TU2021/D2Skill-AgenticRL
Area: Large Language Model Agent (LLM Agent)
Keywords: Agentic Reinforcement Learning / Dual-Granularity Skill Bank / Hindsight Utility Estimation / Dynamic Memory Management / Embodied & Web Decision-Making
TL;DR¶
To overcome the limitations of trajectory-only guidance and unbounded memory degradation in agentic reinforcement learning, D2Skill organizes reusable experience into macro task skills and micro step skills, leveraging paired on-policy rollouts for hindsight utility estimation, two-stage UCB retrieval, and capacity-aware pruning.
Background & Motivation¶
Agentic reinforcement learning (RL) has emerged as a promising paradigm for training language-based agents on complex, long-horizon decision-making tasks spanning embodied interactive environments, web-based tool invocation, and multi-hop search. In these domains, agents interact through text-based observations and bounded interaction history windows, rendering the decision space severely partially observable. Compounded by sparse terminal rewards and vast action spaces, conventional RL agents suffer from inefficient exploration and intractable credit assignment. To break this impasse, non-parametric memory stores and reusable skill banks have emerged as vital vehicles for distilling experiential trajectories into transferable knowledge across tasks.
However, existing skill-based and memory-augmented agentic architectures (such as SkillRL and ExpeL) remain constrained by two fundamental bottlenecks. First, existing methods typically generate skills by reflecting over entire completed trajectories, yielding macro-level guidance that outlines global plans or general workflows. While such high-level guidance keeps global exploration on track, it fails to provide concrete corrective cues when agents encounter unexpected observation anomalies at individual decision stepsโsuch as repeatedly checking an empty cabinet in ALFWorld after the target object is not found. Second, as training proceeds across thousands of steps, external skill banks expand monotonically, accumulating hyper-specific, redundant, or obsolete skills. Without principled utility evaluation and eviction mechanisms, this unbounded memory expansion degrades retrieval quality, clutters prompt budgets with low-utility guidance, and misguides policy updates.
Fundamentally, failures in long-horizon agentic decision-making naturally exhibit hierarchical duality: global planning drift and local step-wise execution breakdowns. Furthermore, externalized memory accumulation cannot remain static or unbounded; it demands disciplined causal utility quantification and active life-cycle governance. Core idea: organize reusable agent experience into macro task skills for global guidance and micro step skills for local error recovery, and deploy on-policy paired baseline and skill-injected rollouts to derive hindsight utility signals that co-evolve policy learning with dynamic retrieval and capacity pruning.
Method¶
Overall Architecture¶
D2Skill couples policy optimization with a dynamic, dual-granularity external skill bank throughout the reinforcement learning lifecycle. At each training step, a sampled task group generates paired parallel rollouts under the identical policy model: half of the trajectories execute in a baseline setting without external skills, while the other half are conditioned on retrieved dual-granularity skills. The terminal performance gap between these two groups supplies unbiased hindsight credit signals to update the empirical utility of task- and step-level skills, while simultaneously providing an intrinsic reward shaping term for policy optimization. When task success drops below a reflection threshold, an introspective reflector synthesizes new task and step skills from failed trajectories, which are deduplicated and indexed. Finally, a two-stage UCB-based retrieval pipeline and capacity-aware pruning protocol dynamically govern skill retention and usage.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Environment Interaction & Context<br/>Task description and history window"] --> B["Paired Rollouts & Hindsight Utility Estimation<br/>On-policy baseline vs. skill-injected comparison"]
B --> C["Failure-Driven Dual-Granularity Skill Generation<br/>Global task workflow and first-error step recovery"]
C --> D["Utility-Aware Retrieval & Dynamic Pruning<br/>Two-stage semantic UCB ranking and capacity eviction"]
D --> E["GRPO Policy Optimization<br/>Skill-augmented returns and advantage normalization"]
Key Designs¶
1. Paired Rollouts & Hindsight Utility Estimation: Intractable Multi-Skill Credit Assignment -> Counterfactual Baseline Comparison -> Low-Overhead Online Utility Valuation
Determining the exact marginal causal contribution of individual language skills injected into an interaction trajectory is computationally prohibitive. D2Skill bypasses heavy value estimation by sampling \(N\) parallel trajectories per task under the active policy, dividing them evenly into a baseline group and a skill-injected group. Because both groups share identical model weights, their terminal outcome divergence provides a direct, low-variance counterfactual signal. This gap yields a task-level hindsight signal \(\Delta_g^{task}\) for general planning skills and an episode-specific credit \(c_i\) for retrieved step skills, which update skill utilities via exponential moving averages while injecting an intrinsic bonus into policy advantages.
2. Failure-Driven Dual-Granularity Skill Generation: Global Reflection Overlooking Local Pitfalls -> Explicit Decomposition of Task and Step Skills -> Complementary Hierarchical Memory
Reflecting solely on global trajectory outcomes tends to produce broad advice that is too vague to resolve local execution bottlenecks. D2Skill structurally bifurcates experience into task skills and step skills. Task skills are indexed by task specifications to prescribe end-to-end workflows and overarching constraints. Step skills are conditioned on composite keys combining task context and the specific observation at the earliest point of divergence (First Error Step). By selectively triggering reflection only on underperforming task groups, the agent extracts targeted recovery rules that prevent loops and search traps without bloating the memory store.
3. Utility-Aware Retrieval & Dynamic Pruning: Memory Bloat Causing Semantic Degradation -> Two-Stage UCB Reranking and Eviction Score -> Compact and High-Quality Skill Bank
Unchecked skill accumulation causes severe prompt clutter and retrieval distraction. D2Skill establishes a two-stage retrieval mechanism: Stage 1 applies cosine similarity filtering over candidate embeddings to retain the top-\(m\) relevant skills; Stage 2 scores and ranks them using a linear combination of semantic similarity and an Upper Confidence Bound (UCB) exploration bonus based on historical retrieval frequency. Periodically, when skill pools exceed their capacity threshold \(N_{max}\), an eviction score removes the lowest-utility, over-explored items while shielding freshly created skills within a protective training window \(T_{prot}\).
Dual-Granularity Skill Evolution and Scheduling Mechanism¶
The mathematical backbone of D2Skill governs how skills co-evolve with the policy. For each task \(g \sim \mathcal{D}\), a group of \(N\) trajectories \(G_g\) is sampled and split into \(G_g^{base}\) and \(G_g^{skill}\) of size \(N/2\). Letting \(Y_i \in \{0, 1\}\) denote the terminal success indicator of trajectory \(i\), the baseline and skill group empirical success rates are computed as: $\(\bar{Y}_g^{base} = \frac{1}{|G_g^{base}|} \sum_{i \in G_g^{base}} Y_i, \quad \bar{Y}_g^{skill} = \frac{1}{|G_g^{skill}|} \sum_{i \in G_g^{skill}} Y_i\)$ These group outcomes produce the task-level hindsight divergence \(\Delta_g^{task} = \bar{Y}_g^{skill} - \bar{Y}_g^{base}\) and trajectory-level step credit \(c_i = Y_i - \bar{Y}_g^{base}\). The utility \(u_m\) of each skill item \(m\) is updated online using an exponential moving average (EMA): $\(u_m \leftarrow (1 - \beta_{task}) u_m + \beta_{task} \Delta_g^{task} \quad (m \in \mathcal{M}_{task})\)$ $\(u_m \leftarrow (1 - \beta_{step}) u_m + \beta_{step} c_i \quad (m \in \mathcal{M}_{step})\)$ To incentivize the policy to effectively heed retrieved skills, skill-injected trajectories receive a hindsight intrinsic reward \(R_{int, i} = \lambda (Y_i - \bar{Y}_g^{base})\), yielding augmented returns \(\tilde{R}_i = R_i + R_{int, i}\) for \(i \in G_g^{skill}\) and \(\tilde{R}_i = R_i\) for \(i \in G_g^{base}\). Policy advantages are normalized across the pooled group \(G_g\) and optimized via clipped GRPO: $\(A_i = \frac{\tilde{R}_i - \text{mean}(\{\tilde{R}_j\}_{j \in G_g})}{\text{std}(\{\tilde{R}_j\}_{j \in G_g})}\)$
During interaction, given a query key \(q\) (\(q=g\) for task skills, \(q_t=(g, o_t)\) for step skills), candidates satisfying \(\text{sim}(q, k_m) \ge \tau_{sim}\) are shortlisted. The top-\(k\) skills injected into the policy prompt are determined by the UCB selection score: $\(\text{score}(m) = \alpha \cdot \widehat{\text{sim}}(m, q) + (1 - \alpha) \left( u_m + \eta \sqrt{\frac{\log(1 + N_r)}{1 + n_m}} \right)\)$ where \(n_m\) is the historical retrieval count of skill \(m\), \(N_r = \sum_{m'} n_{m'}\) is the total pool retrieval count, and \(\eta\) controls exploration. If total bank size \(|\mathcal{M}|\) exceeds capacity limit \(N_{max}\), skills outside the retention window (\(t - t_{create} \ge T_{prot}\)) are ranked by the eviction score: $\(\text{evict}(m) = u_m + \eta \sqrt{\frac{\log(1 + N_r)}{1 + n_m}}\)$ and the lowest-scoring entries are permanently pruned until \(|\mathcal{M}| \le N_{max}\).
Key Experimental Results¶
Main Results¶
D2Skill is thoroughly evaluated across ALFWorld (embodied household task completion), WebShop (e-commerce goal-directed purchasing), and Search-Augmented QA (single- and multi-hop reasoning) utilizing Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507. During evaluation, the learned skill bank is frozen, requiring zero test-time reflection or API overhead.
| Benchmark & Model Backbone | Method | Primary Metric (Success / Accuracy) | Baseline (GRPO) | Absolute Gain |
|---|---|---|---|---|
| ALFWorld (Qwen2.5-7B) | D2Skill (Gemini-3-Flash) | 90.6% | 72.6% | +18.0% |
| ALFWorld (Qwen2.5-7B) | D2Skill (O3) | 87.8% | 72.6% | +15.2% |
| ALFWorld (Qwen2.5-7B) | D2Skill (Self) | 82.8% | 72.6% | +10.2% |
| ALFWorld (Qwen2.5-7B) | SkillRL (O3)* | 89.1% | 72.6% | +16.5% |
| WebShop (Qwen2.5-7B) | D2Skill (O3) | 84.4% (Score: 90.1) | 72.6% (Score: 86.0) | +11.8% |
| WebShop (Qwen2.5-7B) | D2Skill (Gemini-3-Flash) | 80.5% (Score: 91.1) | 72.6% (Score: 86.0) | +7.9% |
| WebShop (Qwen2.5-7B) | D2Skill (Self) | 81.3% (Score: 89.9) | 72.6% (Score: 86.0) | +8.7% |
| ALFWorld (Qwen3-4B) | D2Skill (O3) | 72.7% | 53.9% | +18.8% |
| ALFWorld (Qwen3-4B) | D2Skill (Gemini-3-Flash) | 69.6% | 53.9% | +15.7% |
| ALFWorld (Qwen3-4B) | D2Skill (Self) | 62.5% | 53.9% | +8.6% |
| WebShop (Qwen3-4B) | D2Skill (Gemini-3-Flash) | 82.8% (Score: 90.0) | 74.2% (Score: 87.0) | +8.6% |
| WebShop (Qwen3-4B) | D2Skill (Self) | 77.4% (Score: 89.6) | 74.2% (Score: 87.0) | +3.2% |
| Search QA (Qwen2.5-7B) | D2Skill (O3) | 44.9% (Hotpot: 47.5%) | 40.9% (ZeroSearch) | +4.0% |
| Search QA (Qwen2.5-7B) | D2Skill (Self) | 43.8% (2Wiki: 45.9%) | 40.6% (StepSearch) | +3.2% |
Ablation Study¶
On ALFWorld with Qwen3-4B-Instruct-2507, ablations systematically isolate the role of task skills, step skills, memory management, paired baseline comparison, and utility-driven retrieval. Complementary experiments evaluate matched self-feedback regimes, out-of-distribution (OOD) transfer, and resilience against synthetic noisy skills.
| Ablation Configuration | Train Peak Skill Group (%) | Train Peak Baseline Group (%) | Val Success (%) | Degradation vs. Full |
|---|---|---|---|---|
| D2Skill Full Model | 70.9 | 65.8 | 72.7 | - |
| w/o task skills | 59.1 | 53.7 | 62.7 | -10.0% |
| w/o step skills | 57.8 | 55.8 | 60.2 | -12.5% |
| w/o skill management | 60.0 | 57.4 | 57.8 | -14.9% |
| w/o baseline group | 63.9 | - | 68.8 | -3.9% |
| w/o utility retrieval | 61.4 | 51.8 | 64.8 | -7.9% |
| w/o utility module | 60.3 | - | 62.5 | -10.2% |
| w/o skills (GRPO Baseline) | - | 50.6 | 53.9 | -18.8% |
Under the zero-adaptation out-of-distribution evaluation (No-adaptation ALFWorld OOD), models are trained exclusively on seen environments and tested on unseen environments with the learned skill bank frozen: - GRPO: Seen 53.9% \(\rightarrow\) Unseen 36.7% (-17.2%) - SkillRL: Seen 67.2% \(\rightarrow\) Unseen 55.5% (-11.7%) - D2Skill: Seen 72.7% \(\rightarrow\) Unseen 63.3% (+7.8% over SkillRL, +26.6% over GRPO)
In the online noisy-skill stress test (where 30% of generated skills are systematically corrupted into misleading variants): - Full D2Skill retains 62.5% validation success, successfully shrinking the final noisy bank proportion to 19.6%; - Disabling utility retrieval and eviction management plunges validation success to 51.6% (-10.9%), while noisy entries accumulate to 28.5%.
Key Findings¶
- Dynamic bank management is the single most critical safeguard: Removing skill pruning and management causes the most severe performance collapse (validation success drops by 14.9 points from 72.7% to 57.8%). An unmanaged bank accumulates redundant and suboptimal rules that dilute prompt attention and mislead the agent.
- Dual-granularity skills exhibit vital synergy: Stripping away task skills (-10.0%) or step skills (-12.5%) induces steep performance drops. High-level planning provides directional guidance, while step-level rules terminate local action loops; neither granularity is dispensable.
- Superior sample efficiency and cost profile: D2Skill completes ALFWorld training on 8x H100 GPUs in 25.6 wall-clock hours, closely tracking skill-free GRPO (20.8 hours) while outpacing SkillRL (49.2 hours) by a factor of 1.7x. Early-stage API diagnostics reveal reflection costs of only $1.12 to $1.32.
Highlights & Insights¶
- Counterfactual Paired Sampling for Credit Assignment: By pitting baseline and skill-augmented rollouts generated under identical policy weights against each other, D2Skill computes an empirical counterfactual advantage signal without training separate value networks.
- Balancing Exploration and Exploitation in Prompt Retrieval: Integrating multi-armed bandit UCB exploration into memory retrieval guarantees that high-utility historical skills are exploited, while unvetted skills receive sufficient trials before facing eviction.
- Implicit Policy Internalization: Even when the skill bank is completely removed at test time, D2Skill-trained policies substantially outperform standard GRPO baselines, demonstrating that externalized skills act as scaffolding that transfers procedural competencies into model weights.
Limitations & Future Work¶
- Coarse Attribution Across Concurrent Skills: When multiple step skills are invoked across an episode, the trajectory-level success gap serves as a heuristic ranking proxy rather than an exact causal Shapley value for each individual skill.
- Benchmark Scope vs. Open-World Complexity: Experiments are grounded in text-based interactive environments (ALFWorld, WebShop) and search QA; validating D2Skill in multimodal robotics, operating system control, and stochastic real-world workflows remains open.
- Future Directions: Investigating token-level process supervision (PRM) for fine-grained credit attribution, and extending dual-granularity skill banks to lifelong open-world web and multimodal agents.
Related Work & Insights¶
- vs. Reflexion (Shinn et al., 2023): Reflexion relies on ephemeral verbal trial-and-error within a single task session without establishing a cross-task persistent skill bank or quantitative utility-based pruning.
- vs. SkillRL (Xia et al., 2026): SkillRL maintains a static, monotonically growing task-level skill repository; D2Skill introduces observation-conditioned step skills for local error recovery and maintains dynamic bank quality via paired hindsight utility estimation.
- vs. Mem0 / SimpleMem (Chhikara et al., 2025; Liu et al., 2026a): Generic agent memory architectures lack reinforcement learning-aligned utility evaluation, whereas D2Skill tightly couples memory dynamics with RL advantage normalization.
Rating¶
- Novelty: โญโญโญโญโญ (Elegant coupling of dual-granularity skill abstraction with paired-rollout hindsight credit assignment)
- Experimental Thoroughness: โญโญโญโญโญ (Comprehensive validation across three benchmarks, two open backbones, matched self-feedback, OOD transfer, and synthetic noise stress testing)
- Writing Quality: โญโญโญโญโญ (Rigorous formulations, crisp motivation narrative, and cohesive structural presentation)
- Value: โญโญโญโญโญ (Establishes a practical, highly scalable blueprint for lifelong memory and skill evolution in agentic RL systems)