Skip to content

APEx: Distillation of Agent Procedural Experience for Adaptive Deep Research Question Answering

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/J-Ding519/APEx
Area: LLM Reasoning
Keywords: Deep Research Agents, Experience Distillation, Procedural Skills, Test-time Reinforcement Learning, Closed-loop Adaptation

TL;DR

APEx introduces a hierarchical experience utilization framework coupled with a three-stage alternating GRPO training paradigm, organizing interaction traces into instance-level memories and category-level procedural skills that serve as regularization priors for online test-time reinforcement learning without ground-truth labels.

Background & Motivation

Deep research agents (DRAs) empower large language models with external tool suites (e.g., text search, web navigation, and visual matching) to tackle complex, long-horizon questions through multi-turn decision-making and cross-modal evidence synthesis. Despite their promising capabilities in open-ended exploratory environments, executing multi-turn tool trajectories over prolonged contexts remains intrinsically fragile: agents repeatedly fall into identical retrieval loops, misinterpret intermediate evidence, and lack mechanisms to consolidate past trial-and-error interactions into generalizable procedural knowledge. Existing memory-augmented systems rely predominantly on episodic memory or reflective memory. Episodic memory archives raw interaction traces and retrieves nearest-neighbor demonstrations at decision time, yet these concrete trajectory tokens impose severe cognitive and context-window burdens on the agent while failing to extract transferable macro-level strategies. Reflective memory compresses historical failures into verbal summaries, but these textual artifacts are typically produced via rigid, hand-crafted prompts and remain decoupled from downstream policy optimization.

A deeper structural limitation emerges when examining procedural memory frameworks that seek to represent experience as reusable action instructions or modular skills. In contemporary agent pipelines, procedural skills are treated as external, static knowledge artifacts injected into the prompt context rather than active policy priors that dynamically govern model adaptation. When deployed in non-stationary test environments without annotated labels, agents cannot adjust their planning policies to novel domain shifts, while naïve test-time self-improvement methods relying on unconstrained pseudo-supervision suffer from catastrophic policy collapse and severe hallucination drift. The critical bottleneck lies in the absence of a closed feedback loop that seamlessly connects runtime execution, experience abstraction, strategic planning, and parameter-level adaptation.

To resolve the disconnect between experience consolidation and online self-evolution, this work reconceptualizes experience-driven agent improvement as a closed-loop system encompassing parameter-level reinforcement learning and non-parametric hierarchical memory evolution. Core idea: organize long-term interaction history into a complementary hierarchy of concrete instance-level memories and abstract category-level procedural skills, optimize the Executor, Distiller, and Planner via a three-stage alternating GRPO paradigm, and leverage confidence-weighted procedural skills as regularization anchors for ground-truth-free test-time reinforcement learning.

Method

Overall Architecture

The APEx framework integrates four mutually reinforcing components: a dual-tier Hierarchical Experience Memory bank, a closed-loop Executor-Distiller-Planner multi-agent coordination loop, a Three-Stage Alternating GRPO optimization schedule, and a Skill-guided Test-Time Reinforcement Learning (TTRL) pipeline. At runtime, task queries are routed into category-modality memory buckets to retrieve concrete past successes/failures alongside abstract procedural guidelines. The Planner formulates structured execution steps, which the Executor operationalizes via external tools. Newly collected trajectories are subsequently distilled into evolving procedural skills and leveraged to regularize online policy updates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Query q"] --> B["Hierarchical Experience Retrieval<br/>Instance Traces RM(q) ⊕ Category Skills RS(q)"]
    B --> C["Executor-Distiller-Planner Closed Loop<br/>Plan generation, tool calling, and replanning"]
    C --> D["Three-Stage Alternating GRPO Optimization<br/>Staggered policy training across modules"]
    D --> E["Skill-Guided Test-Time Adaptation<br/>Multi-judge no-GT reward + Skill-alignment anchor"]
    E -->|Trajectory consolidation & Skill evolution| B
    E --> F["Final Grounded Response & Output"]

Key Designs

1. Hierarchical Experience Retrieval: Resolving the Trade-Off Between Verbose Traces and Actionable Abstraction
Flattened memory architectures that inject complete raw trajectories directly into the agent context flood the attention window with noisy entity tokens and task-specific idiosyncrasies, whereas purely abstract textual summaries lack concrete procedural anchors. To strike an optimal balance, APEx stratifies experience into two complementary tiers: an instance-level memory bank \(\mathcal{M} = \bigcup_{c \in \mathcal{C}, d \in \mathcal{D}} \mathcal{M}_{c,d}\) where each trace is recorded as a compact tuple \(m_i=(q_i, \xi_i, y_i, w_i)\) encapsulating query, workflow summary, binary outcome, and empirical reuse win rate \(w_i\); and a category-level procedural skill bank \(\mathcal{S} = \{s_{c,d}\}\) where the Distiller synthesizes bucketed traces into an evolving structured document \(s_{c,d} = (P_{c,d}, F_{c,d}, w_{c,d}, n_{c,d}, \text{desc}_{c,d})\). During online retrieval, instance memories are ranked via dual-channel semantic similarity combined with historical win rate: $\(f(q, m_i) = \alpha \cdot \widetilde{\text{sim}}(q, m_i) + (1-\alpha) \cdot w_i\)$ separately extracting top-\(k\) successful and failed exemplars. Concurrently, category skills are assigned a confidence score \(\gamma_{c,d} = w_{c,d} \cdot \min(n_{c,d}/N_{\text{threshold}}, 1)\) and injected via structured concatenation \(E(q) = R_S(q) \oplus R_M(q)\) only when surpassing confidence threshold \(\epsilon_s=0.1\). This ensures that the Planner receives concise, battle-tested procedural guidance without redundant context bloat.

2. Executor-Distiller-Planner Closed Loop: Coordinating Tool Execution, Knowledge Distillation, and Dynamic Replanning
To prevent role confusion and catastrophic attention degradation that occur when a single language model simultaneously acts as investigator, researcher, and meta-evaluator, APEx decouples responsibilities across three specialized sub-agents. The Executor \(\pi_{\text{exec}}^\theta\) (built upon Qwen2.5-VL-7B) focuses exclusively on executing atomic plan instructions through multi-turn tool interaction (utilizing text retrieval via E5-base-v2/wiki25 and image search via Serper). The Distiller \(\pi_{\text{dist}}^\phi\) (based on Qwen3-8B) periodically analyzes accumulated instance memories within each category-modality bucket to execute one of four evolutionary operations—synthesize, refine, create, or skip—under a conservative editing doctrine: successful memories define operational invariants, while failed memories pinpoint defective sub-steps. The Planner \(\pi_{\text{plan}}^\psi\) (based on Qwen3-8B) conditions on \(q\) and \(E(q)\) to produce initial plan \(p_1\), observes execution trajectory \(\text{Trace}(\tau)\), and predicts a binary replanning decision \(d \in \{0, 1\}\). If \(d=1\), it issues a focused revision plan \(p_2\), providing an adaptive self-correction mechanism against unexpected retrieval dead-ends.

3. Three-Stage Alternating GRPO Optimization: Overcoming Multi-Agent Credit Assignment and Static Skill Limitations
Joint end-to-end training of multiple interacting agents routinely triggers severe gradient instability and diffuse credit assignment, while traditional memory compression systems rely on heuristic, fixed prompts that cannot improve via reinforcement feedback. APEx addresses this via a three-stage alternating Group Relative Policy Optimization (GRPO) framework built on veRL. In Stage 1, with Planner and Distiller frozen, the Executor is trained on FVQA-train to master multi-turn tool calls using a composite reward incorporating answer correctness \(R_{\text{acc}}\), valid tool invocation \(R_{\text{tool}}\), and structural format conformance \(R_{\text{fmt}}\). In Stage 2, with Executor and Planner frozen, the Distiller is optimized via GRPO using an LLM Judge that rates procedural clarity, pitfall grounding, and evolutionary rationality \(R_{\text{dist}} = \mu_1 R_{\text{quality}} + \mu_2 R_{\text{fmt}} + \mu_3 R_{\text{evolve}}\), effectively converting skill distillation from a static prompt into a reward-guided policy. In Stage 3, the Planner is optimized through a closed plan-execute-replan loop driven by a reward balancing initial execution, replanned execution, and decision fidelity: $\(R_{\text{plan}} = \alpha_1 R_{\text{acc}}(\hat{y}_2) + \alpha_2 R_{\text{acc}}(\hat{y}_1) + \alpha_3 R_{\text{fmt}} + \alpha_4 R_{\text{dec}}\)$ which explicitly incentivizes the Planner to terminate early when the first trajectory succeeds and trigger decisive replanning when failure occurs.

4. Skill-Guided Test-Time Adaptation: Ground-Truth-Free Self-Evolution with Policy Drift Regularization
During open-domain test-time deployment, true supervision \(y^*\) is completely unavailable, exposing unconstrained test-time reinforcement learning to severe reward hacking and degenerative policy drift. APEx introduces Skill-guided TTRL, performing online GRPO updates exclusively on the Planner parameters \(\psi\). The system establishes a multi-judge panel comprising three LLM evaluators (\(E_1\) for reasoning consistency, \(E_2\) for evidence grounding against retrieved tool outputs, and \(E_3\) for answer validity), aggregated by an arbiter into a no-ground-truth reward \(R_{\text{nogt}}\). To anchor exploration, the retrieved category skill \(s_{c,d}\) acts as a procedural prior: an alignment reward \(R_{\text{align}} = \mathcal{J}_{\text{align}}(p, P_{c,d}, F_{c,d})\) penalizes deviation from proven steps and warns against known failure modes: $\(R_{\text{final}} = (1 - \lambda_s) R_{\text{nogt}} + \lambda_s R_{\text{align}}, \quad \text{where } \lambda_s = \lambda_{\text{base}} \cdot w_{c,d} \cdot \min\left(\frac{n_{c,d}}{N_{\text{threshold}}}, 1\right)\)$ For highly mature skills with high confidence, an optional skill-gating mechanism forces all rollout rewards to equalize (\(\hat{A}_i = 0\)), preserving verified knowledge while conserving inference compute. The updated Planner generates higher-quality trajectories, which in turn are consolidated into memory to evolve stronger skills, establishing an autonomous self-improving flywheel.

Loss & Training

All three stages and test-time adaptation leverage Group Relative Policy Optimization (GRPO). For a prompt \(x\), the active policy samples a cohort of \(G\) responses \(\{o_i\}_{i=1}^G\) and computes group-normalized advantages: $\(\hat{A}_i = \frac{R(o_i) - \text{mean}_j[R(o_j)]}{\text{std}_j[R(o_j)] + \epsilon}\)$ The parameter objective minimizes clipped policy ratios: $\(\mathcal{L}_{\text{GRPO}}(\eta) = - \mathbb{E}_{x, \{o_i\}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\left( r_{i,t}(\eta)\hat{A}_i, \, \text{clip}(r_{i,t}(\eta), 1-\epsilon, 1+\epsilon)\hat{A}_i \right) \right]\)$ where \(r_{i,t}(\eta) = \pi_\eta(o_{i,t}|x, o_{i,<t}) / \pi_\eta^{\text{old}}(o_{i,t}|x, o_{i,<t})\). Training is conducted on 8 NVIDIA A100 GPUs using FSDP with parameter and optimizer offload. Hyperparameters specify AdamW optimizer and PPO clip range \(\epsilon=0.2\). The Executor is trained with learning rate \(1 \times 10^{-6}\), batch size 128, and group size \(G=8\) over 8 epochs; the Distiller with learning rate \(1 \times 10^{-6}\), batch size 64, and \(G=8\) over 8 epochs; the Planner with learning rate \(1 \times 10^{-6}\), batch size 48, and \(G=4\) over 5 epochs. Online TTRL adapts the Planner over test queries with learning rate \(2 \times 10^{-7}\), batch size 8, and group size \(G=4\) for 1 epoch.

Key Experimental Results

Main Results

APEx is evaluated across 7 diverse benchmarks spanning multimodal image-text QA (FVQA-test, SimpleVQA, LiveVQA, InfoSeek, MMSearch) and complex multi-hop text-only reasoning (HotpotQA, 2WikiMultiHopQA), compared against direct LLMs, search agents, and leading memory-augmented architectures:

Dataset / Benchmark Modality GPT-5.4 Gemini-3-Flash Qwen2.5-VL-7B (Base) MMSearch-R1 MIA (Strongest Memory Baseline) APEx (Ours) Relative Gain over Base
FVQA-test (In-Domain) Image-Text 51.3 68.4 20.7 58.3 65.3 68.7 +48.0
HotpotQA (In-Domain) Text-Only 60.2 66.2 12.6 39.2 64.0 67.8 +55.2
2WikiMultiHopQA Text-Only 64.9 68.4 20.5 48.3 71.6 75.2 +54.7
SimpleVQA (OOD) Image-Text 55.9 72.3 30.2 55.2 63.8 66.3 +36.1
LiveVQA (OOD) Image-Text 22.8 25.7 8.3 28.6 39.7 42.4 +34.1
InfoSeek (OOD) Image-Text 45.6 66.5 23.6 49.5 64.7 66.9 +43.3
MMSearch (OOD) Image-Text 47.4 67.9 7.2 43.3 60.6 63.8 +56.6
Overall Average All Scenarios 49.7 62.2 17.6 46.1 61.4 64.4 +46.8

Ablation Study

Ablation experiments across four representative benchmarks isolate the impact of hierarchical experience components, GRPO post-training, and test-time adaptation:

Configuration / Variant FVQA (Multimodal) SimpleVQA (Multimodal) HotpotQA (Text) 2Wiki (Text) Average Accuracy Relative Drop (↓)
APEx Full Pipeline 68.7 66.3 67.8 75.2 69.5 —
Experience Utilization Ablation
w/o Instance Memory (w/o Mem.) 67.4 64.8 65.7 72.9 67.7 ↓1.8
w/o Procedural Skills (w/o Skill) 64.9 62.7 65.6 72.3 66.4 ↓3.1
w/o Both Memories (w/o Both) 58.6 55.0 58.4 65.5 59.4 ↓10.1
Post-Training GRPO Ablation
Distiller unoptimized (w/o Dis.) 66.4 63.7 65.8 72.7 67.2 ↓2.3
Planner unoptimized (w/o Plan.) 66.8 63.9 66.2 74.0 67.7 ↓1.8
Executor unoptimized (w/o Exec.) 61.0 58.0 58.2 67.2 61.1 ↓8.4
Closed-loop Adaptation Ablation
w/o Test-time RL (w/o TTRL) 64.6 60.8 63.5 72.4 65.3 ↓4.2
w/o Iterative Distillation (w/o Iter.) 66.0 64.1 64.7 73.3 67.0 ↓2.5

Key Findings

  • Procedural skills provide greater strategic guidance than raw instance retrieval: Removing category-level skills causes a 3.1-point drop compared to only 1.8 points when removing instance-level memories. Furthermore, the APEx (Skill Only) variant delivers 67.7% average accuracy, outperforming the full MIA memory baseline (66.2%) while slashing per-step injected memory tokens by 45.6% and total reasoning token overhead by 36.8%, proving that distilled procedural knowledge substantially improves context efficiency.
  • Executor optimization establishes the functional foundation, while Distiller GRPO elevates skill abstraction: Disabling GRPO training on the Executor induces a catastrophic 8.4-point plunge, showing that multi-step tool grounding is indispensable; optimizing the Distiller via GRPO rather than static prompting yields an indispensable 2.3-point performance gain, confirming the utility of reward-guided skill distillation.
  • TTRL and dynamic skill consolidation form an essential positive feedback loop: Ablating online test-time reinforcement learning degrades overall accuracy by 4.2 points, while freezing skill evolution after an initial pass degrades performance by 2.5 points. Iteratively recycling newly consolidated test traces into refined skills directly accelerates continuous self-improvement.

Highlights & Insights

  • Reinforcement-Optimized Experience Distillation: While prior methods treat skill synthesis as a static prompting task, APEx explicitly formulates distillation as a trainable policy optimized by GRPO with multi-perspective quality rewards, ensuring that distilled guidelines are directly actionable and tailored for downstream planning.
  • Skill-Regularized Test-Time Anchoring: To overcome the vulnerability of unsupervised test-time RL to reward hacking and hallucination drift, APEx grounds online policy gradient updates with an adaptive skill-alignment penalty, achieving safe exploration under zero-ground-truth conditions.
  • Plug-and-Play Generalization to Unseen Closed-Source Backbones: When substituting the 7B open-source Executor with proprietary frontier engines (GPT-5.4, Claude-Sonnet-4-6, and Gemini-3-Flash) without retraining, APEx consistently outperforms standard ReAct by 11.0% on HotpotQA and 7.0% on LiveVQA, verifying that its hierarchical experience guidance generalizes across model families.

Limitations & Future Work

  • Author-Acknowledged Limitations: The framework was primarily evaluated on compact open-source models (7B/8B scale); whether the performance gains scale proportionately when applied to massive frontier backbones remains an open question. Furthermore, the three-stage alternating optimization process is strictly sequential, which may underutilize cross-module synergies compared to end-to-end multi-agent joint training.
  • Additional Considerations: The test-time multi-judge framework introduces substantial computational latency and token cost during inference, relying on a 32B model arbiter. In specialized, low-resource domains lacking reliable external retrieval corpora, the judge's verification accuracy could degrade, potentially causing overly conservative skill regularization.
  • Future Directions: Promising extensions include compressing the multi-judge evaluation stack into a distilled lightweight critic, expanding procedural skills into executable programmatic sub-agent graphs, and exploring decentralized cross-agent skill sharing across heterogeneous fleets.
  • vs Reflexion / Mem0 / ExpeL: Conventional episodic and reflective agents retrieve flat interaction records or unstructured natural language reflections, flooding the context window and leaving skills decoupled from policy updates; APEx introduces hierarchical structuring and optimizes skill generation directly through downstream RL signals.
  • vs Search-R1 / MMSearch-R1 / Deepeyes: Traditional agentic search models train tool-use reasoning within parametric weights but possess no cross-task memory to retain lessons learned from prior sessions; APEx bridges this divide by coupling non-parametric procedural memory with parametric test-time reinforcement learning.
  • vs Standard Majority-Voting TTRL: Standard test-time RL relies heavily on self-consistency or majority voting across multiple sampled answers, which breaks down on open-ended knowledge-seeking benchmarks; APEx formulates an anchored reward structure combining multi-faceted evidence judging with skill-alignment regularization.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers the integration of reward-guided procedural skill distillation with skill-regularized test-time reinforcement learning.]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 7 recognized multimodal and text-only benchmarks with exhaustive baselines, token efficiency analyses, and modular ablations.]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formalisms, intuitive visual flowcharts, and fully transparent appendix prompts for all agent modules.]
  • Value: ⭐⭐⭐⭐⭐ [Delivers a robust, context-efficient blueprint for enabling deep research agents to continually learn and self-correct across complex long-horizon tasks.]