Retrieval-Augmented LLM Agents: Learning to Learn from Experience¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/naver/RAGents
Area: LLM Agent
Keywords: LLM agent, experience retrieval-augmented generation, episodic memory, retrieval-aware fine-tuning, embodied decision-making
TL;DR¶
This paper introduces EXPRAG (an experience retrieval-augmented generation framework) and EXPRAG-LoRA (a retrieval-aware fine-tuning paradigm), showing that integrating episodic trajectory retrieval directly into LoRA fine-tuning substantially boosts generalization on unseen tasks and enables agents to learn from their own failures in-context without test-time parameter updates.
Background & Motivation¶
Large language model (LLM) agents have demonstrated substantial promise in multi-step interactive environments and complex task planning. However, robust out-of-distribution generalization to unseen tasks remains a primary bottleneck. Current adaptation paradigms broadly bifurcate into two camps: parametric supervised fine-tuning (SFT/LoRA) and training-free memory-augmented prompting. While supervised fine-tuning excels at fitting in-distribution tasks, it severely over-specializes the policy to the training distribution, leading to catastrophic compounding errors and policy collapse when tested on task groups requiring novel sub-plans (for instance, evaluating on heating objects when the agent was trained only on cleaning). Conversely, training-free agent memory systems have introduced increasingly intricate designs (such as dynamic read-write memory controllers, reflection pipelines, and heuristic forgetting), yet often underperform a strong supervised baseline and struggle to filter out noisy context on moderately sized open-source models.
The tension between these paradigms stems from an underlying disconnect: pure parametric updates strip the agent of dynamic contextual adaptation, whereas purely inference-time episodic retrieval expects a model to effectively utilize complex trajectory memories that it was never trained to interpret. Consequently, frozen backbones suffer from context rot, context churn, and distractor noise. Furthermore, standard agent benchmarks often evaluate on held-out tasks drawn from the exact same distributions seen during training, artificially obscuring the severe degradation that occurs under realistic task-distribution shifts.
This paper tackles this gap by unifying episodic retrieval and policy adaptation, investigating how agents can be explicitly trained to learn from retrieved trajectories in-context. Core idea: maintain a simple, fixed, read-only episodic bank of raw multi-turn trajectories, retrieve relevant top-K experiences at decision time, and crucially inject this retrieval memory block directly into the context during LoRA fine-tuning (EXPRAG-LoRA), thereby teaching the policy how to identify and execute actionable subgoals from retrieved experiences across unseen tasks and imperfect demonstrations.
Method¶
Overall Architecture¶
The EXPRAG system models sequential decision-making in textual environments across three core phases: offline episodic index construction, task-level experience retrieval, and memory-conditioned autoregressive action generation. In the offline phase, expert rollouts (containing both successes and failures) are serialized into standard multi-turn chat dialogues and encoded into dense vector embeddings using a text embedding model. During execution, given the task description or partial trajectory history, the system retrieves the top-K nearest-neighbor trajectories, formats them into a structured memory block distinguishing successes from failures, and feeds this block into the system prompt to condition the policy network's next action prediction.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Environment rollouts<br/>success and failure trajectories"] --> B["Multi-turn chat serialization<br/>map observations to user & actions to assistant"]
B --> C["Dense episodic memory index<br/>Qwen3-Embedding offline encoding"]
D["Task description and interaction context"] --> E["Experience retrieval<br/>Top-K nearest-neighbor search"]
C --> E
E --> F["Structured memory block construction<br/>explicit success vs. failure markers"]
F --> G["Retrieval-aware fine-tuning & policy inference<br/>EXPRAG-LoRA conditioned generation"]
D --> G
G --> H["Execute action in environment & update state"]
Key Designs¶
1. Multi-turn chat trajectory serialization: aligning memory format with native LLM conversations
Prior agent architectures frequently decompose trajectories into independent stepwise state-action pairs \((h_t, a_t)\), re-encoding the full history at every step. This practice fragments temporal continuity and incurs excessive computational and KV-cache overhead. Instead, EXPRAG serializes full trajectories \(\tau = (T, o_1, a_1, \dots, o_T)\) directly using the base model's native chat template, treating environment observations \(o_t\) as user turns and action tokens \(a_t\) as assistant turns. This design preserves conversational coherence, guarantees that supervised cross-entropy is calculated strictly on assistant action tokens, and maximizes KV-cache reuse during training and inference.
2. Strict episodic indexing and retrieval: mitigating action-space leakage and context churn
The retrieval pipeline indexes chat-formatted trajectories using Qwen3-Embedding-0.6B to produce key vectors \(e_i = \phi(\tau_i)\). Two query paradigms are analyzed: static retrieval (querying once at \(t=0\) using task description \(T\)) and dynamic retrieval (re-querying at every step \(t\) using partial history \(h_t\)). To prevent ground-truth action leakage, the paper enforces a realistic evaluation protocol: unlike prior works that provide valid-action candidate lists at each timestep, the agent receives only static command grammar templates in the system prompt. Candidate retrieval follows standard inner-product ranking:
3. Retrieval-aware fine-tuning (EXPRAG-LoRA): learning to leverage in-context experience
Providing retrieved trajectories purely at inference time fails when models have not learned how to parse and ground actions against in-context demonstrations. EXPRAG-LoRA resolves this by mirroring the inference setup during supervised fine-tuning: training instances are augmented with retrieved top-K memory blocks \(m_t = \mathrm{system}(\tau^1, \dots, \tau^K)\) pulled from the training split. This trains the adapter weights to actively reference and extract reusable subgoals from retrieved demonstrations rather than merely memorizing fixed environment trajectories, unlocking superior generalization to unseen hard tasks and noisy experience.
Loss & Training¶
Supervised fine-tuning is implemented with TorchTune using LoRA adapters applied to the attention projection layers (q_proj, k_proj, v_proj, output_proj), with rank \(r=8\), scaling \(\alpha=16\), and dropout \(0.1\). Optimization uses PagedAdamW8bit with a constant learning rate of \(5 \times 10^{-5}\) on a single 80 GB NVIDIA A100 GPU. Fine-tuning is conducted solely on Easy tasks for 10 epochs on ALFWorld and 30 epochs on ScienceWorld. Crucially, empirical training dynamics reveal that task-rollout success on unseen tasks continues to climb long after validation cross-entropy loss has bottomed out and begun to rise, demonstrating that standard validation loss early stopping is counterproductive for long-horizon agent decision-making.
Key Experimental Results¶
Main Results¶
Success rates (%) across in-distribution (in-d) easy tasks and out-of-distribution (oo-d) unseen hard tasks under different adaptation regimes:
| Benchmark & Split | Metric | EXPRAG (Zero-shot) | LoRA (No Retrieval) | LoRA + EXPRAG (Inference RAG) | EXPRAG-LoRA (Ours) |
|---|---|---|---|---|---|
| ALFWorld (Ministral 3-8B) Easy (in-d) | Success (%) | 54.8 | 98.6 | 97.3 | 97.3 |
| ALFWorld (Ministral 3-8B) Hard (oo-d) | Success (%) | 47.5 | 34.4 | 67.2 | 88.5 |
| ScienceWorld (Ministral 3-8B) Easy (in-d) | Success (%) | 43.5 | 38.8 | 54.1 | 58.8 |
| ScienceWorld (Ministral 3-8B) Hard (oo-d) | Success (%) | 18.8 | 15.6 | 28.1 | 42.2 |
| ALFWorld (Qwen 2.5-7B) Hard (oo-d) | Success (%) | 70.5 | 21.3 | 70.5 | 90.2 |
| ScienceWorld (Qwen 2.5-7B) Hard (oo-d) | Success (%) | 6.2 | 7.8 | 23.4 | 29.7 |
| ALFWorld (Qwen 2.5-7B-1M) Hard (oo-d) | Success (%) | 54.1 | 23.0 | 68.9 | 91.8 |
| ScienceWorld (Qwen 2.5-7B-1M) Hard (oo-d) | Success (%) | 15.6 | 14.1 | 12.5 | 29.7 |
Ablation Study¶
Robustness to imperfect memory and learning from failures on unseen hard tasks (Ministral 3-8B success rate % on ALFWorld / ScienceWorld):
| Experience Index Condition | Index Characteristics & Settings | EXPRAG (Zero-shot) | LoRA + EXPRAG | EXPRAG-LoRA (Ours) |
|---|---|---|---|---|
| Matched Expert (Reference) | Full hard-task scripted expert index (1,805 / 1,254 traces) | 44.3 / 20.3 | 68.3 / 31.2 | 91.8 / 47.9 |
| Empty (No Retrieval) | No external trajectories available | 0.0 / 4.7 | 31.1 / 14.6 | 30.6 / 3.1 |
| Mismatched Expert | Easy-task expert trajectories only (cross-task mismatch) | 8.7 / 15.6 | 29.5 / 12.5 | 41.5 / 9.4 |
| Low-Coverage Expert | Only 5 trajectories per hard task type (sparse memory) | 36.1 / 22.9 | 50.8 / 22.9 | 83.6 / 54.7 |
| Matched Strong-LLM | Traces generated by Qwen2.5-70B (suboptimal actions) | 44.3 / 26.6 | 65.6 / 33.9 | 90.2 / 38.0 |
| Self-Experience | First attempt trajectory (successful or failed) without parameter updates | 0.0 / 7.8 | 31.1 / 6.2 | 60.7 / 4.7 |
| Self-Exp + Matched Expert | Combined self-attempt with matched expert demonstrations | 34.4 / 17.2 | 67.2 / 21.9 | 93.4 / 29.7 |
Key Findings¶
- Conventional LoRA suffers severe out-of-distribution performance collapse on hard tasks (plunging from 98.6% to 34.4% on ALFWorld with Ministral, and from 86.3% to 21.3% with Qwen 2.5-7B), dropping well below training-free EXPRAG baselines. This confirms that pure behavior cloning over-fits to training subgoals.
- EXPRAG-LoRA consistently and reliably outperforms standard LoRA and inference-only RAG across backbones, achieving 88.5%โ91.8% on ALFWorld hard tasks (a 54โ70 percentage point gain over LoRA, statistically significant under paired McNemar tests at \(p < 0.001\)).
- Agents can learn from their own failures in-context: on ALFWorld hard tasks where all zero-shot first attempts fail (0% success), using these failed trajectories as the Self-Experience memory index enables EXPRAG-LoRA to reach 60.7% success on the second attempt without any test-time parameter updates, vastly exceeding LoRA + EXPRAG (31.1%).
- While dynamic retrieval updates context step-by-step, it induces context churn and instability when index mismatch occurs; static retrieval (\(K=2\)) achieves a significantly more robust accuracy-efficiency trade-off across environments.
Highlights & Insights¶
- Simplicity over complexity: A straightforward read-only bank of raw trajectories, when paired with retrieval-aware fine-tuning, consistently outperforms sophisticated agentic memory architectures (such as Reflexion, A-MEM, and MemoryBank).
- Retrieval-aware training is necessary: Having external memory available at test time is insufficient if the underlying model was never trained to ground its multi-step actions in retrieved demonstrations.
- Remarkable tolerance to low-quality memory: Even when restricted to ultra-sparse memory (5 trajectories per task) or teacher traces with suboptimal actions, EXPRAG-LoRA preserves high performance (83.6%โ90.2% on ALFWorld), proving its viability in data-constrained scenarios.
Limitations & Future Work¶
- The empirical evaluations focus on simulated text-based embodied benchmarks (ALFWorld and ScienceWorld) with structured expert trajectories, rather than open-ended, non-stationary web or OS environments.
- Self-experience gains depend on the meta-learning environment: while learning from failed attempts succeeds markedly on ALFWorld, it does not generalize to ScienceWorld due to the higher action space complexity and lack of exploratory diversity in failed traces.
- The memory system remains fixed and read-only, without online memory consolidation, abstraction, or long-horizon lifelong curation.
Related Work & Insights¶
- vs Reflexion / A-MEM: These systems rely on LLM reflection loops to synthesize abstract summaries or update dynamic read-write memory banks. In contrast, EXPRAG maintains raw interaction traces and shows that training the policy to read retrieved traces directly yields superior or competitive results with much lower architectural complexity.
- vs ETO / NAT: Prior methods incorporate failed exploration as negative supervision for gradient updates. EXPRAG-LoRA operates orthogonally: it trains the policy to condition on both positive and negative demonstrations in-context, enabling instant test-time adaptation without parameter modification.
Rating¶
- Novelty: โญโญโญโญ [Pioneers the retrieval-aware fine-tuning framework for episodic agent memory]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive OOD hard task evaluation across 4 open LLM backbones with rigorous paired McNemar and bootstrap tests]
- Writing Quality: โญโญโญโญโญ [Exceptionally clear and precise, with acute attention to benchmarking pitfalls and action-space leakage]
- Value: โญโญโญโญโญ [Provides an essential, highly reproducible foundation for deploying robust embodied LLM agents]