Skip to content

Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ictnlp/unified-memory-agent.git
Area: LLM Reasoning
Keywords: memory agents, long-context reasoning, reinforcement learning, state tracking, unified policy

TL;DR

Unified Memory Agent (UMA) formulates query-agnostic memory management as an end-to-end Markov Decision Process using Task-Stratified GRPO, enabling a single policy to actively maintain structured key-value stores across long-horizon text streams and achieve substantial gains across diverse reasoning and state-tracking benchmarks.

Background & Motivation

Modern large language models increasingly support extended context windows spanning hundreds of thousands of tokens, yet relying solely on standard self-attention over raw long sequences remains computationally prohibitive and cognitively brittle. Standard attention scales quadratically with input length, driving up inference latency and memory footprints. More fundamentally, scaling context windows does not resolve attention dilution; models frequently suffer from lost-in-the-middle degradation and general context rot, struggling to locate, combine, and aggregate critical evidence scattered across hundreds of pages. In parallel, standard Retrieval-Augmented Generation (RAG) circumvents full-context attention by retrieving candidate chunks at query time. However, RAG treats input context as passive, static raw text and defers all state tracking, evidence consolidation, and conflict resolution to the point when a query arrives, leading to severe computational redundancy and compounded retrieval errors when answers depend on dynamic or latent states.

This query-time deferral exposes a fundamental deficiency in existing memory-augmented frameworks. In real-world conversational or analytical streams, an agent must process a sequence of updates once and support multiple subsequent queries across diverse topics—a one-to-many setup. Existing methods either rely on brittle, prompt-engineered heuristic rules without direct policy optimization, or adopt query-conditioned memory generation where the entire long text must be repeatedly scanned for every specific query. Furthermore, attempting to jointly optimize query-agnostic memory construction alongside downstream question-answering using standard reinforcement learning algorithms (such as PPO or vanilla GRPO) fails due to severe gradient variance: different questions have vastly disparate intrinsic difficulties, and actions taken during memory construction lack immediate per-step reward signals.

To overcome the dichotomy between passive retrieval and query-specific rescanning, this work formulates long-horizon information processing as a dual-loop Markov Decision Process where a single policy autonomously maintains an external structured store and leverages both structured and unstructured retrieval at query time. Core idea: train a Unified Memory Agent (UMA) via Task-Stratified GRPO, where the upstream memory maintenance policy is supervised by the mean reward of multiple branching downstream question-answering trajectories while normalizing advantages separately across memory rollouts and question-specific groups to achieve durable, reusable state tracking.

Method

Overall Architecture

UMA models long-context reasoning over an incoming stream of text chunks \(C = \{c_1, \dots, c_N\}\) via a two-phase decision process: Phase I handles sequential memory maintenance, and Phase II performs hybrid retrieval-augmented question answering. The agent maintains a dual-component memory state \(M_t = (m^\text{core}_t, B_t)\), where \(m^\text{core}_t\) represents a high-level concise summary capturing global topics, and \(B_t\) denotes an external key-value structured Memory Bank. At each decision step \(t\), the unified policy \(\pi_\theta(a_t|s_t)\) consumes a four-part concatenated input: system instructions \(I_\text{sys}\), current core memory \(m^\text{core}_t\), current focus \(x_t\) (chunk \(c_k\) during Phase I, user query \(q\) during Phase II), and the interaction history \(h_t\) buffering recent tool calls and execution observations.

During Phase I, the agent processes chunks sequentially through an inner loop of structured CRUD operations until issuing a plain-text response that is interpreted as an implicit UpdateCore terminal action, advancing the stream and clearing the short-term buffer. Once all chunks are ingested, the system switches to Phase II where \(x_t\) is bound to incoming query \(q\). The agent issues read-only operations across both the structured memory bank and the raw-context index (via BM25 and dense embeddings) before returning a plain-text response mapped to the Answer terminal action.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Streaming Context / Document Chunks"] --> B["Sequential Memory Maintenance (Phase I)"]
    B --> C["Dual-Loop Dynamics & CRUD Terminal Actions"]
    C --> D["Hybrid Retrieval-Augmented QA (Phase II)"]
    D --> E["Task-Stratified GRPO Optimization"]
    E --> F["Downstream Reasoning & Continuous State Tracking"]

Key Designs

1. Sequential Memory Maintenance (Phase I): solving query-time deferral and unorganized context accumulation through proactive key-value state persistence

Deferring information extraction to query time forces models to repeatedly reprocess raw logs, which inevitably fails when intermediate states require incremental aggregation. UMA addresses this by digesting context chunks \(c_k\) proactively before queries are revealed. In Phase I, the agent accesses a dedicated write-and-read toolset \(\mathcal{A}_\text{mem}\) comprising memory_add, memory_update, memory_delete, memory_key_retrieve, and memory_list. When processing each chunk, the policy actively identifies key entities, state modifications, or numerical records, checks existing keys to prevent duplicates, and executes CRUD operations to update \(B_t\). Simultaneously, the policy maintains the evolving core memory \(m^\text{core}_t\) as a high-level summary of 1–3 sentences focusing strictly on macro-level themes without cluttering detailed attributes. This decoupled representation keeps global context manageable while offloading granular, evolving facts to a reliable external store.

2. Dual-Loop Dynamics & CRUD Terminal Actions: solving context explosion and multi-turn boundary confusion via hierarchical loop transitions and implicit terminal parsing

Naively concatenating multi-turn agent tool traces across long documents rapidly overwhelms the model's context window. UMA formulates system evolution through nested transitions. Within an individual chunk \(c_k\), the agent executes an Inner Loop: tool calls and environment observations append to short-term interaction history \(h_t\) while \(x_t = c_k\) remains fixed. Crucially, terminal actions are not exposed as executable tools but are handled by controller-level parsing conventions. When the model outputs a structured function call within XML tags, the tool executor runs the command; when the model outputs natural language without tool tags, the controller treats this as an implicit UpdateCore terminal action. This triggers the Outer Loop: the text commits directly to \(m^\text{core}_t\), the interaction history \(h_t\) is flushed to \(\emptyset\), and the focus advances to \(c_{k+1}\). In Phase II, plain-text output identically maps to the Answer terminal action. This design bounds memory usage, prevents context drift, and enforces clean operational boundaries.

3. Hybrid Retrieval-Augmented QA (Phase II): solving information bottlenecks of structured-only memory and unstructured noise via dual-channel evidence selection

Relying entirely on pre-structured memory risks catastrophic failure if an unmodeled fact was omitted during Phase I, whereas searching only raw text suffers from retrieval noise and lacks state aggregation. During Phase II, when confronted with user query \(q\), UMA provides the agent with an expanded retrieval toolset \(\mathcal{A}_\text{qa}\). The policy can execute structured point queries (memory_key_retrieve and memory_list) over the Memory Bank \(B_N\), and simultaneously invoke lexical and semantic retrievers (memory_bm25_retrieve and memory_embedding_retrieve using Reciprocal Rank Fusion over MiniLM-L6 embeddings) over the raw context chunks. Empirical tracing reveals that for state-tracking queries, the agent accesses the structured memory bank in 100% of cases, with 92.2% relying exclusively on structured records and 7.8% supplementing with raw context chunks. This demonstrates that structured memory acts as the primary deterministic backbone while raw-context search provides a safety net for edge-case details.

4. Task-Stratified GRPO Optimization: solving delayed reinforcement and cross-query difficulty variance in one-to-many settings via multi-QA mean outcome backpropagation and stratified advantage grouping

Training a query-agnostic memory policy presents a major credit assignment challenge because memory maintenance occurs long before queries are observed. UMA introduces Task-Stratified GRPO with a nested rollout scheme: for each context \(C\), the system samples \(G=16\) independent memory trajectories \(\tau^i\), each yielding a finalized memory bank \(M^i_N\). Next, a shared set of \(M\) diverse questions \(\{q_j\}_{j=1}^M\) is sampled, and each memory bank branches into \(M\) independent multi-turn QA sessions \(s^\text{qa}_{i,j}\). QA session rewards combine tool-call validity and final answer correctness: \(R^\text{qa}_{i,j} = \lambda r^\text{tool}_{\text{qa},i,j} + r^\text{outcome}_{i,j}\) (with \(\lambda=0.5\)). Each upstream memory session \(s^\text{mem}_{i,k}\) receives its own tool validity reward augmented by the average outcome across all \(M\) downstream QA sessions sharing that trajectory:

\[\bar{r}^\text{outcome}_i = \frac{1}{M} \sum_{j=1}^M r^\text{outcome}_{i,j}, \quad R^\text{mem}_{i,k} = \lambda r^\text{tool}_{\text{mem},i,k} + \bar{r}^\text{outcome}_i\]

To prevent cross-question variance from destabilizing training, advantage estimation is stratified: QA advantages \(A^\text{qa}_{i,j}\) are normalized strictly within the group of \(G\) rollouts answering the identical question \(q_j\) across different memory states:

\[A^\text{qa}_{i,j} = \frac{R^\text{qa}_{i,j} - \bar{R}^\text{qa}_{\cdot,j}}{\sigma(R^\text{qa}_{\cdot,j}) + \epsilon_\text{std}}\]

In contrast, memory advantages \(A^\text{mem}_{i,k}\) are normalized globally across all \(G \times N\) memory sessions sampled in the update. The shared policy \(\pi_\theta\) is then optimized end-to-end via a unified surrogate objective:

\[\mathcal{J}(\theta) = \frac{1}{GN + GM} \left( \sum_{i=1}^G \sum_{k=1}^N \frac{1}{|s^\text{mem}_{i,k}|} \sum_{t} \ell(\rho_t, A^\text{mem}_{i,k}) + \sum_{i=1}^G \sum_{j=1}^M \frac{1}{|s^\text{qa}_{i,j}|} \sum_{t} \ell(\rho_t, A^\text{qa}_{i,j}) \right)\]

where \(\ell(\rho_t, A)\) applies standard clipped PPO importance weighting penalized by KL divergence from a frozen reference model (\(\beta=0.001\)). This directly aligns query-agnostic memory writing with multi-task downstream utility.

Loss & Training

The framework builds on a customized veRL system and initializes from Qwen/Qwen3-4B-Instruct-2507. Training runs on 32 NVIDIA H200 GPUs using AdamW with a learning rate of \(1 \times 10^{-6}\), reward balance \(\lambda = 0.5\), and KL penalty \(\beta = 0.001\). Two checkpoints are developed: - UMA-Generalist: Trained for 2 epochs on modified HotpotQA (with 200 injected distractors) and a filtered Mem-\(\alpha\) corpus (covering SQuAD, PerLTQA, NLU, TREC-Coarse, PubMed-RCT, and BookSum, while strictly excluding HotpotQA and LongMemEval test sets to prevent contamination). It serves as the generalist for TTL/AR benchmarks and tests zero-shot transfer on Ledger-QA. - UMA-Specialist: Trained for 6 epochs on the synthetic Ledger-QA training split (\(H=10\) horizon, 200 instances, >6,000 questions) to assess dedicated state-tracking adaptation. Total multi-turn session token budget is fixed at 16,384 tokens (8,192 allocated to prompt, 8,192 shared cumulatively across tool calls and outputs).

Key Experimental Results

Main Results

On 13 established long-context benchmarks comprising 6 Test-Time Learning (TTL) and 7 Accurate Retrieval (AR) tasks, UMA-Generalist was evaluated at a 16k context budget using DeepSeek-V3.2 as LLM-as-a-Judge:

Category / Benchmark Concat RAG (top-20) MemAgent MemAgent-woq Mem-α Mem-T UMA-Generalist (16k)
TTL: Banking77 72.00 80.00 25.00 27.00 83.00 67.00 87.00
TTL: Clinic 79.00 70.00 13.00 41.00 77.00 52.00 91.00
TTL: NLU 71.00 65.00 22.00 34.00 68.00 58.00 82.00
TTL: PubMed-RCT 57.80 51.20 49.90 44.40 56.50 46.80 67.20
TTL: TREC-Coarse 71.00 9.00 61.00 70.00 78.00 15.00 94.00
TTL: TREC-Fine 18.00 14.00 37.00 44.00 62.00 7.00 75.00
AR: ConvoMem 2.46 51.12 59.60 1.56 7.14 51.12 87.05
AR: HotpotQA 63.28 78.12 51.56 11.72 79.69 66.41 76.56
AR: LoCoMo 30.01 54.03 60.37 25.43 40.85 41.04 46.58
AR: LongMemEval 31.60 59.60 55.40 7.60 53.80 52.40 51.60
AR: MSC 58.40 54.60 50.20 29.40 56.80 65.80 66.20
AR: PerLTQA 36.00 83.75 65.25 23.50 74.50 56.75 74.50
AR: SQuAD 63.38 25.35 77.02 25.01 81.92 67.03 87.53
TTL 6-Task Average 61.47 48.20 34.65 43.40 70.75 40.97 82.70
AR 7-Task Average 40.73 58.08 59.91 17.75 56.39 57.22 70.00
13-Task Overall Average 50.30 53.52 48.25 29.59 63.02 49.72 75.86

On the Ledger-QA state-tracking benchmark, models were tested across session horizons \(H \in \{2, 5, 10, 20, 30, 40, 50, 100, 500\}\):

Model / Variant LQA Trained \(H=2\) \(H=10\) \(H=50\) \(H=100\) \(H=500\) Overall Avg (Judge) Overall Avg (EM)
Concat No 63.11 21.36 12.38 7.69 1.92 17.76 7.92
RAG No 80.58 21.36 10.48 8.65 5.77 23.59 0.11
MemAgent No (Query-conditioned) 81.55 42.72 32.38 29.81 11.54 40.52 19.28
MemAgent-woq No (Query-agnostic) 82.52 9.71 7.62 4.81 2.88 19.06 0.61
Mem-α No 60.19 34.95 22.86 14.42 9.62 26.18 0.11
UMA (w/o RL) No 58.25 29.13 32.38 25.00 9.62 30.45 19.09
UMA-Generalist (16k) No (Zero-shot) 73.79 42.72 44.76 21.15 11.54 38.18 34.39
UMA-Specialist (16k) Yes (Task-adapted) 87.38 63.11 45.71 34.62 25.00 53.02 49.39

Ablation Study

The table below isolates the individual contributions of architectural choices, RL optimization, advantage grouping, and token budgets (evaluated at the 16k budget using Judge scores):

Ablation Variant TTL Avg AR Avg 13-Task Avg Drop vs Full Ledger-QA (Judge Avg)
UMA-Generalist (Full) 82.70 70.00 75.86 - 38.18
w/o RL (untrained backbone policy) 56.37 62.58 59.71 -16.15 30.45
w/o Phase I (no sequential memory maintenance) 65.83 59.85 62.61 -13.25 -
w/o RL & Phase I (neither RL nor maintenance) 23.38 58.28 42.17 -33.69 11.99
w/o Raw-Context (disable BM25/Dense raw retrieval) 58.38 57.19 57.74 -18.12 52.51 (under Specialist)
Global Group (no task stratification in GRPO) 74.08 66.42 69.96 -5.90 -
Two-Stage (disjoint memory and QA training) 69.50 60.50 64.65 -11.21 -
Session budget: UMA-Generalist (8k) 82.10 62.33 71.45 -4.41 44.74 (under Specialist)
Session budget: UMA-Generalist (32k) 85.67 69.13 76.76 +0.90 53.34 (under Specialist)

Key Findings

  • Compounding impact of structured maintenance and RL: Removing Phase I memory maintenance drops performance by 13.25 points. Even without RL, structured maintenance outperforms pure retrieval by 17.54 points (59.71 vs 42.17), indicating that structured CRUD state storage provides an inherent inductive bias for retention. End-to-end RL amplifies this, driving an overall 16.15 point gain by learning to actively update existing records (Update operations account for 32.2% of write events).
  • Necessity of task-stratified advantage normalization: Merging memory rollouts and QA rollouts into a single unstratified group (Global Group) decreases overall performance by 5.90 points. This confirms that normalizing within question groups is essential to decouple the intrinsic variation of question difficulty from the evaluation of memory quality.
  • Superiority over two-stage decoupled optimization: Training memory maintenance and question answering as separate pipelines (Two-Stage) drops the overall score by 11.21 points. Joint optimization ensures that memory write decisions directly optimize downstream multi-question utility.
  • Robustness at ultra-long horizons: Under the extreme \(H=500\) horizon on Ledger-QA (>400k tokens), Concat and RAG drop to 1.92 and 5.77 respectively, and MemAgent falls to 11.54. UMA-Specialist maintains a score of 25.00, demonstrating that structured memory prevents the catastrophic decay observed in sliding-window or passive retrieval methods.
  • Dramatic inference efficiency: On a 528k-token ConvoMem context with 10 queries, UMA-Generalist achieves an inference latency of 107.32s, running 44.3x faster than MemAgent (4753.26s) and 80.2x faster than Mem-T (8611.18s). Because UMA builds memory once and amortizes it over subsequent questions, QA latency remains nearly constant (~17–22s) as context grows from 20 to 500 sessions.

Highlights & Insights

  • Downstream mean utility feedback: Supervising upstream query-agnostic memory writing via the average outcome \(\bar{r}^\text{outcome}_i\) across branched downstream QA sessions provides an elegant solution to the fundamental lack of step-wise labels in proactive memory construction.
  • Stratified advantage normalization: Decoupling question-level normalization from memory-level normalization prevents difficult questions from penalizing competent memory states, stabilizing hierarchical reinforcement learning without an explicit critic network.
  • Dual-channel evidence access: Providing both structured point lookups and dense/sparse text retrieval resolves the trade-off between structured state tracking and raw unstructured evidence retention; in practice, 92.2% of complex tracking queries are solved purely through structured lookups.
  • Arithmetic control validation: A 100,000-sample isolated addition control test confirmed that UMA-Specialist does not alter the underlying model's raw arithmetic accuracy (25.14% vs 25.41% baseline), conclusively proving that improvements stem from state tracking and retrieval policies rather than mathematical overfitting.

Limitations & Future Work

  • Single-agent fixed API boundary: The framework is currently constrained to a single agent operating over a fixed tool schema; scaling to multi-agent collaborative memory spaces with evolving tool registries remains an open challenge.
  • Session-level credit attribution: Downstream rewards are averaged across entire sessions, meaning all CRUD operations within a single chunk share identical advantage values, lacking fine-grained counterfactual attribution for individual edits.
  • Synthetic-to-real transfer gap: Ledger-QA is a clean synthetic diagnostic with programmatic transaction semantics; validating proactive state maintenance on real-world messy dialogs with conversational ambiguities and user memory revocations requires further study.
  • vs MemAgent (Yu et al., 2026a): MemAgent uses query-conditioned recurrent summarization, forcing the model to rescan the full context for each new question (\(O(M \times N)\)). UMA constructs a persistent query-agnostic state once and answers queries in parallel (\(O(N + M)\)), resulting in a 44x inference speedup and superior multi-task retention.
  • vs Mem-α (Wang et al., 2025b): While Mem-α also considers shared memory, it optimizes the memory writer using a fixed external RAG reader. UMA unifies both memory maintenance and QA into a single trainable policy optimized end-to-end via Task-Stratified GRPO.
  • vs A-MEM (Xu et al., 2025) & MEM1 (Zhou et al., 2026): A-MEM relies on prompt-engineered memory evolution without RL training, and MEM1 interleaves retrieval within a single trace without persistent bank consolidation. UMA demonstrates that reinforcement learning over simple, explicit CRUD operations produces substantially more resilient state tracking over long horizons.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates query-agnostic memory management as an end-to-end MDP with Task-Stratified GRPO, elegantly solving the credit assignment and variance challenges of one-to-many memory systems.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 13 diverse benchmarks, a new long-horizon state tracking suite (Ledger-QA), extensive ablation studies, and a 100k arithmetic control experiment.
  • Writing Quality: ⭐⭐⭐⭐⭐ Crisp mathematical formulation, intuitive framework diagrams, comprehensive appendix documentation, and insightful error breakdowns.
  • Value: ⭐⭐⭐⭐⭐ Provides a highly practical, open-source architectural blueprint for building scalable memory agents that transcend the limitations of ultra-long context windows and naive RAG.