DeltaMem: Towards Agentic Memory Management via Reinforcement Learning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Agent
Keywords: Agentic Memory Management, Reinforcement Learning, Single-Agent Architecture, State Transition Evaluation, Memory Evolution Synthesis
TL;DR¶
DeltaMem replaces fragile multi-agent memory pipelines with an end-to-end single-agent ReAct framework, leveraging human-inspired memory evolution synthesis and an optimal-transport memory Levenshtein distance reward to train long-term persona memory managers via GRPO.
Background & Motivation¶
Building large language model agents capable of long-term personal interaction requires robust mechanisms to extract, retrieve, and update user-centric memories across temporal horizons. Early systems like MemGPT and MemoryBank drew inspiration from operating systems, maintaining external storage through atomic create, read, update, and delete (CRUD) operations via static retrieval-augmented generation. Contemporary production systems, notably Mem0, transitioned to multi-agent architectures that orchestrate specialized agents across a rigid three-step pipeline: memory extraction, memory retrieval, and memory updating. However, real-world long-term dialogues are inherently fragmented across sliced interaction sessions. Enforcing discrete message-passing handovers across decoupled agents atop already-fragmented data acts as a lossy compression layer, stripping away subtle contextual dependencies and causing cascading information attenuation.
Furthermore, efforts to optimize memory managers through machine learning remain severely constrained by the lack of operation-level supervision in long-term conversational datasets. Prior attempts, such as Memory-R1, resort to training updating modules in multi-agent pipelines using downstream question-answering accuracy as a proxy reward in reinforcement learning. This formulation suffers from severe sparse reward bottlenecks: task questions typically probe only a tiny subset of memory entries from a session, and downstream answer models are themselves prone to reasoning failures. Consequently, using end-task correctness to backpropagate credit to intermediate memory operations yields noisy, high-variance optimization that struggles to converge.
To overcome both cascading pipeline loss and indirect reward sparsity, this work re-envisions memory management from first principles along two axes: system architecture and optimization objectives. Structurally, DeltaMem consolidates memory management into an autonomous single-agent decision process under the ReAct paradigm, empowering a unified agent to dynamically determine what to search, what to integrate, and how to update. Methodologically, inspired by the temporal evolution and cognitive plasticity of human memory, the framework synthesizes high-fidelity longitudinal dialogues paired with ground-truth operation sets, shifting the evaluation paradigm from fragile operation matching to holistic memory state transitions. Core idea: reformulate persona memory management as a single-agent end-to-end decision task, generating causally grounded interaction trajectories via human-inspired memory evolution and optimizing trajectory-level policies under GRPO using a memory-based Levenshtein distance reward grounded in optimal transport and local lexical fidelity.
Method¶
Overall Architecture¶
DeltaMem conceptualizes long-term memory maintenance as an end-to-end state transition system. The memory bank is formalized as a finite partial function \(S_t: \mathcal{I} \rightharpoonup \mathcal{M}\) mapping unique memory identifiers \(\mathcal{I}\) to factual atomic statement-timestamp tuples \(\mathcal{M}\). Upon ingesting a chronological dialogue session \(d_{t+1}\) at timing \(t\), rather than dispatching tasks across separate extraction, retrieval, and updating subagents, a single memory manager executes an autonomous ReAct loop. It actively reasons over current dialogue context, generates search queries to retrieve relevant historical entries via vector search, and outputs a consolidated set of memory operations \(O \subseteq \{\text{add}, \text{update}\}\) that deterministically transitions the memory state from \(S_t\) to \(S_{t+1} = f(S_t, d_{t+1})\).
To train this agentic manager without relying on noisy downstream QA feedback, DeltaMem couples a four-stage memory evolution data synthesis pipeline with an outcome-based reinforcement learning algorithm. The policy generates rollouts of memory operations against the current state, and the resulting candidate memory states are evaluated against synthetic target states using a novel Memory-based Levenshtein Distance. This distance computes the semantic transition cost via optimal transport and keyword coverage, providing a dense soft F1 reward that guides Group Relative Policy Optimization (GRPO) to iteratively refine memory management policies.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Dialogue Session & Current Memory State"] --> B["End-to-End Single-Agent Memory Management<br/>Autonomous ReAct reasoning and search tool invocation"]
B --> C["Memory Evolving Guided Data Synthesis<br/>Profile enrichment โ event generation โ memory evolution โ dialogue"]
C --> D["Memory-based Levenshtein Distance & State Evaluation<br/>Optimal transport semantic matching + local lexical fidelity"]
D --> E["Homologous Reinforcement Learning Optimization<br/>Format reward + retrieval reward + state transition soft F1 + GRPO"]
Key Designs¶
1. End-to-End Single-Agent Memory Management: Eliminating Cascading Pipeline Loss and Heavy Retrieval
Conventional multi-agent frameworks suffer from inter-agent context truncation and rigid query heuristics, triggering an automatic retrieval lookup for every single extracted memory candidate regardless of whether it requires updating. DeltaMem unifies extraction, retrieval reasoning, and update decisions within a single agentic ReAct workflow. When processing a new conversation session, the agent analyzes conversational context within a single context window, dynamically formulates batched search queries via search_memory, and observes retrieved entries before emitting structured operations (ADD, UPDATE, or NONE). This end-to-end reasoning preserves subtle cross-turn temporal context and drastically cuts retrieval overhead, avoiding redundant queries on stable factual memories.
2. Memory Evolving Guided Data Synthesis: Creating Operation-Level Ground Truth via Cognitive Principles Because real-world conversational corpora lack granular, operation-level annotations for memory bank updates, DeltaMem introduces a four-step cognitive synthesis pipeline spanning a 1-3 year horizon across 300 personas. The process progresses through Profile Enrichment (seeding immutable traits and initial dynamic facts), Event Generation (synthesizing \(N=25\) chronological events with state-delta vectors across career, health, relationships, and preferences), Memory Generation, and Dialogue Generation. Memory Generation explicitly models human cognitive plasticity via two complementary mechanisms: Intra-Event Evolving modifies user attributes using expand, adjust, shift, and partial deletion operators under an information-preservation invariant that requires reproducing the exact prior string; Inter-Event Evolving retroactively recontextualizes stable memory anchors from earlier events via emotional updates, semantic reinterpretation, and self-belief shifts. Finally, a Memory Filling step populates non-redundant factual additions, and a Dialogue Generation module translates these knowledge triplets into multi-turn conversations under a strict "user-first" information flow and a pre-calculated fact introduction schedule.
3. Memory-based Levenshtein Distance: State-Centric Semantic Transition Metric Direct operation-level matching is brittle because natural language allows multiple equivalent operation sequences to achieve identical memory states. DeltaMem shifts evaluation to the induced memory states by computing relative complements between the predicted state \(S_{\text{pred}}\) and target state \(S_{\text{target}}\): \(\Delta_{\text{pred}} = S_{\text{pred}} \setminus S_{\text{target}}\) (redundant predicted memories) and \(\Delta_{\text{target}} = S_{\text{target}} \setminus S_{\text{pred}}\) (missing target memories). To accommodate lexical variance while enforcing semantic alignment, the method formalizes mapping between \(\Delta_{\text{pred}}\) and \(\Delta_{\text{target}}\) as an Optimal Transport problem over a cosine similarity matrix \(\Phi\): $\(\gamma^* = \arg\max_{\gamma \in \Gamma} \sum_{(i,j) \in \gamma} \Phi_{ij}\)$ To prevent high embedding similarity from rewarding factually incomplete statements, each aligned pair is gated by a local lexical fidelity score \(s_{ij}\), measuring the exact coverage of target keyword set \(K_j\) in predicted entry \(x_i\): $\(s_{ij} = \frac{\sum_{k \in K_j} \mathbb{I}(k \in x_i)}{|K_j| + \epsilon}\)$ Pairs below threshold \(\tau\) are filtered. The resulting bidirectional Memory-based Levenshtein Distance yields soft precision \(P_{\text{soft}}\) and soft recall \(R_{\text{soft}}\), combining into a state-transition soft F1 reward \(r_{\text{trans}} = \frac{2 P_{\text{soft}} R_{\text{soft}}}{P_{\text{soft}} + R_{\text{soft}} + \epsilon}\) that penalizes both under-generation and hallucinated over-generation.
4. Homologous Reinforcement Learning Optimization: Direct Trajectory Credit Assignment via GRPO
To optimize the policy model without downstream task dependencies, DeltaMem defines a composite reward function balancing syntactical protocol adherence, memory grounding, and state transition accuracy:
$\(r = 0.1 \cdot r_{\text{format}} + 0.1 \cdot r_{\text{retrieval}} + 0.8 \cdot r_{\text{trans}}\)$
Here, \(r_{\text{format}}\) verifies valid ReAct syntax, while \(r_{\text{retrieval}}\) ensures that existing memory IDs referenced in update and none operations were actively recalled from the database. Training is conducted using Group Relative Policy Optimization (GRPO) on 8 rollouts per session, normalizing advantage estimates within the sampled group and constraining updates via a KL divergence penalty against the reference model. This design provides immediate, dense credit assignment directly tied to memory bank quality.
A Worked Example¶
Consider an interaction session featuring user Christopher Martinez discussing updates to his personal routine:
- Input Context: The user mentions, "Christopher adjusts his wardrobe, swapping the navy-and-white ASU hoodie for a charcoal bomber jacket on cooler evenings."
- ReAct Autonomous Reasoning & Tool Use: DeltaMem inspects the conversation in its internal thinking trace and autonomously invokes search_memory with a targeted query list ("Christopher Martinez navy and white Appalachian State hoodie", "Christopher Martinez charcoal bomber jacket", "Christopher Martinez wardrobe adaptation seasons").
- Retrieval Feedback: The database returns existing historical entries, including memory ID 0 (initial league setup) and past wardrobe records.
- Consolidated Operation Output: Inside <outputs>, DeltaMem emits an explicit operation set: marking irrelevant historical memories with "event": "NONE", issuing an UPDATE on prior apparel preferences, and outputting an ADD for "Christopher Martinez likes charcoal bomber jacket on cooler evenings."
- State Transition Reward: The resulting pseudo-memory state is compared to the synthetic ground-truth state; optimal transport pairs the predicted jacket fact with target annotations, and local keyword matching confirms the presence of "charcoal" and "bomber jacket", awarding a high \(r_{\text{trans}}\) reward that reinforces this policy trajectory.
Key Experimental Results¶
Main Results¶
DeltaMem is evaluated on three major long-term memory benchmarks: LoCoMo (long-range dialogues averaging 300 turns and 9K tokens), HaluMem (operation-level benchmark decomposing extraction, updating, and QA hallucination), and PersonaMem (evaluating longitudinal preference tracking). All training-free baselines and downstream QA answer modules utilize GPT-4o-mini or GPT-4o.
Table 1 presents the primary performance on LoCoMo across four question categories:
| Method | Single Hop (LJ) | Multi Hop (LJ) | Open Domain (LJ) | Temporal (LJ) | Overall B1 | Overall F1 | Overall LJ |
|---|---|---|---|---|---|---|---|
| A-Mem | 54.05 | 39.79 | 18.85 | 49.91 | 24.82 | 29.96 | 48.38 |
| Mem0 (Multi-Agent) | 60.88 | 52.84 | 40.62 | 59.50 | 33.79 | 42.55 | 57.86 |
| LangMem | 71.12 | 62.23 | 47.92 | 23.43 | 30.06 | 36.88 | 58.11 |
| Zep (Proprietary) | 76.60 | 61.70 | 41.35 | 49.31 | 33.64 | 43.57 | 65.99 |
| LightMem | 72.06 | 53.90 | 44.79 | 69.16 | 34.17 | 44.81 | 66.43 |
| Memory-R1-7B (RL) | 67.81 | 49.61 | 20.71 | 69.16 | 26.06 | 33.64 | 62.34 |
| DeltaMem-4B (Training-free) | 75.62 | 61.35 | 47.92 | 70.09 | 37.73 | 47.51 | 70.13 |
| DeltaMem-8B (Training-free) | 77.41 | 61.70 | 46.88 | 69.78 | 36.63 | 47.27 | 71.04 |
| DeltaMem-4o-mini (Training-free) | 80.38 | 58.87 | 47.92 | 73.21 | 39.81 | 50.32 | 72.92 |
| DeltaMem-4B-RL | 80.02 | 65.60 | 51.04 | 72.59 | 39.29 | 50.13 | 74.03 |
| DeltaMem-7B-RL | 78.95 | 62.77 | 44.79 | 71.03 | 38.86 | 49.81 | 72.21 |
| DeltaMem-8B-RL | 82.05 | 64.18 | 51.04 | 73.83 | 39.78 | 50.72 | 75.13 |
On the fine-grained HaluMem benchmark (Table 3), DeltaMem demonstrates superior balance across memory extraction, memory updating, and question answering:
| Method | Memory Extraction (%) | Memory Updating Correct (%) | Question Answering Correct (%) |
|---|---|---|---|
| Zep | - | 47.28 | 55.47 |
| Mem0 | 57.31 | 25.50 | 53.02 |
| Mem0-Graph | 57.85 | 24.50 | 54.66 |
| Memobase | 25.13 | 5.20 | 35.33 |
| Supermemory | 56.90 | 16.37 | 54.07 |
| LightMem | 70.80 | 7.43 | 58.38 |
| DeltaMem-4B (Training-free) | 65.06 | 25.40 | 60.14 |
| DeltaMem-8B (Training-free) | 68.02 | 31.77 | 62.50 |
| DeltaMem-4o (Training-free) | 73.13 | 35.11 | 60.89 |
| DeltaMem-4B-RL | 79.26 | 44.55 | 67.93 |
| DeltaMem-8B-RL | 80.32 | 48.81 | 68.62 |
Ablation Study¶
The authors ablated the synthetic data generation modules and the semantic matching threshold \(\tau\).
Table 5 evaluates the impact of removing individual memory evolution modules during data synthesis (evaluated on DeltaMem-8B-RL):
| Configuration | LoCoMo B1 | LoCoMo F1 | LoCoMo LJ | PersonaMem Overall | Preference Evolution (Pref-Evol) | Note |
|---|---|---|---|---|---|---|
| DeltaMem-8B-RL (Full) | 39.78 | 50.72 | 75.13 | 65.03 | 57.55 | Full pipeline with intra- and inter-event evolution |
| w/o Inter-Event Evolving | 39.65 | 50.48 | 73.83 | 61.46 | 54.68 | Drops 3.57 on PersonaMem; misses retrospective anchor updates |
| w/o Intra-Event Evolving | 38.16 | 49.22 | 73.51 | 62.82 | 52.52 | Drops 5.03 on preference tracking; lacks fine-grained state deltas |
Table 4 reports sensitivity to the optimal transport similarity threshold \(\tau\) on LoCoMo:
| Threshold \(\tau\) | 0.00 | 0.50 | 0.60 | 0.70 | 0.75 (Default) | 0.80 | 0.90 |
|---|---|---|---|---|---|---|---|
| BLEU-1 | 38.40 | 37.12 | 38.35 | 39.24 | 39.87 | 39.74 | 38.32 |
| F1 | 48.92 | 47.93 | 48.75 | 50.37 | 50.72 | 50.78 | 48.75 |
Key Findings¶
- Single-Agent Paradigm Outperforms Complex Pipelines: Even without RL training, the training-free DeltaMem-8B achieves 71.04 LJ on LoCoMo, comfortably outperforming production-grade multi-agent architectures including Mem0 (57.86) and Zep (65.99). This confirms that discrete inter-agent message passing induces severe cascading information attenuation.
- RL Training Unlocks Substantial Updating and Generative Gains: RL fine-tuning with the state-transition reward boosts memory updating accuracy on HaluMem from 31.77% to 48.81%, while Question Answering rises to 68.62%. On PersonaMem, DeltaMem-8B-RL achieves a 65% relative improvement in generating personalized recommendations (New-Ideas rises from 24.73 to 40.86) while maintaining rigorous factual recall (Recall-Facts reaches 82.35 in DeltaMem-4B-RL).
- Drastic Mitigation of Heavy Retrieval: On LoCoMo, traditional multi-agent systems incur 1.00 queries per memory entry (QPM) due to static lookup rules. In contrast, RL-trained DeltaMem autonomously decides when retrieval is necessary, dropping QPM to 0.13 while achieving state-of-the-art performance.
Highlights & Insights¶
- Unified End-to-End State Machine: Unifying extraction, search reasoning, and update decisions within a single ReAct agent eliminates the semantic boundary mismatches of multi-agent handoffs while enabling joint credit assignment.
- Optimal Transport State-Transition Reward: Mapping discrete natural language memory sets into continuous semantic distance via optimal transport and keyword coverage solves the notorious reward-sparsity bottleneck of training memory agents with downstream task rewards.
- Longitudinal Synthetic Evolution: Designing causal invariants for synthetic dialoguesโsuch as requiring models to reproduce prior state strings and enforcing user-first information flowsโprovides an effective recipe for generating clean, privacy-free training data for memory agents.
Limitations & Future Work¶
- Domain Gap of Synthetic Dialogues: The training dataset relies entirely on synthetic dialogues generated by high-capacity LLMs (gpt-oss-120b). Real user interactions often feature colloquial abbreviations, erratic turn interruptions, conflicting preferences, and emotional ambivalence that synthetic data may under-represent.
- Sensitivity to Embeddings and Hyperparameters: The reward function relies on semantic embeddings and a tuned threshold \(\tau=0.75\). Suboptimal embedding spaces or out-of-distribution technical jargon could misalign optimal transport assignments.
- Future Directions: Extending the framework to multi-modal personal interactions (e.g., embodied visual lifelogs and audio streams) and investigating hybrid semi-supervised adaptation with human feedback.
Related Work & Insights¶
- vs MemGPT & MemoryBank: While earlier systems pioneered OS-like CRUD operations over external vector databases, they relied on rigid, reactive retrieval rules; DeltaMem introduces active ReAct reasoning and RL-optimized dynamic state transitions.
- vs Mem0 & LightMem: Modern multi-agent pipelines decouple extraction, retrieval, and updating, incurring cascading information loss and exhaustive retrieval overhead; DeltaMem's single-agent architecture reduces queries per memory by 87% while achieving higher temporal reasoning accuracy.
- vs Memory-R1: Memory-R1 optimizes modular memory updating using downstream answer correctness as a sparse proxy reward; DeltaMem formalizes intrinsic, dense memory-based Levenshtein distance rewards directly over memory states, achieving stable convergence without downstream QA model noise.
Rating¶
- Novelty: โญโญโญโญโญ Formulates persona memory management as a single-agent end-to-end ReAct task with an optimal-transport state-transition reward.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across LoCoMo, PersonaMem, and HaluMem benchmarks, featuring multiple model scales, ablation studies, query frequency analysis, and temporal stability tracking.
- Writing Quality: โญโญโญโญโญ Mathematically formal, lucidly structured, and supported by concrete case studies and clear architectural figures.
- Value: โญโญโญโญโญ Offers a transformative architectural and algorithmic blueprint for training lightweight, production-ready, personalized LLM memory systems.