Grounding Memory Summarization in Utility Intent¶
Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.358.pdf
Code: https://github.com/LzyFischer/MemSuit
Area: LLM Agent
Keywords: memory systems, memory summarization, privileged information self-distillation, retrieval calibration, agent long-horizon interaction
TL;DR¶
Addressing the core issue where conversational memory summarization optimizes human-facing criteria at the expense of fine-grained facts, MemSuit introduces privileged query-answer conditioning into an adaptive teacher, distills utility-aware multi-entry extraction into a query-agnostic student, and contrastively calibrates the dense retriever.
Background & Motivation¶
Large language model agents deployed in long-horizon interactive environments rely heavily on external memory systems to store past conversational history and retrieve pertinent fragments on demand. Verbatim storage of raw dialogues quickly exhausts context windows and incurs prohibitive latency, especially given that the vast majority of colloquial exchanges are rarely relevant to subsequent user tasks. Consequently, modern agent pipelines universally deploy memory summarizers to compress historical conversational blocks into concise representations. However, existing extraction heuristics and prompted LLM summarizers suffer from a critical flaw: they tend to condense dialogues into high-level topical gist, routinely discarding granular dates, numerical values, and nuanced causal entities that downstream reasoning tasks strictly require. This omission causes severe performance degradation, particularly on temporal and open-domain queries that hinge on specific past facts.
The root cause of this failure is a fundamental objective misalignment. Standard LLM summarizers are optimizedโeither via human reference summaries or instruction tuningโfor human-facing qualities such as topical faithfulness, grammatical coherence, and broad coverage. In a functioning agent memory pipeline, however, the summary is not consumed by a human reader; it is ingested by a downstream retriever-and-reader system. A memory entry's true value is strictly determined by whether it preserves the precise evidentiary facts necessary to answer future queriesโa criterion of downstream utility rather than human-facing fidelity. A privileged oracle study demonstrates that conditioning summarizers on observed downstream query-answer pairs nearly doubles the downstream QA F1 score. Yet, such an oracle setup is unfeasible at test time because memory entries must be committed to storage hundreds or thousands of turns before future queries are ever formulated.
Crucially, empirical investigations reveal that utility-aware summarization behaviors and evidentiary criteria exhibit strong transferability across diverse query types and conversational scenarios. The core idea is to operationalize utility-grounded memory summarization through a self-distillation framework where a privileged teacher conditioned on observed query-answer pairs adaptively decomposes dialogue blocks into self-contained factual entries, a student model learns to internalize this utility-aware decomposition from raw dialogue alone, and the dense retriever is contrastively calibrated using teacher entries to eliminate the representation gap.
Method¶
Overall Architecture¶
MemSuit resolves the objective misalignment, prevents collateral erasure of non-target facts, and closes the stylistic gap between dense retrievers and compact memory entries. During the training phase, the architecture operates through three unified stages: first, a teacher summarizer \(f_T\), equipped with privileged query-answer pairs \((q, a)\), performs adaptive entry decomposition on dialogue block \(b\) to generate a variable-length set of self-contained memory entries; second, a student summarizer \(f_{S\theta}\) observing only raw dialogue \(b\) is trained via token-level cross-entropy self-distillation to anticipate utility-relevant facts and reproduce the multi-entry structure; third, an LLM judge labels grounding entries to supervise the dense retriever \(g_\phi\) via an InfoNCE contrastive objective. At inference time, both the teacher and the privileged queries are completely discarded; the student summarizer ingests incoming conversation blocks, emits modular, self-contained entries into LanceDB vector storage, and the calibrated retriever fetches top-\(K\) entries for the fixed reader to generate grounded answers.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Dialogue History<br/>Contiguous 20-turn block b"] --> B["Adaptive Entry Decomposition<br/>Teacher conditioned on (q, a) performs joint segmentation & verification"]
B --> C["Utility-Aware Self-Distillation<br/>Student learns to generate structured entries from raw block b alone"]
B --> D["Entry-Aware Retrieval Calibration<br/>LLM judge labels positives; InfoNCE contrastive alignment"]
C -->|Inference Storage| E["Compact Memory Store<br/>LanceDB vector database of decoupled entries"]
D -->|Calibrated Retrieval| F["Top-K Memory Retrieval"]
E --> F
F --> G["Fixed Reader Model<br/>Produces concise, fact-grounded answer"]
Key Designs¶
1. Adaptive Entry Decomposition: Eliminating Collateral Erasure Under Single-Query Conditioning Naively compressing an entire conversation block \(b\) into a single monolithic summary conditioned on a specific query-answer pair \((q, a)\) heavily biases the summary toward that single query. Consequently, crucial evidence supporting other plausible future queries \((q', a') \in \mathcal{Q}_b \setminus \{(q, a)\}\) is discarded as incidental noiseโa failure mode termed collateral erasure. MemSuit resolves this by enabling the teacher to jointly segment block \(b\) and emit a variable-length sequence of self-contained entries in a single forward pass: $\(M_{b,q}^T = \{m_1^T, \dots, m_{n_{b,q}}^T\} = f_T(b, q, a), \quad n_{b,q} \ge 1\)$ Within this entry set, at least one entry \(m_{j^\star}^T\) is explicitly anchored to the observed query-answer pair \((q, a)\) to preserve the necessary grounding evidence, while the remaining entries \(\{m_j^T\}_{j \ne j^\star}\) independently summarize orthogonal semantic regions of \(b\) without query conditioning. The prompt enforces an Internal Coverage Check: the teacher drafts entries across all dialogue turns as if no query existed, and subsequently verifies that the specific evidence turn is retained verbatim with named entities and absolute ISO 8601 timestamps, entirely eliminating ambiguous pronouns (he, she, they) and relative temporal terms.
2. Utility-Aware Self-Distillation: Internalizing Downstream Utility into a Deployable Policy Because future queries are strictly unavailable at deployment, the student summarizer \(f_{S\theta}\) must generate utility-aware entries conditioned solely on raw dialogue block \(b\). The teacher's decomposed entries are serialized into a single target sequence delimited by special separator tokens, \(y_{b,q} = m_1^T \mathbin{\Vert} m_2^T \mathbin{\Vert} \dots \mathbin{\Vert} m_{n_{b,q}}^T\). The student model is trained with standard autoregressive sequence cross-entropy: $\(\mathcal{L}_{\text{distill}}(\theta) = -\mathbb{E}_{(b, q, a) \sim \mathcal{D}_{\text{tr}}} \left[ \sum_{t=1}^{|y_{b,q}|} \log p_\theta \left( y_{b,q}^{(t)} \;\middle|\; y_{b,q}^{(<t)}, b \right) \right]\)$ Through this distillation objective, the student internalizes the teacher's selection criteria across the empirical query distribution. It learns not only what fine-grained details to retain in anticipation of future questions, but also how to partition the dialogue into independently retrievable, self-contained units.
3. Entry-Aware Retrieval Calibration: Bridging the Stylistic Gap in Representation Space Off-the-shelf embedding models (such as all-MiniLM-L6-v2) are primarily pre-trained on verbose, naturally redundant prose. Consequently, they suffer from severe retrieval mismatch when searching over the compact, terse, pronoun-free entries generated by the student. MemSuit closes this gap by recycling the teacher's entries as supervision for the retriever. An LLM judge \(\psi\) identifies the subset of entries that directly ground the reference answer, establishing positive set \(\mathcal{P}_q = \psi(b, q, a, M_{b,q}^T) \subseteq M_{b,q}^T\). Negative candidates \(\mathcal{N}_q\) are mined from unrelated blocks within the training mini-batch, forming candidate pool \(\mathcal{S}_q = \mathcal{P}_q \cup \mathcal{N}_q\). With cosine similarity \(s_\phi(q, m) = \frac{\langle h_\phi(q), h_\phi(m) \rangle}{\|h_\phi(q)\| \|h_\phi(m)\|}\), the encoder is optimized via InfoNCE loss: $\(\mathcal{L}_{\text{ctr}}(\phi) = -\mathbb{E}_{q \sim \mathcal{Q}_{\text{batch}}} \left[ \sum_{m^+ \in \mathcal{P}_q} \log \frac{\exp(s_\phi(q, m^+) / \tau)}{\sum_{m \in \mathcal{S}_q} \exp(s_\phi(q, m) / \tau)} \right]\)$ This contrastive calibration pulls queries directly toward their factual grounding entries, substantially elevating the low-similarity positive tail where surface lexicons diverge but semantic utility aligns.
Loss & Training¶
The framework adopts a sequential training protocol: the teacher first runs offline over training dialogue blocks \(\mathcal{D}_{\text{tr}}\) to generate supervision entries and judge-labeled positives. The student model is fine-tuned using LoRA (\(r=16, \alpha=32\), dropout 0.05, per-device batch size 2 with gradient accumulation 8, effective batch size 16, warmup ratio 0.03 with linear decay over 3 epochs) with learning rates \(2 \times 10^{-5}\) for Llama-3.1-8B-Instruct and \(2 \times 10^{-4}\) for Qwen2.5-3B-Instruct. The embedding model all-MiniLM-L6-v2 is fine-tuned for 3 epochs under bf16 mixed precision with a batch size of 32 and learning rate \(2 \times 10^{-4}\). All experiments train efficiently on a single NVIDIA A100 80GB GPU.
Key Experimental Results¶
Main Results¶
Evaluation is conducted on the LoCoMo benchmark featuring multi-hop, temporal, open-domain, and single-hop reasoning queries across long-horizon dialogues. MemSuit is benchmarked against LoCoMo, MemoryBank, A-Mem, Mem0, and SimpleMem.
| Model | Method | Multi-hop F1 | Temporal F1 | Open-domain F1 | Single-hop F1 | Avg F1 | Avg BLEU |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | LoCoMo | 21.33 | 17.46 | 10.60 | 30.82 | 20.05 | 15.23 |
| Llama-3.1-8B-Instruct | MemoryBank | 24.60 | 12.01 | 10.47 | 38.45 | 21.38 | 17.89 |
| Llama-3.1-8B-Instruct | A-Mem | 17.08 | 24.00 | 11.93 | 29.62 | 20.66 | 15.81 |
| Llama-3.1-8B-Instruct | Mem0 | 28.98 | 19.93 | 19.73 | 45.76 | 28.60 | 23.68 |
| Llama-3.1-8B-Instruct | SimpleMem | 26.92 | 35.11 | 22.41 | 35.92 | 30.09 | 22.24 |
| Llama-3.1-8B-Instruct | MemSuit | 31.18 | 40.77 | 25.42 | 44.89 | 35.56 | 27.73 |
| Qwen2.5-3B-Instruct | LoCoMo | 16.38 | 16.45 | 4.75 | 18.57 | 14.04 | 10.31 |
| Qwen2.5-3B-Instruct | MemoryBank | 20.00 | 15.62 | 8.08 | 27.22 | 17.73 | 13.93 |
| Qwen2.5-3B-Instruct | A-Mem | 18.18 | 19.79 | 9.60 | 29.51 | 19.27 | 15.09 |
| Qwen2.5-3B-Instruct | Mem0 | 22.24 | 23.23 | 10.02 | 29.76 | 21.31 | 17.27 |
| Qwen2.5-3B-Instruct | SimpleMem | 18.83 | 16.55 | 12.44 | 26.92 | 18.68 | 15.02 |
| Qwen2.5-3B-Instruct | MemSuit | 22.63 | 28.66 | 15.40 | 33.48 | 25.04 | 19.97 |
Ablation Study¶
The ablation study validates the individual and combined contributions of Entry-Aware Retrieval Calibration (Rtr-Cal) and Utility-Aware Self-Distillation (Self-Dist).
| Model | Variant | Multi F1 | Temp F1 | Open F1 | Single F1 | Avg F1 | Avg BLEU | Note |
|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | MemSuit full model | 31.18 | 40.77 | 25.42 | 44.89 | 35.56 | 27.73 | Full model |
| Llama-3.1-8B-Instruct | w/o Rtr-Cal | 26.02 | 41.03 | 21.16 | 38.86 | 31.77 | 24.25 | -3.79 Avg F1 without retrieval calibration |
| Llama-3.1-8B-Instruct | w/o Self-Dist | 30.92 | 40.24 | 21.36 | 40.92 | 33.36 | 25.70 | -2.20 Avg F1 without self-distillation |
| Llama-3.1-8B-Instruct | w/o Cal & Dist | 23.16 | 30.88 | 17.53 | 33.38 | 26.24 | 19.98 | -9.32 Avg F1 without both components |
| Qwen2.5-3B-Instruct | MemSuit full model | 22.63 | 28.66 | 15.40 | 33.48 | 25.04 | 19.97 | Full model |
| Qwen2.5-3B-Instruct | w/o Rtr-Cal | 21.23 | 26.65 | 14.58 | 30.09 | 23.14 | 18.23 | -1.90 Avg F1 without retrieval calibration |
| Qwen2.5-3B-Instruct | w/o Self-Dist | 21.51 | 23.86 | 10.80 | 29.84 | 21.50 | 17.17 | -3.54 Avg F1 without self-distillation |
| Qwen2.5-3B-Instruct | w/o Cal & Dist | 21.05 | 20.02 | 10.36 | 29.42 | 20.21 | 16.35 | -4.83 Avg F1 without both components |
Key Findings¶
- Prominent gains on temporal and open-domain queries: On Llama-3.1-8B, MemSuit achieves 40.77 F1 on Temporal questions (+5.54 over SimpleMem, +20.84 over Mem0) and 25.42 F1 on Open-domain queries (+3.01 over SimpleMem), confirming that utility-aware distillation successfully recovers fine-grained facts omitted by vanilla summarizers.
- Retrieval calibration lifts low-similarity positive tails: On validation sets, calibration improves Llama-3.1 Recall@1 from 25.77% to 35.05% and Recall@10 from 64.43% to 69.59%. Cosine similarity distribution analysis shows that the negative distribution remains stable while the lower tail of the positive distribution shifts sharply upward.
- Adaptive decomposition outperforms fixed turn-based chunking: Replacing adaptive decomposition with fixed chunk sizes of 1, 5, or 10 turns consistently degrades F1 across all question categories. Fine granularities fragment relational facts and inject noise, whereas coarse granularities trigger collateral erasure.
- Superior storage token efficiency: MemSuit generates substantially more compact summaries than verbose baselines such as Mem0 while simultaneously achieving the highest QA accuracy, demonstrating that high evidence recall does not necessitate bloated storage budgets.
Highlights & Insights¶
- Principled Learning Under Privileged Information (LUPI): Formulating memory summarization as privileged-information self-distillation elegantly resolves the temporal paradox between early storage and late querying.
- Internal Coverage Check against collateral erasure: The prompting structure treats the conditioning query-answer pair strictly as a post-hoc verification constraint rather than a topical focus, preserving non-target conversational facets.
- Holistic memory-retrieval co-design: Recognizing that compact, factual entries break standard dense embedding models and reusing teacher entries for contrastive calibration establishes an efficient, coherent pipeline.
Limitations & Future Work¶
- Sensitivity to query distribution shift: When evaluated out-of-domain on LongMemEval without adaptation, MemSuit's performance edge narrows (28.16 F1 vs. SimpleMem's 28.91), as teacher selection reflects training query distributions; generating synthetic queries via LLMs offers a promising path forward.
- Exploration limited to small/medium backbones: The study investigates 1.7B, 3B, and 8B backbones due to GPU budgets; scaling to 32B or 70B models remains an open question to assess capacity scaling laws.
- Sequential rather than joint training: The summarizer and retriever are trained sequentially; joint optimization could enable mutual representation co-adaptation.
Related Work & Insights¶
- vs Mem0 / MemoryBank / SimpleMem: Prior systems prompt LLMs to extract facts under human-facing criteria or manage lifecycle updates without aligning with downstream reader utility; MemSuit grounds summarization directly in downstream QA performance.
- vs A-Mem / Ret-LLM: Graph-based or key-value memory architectures restructure relations but do not fix the upstream factual loss in unaligned summarization; MemSuit demonstrates that plain-text modular entries, when utility-grounded, achieve superior evidence preservation.
Rating¶
- Novelty: โญโญโญโญโญ Elegant adaptation of privileged information self-distillation to conversational memory summarization.
- Experimental Thoroughness: โญโญโญโญโญ Thorough evaluation spanning four query categories, multiple backbones, ablations, retrieval diagnostics, and OOD benchmarks.
- Writing Quality: โญโญโญโญโญ Rigorous formalization, compelling analytical narratives, and highly informative figures.
- Value: โญโญโญโญโญ Provides an impactful design paradigm and practical toolkit for long-term interactive LLM agents.