UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/peijunallin/UtilMem
Area: LLM Reasoning
Keywords: Long-Term Conversational Memory, Evidence Utilization, Distractor Filtering, Dense Multi-Session Reasoning, Memory Benchmark
TL;DR¶
This work introduces UTILMEM, a 1,717-instance diagnostic benchmark across five real-world domains that shifts the evaluation of long-term conversational memory from pointwise factual recall to comprehensive evidence utilization, revealing severe degradation in state-of-the-art memory systems when synthesizing multi-session evidence under adversarial retrieval interference.
Background & Motivation¶
Large language model assistants are increasingly deployed across long-horizon, multi-session interactive scenarios such as personal tutors, financial advisors, mental health companions, and fitness coaches. In these long-term engagements, users expect agents to seamlessly retain personal contexts, recall past discussions across weeks or months, and actively synthesize this accumulated history to formulate coherent, personalized, and actionable downstream responses. In response to this demand, the community has seen rapid emergence of memory-augmented architectures ranging from vector-based retrieval pipelines to graph-structured indices and write-time memory compressors, accompanied by benchmark suites like LoCoMo, LongMemEval, and MemoryBank designed to evaluate long-term conversational interaction.
However, existing long-term memory benchmarks remain heavily dominated by a narrow evaluation paradigm: pointwise factual recall against atomic gold answers. In these evaluations, supporting evidence is almost universally localized within one or two isolated sessions, distractors are either absent or topically trivial, and expected outputs are short-form factual extractions. Such an evaluation framework inadvertently rewards shortcut-friendly strategies—such as aggressive write-time slot extraction, superficial entity lookups, or simple top-\(k\) recall expansion—while completely obscuring critical failure modes encountered in real deployments. In practice, genuine long-term memory tasks require integrating dense, temporally dispersed evidence across disparate sessions, extracting latent user preferences and evolving physiological or financial states, producing structured long-form analyses or plans, and robustly filtering out adversarial distractors that share prominent entities and keywords with genuine memories.
Consequently, there is an urgent need to decouple memory access from memory utilization and systematically evaluate whether existing systems can genuinely leverage retrieved long-term information under realistic contextual interference. Core idea: construct UTILMEM, a 1,717-instance diagnostic benchmark spanning five real-world domains with 80k–220k token contexts, featuring counterfactual evidence-sensitivity degradation filtering to enforce strict multi-session dependence and adversarial semantic distractor injection to evaluate dense multi-session reasoning, implicit retrieval, long-form composition, and distractor filtering.
Method¶
Overall Architecture¶
A benchmark instance in UTILMEM is formalized as a tuple \((H, q, E)\), where the user's complete interaction history \(H\) is partitioned into two disjoint subsets: the relevant evidence sessions \(E = \{e_1, \dots, e_K\}\) and the adversarial haystack sessions \(D = \{d_1, \dots, d_L\}\), such that \(H = E \cup D\) and \(E \cap D = \emptyset\). Given the full noisy history \(H\) and an open-ended multi-session query \(q\), the evaluated memory system retrieves context to generate response \(\hat{a}\). The evaluation strictly isolates context quality from generator capability: an oracle reference answer \(a_{oracle}\) is generated by conditioning a frozen deterministic generator solely on the clean evidence set \(E\) and query \(q\), anchored at a score of 10. The noisy output \(\hat{a}\) is scored against \(a_{oracle}\) by two independent frontier judge models using a five-dimensional rubric that enforces anchor-based capping.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-Source Real Interaction Corpora<br/>Tutoring/Finance/Mental/Fitness/10-K"] --> B["Multi-Session Evidence Synthesis & Persona Consistency<br/>4-step pipeline·Timeline anchoring·Contradiction rejection"]
B --> C["Cross-Session Dependent & Deterministic Question Generation<br/>10 candidate Qs/bundle·Objective synthesis instructions"]
C --> D["Counterfactual Sequential Degradation Filtering<br/>E1→E2→E3 monotonic decay·Concordant 4-verdict dual judges"]
D --> E["Two-Stage Adversarial Distractor Synthesis & Haystack Assembly<br/>Shared keywords + Intent shift·80k-220k token multi-domain haystacks"]
E --> F["Five-Dimensional Anchor-Based Judge Protocol<br/>Factual fidelity·Completeness·No hallucination·Equivalence·Groundedness"]
F --> G["UTILMEM Benchmark Output<br/>Normalized Robustness NR · Degradation Rate DR"]
Key Designs¶
1. Multi-Session Evidence Synthesis and Persona Consistency: Harmonizing heterogeneous seeds into coherent longitudinal dialogues
To ensure high ecological validity and diversity, UTILMEM seeds its evidence histories from five public interaction datasets: StudyChat (undergraduate AI tutoring), Personal-Finance-Queries (Reddit budgeting and investing), Reddit mental health corpora (depression, ADHD, OCD, PTSD), Fitness Q&A (longitudinal athletic coaching), and S&P 500 10-K filings. Direct concatenation of disjoint sources risks introducing glaring contradictions in basic persona attributes (such as mutually exclusive ages, conflicting life events, or inconsistent medical histories). The pipeline groups sampled topics into user bundles \(B = \{s_1, \dots, s_K\}\) and subjects them to an LLM consistency filter that rejects any contradictory configurations. Accepted items are rewritten into natural conversational exchanges through alternating multi-turn agent-user rollouts, calibrated with realistic chronological timestamps anchored on longitudinal fitness logs, producing realistic multi-session user histories.
2. Cross-Session Dependent and Deterministic Question Generation: Forcing information aggregation while preventing subjective evaluation drift
Evaluation validity in open-ended generation depends critically on query design. For each multi-session bundle, an LLM generator emits candidate questions that strictly mandate the aggregation of information distributed across all sessions in \(B\) (such as synthesizing code debugging notes across multiple homework assignments into a unified study guide, or coordinating debt cleanup, disputed hospital fees, and house savings into a single financial roadmap). Queries are explicitly prompted to employ natural lexical surface terms present in the dialogue history to ensure fair retrieval comparability. Crucially, questions are constrained to be deterministic, non-judgmental synthesis tasks (e.g., structuring, listing, cross-sectional planning, or timeline alignment) rather than open-ended diagnostic opinions (e.g., assessing personal weaknesses or subjective takeaways), guaranteeing highly stable, grounded reference answers across independent generator runs.
3. Counterfactual Sequential Degradation Filtering: Enforcing strict evidence indispensability through monotonic quality decay
Candidate multi-session questions often suffer from hidden single-session shortcuts where an LLM can formulate a high-quality answer using only a fraction of the provided context. To weed out these false multi-session queries, UTILMEM implements a rigorous counterfactual degradation filter. For each bundle containing \(K\) evidence sessions, three context tiers are constructed via progressive random removals: \(E^{(1)} = E\), \(E^{(2)} = E^{(1)} \setminus R_1\), and \(E^{(3)} = E^{(2)} \setminus R_2\), targeting approximately 50% cumulative removal while guaranteeing \(|E^{(3)}| \ge 1\). A frozen target model generates answers across all three tiers: \(\hat{a}^{(1)}, \hat{a}^{(2)}, \hat{a}^{(3)}\). Two independent judge models perform blind pairwise comparisons across adjacent levels. A question is accepted if and only if both judges unanimously verify strict monotonic degradation:
$\(\text{quality}(\hat{a}^{(1)}) > \text{quality}(\hat{a}^{(2)}) > \text{quality}(\hat{a}^{(3)})\)$
Requiring four concordant positive verdicts across two independent judges effectively guarantees that answer quality strictly suffers whenever any sampled evidence session is removed.
4. Two-Stage Adversarial Distractor Synthesis and Haystack Assembly: Constructing high-interference context with shared vocabulary and shifted intent
Weak distractors drawn from random web corpora or distant topics can be trivially sidestepped by expanding retrieval top-\(k\). To rigorously stress-test memory filtering under realistic interference, UTILMEM deploys a two-stage adversarial distractor synthesis pipeline. For each ground-truth evidence session \(e_k \in E\), an LLM first reads \(e_k\) and extracts 5–10 core domain entities and keywords while formulating \(N\) adversarial directions mapped to distinct cross-domain families (for example, transplanting marathon running fatigue terminology like "2.5-hour run", "fatigue", and "electrolyte fueling" into a conversation detailing the battery drain and IMU drift of an autonomous planetary rover; or framing real debt management vocabulary within a Monopoly board game session). Second, each direction is synthesized into a multi-turn conversation where shared keywords appear naturally while the functional intent is fundamentally irrelevant to the query. In-domain adversarial sessions are then aggregated with cross-domain weak distractors into haystacks averaging 60–70 sessions (80k–220k tokens, mean 120k tokens, closely matched with LongMemEval-S).
5. Five-Dimensional Anchor-Based Judge Protocol: Capping scores via a weakest-link penalty scheme
Evaluating multi-page structured syntheses precludes exact string-matching metrics. UTILMEM anchors the clean oracle answer \(a_{oracle}\) at a protocol score of 10 and evaluates the retrieved response \(\hat{a}\) via two frontier models (GPT-5.2 and GPT-5.4). The judge first classifies \(\hat{a}\) along five distinct dimensions into discrete severity tiers (intact, mild, clear, severe): Factual Fidelity (consistency with oracle claims), Completeness Preservation (coverage of question-relevant facts), Hallucination Absence (absence of fabricated or distractor-sourced details), Semantic Equivalence (actionable takeaway alignment), and Ungrounded Inference (absence of ungrounded logical leaps). Crucially, the final integer score \(s \in [1, 10]\) is determined by explicit anchors that cap the score at the worst-performing dimension (e.g., any severe factual contradiction or refusal caused by distractor contamination immediately caps the score at 1–2), preventing superficial fluency or verbose padding from inflating evaluation scores.
Benchmark & Evaluation Protocol¶
System performance is reported via two primary metrics across the \(N=1,717\) benchmark instances: Normalized Robustness (\(NR \in [0, 100]\)), computed as \(NR = \frac{RS - 1}{9} \times 100\) where \(RS = \frac{1}{N} \sum_i s_i\) measures the percentage of clean reference quality retained; and Degradation Rate (\(DR \in [0, 100]\)), representing the percentage of instances scoring \(s_i \le 6\) (the threshold where noticeable, user-relevant quality loss occurs). The benchmark evaluates two broad architectural classes: standard retrieval backbones using NaiveRAG paired with Nomic-embed-text-v1.5, Qwen3-Embedding-4B, and Qwen3-Embedding-8B; and advanced structured / agentic memory systems including A-MEM, Mem0, Mem0+Graph, MemOS, LangMem, and EverMemOS. All evaluations freeze the QA generator as Qwen3-235B-A22B-Instruct-2507 at temperature 0 to ensure strict causal isolation of memory context.
Key Experimental Results¶
Main Results¶
Evaluation across the 1,717 instances reveals an acute performance discrepancy across memory architectures (Table 2): systems relying on aggressive write-time summarization or slot extraction fail precipitously, while raw turn-preserving retrieval systems consistently outpace complex structured memory managers.
| System / Architecture | Learning Support (NR/DR) | Finance Guidance (NR/DR) | Mental Wellness (NR/DR) | Fitness Coaching (NR/DR) | Document Analysis (NR/DR) | Overall Average NR (↑) | Overall Average DR (↓) |
|---|---|---|---|---|---|---|---|
| Retrieval-Only Baselines | |||||||
| NaiveRAG + Nomic-embed-1.5 | 48.0 / 74.4 | 69.7 / 25.4 | 54.9 / 62.7 | 67.0 / 32.6 | 55.6 / 52.9 | 58.9 | 49.6 |
| NaiveRAG + Qwen3-Embed-4B | 49.2 / 73.1 | 68.4 / 26.2 | 55.7 / 61.5 | 66.1 / 33.8 | 56.8 / 51.7 | 59.2 | 49.3 |
| NaiveRAG + Qwen3-Embed-8B | 48.7 / 73.8 | 69.7 / 24.9 | 55.2 / 62.1 | 67.6 / 31.9 | 55.1 / 53.4 | 59.3 | 49.2 |
| Structured & Agentic Systems | |||||||
| A-MEM | 45.6 / 78.2 | 67.4 / 28.6 | 51.8 / 68.9 | 64.2 / 36.8 | 50.9 / 59.4 | 56.0 | 54.4 |
| Mem0 | 7.4 / 99.7 | 22.1 / 99.4 | 30.8 / 93.7 | 24.4 / 98.9 | 1.7 / 100.0 | 17.3 | 98.3 |
| Mem0 + Graph | 8.9 / 99.4 | 24.7 / 98.7 | 32.1 / 92.8 | 26.2 / 98.1 | 2.8 / 99.8 | 18.9 | 97.8 |
| MemOS | 39.8 / 80.7 | 57.2 / 55.4 | 47.9 / 72.6 | 54.1 / 61.8 | 38.5 / 78.9 | 47.5 | 69.9 |
| LangMem | 33.2 / 87.4 | 49.6 / 69.2 | 42.8 / 80.1 | 47.3 / 75.8 | 30.7 / 90.6 | 40.7 | 80.6 |
| EverMemOS | 41.7 / 82.9 | 58.4 / 51.6 | 49.2 / 70.4 | 55.9 / 58.7 | 36.8 / 81.3 | 48.4 | 69.0 |
Ablation Study: Noise Sensitivity at Perfect Recall¶
By fixing unit-level recall at 1.0 (all ground-truth evidence sessions guaranteed in context) and systematically varying the number of injected distractors \(k\), distractor adversariality, and generator model scale (Figure 4), the authors isolate downstream evidence utilization failures from retrieval omissions:
| Generator Model & Noise Type | \(k=0\) (Oracle Anchor) | \(k=10\) (Weak Noise) | \(k=10\) (Strong Noise) | \(k=40\) (Strong Noise) | \(k_{75\%}\) Noise Tolerance | Peak Marginal Damage Interval |
|---|---|---|---|---|---|---|
| Qwen3-30B (Strong Noise) | 100.0 (DR: 0.0) | 66.8 (DR: 38.2) | 48.2 (DR: 61.4) | 47.4 (DR: 73.6) | 1.5 distractors | \(k \in [2, 8]\) |
| Qwen3-235B (Strong Noise) | 100.0 (DR: 0.0) | 69.6 (DR: 32.5) | 59.2 (DR: 47.8) | 56.0 (DR: 57.6) | 1.7 distractors | \(k \in [2, 8]\) |
| Qwen3-Max (Strong Noise) | 100.0 (DR: 0.0) | 71.3 (DR: 29.8) | 63.1 (DR: 41.2) | 57.2 (DR: 53.2) | 2.2 distractors | \(k \in [2, 8]\) |
Furthermore, when isolating instances where retrieval was already highly successful (\(\text{Recall} \ge 0.8\), Figure 3b), the average Robustness Score remains far below the oracle ceiling of 10 (ranging from 6.4 in Mental Wellness to 7.9 in Finance Guidance), with 53% of high-recall instances in Mental Wellness and 29% in Document Analysis still collapsing below the degradation threshold (\(RS \le 6\)). Inter-judge agreement between GPT-5.2 and GPT-5.4 achieves a Pearson correlation of 0.848 and Spearman of 0.843, with 82.94% of scores differing by at most 1 point (Table 4), validating the metric's empirical stability.
Key Findings¶
- Write-time factual compression causes catastrophic and irreversible information loss: Mem0 collapses to 17.3 NR and 98.3 DR across the benchmark, and adding graph relations (Mem0+Graph, 18.9 NR / 97.8 DR) yields negligible relief. Qualitative failure audit confirms non-empty retrieval, proving that stripping contextual prose into discrete atomic facts irrevocably discards the connective reasoning tissue required for multi-session synthesis.
- High recall is necessary but fundamentally insufficient for utilization: Even under perfect recall (\(\text{Recall} = 1.0\)), injecting just 10 adversarial distractors degrades Qwen3-235B's NR from 100 to 59.2 while spiking the failure rate to 47.8%. Downstream LLMs are easily led astray by superficial lexical overlap, generating confident but ungrounded answers.
- Interference damage is heavily front-loaded: Noise tolerance \(k_{75\%}\) (the distractor count required to lose 25% of clean quality) is astonishingly low—only 1.5 to 2.2 distractors across all generators. Marginal degradation peaks within the first \(k \in [2, 8]\) distractors and plateaus thereafter once the context is contaminated.
- Scaling the generator model does not eliminate noise vulnerability: While moving from Qwen3-30B to Qwen3-Max elevates absolute robustness under 40 adversarial distractors from 47.4 to 57.2, the steep degradation trajectory remains identical, proving that parameter scaling alone cannot resolve retrieval interference.
Highlights & Insights¶
- Formulating "Evidence Utilization" as an essential frontier in conversational memory: Demonstrates that passing recall-oriented QA probes does not translate to robust long-horizon intelligence, shifting benchmark focus to multi-session cross-examination and contextual synthesis.
- Rigorous counterfactual degradation filtering protocol: Validating monotonic quality decline across sequentially ablated contexts (\(E^{(1)} \to E^{(2)} \to E^{(3)}\)) with dual-judge consensus establishes a reproducible template for purging spurious multi-hop and multi-session benchmark instances.
- Adversarial semantic distractor generation: Constructing distractors with shared terminology but disjoint conversational intent exposes the profound vulnerability of embedding matching and attention mechanisms to lexical traps.
Limitations & Future Work¶
- Reliance on LLM-driven synthesis and evaluation: Conversational rewriting, distractor generation, reference creation, and judging rely on frontier LLMs (Qwen3 and GPT-5 families), potentially inheriting stylistic artifacts and alignment biases; broader human-expert validation remains necessary.
- Sampled sequential removal vs. exhaustive minimal subset proof: The evidence-sensitivity filter confirms quality degradation along one sampled elimination path rather than exhaustively proving that every single session is strictly minimal and indispensable.
- Future architectural avenues: Emphasizes the need for noise-resilient conversational memory systems capable of dynamic context pruning, hypothesis verification, and query-guided contextual reconstruction to suppress functionally irrelevant history.
Related Work & Insights¶
- vs. LoCoMo / LongMemEval: Prior benchmarks concentrate evidence within 1–2 sessions and emphasize atomic entity extraction or refusal checks under mild noise; UTILMEM evaluates 3–7 evidence sessions per sample under adversarial multi-session interference, requiring long-form synthesis.
- vs. Commercial Memory Frameworks (Mem0, LangMem): Widely used agent memory tools prioritize aggressively compressed profiles and entity-relation triples; UTILMEM demonstrates that such write-time compression severely cripples long-form contextual planning and multi-session reasoning.
- vs. RAG Noise Robustness Research: Extends paragraph-level distractor studies to full multi-turn conversational trajectories with explicit lexical overlap and semantic divergence.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the critical gap between memory retrieval and memory utilization; novel counterfactual degradation and adversarial distractor synthesis pipelines.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ 1,717 instances across 5 real-world domains, rigorous comparisons spanning retrieval backbones and structured memory operating systems, complemented by controlled noise-scaling studies.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical formalization, comprehensive error analyses, and detailed appendix documentation.
- Value: ⭐⭐⭐⭐⭐ Delivers a vital diagnostic benchmark and reality check for the LLM agent and long-term memory research communities.