Skip to content

HaluMem: Evaluating Hallucinations in Memory Systems of Agents

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Hallucination Detection
Keywords: Agent Memory, Memory Hallucination, Operation-Level Benchmark, Long Context, Dynamic Updating

TL;DR

HaluMem establishes the first operation-level hallucination evaluation benchmark for LLM agent memory systems, decoupling traditional black-box QA into memory extraction, updating, and question answering stages across dialogues reaching one million tokens, revealing how upstream extraction omissions cascade into downstream updating and QA failures.

Background & Motivation

Large language models and autonomous agents increasingly serve as personalized companions and persistent task assistants across long-term, multi-turn interactions. However, the stateless nature of standard context windows impedes their ability to preserve user profiles, personal histories, and evolving preferences over time. To endow agents with lifelong learning and continuous adaptation, modern architectures incorporate external memory systems such as MemOS, Mem0, Zep, Supermemory, and Memobase. These frameworks extract, revise, and retrieve persistent memory items throughout user interactions to sustain long-term behavioral consistency. Nonetheless, external memory storage and lifecycle maintenance remain highly vulnerable to hallucinations—manifesting as fabrication, factual errors, outdated version conflicts, and critical omissions that degrade downstream interactions.

Existing benchmarks for agent memory systems—including LoCoMo, LongMemEval, PrefEval, and PersonaMem—predominantly rely on end-to-end question answering paradigms. These protocols feed full dialogue sessions into the memory system as a black box and evaluate only the final generated answers using accuracy or F1 scores. The fundamental limitation of this black-box approach is its complete lack of operational traceability: when an agent generates an inaccurate response, it is impossible to determine whether the error originates from flawed extraction (storing distorted or fabricated facts), failed updating (failing to overwrite obsolete information or introducing version conflicts), or downstream generative hallucination during retrieval and question answering. This diagnostic opacity prevents researchers from identifying specific vulnerabilities in memory architectures.

To overcome the blind spots of black-box evaluations, the evaluation paradigm must penetrate the internal mechanics of memory lifecycles, establishing verifiable gold standards for each atomic operation. Core idea: Decouple agent memory evaluation into three independent operational stages—memory extraction, memory updating, and memory question answering—supported by a six-stage life-trajectory synthesis pipeline that traces the generation, accumulation, and cascading amplification of hallucinations across million-token conversational contexts.

Method

Overall Architecture

The HaluMem evaluation framework operates via operational decoupling and progressive long-term interaction synthesis. Virtual user conversations spanning multi-year timelines are streamed sequentially into the evaluated memory system via three standardized interfaces: Add Dialogue API (ingestion and extraction), Get Dialogue Memory API (inspecting memories stored per session), and Retrieve Memory API (query-driven memory recall). Following each dialogue session containing annotated memory points, HaluMem triggers stage-specific evaluation routines comparing system actions against fine-grained ground truths, culminating in aggregated multi-stage performance diagnostics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Turn Dialogue Input<br/>Session Stream D_1, ..., D_S"] --> B["Six-Stage Progressive Data Construction<br/>Persona→Life Skeleton→Event Flow→Sessions & Memory Points"]
    B --> C["Operation-Level Gold Standards<br/>Target Set G_ext + Update Pairs G_upd + Diagnostic QA Set"]
    C --> D["Three-Stage Decoupled Evaluation<br/>Memory Extraction E → Memory Updating U → Memory QA Q"]
    D --> E["Diagnostic Metric Architecture<br/>FMR Resistance + Extraction F1 + Update Omission/Error + QA Hallucination"]
    E --> F["Hallucination Traceability & Cascades<br/>Quantifying Error Propagation from Upstream Extraction to Downstream QA"]

Key Designs

1. Six-Stage Progressive Data Construction: From persona seeds to million-token adversarial dialogues

Operational evaluation demands conversational datasets that combine realistic multi-year state evolutions with deterministic, conflict-free factual ground truths. HaluMem implements a disciplined top-down generation procedure. Stage 1 initializes virtual user personas from Persona Hub seeds, combining static core profiles (demographics, MBTI, lifelong aspirations), evolving dynamic states (employment, health status, interpersonal ties), and 6–8 randomized preferences. Stage 2 formulates a life skeleton detailing major career milestones and probabilistic preference evolutions. Stage 3 eventifies this skeleton into structured event flows categorized into initial events (baseline profile introduction), career events (narrative anchors driving socioeconomic changes), and daily events (lifestyle and preference revisions).

Stage 4 synthesizes conversational scenarios with atomic memory points, specifying memory types (persona, event, relationship), importance weights (\(w_i\)), and historical update links (\(m_{old} \rightarrow m_{new}\)). Stage 5 converts event scenarios into context-dense multi-turn interactions. To rigorously simulate conversational noise and contamination, this stage injects adversarial "distractor memories"—unconfirmed false premises casually suggested by the assistant to which the user remains passive—alongside long-range coreference and temporal contradictions. Stage 6 synthesizes diagnostic QA pairs spanning six reasoning categories mapped directly to underlying memory points. This process produces HaluMem-Medium (20 users, 30,073 turns, ~160k tokens/user, 14,948 memory points, 3,467 QA pairs) and HaluMem-Long, which expands contexts to 1M tokens (53,516 turns) by interleaving diverse domain-general dialogues (ELI5, math reasoning, historical and scientific QA).

2. Operation-Level Decoupled Evaluation: Pinpointing stage-specific failure points

Rather than evaluating the system purely on final outputs, HaluMem establishes distinct assessment protocols for the three core operations: memory extraction (\(E\)), memory updating (\(U\)), and memory question answering (\(Q\)). In extraction, the benchmark examines the set of new memory points \(\hat{M}_{ext}^s\) generated by the system for session \(D_s\) against the gold target \(G_{ext}^s\), verifying whether critical information was captured while filtering noise. In updating, sessions involving fact revisions evaluate whether the system correctly replaces or modifies existing records \((m_{old} \rightarrow m_{new})\) rather than creating conflicting duplicate memories. In question answering, top-retrieved memories serve as context for standardized LLM generators (GPT-4o) answering probes. By isolating these stages, HaluMem disentangles whether a downstream failure stems from incomplete extraction, failed overwriting, or generative distortion.

3. Anti-Hallucination and Dynamic Metric Architecture: Measuring integrity versus false memory resistance

HaluMem establishes a comprehensive metric suite to evaluate both completeness and fidelity. In memory extraction, beyond standard recall, it defines Weighted Memory Recall based on importance scores, and introduces False Memory Resistance (FMR) to quantify a system's resilience against unconfirmed conversational distractors:

\[\text{FMR} = \frac{N_{miss}}{N_D}\]

where \(N_D\) represents total injected distractor points and \(N_{miss}\) represents distractors successfully ignored. Extraction F1 is computed as the harmonic mean between Memory Recall (\(R_{mem}\)) and Target Memory Precision (\(P_{tgt}\)). In memory updating, HaluMem reports Update Accuracy (Correct Rate, C), Update Hallucination Rate (H), and Update Omission Rate (O):

\[\text{Omission Rate} = \frac{N_{missed-upd}}{N_{target-upd}}, \quad \text{Hallucination Rate} = \frac{N_{wrong-upd}}{N_{target-upd}}\]

The QA stage similarly records accuracy, hallucination, and omission rates, establishing a unified lens to assess the fundamental trade-off between amnesia and fabrication across all operational phases.

Key Experimental Results

Main Results

The benchmark evaluates six prominent agent memory systems: standard Mem0, graph-augmented Mem0-Graph, lightweight profile-oriented Memobase, operating-system-level MemOS, contextual-graph-based Supermemory, and Zep (whose proprietary interface does not expose a per-session dialogue memory extraction API, omitting extraction metrics). Testing spans HaluMem-Medium and HaluMem-Long using GPT-4o as the primary evaluator (with findings fully verified under GPT-5.4-mini).

Dataset System Extraction Recall R↑ Weighted Recall W-R↑ Target Precision P↑ Extraction Acc↑ Distractor Resistance FMR↑ Extraction F1↑ Update Correct C↑ Update Halu H↓ Update Omission O↓ QA Correct C↑ QA Halu H↓ QA Omission O↓
Medium Mem0 42.91% 65.03% 86.26% 60.86% 56.80% 57.31% 25.50% 0.45% 74.02% 53.02% 19.17% 27.81%
Medium Mem0-Graph 43.28% 65.52% 87.20% 61.86% 55.70% 57.85% 24.50% 0.26% 75.24% 54.66% 19.28% 26.06%
Medium Memobase 14.55% 25.88% 92.24% 32.29% 80.78% 25.13% 5.20% 0.55% 94.25% 35.33% 29.97% 34.71%
Medium MemOS 74.07% 84.81% 86.25% 59.55% 44.94% 79.70% 62.11% 0.42% 37.48% 67.23% 15.17% 17.59%
Medium Supermemory 41.53% 64.76% 90.32% 60.83% 51.77% 56.90% 16.37% 1.15% 82.47% 54.07% 22.24% 23.69%
Medium Zep - - - - - - 47.28% 0.42% 52.31% 55.47% 21.92% 22.62%
Long Mem0 3.23% 11.89% 88.01% 46.01% 87.65% 6.22% 1.45% 0.03% 98.51% 28.11% 17.29% 54.60%
Long Mem0-Graph 2.24% 10.76% 87.32% 41.26% 88.36% 4.36% 1.47% 0.04% 98.40% 32.44% 21.82% 45.74%
Long Memobase 6.18% 14.68% 88.56% 25.61% 85.39% 11.55% 4.10% 0.36% 95.38% 33.60% 29.46% 36.96%
Long MemOS 81.90% 89.56% 82.32% 43.77% 28.85% 82.11% 65.25% 0.29% 34.47% 64.44% 16.61% 18.95%
Long Supermemory 53.02% 70.73% 85.82% 29.71% 36.86% 65.54% 17.01% 0.58% 82.42% 53.77% 22.21% 24.02%
Long Zep - - - - - - 37.35% 0.48% 62.14% 50.19% 22.51% 27.30%

Ablation & Fine-Grained Analysis: Memory Type Accuracy

The authors dissected extraction and update accuracy across three distinct semantic memory categories: Event, Persona, and Relationship memories (excluding distractor noise).

Dataset System Event Memory Persona Memory Relationship Memory
Medium Mem0 29.69% 33.74% 27.77%
Medium Mem0-Graph 30.02% 33.71% 26.60%
Medium Memobase 5.12% 13.38% 6.79%
Medium MemOS 63.41% 59.77% 62.40%
Medium Supermemory 28.66% 32.11% 20.67%
Medium Zep* 44.83% 49.75% 38.81%
Long Mem0 0.92% 3.01% 2.18%
Long Mem0-Graph 1.10% 2.00% 1.59%
Long Memobase 4.09% 5.32% 4.21%
Long MemOS 70.92% 68.35% 71.68%
Long Supermemory 38.48% 40.85% 32.61%
Long Zep* 35.76% 39.07% 31.16%

*Note: Zep statistics reflect only the memory updating task entries.

Key Findings

  • Upstream Extraction Collapse in Million-Token Contexts: Moving from medium to million-token contexts (HaluMem-Long) causes catastrophic extraction failure in widely adopted frameworks. Mem0's recall plummets from 42.91% to 3.23%, with F1 dropping from 57.31% to 6.22%. Current memory architectures lack robust noise filtering and value discrimination over long conversational histories.
  • Upstream Omission Cascading into Downstream Updating: Memory updating exhibits severe omission across almost all systems (exceeding 74% in Medium and 98% for Mem0 in Long). Non-extracted facts cannot be updated; upstream amnesia artificially depresses the measured update hallucination rate to under 2% simply because updates never trigger.
  • The Recall vs. Distractor Resistance Trade-off: Systems emphasizing aggressive extraction (e.g., MemOS achieving >74% recall) exhibit significantly lower False Memory Resistance (44.94% in Medium, dropping to 28.85% in Long), frequently absorbing unverified assistant statements. Conversely, conservative filters (Memobase FMR >80%) miss most critical facts, paralyzing subsequent reasoning.
  • Vulnerability in Relational and Dynamic Tracking: Systems consistently extract static persona traits more accurately than transient events or evolving interpersonal relationships. In QA probes, models handle boundary and conflict detection relatively well but falter on multi-hop temporal inference and dynamic preference tracking.

Highlights & Insights

  • Decoupled Operation-Level Diagnosis: Shifting evaluation from black-box response scoring to atomic extraction, updating, and QA operations provides actionable attribution for memory failures.
  • Million-Token Consistent Trajectory Synthesis: The six-stage progressive construction framework successfully combines multi-year causal coherence with adversarial distractor injections, setting a new standard for long-context agent benchmarking.
  • Unmasking Omission-Masked Hallucinations: Demonstrating that artificially low updating hallucination rates stem from massive upstream omission alerts the community against evaluating memory updates in isolation from extraction completeness.

Limitations & Future Work

  • Heterogeneous Interface Bottlenecks: Commercial systems often lack standardized APIs for session-level memory extraction (exemplified by Zep), forcing fallback database inspections that hinder seamless comparative auditing.
  • Scope Limited to Textual User Memories: The benchmark currently focuses on persona, event, and relational memories, leaving tool-use execution trajectories and executable skill memories (Skill Memory) for future expansions.
  • Future Directions: Developing constrained memory mutation architectures with pre-write semantic conflict arbitration to reconcile high extraction recall with robust distractor resistance.
  • vs LoCoMo / LongMemEval: Early long-context benchmarks relied on holistic end-to-end question answering, obscuring whether errors arose during extraction, overwriting, or retrieval; HaluMem provides the first operation-level stage-by-stage gold standard.
  • vs PersonaMem / PrefEval: Prior user-centric profiling benchmarks evaluated static persona attributes or short-term preferences; HaluMem integrates multi-year life skeletons, career evolutions, and adversarial conversational distractors.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first operation-level hallucination diagnostic benchmark for agent memory systems.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 6 leading memory systems, dual context scales, and multiple judge LLMs.
  • Writing Quality: ⭐⭐⭐⭐⭐ Precise mathematical formulations, thorough empirical analysis, and transparent error taxonomies.
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable diagnostic framework for building trustworthy, long-term autonomous agents.