Skip to content

MemDefrag: Latent Memory Defragmentation for Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ryehr/MemDefrag
Area: LLM Agent
Keywords: Latent Memory, Memory Defragmentation, Attention Density, LLM Agents, Training-Free Evolution

TL;DR

Addressing the long-standing challenges of accumulated positional encoding misalignment and the lack of retrieval tracing in long-term latent memory for LLM agents, this paper discovers an intrinsic tracing signal in middle Transformer layers and introduces MemDefragโ€”a training-free framework that performs inference-time defragmentation (density-based reordering and Top-\(K\) filtering) coupled with informativeness-guided proportional forgetting, boosting 50-step knowledge retention on NaturalQA from 17.4% to 43.0%.

Background & Motivation

As large language model agents advance toward long-horizon task execution and continuous environmental interaction, enabling models to continuously absorb fresh knowledge while faithfully retaining previously acquired facts remains a core challenge. Prior memory paradigms largely fall into three categories: token-level memory is modular and interpretable but strictly bottlenecked by the context window and highly vulnerable to the lost-in-the-middle phenomenon; parametric memory stores knowledge in base model weights or auxiliary adapters without incurring inference latency, yet entails prohibitive continual training costs and suffers from catastrophic forgetting; latent memory represents past interactions as persistent pools of per-layer hidden states, circumventing repetitive context re-encoding and offering higher information density than discrete tokens, thereby providing a compelling foundation for continually updatable agents.

However, existing long-term latent memory systems (such as MemoryLLM and M+) experience drastic performance degradation under repeated memory updates, driven by two fundamental failure modes. First, positional encoding misalignment severely distorts sequence modeling: when new knowledge fragments are consecutively appended into the latent memory pool, the concatenated hidden states are assigned positional encodings that diverge from their original prefilling coordinates, causing attention calculations during inference to warp and triggering a performance collapse once memory length exceeds roughly 2,000 tokens. Second, there is a total absence of a tracing mechanism: current methods treat all accumulated latent states as a flat, undifferentiated prefix, lacking any principled way to filter out semantic noise or single out the specific memory fragment relevant to the current query.

To understand this degradation, the authors probed the layer-wise attention density that stored memory fragments receive with respect to incoming queries. Diagnostic investigations revealed a remarkably robust intrinsic phenomenon: within middle Transformer layers (e.g., layers 13โ€“20 in Llama-3.1-8B-Instruct), query attention density concentrated on the target fragment exhibits a near-perfect monotonic correlation (Pearson and Spearman coefficients exceeding 0.95) with downstream QA accuracy, consistently pinpointing the target fragment regardless of where it was placed in the concatenated sequence. Core idea: exploit intrinsic attention density within middle Transformer layers as a training-free tracing signal to perform inference-time memory defragmentation (reordering fragments to place relevant ones closest to the prompt and filtering out irrelevant ones), complemented by an informativeness-guided proportional forgetting mechanism when memory capacity is exceeded.

Method

Overall Architecture

MemDefrag is a completely training-free, model-agnostic lifecycle management framework for long-term latent memory. It cleanly decouples continuous memory evolution from query-time inference: during memory formation, incoming knowledge fragments are prefilled through the frozen model to yield per-layer hidden states and appended to the persistent memory pool; when total memory exceeds the capacity limit \(N_{\max}\), an informativeness-based proportional forgetting mechanism prunes redundant hidden states coherently across all layers; during inference, given a prompt \(p\), MemDefrag probes attention density at a designated tracer layer \(l^*\), sorts fragments by relevance to move target memories directly adjacent to the prompt (capitalizing on causal recency bias), and filters the Top-\(K\) fragments to construct a clean, defragmented temporary prefix for generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query Prompt p & Accumulated Latent Memory ฮธn"] --> B["Tracer Layer Attention Density Scoring<br/>Compute per-fragment attention density at layer l*"]
    B --> C["Inference-Time Memory Defragmentation<br/>Reorder layers by density ascending and filter Top-K"]
    C --> D["Defragmented Prefix Inference<br/>Place relevant fragments near prompt for causal generation"]
    E["New Knowledge Fragment xn & Capacity Check"] --> F["Informativeness-Guided Proportional Forgetting<br/>Allocate quotas and prune states by self-information"]
    F --> G["Updated Latent Memory Pool ฮธn"]

Key Designs

1. Tracer Layer Attention Density Scoring: Exposing intrinsic relevance signals without tuning

To overcome the inability of flat latent prefixes to differentiate target knowledge from irrelevant background noise, the paper formally defines fragment-level attention density. In a forward pass where the input sequence at layer \(l\) is formed as \(H_l = [\text{bos}_l; m_{1}^l; \dots; m_{n}^l; H_l^p] \in \mathbb{R}^{S \times d}\), multi-head attention is computed under a causal mask, yielding head-averaged attention matrix \(\bar{A}^l \in \mathbb{R}^{|p| \times S}\). Extracting the final row \(\bar{a}_{lt} = \bar{A}^l_{|p|, t}\) (representing attention emitted by the last prompt token), the attention density of stored fragment \(x_i\) occupying retained position set \(P_i\) is computed as: $\(\rho_i^l = \frac{1}{L_i} \sum_{t \in P_i} \bar{a}_{lt}, \quad i = 1, \dots, n\)$ Probing demonstrates that attention density is not merely an indicator of accuracy degradation (correlating with QA performance above 0.95 across layers 13โ€“20), but acts as an exceptionally reliable tracing signal: at layer 13 of Llama-3.1-8B-Instruct, the average rank of the target fragment across all positions is 1.66, yielding a Top-1 retrieval accuracy of 85.6% and Top-3 accuracy of 95.7%. This intrinsic capability stems from intermediate representations capturing rich semantic features optimal for relevance extraction.

2. Inference-Time Memory Defragmentation: Dual action of reordering and Top-K filtering

To mitigate accumulated positional distortion and semantic interference during inference, MemDefrag dynamically executes \(f_{\text{defrag}}(\theta_n, p)\). Using the attention density vector from tracer layer \(l^*\), it computes an ascending permutation \(\pi = \text{argsort}(\rho_1^{l^*}, \dots, \rho_n^{l^*})\) and synchronizes this ordering across all \(L\) layers: $\(\tilde{\theta}_n^l = [m_{\pi(1)}^l; m_{\pi(2)}^l; \dots; m_{\pi(n)}^l], \quad l = 1, \dots, L\)$ Reordering inherently places the most relevant fragments at the end of the memory prefix, immediately preceding prompt tokens, thereby directly exploiting the natural recency bias of causal attention. Next, MemDefrag discards irrelevant memory chunks by retaining only the Top-\(K\) fragments: \((\theta_n^p)^l = [m_{\pi(n-K+1)}^l; \dots; m_{\pi(n)}^l]\), shrinking prefix length to \(N_K = \sum_{k=n-K+1}^n L_{\pi(k)}\). Temporarily removing non-target fragments eliminates accumulated positional distortion for the prompt, significantly boosting attention density concentrated on the target fragment.

3. Informativeness-Guided Proportional Forgetting: Coherent cross-layer state pruning

To sustain continuous memory evolution under bounded storage and compute budgets, MemDefrag replaces the layer-independent random state dropping used in MemoryLLM and M+ (which tears semantic coherence across depth) with a cross-layer coherent eviction strategy. When accumulated length exceeds maximum capacity \(N_{\max}\) by \(N_{\text{forget}}\) tokens, the pruning quota is distributed proportionally across all fragments based on their current retained lengths: \(N_{\text{forget}, i} = \lfloor N_{\text{forget}} \cdot \frac{L_i}{N_{n-1}} \rfloor\). Within each fragment \(x_i\), token informativeness is estimated via self-information (negative log-likelihood): $\(I(x_i^{(t)}) = -\log p_\phi(x_i^{(t)} \mid x_i^{(<t)})\)$ The system identifies the index set \(D_i\) containing the \(N_{\text{forget}, i}\) positions with the lowest self-information and removes the corresponding hidden states synchronously across all \(L\) layers. This guarantees that preserved memory states maintain complete cross-layer semantic integrity, yielding vastly superior long-term retention.

Loss & Training

MemDefrag requires zero training, fine-tuning, or gradient optimization. Model parameters \(\phi\) remain strictly frozen, and no auxiliary modules or co-trained retrievers are attached. The system only requires selecting a single tracer layer index \(l^*\); the paper demonstrates that across different model families and parameter scales, the optimal tracer layer universally falls within the narrow band \([L/3, L/2]\) (e.g., layer 13 for Llama-3.1-8B, layer 14 for Qwen2.5-7B, layer 15 for Mistral-7B and Gemma-2-9b). The maximum memory capacity is set to \(N_{\max} = 12,800\) tokens to ensure fair comparison with MemoryLLM and M+, and Top-\(K\) filtering defaults to \(K \in \{1, 2, 3\}\).

Key Experimental Results

Main Results

In the 50-step knowledge retention benchmark (where a new knowledge passage is injected at each step and the model is continuously queried on the first passage \(x_1\), triggering forgetting once capacity exceeds 12,800 tokens), MemDefrag built on Llama-3.1-8B-Instruct substantially outperforms MemoryLLM-8B and M+-8B across NaturalQA and SQuAD:

Method NaturalQA (n=10) NaturalQA (n=20) NaturalQA (n=30) NaturalQA (n=40) NaturalQA (n=50) SQuAD (n=10) SQuAD (n=20) SQuAD (n=30) SQuAD (n=40) SQuAD (n=50)
MemoryLLM-8B 23.4 18.2 18.4 18.0 17.4 29.0 24.2 20.2 19.0 15.4
M+-8B 26.2 19.6 21.0 17.2 17.6 44.0 36.4 33.2 29.6 20.2
MemDefrag (Top-1) 66.0 67.2 63.2 53.2 43.0 76.6 67.6 54.2 35.0 21.6
MemDefrag (Top-2) 66.8 67.4 62.2 50.0 37.8 78.2 74.0 60.8 40.0 27.2
MemDefrag (Top-3) 62.8 62.0 59.2 44.6 36.2 76.8 73.8 60.2 41.6 25.0
MemDefrag (Top-4) 59.0 59.2 52.6 41.6 33.6 72.0 73.0 59.2 37.2 23.2

On LongBench evaluating long-context multi-document and narrative QA (maximum sequence length 16,384 tokens), MemDefrag (Top-2) achieves the best performance across 20 out of 24 dataset-metric combinations:

Dataset Metric MemoryLLM-8B M+-8B MemDefrag (Top-2) Note / Analysis
2WikiMultihopQA String Match / BERTScore 35.00 / 82.32 35.60 / 81.81 36.00 / 83.82 Accurate multi-hop reasoning with higher semantic fidelity
HotpotQA String Match / F1 37.50 / 8.10 38.00 / 8.56 39.00 / 8.88 +1.00% absolute string match gain over M+
MuSiQue String Match / F1 16.00 / 5.08 19.50 / 6.03 20.00 / 6.16 Outperforms co-trained retriever in complex multi-hop QA
MultiFieldQA-en String Match / F1 3.33 / 14.76 6.00 / 14.11 6.00 / 15.74 +1.63% higher F1 on single-document long context
NarrativeQA String Match / F1 9.84 / 3.37 4.92 / 3.07 11.48 / 7.81 More than doubles the F1 score of M+ (7.81 vs. 3.07)
Qasper String Match / F1 17.77 / 8.58 17.26 / 8.34 18.27 / 9.29 Consistent lead on academic research paper comprehension

Ablation Study

Ablation analysis on defragmentation reordering, Top-\(K\) filtering, and forgetting strategies on NaturalQA:

Configuration / Variant NaturalQA (n=20) NaturalQA (n=50) Peak GPU Mem (GB) Runtime / Group (s) Description
Vanilla Latent Memory 9.4 Collapsed (<4.4) 23.03 59.30 Unchecked positional distortion destroys retention
Top-all (Reorder Only) 22.6 18.2 23.03 103.67 Demonstrates benefits of recency bias without truncation
MemDefrag (Top-2) + Random Forgetting 67.4 20.8 19.73 76.49 Random dropping breaks cross-layer representations
MemDefrag (Top-2) + Info-Based Forgetting 67.4 37.8 19.73 76.49 +81.73% relative accuracy gain over random eviction
MemDefrag (Top-1) + Info-Based Forgetting 67.2 43.0 18.92 75.07 Maximal attention density focus at step 50

Key Findings

  • Near-perfect monotonic correlation between attention density and QA accuracy: In intermediate layers (13โ€“20), correlation coefficients exceed 0.95. Temporarily removing non-target fragments directly boosts target attention density and restores generation quality.
  • Small \(K\) is universally superior: Top-1 and Top-2 consistently achieve the highest accuracies. Expanding \(K\) (e.g., Top-6 or Top-all) increases the likelihood of capturing the target fragment but dilutes its allocated attention density, degrading output accuracy.
  • Tracer layers are universally located in \([L/3, L/2]\): Probing Llama-3.1-8B, Qwen2.5-7B, Mistral-7B, and Gemma-2-9b reveals that the optimal tracer layer always resides in the middle third of the network, achieving Top-2 tracing accuracies above 90% (up to 99.3% on Gemma-2-9b).
  • Reduced memory footprint and superior efficiency: Because generation operates solely over Top-\(K\) states rather than the full 12,800 states, MemDefrag achieves a peak memory of 19.73 GBโ€”saving 3.30 GB (14%) compared to vanilla latent memoryโ€”while running faster than M+ (76.49s vs. 87.24s).

Highlights & Insights

  • Representation engineering enables zero-cost tracing: Instead of training complex external dense retrievers, the authors discovered that Transformer middle layers naturally function as high-precision semantic probes, turning fragment retrieval into a lightweight single-pass operation.
  • Operating-system defragmentation applied to latent space: Analogous to disk defragmentation, reordering and filtering restores spatial and temporal locality for latent memory vectors, neutralizing accumulated positional misalignment via causal recency bias.
  • Cross-layer coherent token pruning: By tying eviction decisions to token self-information across all layers simultaneously, the forgetting mechanism avoids breaking vertical semantic representations throughout the Transformer stack.

Limitations & Future Work

  • Static \(K\) selection across update horizons: Top-\(K\) is fixed empirically throughout evaluation. While \(K=1\) or \(K=2\) is optimal for most models, on architectures like Gemma-2-9b Top-1 accuracy drops past step 20 while Top-2 remains stable, indicating that dynamic entropy-based \(K\) selection could further improve robustness.
  • Single/dual target fragment assumption: Current defragmentation assumes the answer can be derived from 1โ€“2 dominant memory fragments. In complex multi-hop synthesis requiring aggregation across dozens of dispersed snippets, small \(K\) truncation could inadvertently prune secondary evidence.
  • vs MemoryLLM (Wang et al., 2024): MemoryLLM relies on fixed 2:1 chunk compression and independent layer-wise random state dropping, leading to severe information loss and positional misalignment; MemDefrag operates training-free, utilizing tracer-guided defragmentation to maintain 43.0% accuracy at step 50 vs. MemoryLLM's 17.4%.
  • vs M+ (Wang et al., 2025): M+ requires a co-trained external retriever to salvage evicted states, adding training complexity and inference latency; MemDefrag relies entirely on intrinsic self-attention mechanisms, consuming 1.44 GB less peak memory without any external modules.
  • vs SnapKV (Li et al., 2024) / QUEST (Tang et al., 2024): SnapKV and QUEST compress intra-sequence KV caches during long-context single-pass generation; MemDefrag introduces attention density to manage multi-round, persistent, inter-update latent memory pools.

Rating

  • Novelty: โญโญโญโญโญ Elegant discovery of intrinsic middle-layer tracing signals in latent memory without extra parameters.
  • Experimental Thoroughness: โญโญโญโญโญ Exhaustive 50-step knowledge retention tests, LongBench QA, efficiency/memory profiling, and scaling to 24Bโ€“34B models.
  • Writing Quality: โญโญโญโญโญ Rigorous motivation, transparent mathematical derivations, and crystal-clear empirical exposition.
  • Value: โญโญโญโญโญ Provides a foundational, plug-and-play paradigm for building persistent, continually updatable LLM agents.