Skip to content

CoreMem: Riemannian Retrieval and Fisher-Guided Distillation for Long-Term Memory in Dialogue Agents

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/CJQ6661999/CoreMem-Code
Area: LLM Reasoning
Keywords: Long-term memory, Riemannian retrieval, Fisher information distillation, Edge computing, Information geometry

TL;DR

Addressing dual bottlenecks of consumer edge hardware (8 GB VRAM) and linear cloud API costs, CoreMem unifies long-term dialogue memory under information geometry via Woodbury-accelerated Fisher-Rao Riemannian retrieval and syntax-protected Fisher-guided token distillation, achieving decisive accuracy gains in open-domain and temporal reasoning.

Background & Motivation

Building personalized conversational agents that maintain continuous, long-term memory across dozens of interaction sessions is central to transitioning conversational AI from server clusters to consumer-grade edge devices, such as personal laptops equipped with 8–16 GB RAM and edge GPUs with 6–12 GB VRAM. However, the dominant industry practice of naive context stuffing—greedily concatenating historical dialogues into prompts—is economically unsustainable and mathematically flawed: API costs scale linearly with session length, while excessively long contexts exacerbate positional attention bias and induce severe "lost-in-the-middle" hallucinations. In a realistic hybrid setup where an edge user runs a local embedding model on an NVIDIA RTX 4060 Laptop GPU (8 GB VRAM) and offloads generation to cloud APIs, an embedding encoder alone occupies approximately 3 GB VRAM, leaving virtually no headroom for context compressors, vector indices, and system overhead. Existing prominent architectures such as MemGPT, MemoryBank, and Mem0 were predominantly designed for server-scale environments, lacking the algorithmic frugality required for consumer-grade devices.

Current edge-oriented memory and compression workflows suffer from three critical theoretical and practical shortcomings. First, ubiquitous isotropic cosine retrieval is vulnerable to the hubness problem in high-dimensional spaces: contextual embedding spaces naturally collapse into narrow cones, causing a tiny subset of central "hub" vectors to dominate nearest-neighbor lists while submerging geometrically peripheral yet factually crucial tail memories. Second, popular context compression algorithms (e.g., LLMLingua2, SelectiveContext) rely on shallow self-information heuristics or empirical classifiers without formal semantic preservation guarantees, turning low-budget edge deployment into an unstable trial-and-error gamble. Third, retrieval and compression are optimized with disjoint objectives; extracted multi-hop facts frequently have their structural connectors (e.g., causal and temporal conjunctions) discarded during subsequent compression, resulting in catastrophic cascade reasoning failures.

To address these challenges, edge memory management demands an algorithmic framework that is simultaneously VRAM-friendly (fitting within 8 GB), time-friendly (sub-second local execution), and token-friendly (substantially reducing transmitted tokens). Core idea: unify retrieval and compression under the lens of information geometry by employing a locally adaptive Fisher-Rao Riemannian metric that suppresses hubness via low-rank Woodbury-accelerated inverse covariance, complemented by a hierarchical Fisher-guided discrete token distillation algorithm that guarantees an explicit KL-divergence bound regularized by structural syntax priors.

Method

Overall Architecture

CoreMem implements an edge-cloud decoupled architecture: On the edge device, incoming user queries and historical memories are encoded into dense vectors. The Riemannian retriever selects the top-\(k\) relevant memories from the local bank via precomputed Woodbury-accelerated inverse covariance tensors, followed by residual fusion with cosine similarity. Next, a frozen-weight local small language model (SLM) executes Fisher-guided discrete token distillation (FDTD) to compress the retrieved context down to a strict capacity budget \(B\). Finally, only the compact, information-dense payload is transmitted across the network to a commercial cloud LLM for answer generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User query and historical memory bank"] --> B["1. Riemannian Retrieval with Woodbury Acceleration<br/>Fisher-Rao metric·SVD inverse covariance·O(Ndr) complexity"]
    B --> C["2. Residual Metric Fusion<br/>z-score normalization·Local geometry and global semantic balance"]
    C --> D["3. Fisher-Guided Discrete Token Distillation<br/>Diagonal FIM trace sensitivity·Theoretical KL-divergence bound"]
    D --> E["4. Structural Syntax Prior and Hierarchical Distillation<br/>Sentence skeleton protection·Keyword/punctuation boosts·Gap filling"]
    E --> F["Edge-to-cloud transport<br/>High-density compressed context·Reduced API costs"]
    F --> G["Cloud LLM deterministic generation"]

Key Designs

1. Riemannian Retrieval with Woodbury Acceleration: Mitigating high-dimensional hubness and surfacing peripheral tail memories

Isotropic cosine similarity assumes all latent dimensions carry equal geometric variance, causing contextual representations to collapse into a cone where a small set of high-variance hub embeddings absorb nearest-neighbor traffic from irrelevant queries. To resolve this, CoreMem models memory embeddings as the mean vectors of Gaussian distributions on a statistical manifold, where the Fisher-Rao metric naturally reduces to the Mahalanobis distance weighted by the empirical inverse covariance \(\Sigma^{-1}\). To bypass the intractable \(\mathcal{O}(d^3)\) matrix inversion cost on edge hardware, embeddings are first \(L_2\)-normalized, and the coordinate variance \(\sigma^2 = \text{Var}(\{h_i\}_{i=1}^N)\) is smoothed via \(D = \text{diag}(\sigma^2 + \lambda)\) using an adaptive ridge parameter \(\lambda = 10 \cdot \text{mean}(\sigma^2)\). Singular Value Decomposition (SVD) is applied to centered representations to capture cross-dimensional correlations, retaining the minimal rank \(r\) (\(r \le 100\)) explaining \(\tau = 0.95\) of total variance. Applying the Woodbury matrix identity enables precomputing the exact inverse: $$ \Sigma^{-1} = D^{-1} - D^{-1} V_r \left( M_r^{-1} + V_r^\top D^{-1} V_r \right)^{-1} V_r^\top D^{-1} $$ At query time, calculating Riemannian distances requires only \(\mathcal{O}(Ndr)\) floating-point operations. The resulting elongated metric tensor stretches collapsed dimensions and penalizes dense clusters, projecting false hubs outward and pulling previously unreachable tail memories into the retrieval neighborhood.

2. Residual Metric Fusion: Balancing local geometric awareness with global semantic bridge facts

While Riemannian distance excels at uncovering isolated, fine-grained tail facts, standard cosine similarity retains strong utility for identifying broad semantic "bridge facts" essential in multi-hop reasoning chains. Relying solely on local geometric distortion risks over-penalizing relevant high-frequency topical context. To combine their strengths, CoreMem introduces a residual fusion mechanism that interpolates between z-score normalized cosine and Riemannian similarity scores: $$ \text{score}(q, h) = \alpha \cdot \text{norm}(\text{sim}{\text{cos}}) + (1 - \alpha) \cdot \text{norm}(\text{sim}) $$ Setting }\(\alpha = 0.5\) yields a balanced convex combination that harmonizes global semantic relevance with local curvature rectification, ensuring neither bridge facts nor subtle temporal/entity details are lost.

3. Fisher-Guided Discrete Token Distillation: Establishing a mathematically bounded compression-KL tradeoff

Given retrieved raw context \(X = (x_1, \dots, x_N)\) and budget \(B\), finding the optimal discrete retention mask \(m \in \{0, 1\}^N\) to minimize generation divergence \(D_{\text{KL}}(p_\theta(Y|X) \| p_\theta(Y|\tilde{X}))\) is NP-hard. Rather than using heuristic pruning, CoreMem approximates the local loss landscape using a first-order Taylor expansion on the diagonal Fisher Information Matrix (FIM) of a local small language model \(p_\phi\). Via a single memory-efficient backward pass through input embeddings with frozen weights, per-token sensitivity is computed from the gradient of the language modeling loss: $$ S(x_i) = |\nabla_{h_i} \mathcal{L} \odot h_i|2^2 \approx \text{Tr}\left( \mathcal{I} H_i H_i^\top \right) $$ Assuming the local distiller satisfies local }\(L\)-Lipschitz continuity, dropping tokens in ascending order of \(S(x_i)\) guarantees that the induced output divergence is bounded: \(D_{\text{KL}} \le \mathcal{O}(L^2 \sum_{m_i=0} S(x_i))\). This provides an explicit mathematical error bound that bridges empirical pruning with formal information theory.

4. Structural Syntax Prior and Hierarchical Distillation: Preventing linguistic fragmentation caused by attention-sparse gradients

Because transformer gradients are attention-sparse, raw Fisher sensitivity scores often assign near-zero importance to syntactic and logical connectors (e.g., "because", "although", punctuation), stripping the grammatical glue required for downstream causal reasoning. FDTD addresses this by superimposing multiplicative structural priors: \(S_{\text{final}}(x_i) = S_{\text{smooth}}(x_i) \times \text{decay}(x_i) \times \max(M_{\text{syntax}}, M_{\text{content}})\). Specifically, logical conjunctions receive a \(1.2\times\) boost, punctuation marks receive $1.3\text}1.5\times$ preservation, and proper nouns/digits receive $1.3\text{1.4\times$ weight, accompanied by a linear turn-level temporal decay from \(1.0\) down to \(0.6\). Furthermore, FDTD operates hierarchically: sentences are segmented, and for any sentence with length \(>8\), the first three and last three tokens are strictly protected to preserve the subject-verb-object skeleton, dropping only middle redundant words. Finally, a local gap-filling step automatically restores dropped intervals of length \(\le 3\) between retained tokens, ensuring grammatical coherence for the cloud LLM.

Loss & Training

All edge components in CoreMem operate in a zero-shot, frozen-weight inference mode requiring no task-specific fine-tuning. The local distiller (e.g., Qwen3-0.6B, SmolLM2-360M, or Llama-3.2-1B) computes loss gradients \(\nabla_H \mathcal{L}\) in a single backward pass through input embeddings while freezing all transformer weights. The low-rank SVD components and Woodbury tensors are updated dynamically as memories are ingested. Local edge processing incurs ~1 second per query, which is virtually negligible compared to cloud network and generation latencies (~140–155 seconds across full benchmarks).

Key Experimental Results

Main Results

CoreMem was evaluated across two comprehensive long-context dialogue benchmarks: LOCOMO (1,542 QA pairs spanning Single-hop, Multi-hop, Temporal, Open-domain, and Adversarial categories) and LongMemEval-S (266 QA pairs assessing long-range cross-session aggregation). Evaluation relies on binary strict LLM-as-a-Judge (GPT-4o-mini, temperature 0.0, outputting yes/no), ROUGE-L, and Hit@10.

End-to-end LOCOMO results across low-dimensional (MiniLM-L6, 384-d) and high-dimensional (gte-Qwen2, 1536-d) embeddings:

System Embedding Model ROUGE-L Hit@10 Open-domain Judge↑ Temporal Judge↑ Overall Judge↑ Prompt Tokens
NaiveRAG all-MiniLM-L6 (384-d) 0.215 0.515 0.678 0.427 0.503 2.05M
Mem0 all-MiniLM-L6 (384-d) 0.216 0.515 0.679 0.406 0.503 2.05M
MemoryOS all-MiniLM-L6 (384-d) 0.215 0.517 0.678 0.417 0.501 2.05M
MemoryBank all-MiniLM-L6 (384-d) 0.079 0.164 0.113 0.229 0.097 2.47M
CoreMem-Fusion all-MiniLM-L6 (384-d) 0.227 0.532 0.723 0.469 0.540 2.09M
NaiveRAG gte-Qwen2 (1536-d) 0.227 0.536 0.715 0.448 0.531 2.36M
Mem0 gte-Qwen2 (1536-d) 0.226 0.536 0.718 0.438 0.533 2.36M
MemoryOS gte-Qwen2 (1536-d) 0.223 0.529 0.710 0.458 0.529 2.36M
MemoryBank gte-Qwen2 (1536-d) 0.076 0.146 0.112 0.250 0.099 2.34M
CoreMem-Fusion gte-Qwen2 (1536-d) 0.222 0.510 0.722 0.458 0.536 2.30M

LongMemEval-S cross-benchmark results (266 QA, MiniLM-L6 encoder, all local systems <100 MB VRAM):

System Overall Judge↑ Multi-Session Judge (MS)↑ Temporal Reasoning Judge (TR)↑ Prompt Tokens Completion Tokens Total Time (s)↓
NaiveRAG 0.436 0.541 0.331 2.68M 13.0K 711.9
Mem0 0.444 0.556 0.331 2.68M 13.4K 703.2
MemoryOS 0.406 0.489 0.323 2.54M 13.0K 633.9
MemoryBank 0.327 0.308 0.346 0.56M 7.9K 5952.3
CoreMem-V3-Pure (\(\alpha=0\)) 0.440 0.556 0.323 2.51M 13.0K 631.7
CoreMem-V3 (\(\alpha=0.9\)) 0.432 0.526 0.338 2.66M 12.9K 632.2
CoreMem-Fusion (\(\alpha=0.5\)) 0.462 0.571 0.353 2.58M 13.1K 659.2

Ablation Study

Structural protection ablations in FDTD on LOCOMO Temporal QA (96 QA, MiniLM-L6 encoder):

Configuration ROUGE-L Hit@1 Hit@10 Notes
Full Model (B=1000) 0.1245 0.0938 0.3750 Complete FDTD pipeline
- SentLvl (B=1000) 0.1263 0.0938 0.3750 Removing hierarchical sentence chunking
- Keyword (B=1000) 0.1283 0.0938 0.3750 Removing logical keyword multiplier (\(1.2\times\))
- Punct (B=1000) 0.1307 0.0938 0.3750 Removing punctuation protection ($1.3\text}1.5\times$)
- GapFill (B=1000) 0.1308 0.0000 0.0521 Disabling local gap filling triggers catastrophic accuracy collapse
Full Model (B=500) 0.1219 0.0208 0.0833 Intermediate compression budget
- Content (B=500) 0.1112 0.0938 0.3750 Removing proper noun/digit content boost ($1.3\text{1.4\times$)
- TurnDecay (B=500) 0.1121 0.0938 0.3750 Removing temporal turn decay
Full Model (B=250) 0.0938 0.0938 0.3750 Extreme token starvation
- Content (B=250) 0.0898 0.0938 0.3750 Loss of named entities severely degrades lexical recall

In the controlled compression stress test under unified NaiveRAG retrieval (MiniLM-L6, Temporal QA, budget 1000, CA = Conditional Accuracy, AR = Agreement Rate): - Truncation: CA 0.9268, AR 0.9167, Retained Ratio 75.3% - SelectiveContext: CA 0.9268, AR 0.9479, Retained Ratio 74.1% - LLMLingua2: CA 0.9512, AR 0.9375, Retained Ratio 68.7% - CoreMem (Qwen3-0.6B): CA 0.9512, AR 0.9479, Retained Ratio 68.6% - CoreMem (SmolLM2-360M): CA 0.9756, AR 0.9688, Retained Ratio 67.4% (highest factual retention with maximum compression)

Key Findings

  • Local gap filling is the discrete linchpin: Eliminating local gap filling (-GapFill) collapses Hit@1 from 0.0938 to 0.0000. Pure gradient sensitivity picks fragmented tokens that destroy autoregressive perplexity; filling short grammatical gaps (\(\le 3\) tokens) restores syntactic fluency.
  • Topological complementarity of metrics: Pure Riemannian retrieval (\(\alpha=0\)) outperforms cosine-dominated matching (\(\alpha=0.9\)) on Open-domain (+3.0 pp) and Multi-hop (+0.9 pp) by unearthing distant peripheral facts. Conversely, cosine similarity holds an advantage in Temporal reasoning (+1.0 pp) where bridge events behave as natural hubs. The hybrid CoreMem-Fusion (\(\alpha=0.5\)) harmonizes both, achieving the top score across all categories.
  • Asymmetric benefits across model sizes: Riemannian correction produces massive gains on compact encoders (+3.70 pp on MiniLM-L6, 384-d), whereas gains on high-capacity encoders (+0.5 pp on gte-Qwen2, 1536-d) are modest. Because gte-Qwen2 consumes ~5.9 GB VRAM alone, MiniLM-L6 (peak total pipeline 3.2 GB VRAM) represents the true edge-viable configuration, demonstrating that Riemannian geometry effectively compensates for limited embedding capacity.

Highlights & Insights

  • Geometric unification of retrieval and compression: Modeling retrieval as geodesic Mahalanobis distance under inverse covariance and compression as curvature preservation via diagonal Fisher traces establishes a cohesive mathematical foundation for dialogue memory.
  • Woodbury acceleration for real-time edge viability: Truncated SVD coupled with the Woodbury matrix identity reduces covariance inversion complexity from \(\mathcal{O}(d^3)\) to an \(\mathcal{O}(Ndr)\) online search, making non-Euclidean information geometry practically deployable on edge laptops.
  • Lexical versus semantic evaluation divergence: The modest correlation (\(r \approx 0.56\)) between ROUGE-L and LLM-as-a-Judge demonstrates that high lexical overlap does not guarantee sound reasoning chains, emphasizing the importance of model-based factual evaluation.

Limitations & Future Work

  • Extreme budget collapse: Under extreme starvation budgets (\(B \le 250\)), all learned neural compressors including FDTD fall below simple truncation, indicating that sub-1B parameter distillers cannot reconstruct contextual semantics under severe token pruning.
  • Linguistic and syntactic boundaries: The structural heuristic masks are currently tailored to English syntax; evaluating their robustness across morphologically rich or non-segmented languages (such as Chinese or Japanese) remains future work.
  • Large-scale memory scaling: While Woodbury acceleration ensures \(\mathcal{O}(Ndr)\) search, memory banks exceeding \(N > 100,000\) items will face memory bandwidth ceilings on consumer devices, motivating future integration with hierarchical clustering or quantized Riemannian indexes.
  • vs MemGPT / MemoryBank / Mem0: Prior systems rely on isotropic cosine matching paired with OS-like disk paging or exponential decay, targeting server hardware; CoreMem explicitly mitigates hubness via Riemannian geometry and compresses contexts to fit strictly within an 8 GB consumer edge budget.
  • vs LLMLingua2 / SelectiveContext: Prior compressors rely on token classification or heuristic self-information without theoretical bounds; CoreMem establishes an explicit KL-divergence bound via the Fisher Information Matrix, augmented with structural syntax priors.
  • vs CSLS / Inverted Softmax: These methods apply post-hoc heuristics to re-rank cosine nearest-neighbor lists; CoreMem replaces the underlying metric tensor itself, achieving fundamentally more robust hubness suppression.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates conversational memory retrieval and discrete compression under a unified information-geometric foundation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-benchmark evaluations, rigorous module ablations, compression stress tests, and failure case analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Cohesive theoretical narrative, clear mathematical formulation, and realistic hardware profiling.
  • Value: ⭐⭐⭐⭐⭐ Resolves practical memory, latency, and API cost bottlenecks for lifelong conversational agents on consumer edge devices.