Beyond Fact Retrieval: Episodic Memory for RAG with Generative Semantic Workspaces¶

Conference: AAAI 2026 Oral
arXiv: 2511.07587
Code: Available
Area: Video Understanding Keywords: Episodic Memory, RAG, Long-context Reasoning, Structured Representation, World Model

TL;DR¶

This paper proposes the Generative Semantic Workspace (GSW), a neuroscience-inspired generative memory framework that constructs structured episodic memory representations for LLMs, achieving an F1 of 0.85 on EpBench while reducing query-time context tokens by 51%.

Background & Motivation¶

Current LLMs face two fundamental challenges in long-context reasoning: (1) documents exceeding limited context windows, and (2) performance degradation with length even within the window (e.g., "context rot" and "lost-in-the-middle" effects). Existing RAG approaches have evolved from semantic embedding retrieval to structured representations such as knowledge graphs, but these methods are primarily designed for fact retrieval and cannot construct narrative representations that track how entities evolve across time and space.

Real-world texts—crime reports, political briefings, corporate documents, war correspondences—describe episodic narratives in which actors continuously evolve across spatiotemporal contexts. Accurately reasoning over such documents requires an internal world model that tracks "who was involved in what, where, when, and how roles changed." Humans achieve this through episodic memory, a capability that existing RAG systems lack.

Method¶

Overall Architecture¶

GSW consists of two core modules inspired by the neocortex–hippocampus architecture of the brain:

Operator: Corresponds to neocortical function; maps input text to intermediate semantic structures, extracting actors, roles, states, verbs, and spatiotemporal coordinates.
Reconciler: Corresponds to hippocampal function; integrates intermediate semantic structures into a persistent workspace while enforcing temporal, spatial, and logical consistency.

Workflow: text is segmented into semantically coherent chunks → the Operator generates a local workspace instance for each chunk → the Reconciler incrementally integrates these into a global memory → at query time, relevant memory segments are retrieved via entity matching.

Key Designs¶

1. Probabilistic Semantic Model in the Operator

A structured representation is extracted for each text input $C_n$:

Actors & Roles: Role labels specify a distribution $\pi_r: \mathcal{A} \times \mathcal{A} \to [0,1]$, describing the probability of an actor taking action toward another actor under a given role.
States: States serve as contextual modifiers for roles, constraining the available action space: $\pi_{r,s}(a_i \to a_j) = \pi_r(a_i \to a_j | s)$.
Verbs & Valences: Verbs encode causal relationships, with valence signals indicating changes in roles or states.
Spatiotemporal Continuity: Actors participating in an interaction are required to share consistent temporal and spatial coordinates.
Forward-Falling Questions: A set of questions anticipating potential future developments is generated based on current roles, states, and spatiotemporal coordinates.

The complete workspace instance is represented as: $\mathcal{M}_n \sim p(\mathcal{A}, \mathcal{R}, \mathcal{S}, \mathcal{V}, \mathcal{T}, \mathcal{X}, \mathcal{Q} | \mathcal{C}_{0:n})$

2. State-Space Recursive Update in the Reconciler

A Markov assumption enables recursive updates:

\[P(\mathcal{M}_n | \mathcal{C}_{0:n}) = \sum_{\mathcal{M}_{n-1}, \mathcal{W}_n} P(\mathcal{M}_n | \mathcal{M}_{n-1}, \mathcal{W}_n) \times P(\mathcal{M}_{n-1} | \mathcal{C}_{0:(n-1)}) P(\mathcal{W}_n | \mathcal{C}_n)\]

where $\mathcal{W}_n$ is the Operator's intermediate representation of the current context $\mathcal{C}_n$. The Reconciler reconciles the semantic graph of the existing workspace with incoming semantic information, enabling incremental memory construction.

3. Query Resolution Pipeline

At query time: (1) named entities in the query are located in the GSW memory via string matching; (2) contextual summaries (episodic reconstructions) are generated for matched entities; (3) summaries are reranked by semantic similarity; (4) reranked results are passed to the LLM for answer generation.

Loss & Training¶

The Operator and Reconciler are implemented via prompting GPT-4o (temperature=0) without additional training.
Maximum context utilization is uniformly capped at 17 chapters per query, consistent with the maximum number of relevant chapters per query in the ground truth.
Answer generation uniformly uses GPT-4o to ensure fair comparison.

Key Experimental Results¶

Main Results¶

Table 1: F1 Scores on EpBench-200 (by Number of Cues)

Method	0 Cues	1 Cue	2 Cues	3-5 Cues	6+ Cues	Overall
Vanilla LLM	0.840	0.709	0.585	0.476	0.325	0.629
Embedding RAG	0.906	0.726	0.723	0.745	0.680	0.771
GraphRAG	0.950	0.625	0.625	0.657	0.607	0.714
HippoRAG2	0.829	0.676	0.762	0.754	0.746	0.753
LightRAG	0.946	0.594	0.587	0.579	0.561	0.678
GSW (Ours)	0.978	0.744	0.807	0.868	0.834	0.850

Table 2: Overall Performance on EpBench-2000

Method	Precision	Recall	F1
Embedding RAG	0.827	0.688	0.675
GraphRAG	0.761	0.548	0.544
HippoRAG2	0.759	0.648	0.635
LightRAG	0.649	0.497	0.494
GSW (Ours)	0.830	0.796	0.773

Table 3: Token Efficiency Comparison

Method	Avg. Token Count	Avg. Cost
Vanilla LLM	~101,120	~$0.2528
GraphRAG	~7,340	~$0.0184
GSW (Ours)	~3,587	~$0.0090

Ablation Study¶

In the most challenging setting (6+ Cues, requiring reasoning across 17 chapters), GSW achieves a Recall of 0.822, approximately 20% higher than the second-best method HippoRAG2.
The Recall of all competing methods degrades as the number of cues increases, whereas GSW remains stable.
On EpBench-2000 (10× scale), GSW still leads by 15% in F1, demonstrating strong scalability.

Key Findings¶

GSW achieves first place in 16 out of 18 individual metric evaluations and second place in the remaining 2.
Query-time context tokens are reduced by 51% compared to the most efficient baseline, substantially lowering inference costs.
The advantage of structured representations is most pronounced on complex queries requiring reasoning across multiple chapters.

Highlights & Insights¶

Elegant Neuroscience-Inspired Design: The Operator–Reconciler architecture maps cleanly onto the functional division between the neocortex and hippocampus, yielding a conceptually coherent framework.
Simultaneous Gains in Efficiency and Accuracy: Achieving best-in-class performance with minimal token usage is a rare combination in the RAG literature.
Actor-Centric World Model: Building memory around entities rather than chunks is more cognitively aligned and better suited to narrative reasoning.
Forward-Falling Questions Mechanism: Generating anticipatory questions to support memory indexing represents an interesting and novel design choice.

Limitations & Future Work¶

Evaluation is conducted exclusively on EpBench, a synthetic dataset; validation on real-world scenarios is lacking.
Both the Operator and Reconciler rely on GPT-4o prompting, which may introduce cost and latency bottlenecks in practical deployment.
Computational overhead during the memory construction phase is not reported in detail; only query-time token efficiency is analyzed.
Entity matching relies on simple string matching, which may fail under entity variation, coreference, or aliasing.

Compared to structured RAG methods such as GraphRAG and HippoRAG2, the core distinction of GSW lies in its actor-centric design rather than fact-triple-centric representation.
Episodic memory modeling can serve as a core memory component for LLM agents, providing a foundation for long-horizon task planning.
The forward-falling question generation mechanism offers transferable inspiration for other memory-augmented systems.

Rating¶

Novelty: ⭐⭐⭐⭐ — Formalizing the neuroscientific concept of episodic memory into a computable LLM memory framework represents a strong contribution.
Technical Depth: ⭐⭐⭐⭐ — Probabilistic modeling is rigorous, and the state-space recursive update is formally complete.
Experimental Thoroughness: ⭐⭐⭐ — Experimental design is sound but limited to a single benchmark (EpBench); multi-scenario validation is absent.
Value: ⭐⭐⭐⭐ — Substantially reducing inference cost while improving performance offers practical guidance for real-world RAG systems.