Skip to content

Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/LordTARN1SHED/SCALE-QA
Area: LLM Reasoning
Keywords: Conversational Memory, Episode Integrity Failure, SCALE-QA, TSIM, Interleaved Multi-Topic Dialogues

TL;DR

Targeting "episode integrity failure" in flat unsegmented multi-turn conversations where local constraints are severed by mechanical chunking, this paper introduces the 3,000-question benchmark SCALE-QA and proposes TSIM, a lightweight multi-view episodic reconstruction and ranking stack that achieves state-of-the-art accuracy with merely ~1k prompt tokens across diverse LLM backends.

Background & Motivation

Conversational assistants in desktop and enterprise deployments increasingly span months of user interaction within a single long-running session. In these environments, discussions regarding compute budgets, software deployment policies, institutional expense reimbursement rules, clinical contraindications, and travel itineraries are routinely interleaved across thousands of turns. However, established long-context and assistant-memory benchmarks (such as LongBench, RULER, and LongMemEval) tend to evaluate either clean documents with exposed session timestamps or straightforward personal fact recall tasks (e.g., retrieving typical working hours). These benchmarks systematically understate the hardest regime encountered in real-world agentic assistance: flat, unsegmented, mixed-topic interaction streams where the system must identify which dormant historical episode renders a later local decision valid.

In this boundary-free setting, memory failures are rarely caused by simple retrieval misses or positional lost-in-the-middle degradation. Instead, models suffer from a fundamental failure mode identified here as episode integrity failure. The decisive evidence is technically present within the dialogue history, but conventional retrieval architectures return fragmented passages, stale defaults, or generic semantic summaries while dropping the critical neighboring turns that define applicability or exceptions. An episode is formally understood as the contiguous set of conversational turns that jointly establish, update, or invalidate a local constraint; if the retrieved memory unit lacks episodic integrity, the downstream language model inevitably falls into plausible but invalid distractor traps regardless of its nominal context window capacity.

To diagnose this challenge and provide an effective reference architecture, conversational memory must shift its core recovery unit from fixed arbitrary token chunks to reconstructed semantic episodes. The core idea is to replace mechanical token chunking with an inferred episodic memory substrate: by leveraging an online streaming semantic-drift segmentation algorithm coupled with an LLM-free hierarchical multi-view indexing stack (TSIM), the system accurately reconstructs complete operative episodes and presents compact ~1k-token prompts that consistently outperform brute-force million-token native contexts.

Method

Overall Architecture

The proposed approach comprises two complementary contributions: the SCALE-QA benchmark, which diagnoses constraint-grounded task decisions across flat mixed-topic histories, and the TSIM (Temporal-Semantic Interleaved Memory Reconstruction) framework. TSIM operates entirely without pre-retrieval LLM calls or future dialogue peeking. The pipeline unfolds across three core stages: first, module M1 streams through raw turns and detects semantic shifts against a local moving center to segment turns into candidate episodes; second, module M2 constructs a three-tier memory representation for each episode (raw turns at L0, deterministic truncated summaries at L1, and semantic cluster routing at L2); finally, module M3 executes evidence-first multi-view ranking, aggregating relevance signals back to episode entities and assembling top-ranked complete episodes into an efficient ~1k prompt context for the answering LLM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Flat Unsegmented Turn Stream<br/>L0: Raw interleaved turn sequence"] --> B["Semantic-Shift Streaming Segmentation<br/>Dynamic local centroid & cosine drift detection"]
    B --> C["Hierarchical Multi-View Episodic Indexing<br/>Build L0 raw / L1 summary / L2 cluster views"]
    C --> D["Evidence-First Multi-View Episode Ranking<br/>Multi-granularity scoring & episode aggregation"]
    D --> E["Operative Episode Prompt Assembly<br/>Pack ~1k tokens of compact evidence"]
    E --> F["Downstream LLM Decision<br/>Strictly grounded deterministic option selection"]

Key Designs

1. Semantic-Shift Streaming Segmentation: Reconstructing Coherent Operative Episodes Online

To eliminate the risk of splitting operative constraints across mechanical token chunk boundaries, TSIM introduces a fully online, streaming segmentation algorithm (M1) that infers episode boundaries without gold timestamps. Given normalized dense embeddings \(x_i \in \mathbb{R}^d\) for incoming turn \(i\), TSIM maintains a dynamic semantic center \(c_i\) over the recent turns \(R_i\) (a sliding window of 4 turns) belonging to the current active episode: $\(c_i = \text{norm}\left(\frac{1}{|R_i|} \sum_{j \in R_i} x_j\right)\)$ The incoming turn is evaluated via its cosine similarity against \(c_i\), incorporating a small transition incentive \(b = 0.03\) whenever a model response transitions back to a user query to capture intent pivots: $\(s_i = \cos(x_i, c_i) + b \cdot \mathbb{I}[z_{i-1}=\text{model}, z_i=\text{user}]\)$ A boundary cut is deterministically triggered via guarded thresholds: $\(\text{cut}(i) = \mathbb{I}[s_i < \theta_s \wedge L_i \ge L_{\min}] \lor \mathbb{I}[L_i \ge L_{\max}]\)$ Using frozen parameters (\(\theta_s = 0.70\), \(L_{\min} = 120\) tokens, and \(L_{\max} = 320\) tokens), M1 ensures that reconstructed episodes remain compact enough for retrieval while preserving the contiguous multi-turn context necessary to interpret local policies.

2. Hierarchical Multi-View Episodic Indexing: Anchoring Multi-Granularity Clues to Uniform Episode Entities

Because raw conversational turns can be too noisy for broad thematic queries while high-level abstracts discard specific parameter names, TSIM's M2 module organizes every reconstructed episode \(e = [s_e, t_e]\) across three parallel retrieval representations that all resolve to the same underlying episode anchor. The Raw View (L0) indexes original turn embeddings into an HNSW vector collection. The Summary View (L1) constructs a deterministic text string by prepending relative recency markers and episode identifiers to turn text truncated to 1,200 characters, indexing its embedding without invoking any external LLM summary generation. The Cluster View (L2) groups neighboring episode summaries into semantic topic clusters using centroid tracking, storing cluster summary embeddings in a routing index. Thus, every hit across fine-grained turns, episode summaries, or thematic clusters directly votes for the parent episode rather than isolated fragment tokens.

3. Evidence-First Multi-View Episode Ranking: Balanced Fusion and Compact Context Assembly

To prevent diffuse high-level cluster representations from diluting decisive technical facts, TSIM executes an evidence-first ranking policy (M3). Given a user query \(q\), retrieval is conducted across the three indexes, and candidate episodes \(e\) are scored via a multi-component objective: $\(\text{Score}(e, q) = w_r R_r(e, q) + w_s R_s(e, q) + w_l R_l(e, q) + R_{\text{sem}}(e, q)\)$ Here, \(R_r(e, q)\) and \(R_s(e, q)\) aggregate similarities from raw turns and episode summaries, while \(R_l(e, q)\) acts as an L2 cluster routing boost via \(\mathbb{I}[e \in \text{Expand}(c)]\) without ever injecting cluster summary text into the LLM prompt. \(R_{\text{sem}}(e, q)\) applies second-order semantic expansion across high-confidence seed episodes. With frozen weights calibrated on development sets (\(w_r = 1.15, w_s = 1.20, w_l = 0.75\)), raw and summary evidence dominate the score. Top-\(k\) complete episodes are ordered chronologically and formatted into a succinct prompt of ~1,000 tokens, giving the downstream answer backend clean, uncorrupted evidence.

4. SCALE-QA Benchmark Protocol: Counterfactual Grounding and Deterministic Auditability

To measure episode integrity rigorously, SCALE-QA introduces a controlled 3,000-question suite spanning 10 task domains (software engineering, networking/hardware, finance, legal, biomedicine, engineering, business operations, social/personal, games/novels, and daily life; 300 questions each). Every scenario incorporates counterfactual constraints (fictional organizational policies, idiosyncratic port allocations, custom budget caps) to eliminate pretraining leakage. Questions are formatted as deterministic four-way multiple-choice questions (MCQs) where distractors are plausible under generic public priors but strictly invalidated by the dormant local conversation episode. The generation pipeline applies strict programmatic filters (accepting only 28.8% of candidates) followed by three-reviewer human verification (84.3% acceptance) and a 300-example blind realism audit achieving 99.0% majority-gold agreement with substantial inter-annotator agreement (\(\kappa = 0.895\)), delivering an uncontaminated benchmark with 4,346 audited evidence snippets.

Benchmark & Evaluation Protocol

The benchmark evaluates conversational histories scaling from 16k to 128k tokens across all 3,000 questions, alongside a 400-question diagnostic subset extending to 1M tokens. Baselines include Standard RAG (top-5 chunks), Hybrid-RRF Chunk RAG, RAPTOR (strict no-block), MEMGPT, HIPPORAG, full native context (Full Context), and a Tuned Hybrid-Rerank Chunk RAG control incorporating BM25, BGE dense reranking, HyDE, and parent-window expansion. Evaluation tracks forced-choice Accuracy (primary metric), closed-loop evidence hit (CL Hit %), rationale cosine similarity (Rat. Sim.), and consumed answer-context tokens (CtxTok).

Key Experimental Results

Main Results

On the full 3,000-question SCALE-QA benchmark evaluated at the 128k context budget, TSIM achieves the highest accuracy across local open-source models (Gemma2:9b), commercial long-window APIs (Gemini 2.5 Flash), and high-throughput systems (GPT-4o-mini), outperforming all chunk-based, hierarchical, and graph memory baselines.

LLM Backend Retrieval / Memory System Accuracy (%) CL Hit (%) Rat. Sim. Context Tokens Serial Latency (s)
Gemma2:9b TSIM (Ours) 69.6 70.7 0.472 1060.8 4.20
Gemma2:9b Standard RAG 24.4 7.7 0.296 929.5 3.35
Gemma2:9b Hybrid-RRF Chunk RAG 31.1 11.2 0.254 1193.6 4.00
Gemma2:9b RAPTOR (no-block) 40.6 35.3 0.403 790.5 20.67
Gemma2:9b MEMGPT 60.1 62.6 0.413 2301.9 3.88
Gemma2:9b HIPPORAG 25.2 12.5 0.380 748.9 6.51
Gemini 2.5 Flash TSIM (Ours) 80.2 67.9 0.445 1275.3 1.46
Gemini 2.5 Flash Standard RAG 29.8 7.6 0.166 912.6 0.98
Gemini 2.5 Flash Hybrid-RRF Chunk RAG 32.8 11.2 0.183 1193.3 1.20
Gemini 2.5 Flash RAPTOR 52.6 34.8 0.305 790.0 9.85
Gemini 2.5 Flash MEMGPT 74.6 62.3 0.398 2349.6 1.19
Gemini 2.5 Flash HIPPORAG 34.4 12.1 0.245 752.7 3.10
GPT-4o-mini TSIM (Ours) 73.8 74.2 0.485 1043.6 2.66
GPT-4o-mini Standard RAG 27.1 7.6 0.238 910.1 1.72
GPT-4o-mini Hybrid-RRF Chunk RAG 31.1 11.3 0.203 1193.5 1.59
GPT-4o-mini Tuned Hybrid-Rerank โ€  56.2 63.5 0.487 3004.2 11.91
GPT-4o-mini RAPTOR 43.0 35.1 0.419 789.2 11.59
GPT-4o-mini MEMGPT 50.2 58.6 0.298 2238.4 1.77
GPT-4o-mini HIPPORAG 31.0 11.9 0.202 753.7 2.85
GPT-4o-mini Full Context 29.8 100.0 โ€” 88425.0 14.61

Note: โ€  indicates the tuned non-episodic retrieval baseline combining BM25, BGE dense retrieval/reranking, HyDE, and parent-window expansion.

Ablation Study

A progressive ablation conducted with GPT-4o-mini at 128k isolates the specific contributions of streaming semantic segmentation and multi-view indexing:

Level Configuration Variant Accuracy (%) CL Hit (%) Rat. Sim. Context Tokens Latency (s) Description
L0 Standard RAG top-5 26.2 5.6 0.246 968.4 1.61 Severe chunk fragmentation
L1 Fixed-Token Direct Window (128 tok) 43.4 35.0 0.416 1269.2 3.52 Arbitrary boundaries lose context
L2 Semantic-Drift Direct Episodes (M1) 55.5 52.2 0.430 1179.8 3.62 Preserves complete local episodes
L3 Full TSIM Memory Stack (M1+M2+M3) 74.2 74.3 0.485 1044.4 3.56 Multi-view ranking maximizes utility

Key Findings

  • Episode fragmentation dominates baseline failures: Standard RAG recovers the necessary evidence in only 7.6%โ€“7.7% of queries at 128k, leading to near-random accuracy. Even the intensely tuned non-episodic pipeline (Tuned Hybrid-Rerank Chunk RAG) reaches only 56.2% accuracy while consuming nearly triple the prompt context tokens (3,004 vs. 1,044 tokens), demonstrating that better chunk retrieval cannot compensate for missing episodic boundaries.
  • Native context scaling suffers severe reliability collapse: Under GPT-4o-mini, Full Context performance drops precipitously from 62.5% at 16k to 29.8% at 128k, despite the prompt containing 100% of the gold evidence. In the 1M-token diagnostic test with Gemini 2.5 Flash, Full Context achieves 87.2% accuracy while paying 1.05M prompt tokens and 23.87 seconds of latency; in contrast, TSIM achieves 96.5% accuracy with only ~1.3k prompt tokens and 2.16 seconds of latency.
  • Robust transfer across benchmarks without boundary metadata: Evaluated on the 500-question LongMemEval-S benchmark without access to official session boundaries, TSIM attains 71.0% judged accuracy, outperforming context-matched fixed-chunk retrieval by 9.8 points and turn-level BGE retrieval by 14.4 points, closely approaching the 74.4% score achieved by session-boundary-assisted methods that consume 2.77ร— more context.

Highlights & Insights

  • From Arbitrary Chunks to Coherent Operative Episodes: Rather than treating dialogue histories like static text corpora split by fixed token budgets, TSIM shows that conversational memory requires semantic episode boundaries that maintain the full lifecycle of a constraint.
  • Zero-LLM Overhead in Ingestion and Indexing: Both episode segmentation (cosine drift over sliding centroids) and multi-view index creation (deterministic truncation and text clustering) operate entirely through local embedding lookups, incurring zero LLM inference cost prior to query answering.
  • Contamination-Resilient Benchmark Methodology: SCALE-QA's pairing of counterfactual local constraints with plausible public distractors guarantees that models cannot cheat via pretraining priors, establishing an auditable gold standard for conversational memory research.

Limitations & Future Work

  • Semi-Synthetic Construction vs. Real In-the-Wild Logs: While validated through a 300-example blind human audit yielding 99.0% gold agreement, SCALE-QA scenarios remain synthetically generated, leaving colloquial messiness, typing noise, and nuanced human conversational habits for future empirical validation.
  • Multiple-Choice Formatting Boundaries: Deterministic MCQ grading provides pristine evidence auditability but does not test open-ended conversational skills such as hedging, clarifying questions, or iterative tool-use follow-ups.
  • Multi-Episode Routing Fragmentation: Qualitative error analysis indicates that when a problem requires synthesizing disjoint constraints established in two widely separated episodes, top-\(k\) packing and cluster routing occasionally fail to co-retrieve both episodes.
  • vs LongMemEval: LongMemEval focuses on multi-session dialogues structured with explicit timestamps and evaluates personal-memory facts. SCALE-QA targets flat, boundary-free, mixed-topic streams with counterfactual operational constraints, isolating episode integrity failure.
  • vs MemGPT: MemGPT manages persistent memory using LLM function calling over tiered buffers; TSIM replaces LLM memory management with streaming mathematical segmentation and multi-view retrieval, halving context token overhead (1.3k vs. 2.3k tokens) while improving accuracy by 5.6โ€“23.6 points.
  • vs RAPTOR: RAPTOR constructs hierarchical tree summaries offline; TSIM performs online streaming segmentation suited for monotonically expanding chat streams, maintaining high signal-to-noise ratios without expensive recursive summarization calls.

Rating

  • Novelty: โญโญโญโญโญ Formulates the critical concept of "Episode Integrity Failure" and develops a clean, streaming, multi-view episodic memory architecture.
  • Experimental Thoroughness: โญโญโญโญโญ Spans 16k to 1M contexts, three distinct model backends, 3,000 fully audited questions, and comprehensive ablations.
  • Writing Quality: โญโญโญโญโญ Rigorous mathematical formalization, clear conceptual definitions, and transparent diagnostic reporting.
  • Value: โญโญโญโญโญ Provides an indispensable evaluation benchmark and a highly practical, token-efficient architecture for long-context assistant development.