Skip to content

COLLABMEM: A Diagnostic Benchmark for Collaborative Memory Operations in Multi-Agent LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/NLPerWS/COLLABMEM
Area: LLM Agent
Keywords: Collaborative Memory, Multi-Agent Systems, Transactive Memory Systems, Access Control, Knowledge Graph Memory

TL;DR

This paper introduces COLLABMEM, the first diagnostic benchmark for collaborative memory operations in role-governed multi-agent LLM systems, alongside TMS-GRAPH++, a scope-aware graph memory framework that combines role-filtered visibility, writer provenance, and temporal [CURRENT] tags to achieve top-tier collaborative retrieval while strictly preventing unauthorized leakage.

Background & Motivation

Multi-agent large language model (LLM) systems are rapidly transforming automated software development, enterprise workflows, clinical coordination, and collaborative problem-solving. As teams of specialized agents tackle extended tasks, they increasingly rely on persistent memory substrates to coordinate role-specific activities and synchronize shared understanding across time. However, existing persistent memory benchmarks—such as LongMemEval, MemBench, and MemoryBank—evaluate memory primarily within dyadic, single-user histories focusing on temporal recall, personalization, and context compression. Conversely, popular multi-agent evaluation frameworks like AgentBench, Sotopia, AgentVerse, and MultiAgentBench assess holistic task completion, conversation dynamics, or tool-use trajectories. In these end-to-end environments, memory errors are deeply confounded with planning failures, communication overhead, tool invocation issues, and downstream reasoning bottlenecks, making it nearly impossible to pinpoint why an agent failed.

In specialized agent teams, correctness goes far beyond whether a given fact was retained in a storage bank. A critical API modification observed by a backend agent must be selectively routed to the QA engineer while remaining strictly hidden from an unprivileged customer support agent; furthermore, an authoritative revision issued by a lead architect must reliably supersede an earlier consensus while preserving the provenance trail explaining why the change occurred. This operational challenge mirrors Transactive Memory Systems (TMS) theory from organizational psychology, which models collective memory as distributed task knowledge coupled with meta-knowledge regarding who knows what and which sources are credible. Contemporary collaborative systems either isolate agents into siloed private stores—blocking collective recall—or collapse context into ungoverned global logs and shared vector databases, which inevitably suffer from severe privacy leaks, stale value confusion, and broken source attribution.

To establish clear diagnostic signals for multi-agent memory engineering, this work isolates collaborative memory from general agent execution loops into a controlled write-index-retrieve problem. Core idea: Grounded in Transactive Memory Systems theory, COLLABMEM establishes a matched, memory-only benchmark that fixes the interaction trace, query role, answering model, and judge across six core competencies; concurrently, TMS-GRAPH++ is introduced as a scope-aware graph memory architecture featuring role-filtered views, writer provenance, and temporal [CURRENT] marking.

Method

Overall Architecture

COLLABMEM formalizes collaborative memory evaluation as a controlled testbed over a role-governed team. Each benchmark scenario is defined by a tuple \((R, A, E, Q)\), where \(R\) denotes the set of specialized agent roles, \(A\) represents the role-to-category access matrix specifying permissions for storing and disclosing facts, \(E\) is an ordered schedule of gold-bearing events annotated with observers, payload attributes, timestamps, credibility weights, and supersession pointers, and \(Q\) is a deterministic probe set derived directly from \(E\) and \(A\). In this matched harness, evaluated memory architectures ingest the identical trace, retrieve a scoped memory context for the designated target role \(r\), and pass this context to a strictly frozen answering LLM evaluated by semantic and evidence-aware judges.

To resolve the architectural trade-off between role isolation and coordinated recall, the proposed TMS-GRAPH++ framework models the collective team memory as a unified visibility graph \(G\). Each ingested event forms an edge containing writer identity, creation timestamp, credibility weight, contradiction flag, and the authorized role subset. At query time, TMS-GRAPH++ executes role-aware scope filtering, computes temporal authority keys to mark current truth, and attaches a specialist registry strictly when answering transactive routing queries. The complete diagnostic evaluation and retrieval pipeline is structured as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Event Schedule E & Access Matrix A"] --> B["Pre-Rendered Gold Anchors & Multi-Agent Trace"]
    B --> C["Matched Memory-Only Harness COLLABMEM<br/>Fixed Trace / Target Role / Answering LLM"]
    C --> D["Role-Aware Scope Filtering<br/>Filter visible edges V per target role r"]
    D --> E["Temporal Authority Tagging [CURRENT]<br/>Lexicographic tuple maximization"]
    E --> F["Conditional Specialist Registry<br/>Expose who knows what only on C4 routing probes"]
    F --> G["Standardized Context Injection & Dual Evaluation<br/>Loose Semantic Judge + Strict Provenance Judge"]

Key Designs

1. Six Transactive Memory Competencies with Audit-Gated Synthesis: Decoupling memory operations into auditable failure modes To map empirical score drops directly to actionable system repairs, COLLABMEM operationalizes the four TMS pillars (Specialization, Credibility, Coordination, Shared Meta-Knowledge) into six distinct, measurable memory competencies: Directed Sharing (C1, non-sensitive facts routed to designated roles), Cross-Role Consistency (C2, synchronized views across roles after updates), Access Control (C3, withholding sensitive facts from unauthorized roles), Transactive Routing (C4, identifying authoritative specialists without leaking facts), Conflict Resolution (C5, adjudicating contradictory updates via credibility and supersession), and Collective Compression (C6, retaining superseded values and update rationales under strict memory budgets). All 58 core seed scenarios are hand-crafted across five collaborative domains (software engineering, research, customer support, clinical coordination, project management). Crucially, the gold fact chains are generated and verified prior to natural-language trace rendering, guaranteeing 100.00% deterministic machine auditability and 96.57% two-annotator human verification agreement.

2. Role-Aware Scope Filtering on Visibility Graphs: Resolving the conflict between global leakage and role isolation Existing memory designs either enforce strict per-role isolation (causing complete failure on shared information) or merge logs into a shared pool (causing rampant privacy leakage). TMS-GRAPH++ models memory as a global directed visibility graph \(G\), where each edge \(e\) encapsulates writer identity, validity timestamp, credibility score, contradiction metadata, and authorized viewer set \(\text{visible}(e)\). Upon receiving a query \(q\) directed to role \(r\), the system performs deterministic scope filtering: $\(V \leftarrow \{e \in G \mid r \in \text{visible}(e)\}\)$ By pruning unauthorized edges at the graph retrieval stage, TMS-GRAPH++ enforces private-by-default access control without sacrificing legitimate cross-role information channels, eliminating the fragility of relying on prompt instructions to safeguard sensitive information.

3. Temporal Authority Tagging with [CURRENT] Anchors: Resolving semantic contradictions through source credibility When multiple roles introduce conflicting statements across time, surface-level LLMs frequently assume the most recent text utterance is correct, failing when an authoritative specification is contradicted by an unverified claim. Within the visible subgraph \(V\), TMS-GRAPH++ computes an authority key for each \((\text{subject}, \text{predicate})\) pair defined as the lexicographic tuple \((\text{contradiction\_flag}, \text{credibility}, \text{ingestion\_time})\). The edge achieving the maximal tuple is explicitly tagged with [CURRENT]. Linearizing visible edges chronologically with writer labels and [CURRENT] markers provides the downstream answering LLM with an unambiguous indicator of current factual consensus while preserving the audit trail required for historical justification.

4. Conditional Specialist Registry for Transactive Routing: Enabling meta-knowledge discovery without content leakage In collaborative teams, knowing which agent possesses specific domain expertise is essential for task routing; however, exposing internal memory contents to all agents breaches confidentiality. TMS-GRAPH++ introduces a segregated Writer and Specialist Registry that tracks role-to-topic ownership. This registry is conditionally surfaced only when the probe query explicitly targets transactive routing (C4). During standard factual queries or access probes (C3), the registry is completely decoupled, preventing information leakage while providing direct meta-knowledge lookup for collaborative handoffs.

Benchmark & Evaluation Protocol

COLLABMEM incorporates a dual-track scoring harness to separate superficial recall from provenance-aware retrieval. The Loose Protocol utilizes an LLM-as-a-judge semantic evaluator accepting valid paraphrases of the target value. The Strict Protocol enforces rigorous criteria: the model must supply the exact entity value, state the correct authoritative role or writer, provide temporal or update evidence, and refrain from unsupported hedging. The dataset comprises five benchmark splits: Core (58 scenarios, 1,118 probes), Scaled (174 scenarios, 3,359 probes), Adversarial (399 probes testing indirect sharing, obfuscated boundaries, and silent contradictions), Compositional (130 multi-hop probes chaining conflict resolution with routing), and a stratified Human Verification sample (204 probes). Evaluated systems span trivial controls (NoMem, FullShare, RAG-Shared), controlled mechanism baselines (Mem0-inspired, Letta-inspired, A-Mem-inspired, Graph-Shared), and official SDK implementations (Mem0 SDK, Letta SDK, A-Mem SDK).

Key Experimental Results

Main Results

Under the uniform answering backbone (gpt-4o-mini), Table 1 displays the primary diagnostic scorecard across the six competencies on the Core Loose split:

System Category System / Architecture C1 (Share) C2 (Consist.) C3 (Access) C4 (Route) C5 (Conflict) C6 (Compress) Core Loose Overall
Trivial Controls NoMem 0.0 0.0 100.0 11.7 0.0 0.0 18.6
FullShare 67.2 83.0 60.0 78.6 80.1 45.7 69.1
RAG-Shared 58.4 68.1 39.0 74.0 69.3 55.6 60.7
Controlled Baselines Mem0-inspired 71.2 77.7 96.8 64.9 82.5 54.3 74.6
Letta-inspired 72.0 76.6 96.8 65.1 77.7 58.0 74.4
A-Mem-inspired 1.6 6.4 100.0 15.7 36.8 55.6 36.0
Graph-Shared 73.6 81.9 97.9 94.9 87.3 51.9 81.2
TMS-GRAPH++ (Ours) 81.6 94.7 96.8 99.7 92.2 49.4 85.7
Official SDKs Mem0 SDK (role) 0.0 0.0 100.0 26.3 0.0 1.2 21.2
Mem0 SDK (role+team) 0.0 1.1 100.0 22.6 0.0 1.2 20.8
Letta SDK (role) 1.6 5.3 100.0 18.3 28.3 64.2 36.3
Letta SDK (role+team) 68.8 78.7 96.8 74.9 72.3 54.3 74.3
A-Mem SDK (role) 1.6 5.3 100.0 16.9 29.5 63.0 36.0
Calibration Oracle 82.4 100.0 100.0 100.0 93.4 49.4 87.5

Cross-split robustness across Core, Strict, Adversarial, Compositional, and Scaled environments is summarized in Table 2:

System Category System / Architecture Core Loose Core Strict Adversarial Compositional Scaled 5-Split Avg.
Trivial Controls NoMem 18.6 18.4 29.0 22.5 18.6 21.4
FullShare 69.1 31.0 58.0 37.8 67.6 52.7
RAG-Shared 60.7 25.2 50.7 42.4 56.9 47.2
Controlled Baselines Mem0-inspired 74.6 30.0 63.2 39.5 71.5 55.7
Letta-inspired 74.4 30.1 63.6 38.4 71.1 55.5
A-Mem-inspired 36.0 21.5 41.4 37.0 35.0 34.2
Graph-Shared 81.2 53.0 73.2 49.5 78.7 67.1
TMS-GRAPH++ (Ours) 85.7 52.9 74.2 52.1 82.9 69.6
Official SDKs Mem0 SDK (role) 21.2 20.2 34.6 22.4 20.9 23.9
Letta SDK (role) 36.3 23.5 39.0 42.7 35.8 35.5
Letta SDK (role+team) 74.3 40.4 65.4 47.0 73.9 60.2
A-Mem SDK (role) 36.0 24.1 38.6 40.9 33.9 34.7
Calibration Oracle 87.5 68.7 75.2 51.2 87.0 73.9

Ablation Study

Ablation experiments on Core Loose (Table 3) isolate the performance gains attributable to each architectural module in TMS-GRAPH++:

Variant Architectural Deviation Core Overall (%) \(\Delta\) (vs. Full Model)
TMS-GRAPH++ Full architecture (Scope filtering + [CURRENT] tags + Conditional registry) 85.7 —
Graph-Shared Removes scope filtering and [CURRENT] tags (shared graph) 81.2 -4.5
TMS-lite Removes role-aware scope filtering 81.1 -4.6
TMS-Graph Removes conditional writer/specialist registry (degrades C4) 80.6 -5.1

Evaluating backbone sensitivity across LLMs indicates that ranking remains completely invariant to answering capacity: TMS-GRAPH++ scores 85.7% on gpt-4o-mini, 87.7% on claude-3.5-haiku, 85.1% on qwen-2.5-72b, and 81.2% on llama-3.1-70b, consistently outperforming Graph-Shared (81.2%, 85.4%, 84.6%, and 78.9% respectively).

Key Findings

  • Isolated role memory blocks collaborative recall: Official SDK role-only baselines achieve 100.0% access security (C3) but virtually fail on cross-role sharing (C1: 0.0%–1.6%), consistency (C2: 0.0%–5.3%), and conflict resolution (C5: 0.0%–29.5%). Adding a non-sensitive team archive (Letta role+team) dramatically boosts overall accuracy to 74.3%, proving that explicit collaborative pathways are mandatory.
  • Ungoverned global sharing leaks protected knowledge: FullShare and RAG-Shared achieve reasonable recall on non-sensitive facts but fail catastrophically on access boundaries, leaking confidential facts on 40.0% and 61.0% of C3 probes.
  • Provenance metadata is indispensable for temporal updates: While unguided graph retrieval (Graph-Shared) aids routing, it degrades under strict evidence requirements and compositional chaining. Explicit [CURRENT] markers and writer metadata provide an essential +5.1 point boost.
  • The benchmark remains un-saturated: Even the Oracle baseline reaches only 49.4% on C6 (Collective Compression) and 68.7% on Strict evaluation, highlighting that collaborative retention of historical update rationale remains an unsolved challenge for agent memory systems.

Highlights & Insights

  • Clean Diagnostic Decoupling: By fixing the dialogue traces, querying roles, answering models, and evaluation judges, COLLABMEM completely removes planning and tool-execution confounding factors, offering the community an auditable testbed for memory architectures.
  • Theoretically Grounded System Primitives: TMS-GRAPH++ translates cognitive Transactive Memory Systems theory into efficient graph operations (\(O(|G|)\) complexity), successfully resolving the privacy-sharing dilemma via dynamic scope filtering and lexicographic authority tagging.
  • Rigorous Strict Provenance Evaluation: Demanding not just the final entity value but the verified source role and chronological update evidence prevents models from getting credit via spurious pattern matching or hallucinations.

Limitations & Future Work

  • Synthetic vs. Organic In-the-Wild Traces: The benchmark relies on controlled grammar-based simulation to guarantee ground-truth auditability; future iterations should incorporate real-world production interaction logs containing conversational disfluencies and multi-party interruptions.
  • SDK Baselines Reflect Minimal Transparent Adapters: Official SDK results evaluate out-of-the-box memory primitives under transparent role policies rather than proprietary, heavily tuned end-to-end multi-agent orchestration frameworks.
  • Need for Specialized Compression Primitives: Current graph architectures lack purpose-built episodic summarization and forgetting algorithms, reflected by the lower performance across systems on C6 Collective Compression.
  • vs. Long-Memory Benchmarks (LongMemEval, MemBench): Prior benchmarks evaluate recall, temporal order, and user personalization within single-user contexts; COLLABMEM introduces multi-agent role access boundaries, authority conflict resolution, and transactive routing.
  • vs. Multi-Agent Task Benchmarks (AgentBench, MetaGPT, MultiAgentBench): Existing suites measure holistic task success, where planning and tool failures mask underlying memory bugs; COLLABMEM isolates the memory layer to pinpoint operational failures.
  • vs. Memory Architectures (Mem0, Letta / MemGPT, A-Mem): Prior frameworks provide tiered buffers, archival stores, or note graphs without native role-governed access matrices; TMS-GRAPH++ establishes that role-filtered graph views and explicit [CURRENT] tags are vital for reliable multi-agent coordination.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering formulation of collaborative memory operations grounded in transactive memory systems theory.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 5 splits, 4 LLM backbones, official SDK integrations, and dual-track scoring protocols.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured, mathematically precise, and provides crystal-clear mappings between theoretical constructs and engineering primitives.
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable foundational benchmark and architectural blueprint for next-generation multi-agent memory infrastructure.