TriggerBench: Investigating Prospective Memory for Large Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/KristenZHANG/TriggerBench-Official
Area: LLM Reasoning
Keywords: Prospective memory, Proactive intervention, Attentional fragility, Cognitive cliff, Spare reasoning budget
TL;DR¶
This work introduces TriggerBench, a five-dimensional benchmark comprising 1,265 tasks that systematically evaluates Prospective Memory (PM)—the capability to spontaneously recall and act on latent constraints without direct prompting—revealing severe attentional fragility and a Cognitive Cliff over long contexts, while establishing PM as a fine-grained behavioral probe of spare reasoning capacity.
Background & Motivation¶
As Large Language Models (LLMs) transition from single-turn chat interfaces to long-horizon conversational assistants and autonomous software agents, systems must move beyond passively answering explicit queries to proactively anticipating latent risks and initiating timely interventions. However, existing benchmarks for long-context interactive memory (e.g., LoCoMo, LongMemEval) and memory-augmented frameworks (e.g., Mem0, A-MEM) evaluate almost exclusively on Retrospective Memory (RM). In RM, models are evaluated purely on passive retrieval triggered by direct user queries (e.g., "Where did I put my keys?"). Such passive mechanisms completely fail to safeguard real-world interactions where latent vulnerabilities are left unaddressed—such as a developer temporarily commenting out an authentication decorator during debugging and forgetting to restore it before deployment, or a passenger placing a valuable musical instrument in a taxi trunk and walking away upon arrival.
In cognitive psychology, the ability to formulate, retain, and spontaneously execute intended actions upon encountering a future triggering event is formalized as Prospective Memory (PM). Developing robust PM in LLMs presents a profound cognitive tension: in realistic human interactions, latent constraints emerge implicitly across conversational turns without explicit reminder tags ("Remind me to..."), requiring autonomous situational awareness rather than delayed instruction following. Concurrently, when the triggering scenario manifests, it is frequently accompanied by distracting, foreground user requests (e.g., asking for nearby restaurant recommendations while stepping out of a cab), which violently hijacks the model's limited attentional bandwidth. Models evaluated solely on positive recall frequently collapse into an "always-remind" false-alarm heuristic, creating an unbearable user experience.
To rigorously address this critical blind spot, this paper formalizes prospective memory as a dynamic, context-driven sequential reasoning process and introduces TriggerBench. Core idea: by constructing multi-turn dialogues with strict lexical and semantic decoupling between latent constraints and future triggers, paired with contrastive positive, overloaded, and negative variants alongside matched retrospective memory controls, this benchmark not only exposes the catastrophic "Cognitive Cliff" of proactive recall under long contexts, but also establishes PM as a non-intrusive behavioral probe of an LLM's spare reasoning capacity.
Method¶
Overall Architecture¶
TriggerBench formalizes prospective memory as an autonomous sequence-level observation and reasoning task: early in the interaction history, the user introduces a declarative statement, environmental state, or personal policy without explicit delegation, establishing a latent Constraint (\(C\)). The dialogue then proceeds through variable-length Orthogonal Thematic turns (\(\Theta\)) that are cohesive yet topic-independent, avoiding shallow information leakage. At the current turn, a benign action or state mutation serves as the Trigger (\(T\)). When combined with \(C\), the trigger creates an active hidden risk, requiring the model to spontaneously execute an Intervention (\(A\)). The benchmark generation pipeline proceeds through Blueprint Design, Base Dialogue Stretching with Variant Derivation (Pos-Clean, Pos-Over, and Neg-Clean), and rigorous Expert-in-the-Loop Quality Control, supplemented by parallel Retrospective Memory (RM) probes.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Scenario Blueprint Design<br/>488 Curated Blueprints · 5 Cognitive Dimensions"] --> B["Dialogue Stretching & Variant Derivation<br/>Orthogonal Turns · Prefix Preservation Overload · State-Flipping Negatives"]
B --> C["Formal Decoupling Validation<br/>Lexical Word Overlap & Dense Embedding Cosine Distance"]
C --> D["Dual Evaluation Protocols & Intent Judge<br/>Slot Match & LLM-as-a-Judge Intent Detector"]
D --> E["Reasoning Budget Probing<br/>Joint PM Probes with AIME-2025 Cognitive Load"]
Key Designs¶
1. Scenario Blueprint Design: Hierarchical Taxonomies Across 5 Cognitive Dimensions
Addressing the narrow scope and artificial nature of prior memory benchmarks, TriggerBench designs 488 curated blueprints across 19 sub-domains, spanning five distinct dimensions of prospective reasoning: (1) State-Tracking: evaluates the model's ability to track implicit physical or digital state mutations (e.g., placing items in a vehicle trunk, opening safety valves) and intervene when subsequent actions create hazards; (2) Temporal Grounding: covers point-in-time commitments across Absolute Time ("at 10:00 AM") and Relative Time ("needs 45 minutes of incubation"), testing autonomous deadline tracking and scheduling collision prevention; (3) Logical Adherence: formalizes dynamic policy compliance across two-hop conditionals (If-Then), mutual exclusions (e.g., strict budget caps, anonymization rules), and prerequisite gates (Only-After); (4) Attention Recovery: evaluates attentional resilience when an ongoing primary task is abruptly suspended due to cognitively intensive interruptions (e.g., competitive math or programming problems), testing whether the model autonomously prompts resumption once the interruption concludes; (5) Safe Coding: grounds PM in professional software engineering, covering 40 Common Weakness Enumerations (CWEs) across 8 vulnerability classes from SafeGenBench (e.g., temporary auth bypasses, relaxed CORS), requiring proactive intervention before code is tagged for production rollout.
2. Dialogue Stretching & Variant Derivation: Isolating False Alarms and Attentional Hijacking
To prevent models from exploiting superficial heuristics or trivial "always-remind" defaults, TriggerBench programmatically derives three contrastive evaluation variants from each blueprint: Positive-Clean (Pos-Clean) serves as the canonical task where the constraint remains active and the trigger arrives cleanly; Positive-Overloaded (Pos-Over) enforces strict Prefix Preservation by appending an explicit, distracting user request (e.g., asking for a culinary recipe) to the verbatim clean trigger, cleanly isolating performance collapses caused purely by concurrent task interference and attention hijacking; Negative-Clean (Neg-Clean) tests precision by introducing an intermediate turn where the user preemptively resolves the hazard (for State-Tracking and Safe Coding) or by mutating the trigger to logically satisfy the constraint (for Logical Adherence). Reliable models must detect context updates and remain appropriately silent, directly penalizing over-intervention.
3. Formal Decoupling Validation: Neutralizing Lexical Overlap and Embedding Search Shortcuts
To ensure models cannot solve TriggerBench via superficial retrieval shortcuts, the benchmark enforces stringent semantic decoupling between constraints (\(C\)), triggers (\(T\)), and orthogonal thematic context (\(\Theta\)). Quantitatively, in State-Tracking, Temporal Grounding, and Attention Recovery, over 94% of constraint-trigger pairs share zero content words (Jaccard coefficient \(\le 0.005\)). Dense embedding analysis using bge-base-en-v1.5 reveals that the pairwise cosine similarity between \(C\) and \(T\) (0.46–0.48) is statistically identical to background conversational noise (\(C \leftrightarrow \Theta\)), while thematic turns remain tightly cohesive (\(\Theta \leftrightarrow \Theta \approx 0.650\)). This structural separation causes conventional embedding-based RAG and semantic long-term memory architectures to catastrophically fail during retrieval, proving that prospective intervention demands genuine working memory maintenance.
4. Dual Evaluation Protocols & Intent Judge: Combining Slot Matching with Semantic Intent Calibration
To objectively measure open-ended proactive behavior without lexical rigidity, TriggerBench establishes two complementary evaluation metrics: rule-based Slot Match provides a transparent, deterministic lower bound by checking case-insensitive substring recall of gold constraint entities; PM Accuracy (PM Acc) configures GPT-4o (\(T=0\)) as a calibrated, deterministic Intent Detector. Grounded in the constraint-trigger-reference context and informed by dimension-specific few-shot exemplars, the judge credits valid semantic paraphrases and contextualized warnings while strictly penalizing superficial keyword repetition devoid of actionable reminders. On negative controls, the evaluator verifies that the model refrains from issuing false alarms. Expert human validation confirms high inter-annotator agreement (\(\kappa = 0.940\)) and 93.3%–94.2% human-judge alignment.
Benchmark & Evaluation Protocol¶
- Evaluated Architectures: Long-Context LLMs (Qwen3-32B, Qwen3-235B-Instruct/Thinking, Gemma3-27B-it, GPT-4o, GPT-4.1, GPT-5.2 with non-thinking, medium, and high reasoning budgets) alongside Memory-Augmented Frameworks on a GPT-4.1 backbone (Standard RAG, A-MEM, Mem0, Letta-Sim).
- Context Scaling: Base dialogues average 2.5K tokens across 16.4 turns; long-context versions scale up to 40K tokens via domain-tailored distractors (MultiWOZ for daily assistant tasks, OpenThoughts for Attention Recovery, and TerminalBench for Safe Coding), extended up to 100K tokens for deep cognitive profiling.
- Prompt Standardization: Evaluated under the unified zero-shot BLENDED prompt, which instructs the model to act as a helpful and attentive conversational assistant without providing task-specific hints or trigger foreknowledge.
Key Experimental Results¶
Main Results¶
Performance across Base (2.5K) and 40K contexts on TriggerBench is summarized below (raw figures from Table 3):
| Model / System | Pos-Clean Slot | Pos-Clean PM Acc | Pos-Over PM Acc | Neg-Clean PM Acc | All-Pos PM Acc | Overall PM Acc |
|---|---|---|---|---|---|---|
| Base Context (2.5K) | ||||||
| Qwen3-32B | 49.32 | 49.80 | 13.11 | 71.97 | 31.46 | 40.71 |
| Qwen3-235B-Instruct | 68.31 | 68.65 | 35.66 | 69.90 | 52.15 | 56.21 |
| Qwen3-235B-Thinking | 88.69 | 92.83 | 64.14 | 62.63 | 78.48 | 74.86 |
| Gemma3-27B-it | 79.00 | 76.43 | 68.85 | 63.67 | 72.64 | 70.59 |
| GPT-4o | 60.38 | 62.91 | 22.34 | 82.70 | 42.62 | 51.78 |
| GPT-4.1 | 72.37 | 73.36 | 36.68 | 76.82 | 55.02 | 60.00 |
| GPT-5.2 (no thinking) | 84.39 | 89.75 | 64.55 | 42.90 | 77.15 | 69.33 |
| GPT-5.2 (reason: medium) | 89.48 | 96.72 | 87.70 | 46.37 | 92.21 | 81.74 |
| GPT-5.2 (reason: high) | 90.88 | 98.16 | 91.60 | 43.94 | 94.88 | 83.24 |
| RAG (GPT-4.1) | 67.76 | 68.85 | 27.87 | 62.63 | 48.36 | 51.62 |
| A-MEM (GPT-4.1) | 50.89 | 51.03 | 22.95 | 56.40 | 36.99 | 41.42 |
| Mem0 (GPT-4.1) | 70.32 | 71.31 | 33.61 | 55.71 | 52.46 | 53.20 |
| Letta-Sim (GPT-4.1) | 79.44 | 75.20 | 53.89 | 54.67 | 64.55 | 62.29 |
| 40K Context | ||||||
| Qwen3-235B-Instruct | 37.77 | 37.50 | 19.06 | 82.70 | 28.28 | 40.71 |
| Qwen3-235B-Thinking | 61.71 | 61.68 | 34.84 | 67.48 | 48.26 | 52.65 |
| GPT-4.1 | 48.12 | 51.23 | 23.15 | 72.32 | 37.19 | 45.22 |
| GPT-5.2 (reason: high) | 86.51 | 92.62 | 78.28 | 39.80 | 85.45 | 75.02 |
| RAG (GPT-4.1) | 38.32 | 38.53 | 19.67 | 69.21 | 29.10 | 38.26 |
Ablation Study: Explicit vs. Implicit Constraints¶
By minimally editing the constraint turn into an explicit delegation template while preserving triggers and dialogues verbatim, the benchmark isolates the cost of autonomous situational awareness compared to delayed instruction following (raw figures from Table 4, GPT-4.1):
| Dimension | Variant | Base Implicit | Base Explicit | Base Gain (\(\Delta\)) | 40K Implicit | 40K Explicit | 40K Gain (\(\Delta\)) |
|---|---|---|---|---|---|---|---|
| State-Tracking | Pos-Clean | 57.4 | 100.0 | +42.6 | 31.5 | 80.6 | +49.1 |
| Pos-Over | 25.0 | 60.2 | +35.2 | 13.0 | 40.7 | +27.8 | |
| Neg-Clean | 82.4 | 73.2 | -9.3 | 83.3 | 73.2 | -10.2 | |
| Temporal Grounding | Pos-Clean | 58.7 | 81.1 | +22.4 | 25.9 | 58.0 | +32.2 |
| Pos-Over | 35.0 | 51.0 | +16.1 | 15.4 | 32.9 | +17.5 | |
| Logical Adherence | Pos-Clean | 96.2 | 99.2 | +3.0 | 90.2 | 98.5 | +8.3 |
| Pos-Over | 64.7 | 87.2 | +22.6 | 57.1 | 83.5 | +26.3 | |
| Neg-Clean | 78.2 | 78.2 | +0.0 | 69.2 | 75.9 | +6.8 | |
| Attention Recovery | Pos-Clean | 96.4 | 98.2 | +1.8 | 73.2 | 89.3 | +16.1 |
| Pos-Over | 23.2 | 73.2 | +50.0 | 0.0 | 23.2 | +23.2 |
Key Findings¶
- The Precision-Recall Trade-off & Always-Remind Heuristics: Advanced reasoning models (e.g., GPT-5.2 high, Qwen3-235B-Thinking) achieve superior positive proactive recall, but their negative clean accuracy plummets below 45% (GPT-5.2 high achieves only 43.94% on Neg-Clean). In contrast, conservative models like GPT-4o reach 82.70% on Neg-Clean but struggle on Pos-Clean. High-capacity reasoning models overfit to an "always-remind" heuristic, frequently warning about risks that were already resolved.
- Attention Hijacking under Overloaded Triggers: When a concurrent user request is appended to the trigger (Pos-Over), proactive accuracy collapses by 25 to 41 percentage points across models. Standard LLMs prioritize the immediate foreground question and completely overlook the background constraint, mirroring dual-task interference in human cognition.
- The Cognitive Cliff: PM vs. RM Disparity: When evaluated on identical contexts up to 100K tokens, Retrospective Memory (RM Acc) remains near-saturated (\(\ge 98\%\)) for frontier models like GPT-4o. In sharp contrast, spontaneous Prospective Memory (PM Acc) plummets from \(>90\%\) to \(<40\%\). This demonstrates that long-context failures stem not from passive retrieval degradation, but from the inability to autonomously recognize and bridge implicit constraint-trigger dependencies without external prompts.
- PM as a Behavioral Probe of Spare Reasoning Capacity: In joint experiments pairing PM probes with 30 complex AIME-2025 math problems, PM accuracy strongly negatively correlates with reasoning token length (\(r = -0.91\)). Crucially, within identical reasoning length bands (30–50K tokens), trajectories that successfully solved the math problem achieved +9.3% higher PM accuracy than failed ones (61.2% vs. 41.5% across all lengths). This proves PM acts as a sensitive behavioral barometer of remaining cognitive bandwidth that raw token counts obscure.
Highlights & Insights¶
- From Reactive Retrieval to Autonomous Situational Proactivity: The work fundamentally expands long-context evaluation from passive "needle-in-a-haystack" retrieval to active situational awareness, providing the first rigorous, comprehensive benchmark for prospective memory in LLMs.
- Zero-Shortcut Constraint-Trigger Decoupling: By proving that constraints and triggers share virtually zero lexical overlap and mirror background semantic noise, TriggerBench eliminates dense retrieval cheating and reveals why conventional RAG architectures fail on proactive tasks.
- Empirical Discovery of the Cognitive Cliff: The stark contrast between near-perfect RM retrieval and plunging PM recall definitively isolates the cost of spontaneous self-initiation, demonstrating that larger context windows do not automatically confer proactive intelligence.
- Non-Invasive Diagnostic Probing of Reasoning Budget: Discovering that PM accuracy separates successful from struggling reasoning paths at matched token budgets offers an entirely new tool for measuring test-time compute efficiency and cognitive exhaustion.
Limitations & Future Work¶
- Language and Cultural Diversity: The current benchmark focuses exclusively on English dialogues; extending the scenarios to multilingual and culturally diverse conversational contexts is necessary to investigate how linguistic structures modulate proactive reasoning.
- Embodied and Agentic Tool-Use Environments: TriggerBench tests textual conversational turns; validating prospective memory within embodied physical environments, multi-agent frameworks, and autonomous continuous execution loops represents an important next milestone.
- Personalized Intervention Thresholds: While the benchmark enforces objective, expert-aligned ground truth for critical risks, real-world user preferences for assistant proactivity vary considerably between conservative silence and proactive assistance.
Related Work & Insights¶
- vs. Retrospective Memory Benchmarks (LoCoMo / LongMemEval): Prior benchmarks evaluate passive retrieval of historical facts upon explicit questioning; TriggerBench measures spontaneous, unprompted intervention driven by latent risk awareness.
- vs. Proactive Agent Benchmarks (ProAgentBench / PROEVENT): Existing proactive benchmarks focus primarily on predictive assistance based on GUI logs, workflow traces, or scheduled calendar events; TriggerBench evaluates latent constraint tracking across semantic disconnects and concurrent distractions.
- vs. Delayed Instruction Following (Mittal, 2026): Delayed instruction following relies on explicit, pre-delegated reminder directives ("remind me later"); TriggerBench forces models to infer the necessity, timing, and content of interventions autonomously from unstructured conversational history.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers the formalization and evaluation of prospective memory in LLMs with a rigorous 5-dimensional framework]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-model evaluations, context scaling up to 100K tokens, explicit-implicit ablations, and joint AIME cognitive load probing]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear problem formulation, rigorous empirical methodology, and compelling cognitive psychology grounding]
- Value: ⭐⭐⭐⭐⭐ [Provides a foundational benchmark for proactive AI assistants, autonomous agent safety, and test-time compute behavioral diagnostics]