Skip to content

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/BUPT-GAMMA/ATRBench
Area: LLM Reasoning
Keywords: Long-lived LLM agents, Proactivity gap, Ask-to-Remember (ATR), Cross-session preference acquisition, ATRBench

TL;DR

This paper formalizes the Proactivity Gap in long-lived personal assistant agents and presents the Ask-to-Remember (ATR) paradigm along with ATRBench, a controlled benchmark decoupling acquisition from application across 8 frontier LLMs, revealing that prompt-based interventions recover at most 15.5% of the 62.1–76.6 point gap to an oracle.

Background & Motivation

As large language model (LLM) agents are increasingly deployed as long-lived personal assistants serving the same user across multiple sessions, their true utility shifts from merely executing isolated tasks to incrementally accumulating latent preferences, standing constraints, and operational guidelines over time. Existing evaluations of personalization (such as LaMP, PrefEval, and LifeSim-Eval) typically operate under the passive assumption that user profiles, past interaction traces, or persistent memories are already available in context. Similarly, clarification and planning benchmarks restrict agent inquiries strictly to the immediate objective at hand. Under these paradigms, an assistant preserves only what the user explicitly volunteers, remaining fundamentally blind to unvoiced preferences that govern future autonomous actions.

This structural blind spot becomes critical when the user delegates recurring responsibilities (e.g., travel rescheduling, calendar management, email triage, and purchasing) and goes offline. In the user's absence, an assistant executing literal instructions without awareness of latent constraints will frequently take actions that violate unstated preferences. Evaluating proactive elicitation for future utility—termed Ask-to-Remember (ATR)—is notoriously difficult due to two fundamental barriers: the target question is severely underdetermined because latent preferences are hidden and unrelated to the current task, and the payoff is delayed and uncertain because its value materializes only if an applicable future task arises. Consequently, raw interaction histories hopelessly entangle the elicitation policy with subsequent memory retention and tool-execution mechanics.

To resolve this evaluation dilemma, this paper introduces a controlled decoupling protocol: the user's standing preferences are fixed as evaluator-known ground truth while remaining strictly hidden from the agent during interaction, and each preference is tied directly to a subsequent offline test task where correct execution strictly requires adherence to that rule. Core idea: formalize proactive cross-session preference elicitation as an Ask-to-Remember (ATR) task and build ATRBench, a controlled benchmark that decouples earlier asking from downstream execution under a fixed memory substrate to measure and diagnose the proactivity gap in frontier LLM agents.

Method

Overall Architecture

ATRBench operationalizes the ATR formulation through a two-phase decoupled protocol separating user-online learning sessions (\(L_u\)) from user-offline test sessions (\(T_u\)). During the learning phase, the agent interacts with a fixed LLM user simulator to resolve ordinary everyday tasks while deciding whether to proactively ask reusable standing-rule questions. All user-facing communications are intercepted by a frozen side-channel Router–Classifier scaffold: the Router identifies whether an utterance constitutes a strict standing-rule inquiry, and the Classifier determines whether it matches an active rule in the persona's hidden pool. Valid inquiries trigger standard first-person answers, while non-matching questions receive neutral deflections. After all learning sessions conclude, the accumulated dialogue transcript is frozen into a cross-session context block. In the test phase, the user is completely offline, and the agent must execute rule-bound tasks where its initial tool calls are deterministically matched against ground-truth behavioral specifications.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Persona Profiles & Environment Ontology<br/>20 Personas · 6 Domains · 74 Tools"] --> B["Counter-Default Grounded Standing Rule Generation<br/>496 Candidates → 2-Axis QC → 340 Retained Rules"]
    B --> C["Rule-Bound Test Session Construction via Bound Verification<br/>Upper/Lower-bound 2-of-3 Majority Verification"]
    B --> D["Rule-Blind Learning Session Generation & Trajectory Alignment<br/>Skeleton Sampling + Fill + Action Surface Overlap"]
    D --> E["Two-Phase Interactive Protocol with Router-Classifier Scaffold<br/>Router Binary & Span Extraction + Classifier Matching"]
    E --> F["Frozen Cross-Session Substrate & Decoupled Diagnostics<br/>RuleAsk / RuleCov / AcqPrec / AppliedRate / TSAcc"]

Key Designs

1. Counter-Default Grounded Standing Rule Generation: Anchoring Elicitation Targets Against Default Bias

To prevent agents from exploiting generic moral or common-sense heuristics to succeed by chance, ATRBench synthesizes standing rules \(r \in \mathcal{R}_u\) conditioned on synthetic profiles from Nemotron-Personas-USA and an executable 74-tool ontology across 6 domains. Crucially, each rule specifies not only a canonical first-person answer \(a_r\) and a tool action requirement \(\rho_r\), but also a "counterfactual default"—the natural action a helpful, rule-blind agent would typically take (e.g., standard agents default to archiving promotional emails, whereas the user rule requires permanent deletion; standard agents accept invitations, while the user avoids crowded venues). Candidate rules undergo two independent LLM-judge verification axes: counter_default verifies that a rule-naive agent would select an alternative action, and binding_sound confirms that the rule maps cleanly onto tool parameters across four action surfaces (param_id, param_enum, tool_identity, and confirm). From 496 candidates across 20 personas, 340 rules (68.5%) pass both axes, establishing a rigorously vetted ground-truth pool that strictly requires proactive acquisition.

2. Rule-Bound Test Session Construction via Bound Verification: Converting Delayed Utility into Deterministic Offline Audits

To eliminate scoring ambiguity in delayed downstream payoffs, the benchmark generates offline test sessions \(t \in \mathcal{T}_u\) working backward from the required tool-level behavior of each retained rule. To ensure discrimination, each candidate test session is subjected to paired upper-bound and lower-bound verification across three independent simulated trials: the upper bound injects the canonical rule answer directly, while the lower bound runs a rule-blind agent without rule access. A test session is accepted only if at least two of three upper-bound trials reach the gold action and at least two of three lower-bound trials fail into the counterfactual decoy. Failed sessions undergo up to two iterative refinement passes before being discarded. This yields 284 high-discrimination rule–test pairs (a 1-to-1 mapping), evaluated programmatically by checking the tool identity and comparison parameters on the agent's very first relevant action batch, completely bypassing noisy LLM evaluators.

3. Rule-Blind Learning Session Generation & Trajectory Alignment: Natural Interaction without Verbal Leakage

To prevent the benchmark from collapsing into superficial reading comprehension, learning sessions \(\mathcal{L}_u\) are generated entirely from persona themes and domain tool specifications without conditioning on the target standing rules. Each filled session is validated via dry-run execution against a live environment instance. The orchestrator then greedily selects \(2|\mathcal{T}_u|\) sessions per persona, prioritizing tasks whose oracle tool trajectories touch the decision surfaces of the retained test rules. This indirect alignment ensures that 72.2% (205/284) of target standing rules have at least one learning session where the agent encounters an opportunity to inquire about that specific action surface. Human audits confirmed 0% verbal leakage of target preferences in session prompts, creating authentic opportunities for proactive elicitation while guaranteeing that success demands active asking.

4. Two-Phase Interactive Protocol with Router-Classifier Scaffold: Non-Intrusive Speech-Act Interception

During learning sessions, the agent communicates with the user simulator via a dedicated control-plane tool send_to_user(output, reason). A frozen two-stage scaffold intercepts every utterance in real time: a Gemini 3 Flash Preview Router performs binary detection to verify if the text is a strict standing-rule inquiry and extracts the verbatim question span \(q_t\), after which a GPT-5.4 Classifier matches \(q_t\) against the persona's rule pool. Upon a confirmed match, the orchestrator injects a transient token into the user simulator, which is dynamically replaced by the canonical rule answer \(a_r\); unconfirmed inquiries receive a standardized deflection ("no strong preference there – your call"). The simulator never sees the master rule list, eliminating leakage risks. Extensive manual verification on held-out trajectories demonstrated 100.0% binary precision for the Router and 90.0% hit precision for the Classifier.

5. Frozen Cross-Session Substrate & Decoupled Diagnostics: Isolating Asking from Memory and Application Mechanics

To ensure performance differences reflect elicitation behavior rather than disparities in proprietary memory retrieval algorithms, ATRBench freezes the verbatim transcript across all learning sessions into a clean <context> block shared across all offline test sessions without dynamic write-backs. The benchmark defines an end-to-end primary metric \(\text{TSAcc}\) (offline test session success rate) alongside an oracle gap \(\text{Gap} = \text{TSAcc}_{\text{orc}} - \text{TSAcc}_{\text{def}}\) and gap recovery rate \(\text{GapRec}\). Crucially, it decomposes failure via fine-grained learning diagnostics: asking volume \(\text{RuleAsk}\) (strict rule inquiries per session), coverage breadth \(\text{RuleCov}\) (fraction of hidden rules hit), asking precision \(\text{AcqPrec}\) (fraction of inquiries targeting true rules), and downstream utilization \(\text{AppliedRate}\) (test accuracy on acquired, learnable rules).

Benchmark & Evaluation Protocol

The benchmark evaluates four controlled diagnostic variants across a 20-persona cohort: - default: Baseline agent operating with zero rule-asking instructions, measuring spontaneous initiative; - atr: Agent provided with generic guidance defining standing rules and interaction trade-offs, without disclosing specific rules; - always_ask: Agent instructed to emit exactly one standing-rule inquiry after completing each learning task; - oracle (oracle_target): Agent directly supplied with the test-bound canonical rule answer at test time, establishing the executable upper bound.

Evaluations span eight frontier LLM models: GPT-5.4, Claude Opus 4.7, Gemini 3 Flash Preview, Gemini 3.1 Pro Preview, Qwen3.6-Plus, MiniMax M2.7, DeepSeek V4 Pro, and DeepSeek V4 Flash. All models run with native tool calling and high reasoning effort / thinking modes enabled, executing across 640 experimental cells comprising 13,632 learning sessions and 9,088 test sessions.

Key Experimental Results

Main Results

The primary scorecard across the 20-persona macro-average is presented below (quoted directly from Table 3 of the paper):

Model TSAcc (%) Def. TSAcc (%) ATR TSAcc (%) Always TSAcc (%) Oracle RuleAsk ATR RuleAsk Always RuleCov (%) ATR RuleCov (%) Always AcqPrec (%) ATR AcqPrec (%) Always AppliedRate (%) ATR AppliedRate (%) Always Oracle Gap (pts)
GPT-5.4 15.0 13.3 24.9 88.6 0.01 1.00 0.4 12.7 5.0 9.4 0.0 80.9 73.7
Claude Opus 4.7 22.2 28.7 32.1 92.5 0.14 1.01 7.1 15.6 22.4 8.9 97.7 83.2 70.3
Gemini 3 Flash Preview 15.6 26.6 25.8 91.0 0.51 1.01 12.0 14.4 13.1 8.7 78.4 70.0 75.4
Gemini 3.1 Pro Preview 20.1 32.0 26.5 96.7 0.80 1.00 13.3 11.3 10.1 6.0 88.9 87.2 76.6
Qwen3.6-Plus 20.3 19.4 25.3 93.0 0.01 0.96 0.7 10.3 10.0 8.1 100.0 73.5 72.7
MiniMax M2.7 20.4 21.3 18.2 82.5 0.00 0.09 0.0 1.8 — 4.2 — 25.0 62.1
DeepSeek V4 Pro 20.0 21.2 23.9 89.5 0.01 0.98 0.6 14.5 5.0 8.6 50.0 70.6 69.5
DeepSeek V4 Flash 23.7 23.2 26.5 93.0 0.05 0.97 2.1 13.3 16.7 10.4 100.0 71.4 69.3

Diagnostic Breakdown and Domain Analysis

Cohort-level acquisition summary across variants and domain-specific execution accuracies (quoted from Table 4 and Table 17 of the paper):

Variant Avg. RuleAsk (per LS) RuleCov (%) AppliedRate (%) Gap Recovery (%) Active Models (Ask \(\ge\) 0.5)
ATR 0.19 4.5 73.6 4.8 2/8
Always Ask 0.88 11.7 70.2 7.8 7/8

Capability TSAcc summaries across domains and action surfaces (TS-micro, Table 17):

Domain / Action Surface (Rules) Default ATR Always Ask Oracle Oracle Gap (pts)
Commerce (53) 8.0 9.9 11.1 89.4 81.4
Reservation (56) 13.2 19.0 17.0 95.5 82.3
Travel (74) 15.2 19.1 21.3 89.9 74.7
Scheduling (40) 15.0 17.5 20.9 85.6 70.6
Workspace (33) 44.3 48.5 51.9 90.9 46.6
Communication (28) 48.2 51.8 60.7 91.1 42.9
param_id (124) 17.1 21.4 21.6 92.9 75.8
param_enum (99) 18.3 21.6 24.4 90.5 72.2
tool_identity (59) 29.9 33.1 38.1 88.1 58.2

Key Findings

  • Substantial Proactivity Gap: Rule-blind default agents achieve only 15.0%–23.7% accuracy on future tasks, while oracle agents reach 82.5%–96.7%. The massive 62.1–76.6 point gap demonstrates that the tasks are fully executable when preferences are known; the fundamental failure lies in the agent's inability to acquire them beforehand.
  • Prompting Barely Triggers Elicitation: Standard ATR prompting fails to activate proactive questioning in six out of eight models (\(\text{RuleAsk} \le 0.14\)). Only the Gemini models exhibit spontaneous asking behavior. Non-oracle prompting recovers at most 15.5% of the oracle gap.
  • Forced Asking Exposes the "What to Ask" Bottleneck: In the always_ask setting, asking volume rises to nearly 1.0 inquiry per session across seven models, yet asking precision (\(\text{AcqPrec}\)) plummets to 6.0%–10.4%, leaving rule coverage below 16%. When timing is fixed, determining what to ask remains the core limiting factor.
  • Indiscriminate Elicitation Degrades Application Accuracy: For the four models demonstrating non-trivial spontaneous acquisition, forcing inquiries via always_ask reduces \(\text{AppliedRate}\) by 1.7 to 28.6 points (e.g., Gemini 3 Flash drops from 78.4% to 70.0%), indicating that cluttering context with uncalibrated inquiries creates cognitive interference during downstream execution.

Highlights & Insights

  • Formalization of Ask-to-Remember (ATR): Isolating proactive preference acquisition from passive recall and immediate clarification grounds long-term personalization into a concrete, observable interaction paradigm with delayed utility.
  • Counterfactual Default and Bound Verification: Generating rules with counterfactual defaults and enforcing a two-of-three majority filter across upper and lower bounds prevents models from succeeding via common-sense bias, providing an unyielding benchmark standard.
  • Shifting the Personalization Frontier from Storage to Exploration: The high baseline application rate (70%–100% on acquired rules) proves that modern LLMs possess adequate memory retention and tool compliance; the critical research gap lies in active exploration policies, suggesting that techniques like reinforcement learning for exploratory dialog or Bayesian active elicitation are vital next steps.

Limitations & Future Work

  • Synthetic Personas and Static Rules: Ground-truth preferences are derived from synthetic profiles and modeled as static, non-conflicting rules. Real human preferences are dynamic, context-dependent, and prone to shifts over time.
  • Standardized Simulator Feedback: Canonical answers are injected via a uniform deterministic hook, bypassing natural human dialogue nuances such as ambiguous phrasing, reluctance to disclose, or spontaneous voluntary revelations.
  • Omission of Interaction Fatigue Penalties: While the conceptual framework models interaction cost, the evaluation does not dynamically penalize agents for annoying or repetitive inquiries through simulator frustration modeling.
  • vs PrefEval / MPT / PersonalAlign: Prior benchmarks evaluate passive memory retrieval or tool execution given pre-populated profiles or past interaction histories; ATRBench requires agents to actively discover and elicit missing preferences from scratch.
  • vs Ask-before-Plan / GATE / PrefDisco: Existing elicitation benchmarks focus on clarifying ambiguous instructions or eliciting preferences to complete the current objective; ATRBench studies inquiries whose payoff is exclusively deferred to separate future sessions.
  • vs Proactive Agent / Pare-Bench / \(\pi\)-Bench: These frameworks evaluate unprompted actions or workflow suggestions ("act"); ATRBench focuses on explicit informational inquiries ("ask"), which entail communication trade-offs and provide verifiable information gain.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes a groundbreaking paradigm for proactive preference acquisition in long-lived agents.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 640-cell evaluation covering 8 frontier models with rigorous bound verification and audit protocols.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical formulation, modular architectural prose, and transparent error reporting.
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable diagnostic testbed for shifting agent personalization from passive memorization to active inquiry.