Skip to content

Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/FKIRSTE/emnlp2026-meeting-delegation
Area: LLM Reasoning
Keywords: Multi-Party Meeting Agent, Situational Awareness, Online Meeting Delegation, Dialogue State Tracking, Predictive Recalibration

TL;DR

Addressing the pervasive "silent abstention" problem in online meeting delegation where LLMs fail to recognize when to speak, this paper proposes CAPA (Collaborative Agent Predictive Architecture) with a perceive-act-recalibrate loop over explicit multi-party state representations, slashing the silence rate from 51.4% to 2.5% and doubling credited idea recovery to 52.2% across 137 AMI meetings.

Background & Motivation

In modern organizational ecosystems and distributed collaboration workflows, ensuring consistent synchronous attendance across cross-functional meetings is increasingly difficult for key stakeholders. Post-hoc transcripts, automated summaries, and recorded sessions offer retrospective overviews, but they structurally prevent absent members from contributing while consensus is actively negotiated and trade-offs are forged. Because commitments and strategic compromises made in the absence of vital domain experts are notoriously expensive and politically difficult to reverse, deploying proactive Large Language Model delegates (proxies) that can authentically represent an absent participant in real-time meetings has become a compelling yet computationally demanding frontier.

However, standard prompt-only LLM delegates exhibit severe behavioral breakdowns when navigating the fluid dynamics of multi-party conversations. In controlled evaluations on the AMI Meeting Corpus, conventional prompt-driven baselines remain completely silent during 51.4% of the absent stakeholder's legitimate speaking opportunities. When they do manage to speak, their behavior falls into two dysfunctional extremes: their intervention cue policy is brittle and surface-bound, firing reliably only upon explicit name-calling while routinely missing implicit handoffs or role-relevant topical opportunities; concurrently, their content policy conflates topical relevance with proposition-level coverage, falsely assuming that because a broad subject was touched upon by another speaker, their own distinct argument has been made and should be suppressed.

These fundamental failure modes originate from an upstream architectural deficit: the total absence of explicit dialogue state tracking. In standard agents, action-relevant variables—such as unresolved decisions, open questions, participant stances, prior proposition coverage, and conversational floor control—remain latent and unobserved in raw transcript histories. Extending the context window does not alleviate this bottleneck, as attention dispersion over long transcripts exacerbates contextual drift rather than resolving multi-party reasoning deficits. Core idea: formulate online meeting delegation as sequential decision-making under partial observability, decoupling floor-taking decisions from surface generation while maintaining an explicit six-field interaction state through a continuous perceive-act-recalibrate loop that transforms silent omissions into transparent, diagnosable proposition selections.

Method

Overall Architecture

CAPA operationalizes online multi-party delegation under a causal replay protocol where the delegate only observes the transcript prefix available at turn \(t\). The architecture coordinates four persistent stores in a centralized Shared Memory partitioned across timescale and origin: Long-term Memory holds the participant profile \(p_{u^*}\) (role, style, expertise); Session Memory stores the pre-extracted briefing \(b_{u^*}\) (candidate talking points); Working Memory holds the dynamic interaction state \(m_t\); and Episodic Memory tracks historical agent actions. At each turn, a single Perceiver step consumes the newly observed turn \(o_t\) to incrementally update \(m_t\). A two-tier action policy then operates: a strategic Controller evaluates helper scores to decide between SILENT and SPEAK, committing to a discrete semantic proposition \(z_t\) before a style-faithful Generator renders surface utterance \(y_t\). Simultaneously, a Predictor forecasts the intent and floor effect of the next turn. Once future turns unfold, Environment and Delegate Judges compare predictions and actions against reality, enabling the Recalibrator to patch state misalignments and adjust future intervention pressure.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: causal dialogue turn stream o_t"] --> B["1. Centralized Shared Memory & State Tracking<br/>4-quadrant stores + explicit 6-field state m_t"]
    B --> C["2. Decoupled Two-Tier Action Strategy<br/>3-helper gate (Silent/Speak) + pre-generation z_t"]
    C -->|Trigger SPEAK| D["Stylized Linguistic Generator<br/>enforce persona voice & render y_t"]
    C -->|Concurrent intent forecast| E["3. Intent-Level Future Turn Predictor<br/>forecast next speaker, intent, & floor effect"]
    D --> F["Causally observed continuation window"]
    E --> F
    F --> G["4. Dual-Judge State Recalibration<br/>isolate perception vs action errors & patch m_t"]
    G -->|State updates & guidance summary| B

Key Designs

1. Centralized Shared Memory & State Tracking: Overcoming Contextual Drift from Latent Interaction Variables Relying on raw transcripts forces LLMs to re-infer social dynamics from scratch at every turn, inevitably inducing contextual drift. CAPA partitions Shared Memory along two orthogonal axes—timescale (static across the meeting vs. dynamic turn-by-turn) and information source (agent-introduced vs. environment-generated)—into four schema-constrained repositories (\(p_{u^*}, b_{u^*}, m_t\), and action traces). In each decision step, the Perceiver maps the newly observed utterance \(o_t\) and prior state \(m_{t-1}\) into an updated state \(m_t\) tracking six explicit fields: macro/meso/micro topic hierarchy, agreed decisions, open questions, participant stances, proposition coverage, and conversational floor control. By maintaining these variables as explicit, queryable fields rather than latent text representations, downstream decision components obtain a persistent, inspectable interaction substrate.

2. Decoupled Two-Tier Action Strategy: Eliminating Silent Abstention and Generation Hallucination Monolithic agents that generate text directly from prompts conflate tactical floor-taking (whether to speak, when, and on what topic) with natural language rendering, leaving no structural point where abstention can be exercised as a deliberate choice on a specific claim. CAPA splits the action policy into a strategic Controller and a linguistic Generator. The Controller consults three specialized helpers: the Candidate Curator ranks briefing points by micro/meso topic fit; the Coverage Evaluator tags whether points are Fresh, Partial, Reinforce, or Skip relative to prior dialogue; and the Speaker Scorer evaluates turn-taking cooldowns and topical suitability. When a viable, uncovered claim exists and floor dynamics permit, the Controller selects SPEAK and commits to a discrete semantic proposition \(z_t\). The Generator then functions as a pure linguistic stylist, translating \(z_t\) into natural speech matching the participant's verbosity, formality, and rhetorical patterns without inventing unverified facts.

3. Intent-Level Future Turn Predictor: Anticipating Conversational Flow to Cure Timing Lag Appropriate floor-taking in multi-party meetings requires anticipating conversational trajectory rather than reacting post-hoc. In parallel with the Controller's decision, CAPA invokes the Predictor. The Predictor consumes the current state snapshot and recent turns to produce an intent-level forecast of the upcoming turn, specifying the predicted speaker ID, a 5-10 word intent summary (e.g., Question, Propose, Agree, Inform), the target micro-topic, and the anticipated floor effect (such as TAKES, YIELDS, or DIRECTED_TO). This forecast is withheld from the current action policy and cached strictly for post-action diagnostics, establishing an objective counterfactual baseline to measure whether the agent accurately comprehended conversational momentum.

4. Dual-Judge State Recalibration: Disentangling Perception from Action Errors When an agent fails to intervene or produces an uncredited utterance, single post-hoc critique mechanisms cannot discern whether the failure stemmed from a misread meeting state or poor candidate selection. CAPA introduces a dual-judge framework evaluated over a causally observed window \(t+1 \dots t+k\): the Environment Judge compares the Predictor's forecast against the ground truth to isolate perception errors, while the Delegate Judge scores the agent's actual intervention against the same window to isolate action and content errors. The Recalibrator fuses both verdicts, updating \(m_t\) with missing key points or resolved questions, and outputs an operational guidance object containing contribution pressure dials (dial-up to encourage participation, dial-down during cooldowns) to guide subsequent turns.

Benchmark & Evaluation Protocol

To overcome the limitations of surface n-gram overlap metrics (ROUGE/BLEU) that ignore conversational timing, the authors formulate an episode-level causal replay protocol anchored on participant-owned "Idea Units"—substantive, proposition-level claims, decisions, and technical inputs introduced by the target stakeholder. - Data & Splitting: Evaluated on 137 full remote-control design scenario meetings from the AMI Meeting Corpus (4 participants per meeting: PM, ID, UI, ME). The initial 60% of each meeting is reserved to build the participant profile and forward-looking briefing, while the terminal 40% serves as the evaluation testbed. - Cues, Anchors, and Windows: Each idea unit defines an episode initiated at a cue turn (information boundary) and anchored at the exact ground-truth turn where the human spoke. The agent operates under causal replay over a \(k=5\) turn post-cue window. - Categorical Outcomes: 1. Strict hit: credited recovery at the exact anchor turn (\(\Delta = 0\)); 2. Loose hit: credited recovery within the \(k\)-turn window (\(0 \le \Delta \le k\)); 3. Uncredited attempt: agent spoke, but utterance lacked essential qualifiers, picked a different proposition, arrived late, or triggered safety filters; 4. Not-attempted: delegate remained entirely silent throughout the opportunity window. - Human Calibration: A stratified sample of 150 idea units and 150 match decisions validated that the schema-constrained LLM judges achieve substantial alignment with human consensus (Cohen's \(\kappa = 0.71\)).

Key Experimental Results

Main Results

Across all 137 AMI meetings (comprising 1,126 idea-unit episodes), CAPA was evaluated against a transcript-only delegate (supplied with a 50-turn raw context window) and a Reflexion-style procedural memory baseline. All metrics represent macro-averaged percentages across meetings with 95% bootstrap confidence intervals (\(B=10,000\)):

System Configuration Strict Recall (%) ↑ Loose Recall (%) ↑ Uncredited Attempt (%) ↓ Not Attempted (%) ↓ Decision F1 (%) ↑ Hallucination (%) ↓
Transcript-only (50-turn) 10.7 [8.7, 12.8] 26.1 [22.1, 28.2] 22.5 [19.7, 25.3] 51.4 [47.8, 55.2] 38.1 1.5
Reflexion Baseline 10.3 24.3 26.1 49.5 47.6 0.3
CAPA (Ours) 25.1 [22.1, 28.2] 52.2 [48.9, 55.6] 45.3 [41.7, 48.9] 2.5 [1.6, 3.7] 63.0 0.6

Ablation Study

On a controlled 20-meeting subset sharing identical episode rosters and briefings, the explicit meeting state \(m_t\) and the post-action Recalibrator were independently ablated:

Experimental Variant Loose Recall (%) ↑ Strict Recall (%) ↑ Decision F1 (%) ↑ Not Attempted (%) ↓ Redundancy (%) ↓ Off-Topic Rate (%) ↓
CAPA Full Architecture 50.5 [40.7, 59.7] 23.0 [15.4, 30.7] 61.1 [56.9, 65.1] 2.1 [0.0, 5.9] 0.0 [0.0, 0.0] 5.2 [2.3, 8.5]
State-Ablated (50-turn raw window) 28.1 [20.1, 36.8] 8.5 [4.1, 13.4] 36.4 [33.2, 39.9] 10.9 [5.9, 16.2] 10.7 [6.7, 15.2] -
Mechanism Delta (\(\Delta\) State) +22.4 +14.5 +24.7 -8.8 -10.7 -
Without Recalibration (State kept) 49.8 [41.7, 57.2] 26.0 [19.1, 32.8] 66.7 [62.3, 71.0] 0.0 [0.0, 0.0] 0.0 7.2 [3.6, 11.4]
With Recalibration (CAPA matched) 53.7 [44.1, 63.6] 28.0 [20.7, 36.0] 66.2 [61.9, 70.4] 0.0 [0.0, 0.0] 0.0 5.2 [2.3, 8.5]
Mechanism Delta (\(\Delta\) Recalib) +3.9 +2.0 -0.5 0.0 0.0 -2.0

Key Findings

  • Eliminating the Silent Abstention Bottleneck: Maintaining an explicit meeting state completely resolves the prompt-only failure mode, reducing silence from 51.4% to 2.5% and doubling credited loose recall (26.1% to 52.2%). Expanding the context window to 50 turns without explicit state fails to close this gap, proving that raw context scaling is fundamentally insufficient for multi-party interaction tracking.
  • Shift from Omission to Traceable Selection: Rather than failing invisibly through silence, CAPA redistributes unsuccessful outcomes into diagnosable "uncredited attempts" (45.3%). Error taxonomy reveals these attempts comprise: incomplete matches (40.5%, Generator realization deficit), different valid propositions (26.6%, Curator selection imprecision), safety/quality flags (19.4%), and late matches (13.5%), enabling localized module refinement.
  • Pre-Anchor Intervention Synchrony: In 93.8% of CAPA's 595 credited interventions, the agent took the floor synchronously with or prior to the human anchor (median lead time of 1.0 turn). This demonstrates that tracking open questions and speaker fit enables the agent to actively seize opportunities during conversational openings rather than passively waiting for turn handoffs.
  • Cross-Corpus Robustness on ICSI: Replaying 10 unstructured ICSI research meetings (with 6 to 10 participants) under identical protocols yielded near-zero silence (1.2%), higher loose recall (73.6%), robust Decision F1 (64.8%), and minimal hallucination (1.8%), validating that the state tracking mechanism generalizes beyond scripted design scenarios.

Highlights & Insights

  • Floor-Taking as an Independent First-Class Decision: By strictly separating the strategic decision of whether to speak from surface lexical realization, the system prevents hallucinations caused when open-ended generation dictates underlying reasoning.
  • State-Directed vs. Text-Directed Correction: Traditional self-refinement loops (e.g., Reflexion) operate strictly on generated text, making them completely blind to silent omission failures where no utterance was produced. CAPA feeds dual-judge critiques back into the latent interaction state, adjusting participation pressure dynamically.
  • Modular Diagnostic Observability: Replacing a black-box prompt with structured perception, curation, and gating modules allows every conversational near-miss to be pinned directly to a specific submodule (Curator, Scorer, or Generator).

Limitations & Future Work

  • High Critical-Path Inference Latency: The floor-taking decision path requires multiple LLM calls (Controller and three helpers), incurring an average provider latency of 21.14 seconds (wall-clock) with GPT-4o. In fast-paced human speech, this delay would cause awkward interruptions, underscoring the urgent need for speculative decoding or small distilled policy models.
  • Dependence on Pre-Meeting Briefings: The agent's substantive interventions are grounded primarily in pre-extracted talking points (\(b_{u^*}\)), limiting its ability to formulate spontaneous, unprompted synthesis when meeting topics diverge sharply into unanticipated technical domains.
  • Lack of Multimodal Prosodic Cues: Operating strictly on text transcripts discards essential acoustic and visual turn-taking signals such as speech pitch, gaze shifts, hesitation pauses, and backchannel vocalizations.
  • vs Meeting Delegate (Hu et al., 2025): Hu et al. established the benchmark exposing prompt-only failure modes (silence, redundancy, mistiming); CAPA provides the first systematic architectural solution through explicit state tracking and POMDP-inspired modularity.
  • vs Reflexion (Shinn et al., 2023): Reflexion relies on critiquing and rewriting drafted text outputs; CAPA demonstrates that in multi-party delegation the predominant error is silent omission, requiring feedback to be routed into the continuous meeting state rather than post-hoc text rewriting.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First end-to-end predictive architecture with explicit situational state tracking for proactive multi-party LLM delegation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive validation on 137 AMI meetings, 10 ICSI meetings, human judge calibration (\(\kappa=0.71\)), and fine-grained error taxonomy.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptional clarity, mathematically and linguistically sound problem formulation, and highly detailed qualitative/ablation breakdowns.
  • Value: ⭐⭐⭐⭐⭐ Establishes a transformative blueprint for autonomous, state-aware agent representation in real-time collaborative enterprise environments.