WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Agent
Keywords: Embodied Agents, Long-Term Memory, State Tracking, Partial Observability, Task Planning
TL;DR¶
Introduces WorldLines, the first benchmark evaluating long-horizon stateful embodied household assistance under partial observability, and proposes ObsMem, an observer-grounded memory framework that separates event evidence, structured state trails, epistemic beliefs, and commitments to prevent state-overwriting hallucinations and enable executable embodied planning.
Background & Motivation¶
To operate reliably over extended periods in real-world human homes, embodied agents require far more than conversational recall of past dialogue; they must maintain an accurate, stateful representation of a constantly evolving physical environment. Genuine assistive requests naturally unfold over multiple days and are deeply entangled with recurring human routines, shifting object placements, smart-device configurations, and unexpected disruptions. For instance, a user might establish a weekend coffee-brewing routine on Friday, accidentally alter the machine's timer on Sunday night due to exhaustion, and verbally request breakfast preparation on Monday morning. To execute such tasks safely and correctly, an assistive robot must seamlessly bridge current language commands with historical schedules, user preferences, physical environment evolutions, and previous execution feedback.
However, existing evaluation benchmarks bifurcate this foundational challenge into two incomplete, mutually exclusive paradigms. On one side, language-centric long-term memory benchmarks—such as LoCoMo, LongMemEval, and RealMem—evaluate cross-session retrieval, temporal reasoning, and memory updating, yet they treat memory purely as text-based question answering completely decoupled from physical state transitions, robotic action feedback, and physical affordance constraints. On the other side, standard embodied AI benchmarks—including ALFWorld, Habitat 3.0, PARTNR, and BEHAVIOR-1K—advance embodied navigation, object rearrangement, and multi-agent coordination, but remain bounded within brief, isolated task episodes. In these environments, world states are reset upon episode completion, preventing evaluation of whether an agent can accumulate persistent state over days and leverage it for downstream decisions.
Furthermore, domestic environments inherently exhibit partial observability: household residents frequently relocate objects outside the robot's field of view, container lids and appliances change states unobserved, and reported verbal statements may conflict with physical reality. Mainstream flat-text agent memory systems (e.g., Mem0, A-mem, MemoryOS) fail catastrophically in this setting because they treat all interactions as undifferentiated text snippets, incapable of distinguishing direct visual observations from second-hand reported dialogue, or tracking sequential state-overwriting events. When queried about current object locations, they frequently retrieve obsolete past states or confuse recurring habits with one-off anomalous updates. Core idea: Redefine embodied memory not as flat text retrieval, but as an infrastructure for persistent world-state maintenance and causal evidence tracking under partial observability, establishing the project-driven WorldLines benchmark and the multi-track ObsMem observer-grounded architecture.
Method¶
Overall Architecture¶
WorldLines and ObsMem form a comprehensive closed-loop framework for stateful embodied intelligence. On the benchmarking side, WorldLines builds upon realistic Habitat/HSSD scenes and executes a four-stage project-driven synthesis pipeline: Project Candidate Ideation \(\rightarrow\) Multi-Day Beat Planning \(\rightarrow\) Session Intent Scheduling \(\rightarrow\) Closed-Loop Director–Actor–Executor Simulation. This pipeline synthesizes grounded multi-day household traces featuring multi-modal dialogue, physical human/robot actions, smart-device state transitions, and execution feedback. By programmatically determining context cutoffs, WorldLines produces evaluation samples paired with explicit evidence chains for two complementary tasks: Memory QA and Embodied Task Planning, strictly precluding future information leakage.
On the architectural side, ObsMem addresses partial observability and mutable world dynamics through three strictly decoupled processing stages. Stage 1 executes observer-grounded trace ingestion, filtering out all events outside the robot's perception window while converting visible dialogue into reported atoms. Stage 2 builds a four-track typed memory state containing immutable event evidence \(E_t\), structured mutable world-state trails \(S_t\), epistemic belief records \(B_t\), and forward-looking commitments \(K_t\), periodically executing progressive episode consolidation to reconcile conflicts. Stage 3 performs typed retrieval and grounded reasoning: incoming questions are transformed into explicit query plans that route retrieval to the corresponding memory views, followed by an LLM-based evidence selector that isolates a compact, complementary evidence subset to guide state-aware QA and precondition-compliant physical action planning.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Cross-Day Household Interaction Stream<br/>(Dialogue / Human Actions / Device Ops / Feedback)"] --> D1["Observer Gating & Multi-Track Typed Ingestion<br/>Filter visibility & decouple events Et / states St / beliefs Bt / commitments Kt"]
D1 --> D2["Epistemic Belief Decay & Conflict Resolution<br/>Track provenance & intervening events, marking fresh / stale / uncertain / contradicted"]
D2 --> D3["Intent-Driven Typed Routing & Evidence Selection<br/>Predict query plan, route to specialized views, select complementary Top-5 evidence"]
D3 --> D4["Precondition-Constrained Embodied Task Planning<br/>Ground actions in persistent world state St & beliefs Bt to enforce valid execution"]
D4 --> OUT["State-Aware Embodied Action & Question Answering<br/>(Memory QA / Embodied Task Planning)"]
Key Designs¶
1. Observer Gating and Multi-Track Typed Ingestion: Decoupling Immutable Event History from Mutable World States
Conventional agent memory architectures store all interaction logs indiscriminately in a flat vector database, creating a severe ontological flaw in embodied scenarios: past occurrences are permanent historical facts, whereas spatial locations of movable items or appliance power states are mutable physical properties subject to frequent overwriting. Conflating these two concepts causes agents to confuse what happened at a historical moment with what is currently true in the physical world. ObsMem addresses this by formalizing an observer gating function: for any physical event \(o_t = (x_t, \tau_t, \ell_t, A_t, V_t)\) occurring at time \(\tau_t\) at location \(\ell_t\) with actors \(A_t\) and observers \(V_t\), robot \(r\) ingests direct physical observations only when \(r \in V_t\), while visible utterances by other actors are ingested as reported atoms: $\(\Phi(o_t) = \begin{cases} \{e_{\text{obs}}^t\}, & r \in V_t \\ \emptyset, & r \notin V_t \end{cases} \;\cup\; \mathbf{1}[utt(o_t) \land actor(o_t) \neq r] \{e_{\text{rep}}^t\}\)$
Subsequently, ObsMem partitions memory into four dedicated typed tracks \(M_t = (E_t, S_t, B_t, K_t)\). The event track \(E_t\) is an append-only ledger preserving historical action sequences. The state track \(S_t\) specifically handles physical property diffs: whenever an executable action alters the environment, a structured update tuple \(\Delta_t = (i, a, v_{\text{old}}, v_{\text{new}}, \tau_t, s_t)\) is logged, recording that attribute \(a\) of entity \(i\) transitioned from \(v_{\text{old}}\) to \(v_{\text{new}}\) at time \(\tau_t\) via source \(s_t\). This track maintains both the latest environment snapshot and the complete historical state evolution chain, resolving the fundamental failure of text-matching models when objects are repeatedly relocated across rooms.
2. Epistemic Belief Decay and Conflict Resolution: Explicit Uncertainty Modeling under Partial Observability
In partially observable home environments, an agent cannot naively assume that an observed state remains true indefinitely; once the robot leaves a room, human residents may move items or modify appliance settings without direct robotic oversight. Flat memory systems lack epistemic representation and inevitably hallucinate certainty over obsolete facts. ObsMem introduces an epistemic belief track \(B_t\) that assigns a dynamically evolving confidence state \(z_t(i, a)\) to each tracked entity attribute. Direct observations initialize beliefs as fresh with high confidence. As unobserved intervening events \(I_t\) and other actors \(A_t\) accumulate over time, confidence decays exponentially according to: $\(c_t = c_0 \cdot \exp(-\alpha n) \cdot (1 - \beta I_{\text{conflict}})\)$ where \(n\) represents the number of intervening events. Based on calibrated thresholds, belief states transition across four discrete epistemic categories: $\(z_t(i, a) = \begin{cases} \text{contradicted}, & |C_t| > 0 \\ \text{uncertain}, & |I_t| \ge \lambda_I \lor (|I_t| \ge \lambda_m \land |A_t| \ge \lambda_A) \\ \text{stale}, & |I_t| > 0 \lor \rho_t \neq \text{observed} \\ \text{fresh}, & \text{otherwise} \end{cases}\)$ At each episode boundary, ObsMem activates a Progressive Episode Consolidation module that synthesizes raw event buffers into structured episode cards, extracts high-level relational facts and future commitments \(K_t\), and resolves cross-session state contradictions. This grants the robot meta-cognitive awareness of its own perceptual blind spots, allowing it to appropriately express uncertainty rather than acting on stale assumptions.
3. Intent-Driven Typed Routing and Evidence Selection: Bridging the Coarse-Session to Fine-Event Retrieval Gap
Extensive empirical probing reveals a critical retrieval bottleneck in agent memory: existing systems frequently achieve high session-level hit rates (Session Any@5) but abysmal event-level evidence recall (Event R@5). When searching through voluminous household logs, naive dense retrieval gravitates toward recurring habits or lexical surface similarities while missing the single event where a critical state change occurred. ObsMem eliminates flat nearest-neighbor search in favor of an intent-driven query planner and an LLM-based evidence selector. Given a query \(q\), the planner first predicts an explicit retrieval plan: $\(p_q = (\eta_q, I_q, A_q, T_q)\)$ specifying the intent \(\eta_q\) (current state, past event, epistemic status, commitment schedule, or planning goal), target entities \(I_q\), target attributes \(A_q\), and temporal boundaries \(T_q\).
The query plan routes retrieval to specialized memory tracks: current-state queries route directly to state trails \(S_t\) to retrieve the latest snapshot; historical episodic queries target the event ledger \(E_t\) and consolidated cards; and epistemic queries inspect belief records \(B_t\). The retrieved candidates across views are then filtered by an LLM Evidence Selector under a strict budget of at most Top-5 records. The selector deliberately prioritizes complementary evidence bundles that span temporal milestones, distinct modalities, and causal dependencies, directly suppressing noisy or obsolete distractors before answer generation.
4. Precondition-Constrained Embodied Task Planning: Translating Stateful Memory into Valid Physical Action Sequences
The ultimate utility of embodied memory lies in guiding physical action execution rather than text-only question answering. In multi-step embodied task planning, an agent must verify a web of physical preconditions—such as container open/closed statuses, device power modes, and hand occupancy—before proposing an action. Lacking structured state tracking, standard agents hallucinate invalid sequences such as picking an object from inside a closed refrigerator or setting device parameters without navigation. ObsMem couples its structured world-state track \(S_t\) and belief track \(B_t\) directly to an executable household action space (comprising navigate_to, inspect, pick, place, open, close, set_device_state, and handoff).
During plan generation, the planner enforces causal consistency: pick and place actions require empty hands and co-location; accessing contained objects enforces prior container-opening steps; and smart-device interactions enforce validated parameter fields. Because \(S_t\) provides an accurate, grounded representation of current object locations and container configurations, the agent avoids pointless exploratory room-sweeps and produces direct, executable action trajectories that respect the cumulative physical changes wrought across preceding days.
Benchmark & Evaluation Protocol¶
WorldLines is grounded in 3D simulated homes from Habitat/HSSD, featuring interactive objects, openable receptacles, and controllable appliances organized into executable scene-graph representations. The trace generator instantiates eight high-level household project themes (e.g., household routines, routine support, home organization, meal preparation, health lifestyle, device coordination). Traces are simulated over multi-day horizons where active projects \(P_d\), accumulated physical states \(S_{d-1}\), and carry-forward notes \(N_{<d}\) dynamically drive agentic director and resident actor dialogues and actions, verified by a deterministic physics executor.
Every benchmark instance is strictly governed by a context cutoff \(c_i\). Evaluated agents observe only the pre-cutoff visible history \(H_{<c_i}\) and task query \(q_i\), while ground-truth world states \(S_{c_i}\) and annotated evidence chains \(E_i\) are sequestered exclusively for automated scoring. The benchmark features 310 rigorous Memory QA samples (divided into StateSingleHop, StateMultiHop, and TemporalMemory) and 21 action-dense Embodied Task Planning instances (averaging 7.6 target actions and 3.1 remembered state constraints per sample). Evaluated systems utilize gemini-1.5-flash for generation under an identical maximum context budget of Top-5 retrieved records. Performance is evaluated using GPT-4o as an independent judge based on a calibrated 0–3 scoring rubric, verified against human consensus across 80 diagnostic samples (Fleiss' \(\kappa = 0.71\), 87.5% human agreement).
Key Experimental Results¶
Main Results¶
On the 310 Memory QA benchmark instances, ObsMem was evaluated against premier open-source long-term memory architectures: Mem0, A-mem, MemoryOS, GraphMem, Graphiti, and HippoRAG2. All systems operated under the identical Top-5 retrieved context constraint. Metrics include the normalized LLM judge score (Judge), the percentage of perfectly answered queries (Perfect Rate), coarse session recall (Session Any@5), fine-grained event recall (Event R@5), and diagnostic breakdowns across multi-hop state tracking (StateMH-J, StateMH-E), single-hop state queries (StateSH-J), and temporal memory (Temp-J).
| Method | Judge ↑ | Perfect ↑ | Sess. Any@5 ↑ | Event R@5 ↑ | StateMH-J ↑ | StateMH-E ↑ | StateSH-J ↑ | Temp-J ↑ |
|---|---|---|---|---|---|---|---|---|
| MemoryOS (Kang et al., 2025) | 0.312 | 29.0% | 0.452 | 0.085 | 0.287 | 0.086 | 0.350 | 0.308 |
| GraphMem (Hu et al., 2026a) | 0.457 | 38.7% | 0.806 | 0.243 | 0.529 | 0.184 | 0.417 | 0.359 |
| Mem0 (Chhikara et al., 2025) | 0.554 | 53.2% | 0.823 | 0.378 | 0.598 | 0.264 | 0.550 | 0.462 |
| A-mem (Xu et al., 2026b) | 0.575 | 53.2% | 0.839 | 0.355 | 0.540 | 0.216 | 0.550 | 0.692 |
| HippoRAG2 (Gutiérrez et al., 2025) | 0.613 | 53.2% | 0.774 | 0.274 | 0.621 | 0.172 | 0.600 | 0.615 |
| Graphiti (He et al., 2025) | 0.642 | 58.1% | 0.806 | 0.331 | 0.356 | 0.158 | 0.383 | 0.627 |
| ObsMem (Ours) | 0.713 | 69.0% | 0.879 | 0.537 | 0.762 | 0.452 | 0.667 | 0.667 |
In downstream Embodied Task Planning across 21 action-dense instances, agents were tasked with translating historical memory into executable physical action plans. Evaluation evaluated overall Plan Judge, physical State Consistency, Precondition Validity, Memory Use, and Action Order.
| Method | N | Plan Judge ↑ | State Consistency ↑ | Precondition Validity ↑ | Memory Use ↑ | Action Order ↑ |
|---|---|---|---|---|---|---|
| MemoryOS | 21 | 0.337 | 0.361 | 0.376 | 0.319 | 0.352 |
| GraphMem | 21 | 0.481 | 0.493 | 0.507 | 0.462 | 0.496 |
| Mem0 | 21 | 0.526 | 0.551 | 0.563 | 0.512 | 0.535 |
| A-mem | 21 | 0.542 | 0.566 | 0.581 | 0.524 | 0.553 |
| ObsMem (Ours) | 21 | 0.684 | 0.721 | 0.702 | 0.690 | 0.676 |
Ablation Study¶
To dissect the specific contributions of ObsMem's architectural components, an ablation study was performed on a 62-sample diagnostic subset (29 StateMultiHop, 20 StateSingleHop, 13 TemporalMemory). A specialized diagnostic metric, Hidden Judge, evaluated 6 instances involving unobserved state changes outside the robot's perception.
| Variant | Ablated Component | Judge ↑ | \(\Delta\)Judge | Perfect ↑ | Event R@5 ↑ | Hidden Judge ↑ | Latency (s) ↓ |
|---|---|---|---|---|---|---|---|
| w/ Full ObsMem | None (Complete Pipeline) | 0.699 | — | 66% | 0.563 | 0.278 | 8.82 |
| w/o Belief | Remove belief-view retrieval | 0.651 | -0.048 | 63% | 0.558 | 0.000 | 8.63 |
| w/o State | Remove world-state retrieval | 0.597 | -0.102 | 56% | 0.532 | 0.111 | 8.95 |
| w/o Consolidation | Disable episode synthesis & conflicts | 0.554 | -0.145 | 53% | 0.419 | 0.167 | 9.16 |
| w/o Selector | Bypass LLM evidence selector | 0.435 | -0.264 | 40% | 0.466 | 0.000 | 5.54 |
To establish upper and lower bounds for the benchmark, context and oracle control baselines were also evaluated:
| Control Baseline | Judge ↑ | Perfect ↑ | Sess. Any@5 ↑ | Event R@5 ↑ | Description |
|---|---|---|---|---|---|
| Grep+LLM | 0.349 | 16.0% | 0.306 | 0.165 | Naive keyword matching returning up to 15 events |
| FullContext | 0.758 | 74.2% | — | — | Unbounded full pre-cutoff visible history passed to LLM |
| Oracle Evidence | 0.952 | 90.3% | 1.000 | 1.000 | Upper bound: only annotated gold evidence records provided |
Key Findings¶
- The Coarse-Session vs. Fine-Event Divergence: Existing memory baselines like A-mem and Mem0 demonstrate strong coarse session retrieval (Session Any@5 of 83.9% and 82.3%), confirming they readily locate the broad conversational episode. However, their event-level recall in complex multi-hop state scenarios (StateMH-E) drops precipitously to 21.6% and 26.4%, compared to ObsMem's 45.2%. In dynamic households where sessions contain multiple object relocations and device tweaks, coarse semantic retrieval is insufficient; isolating the exact state-changing event is mandatory.
- Evidence Selection Prevents Distractor Contamination: Ablating the LLM Evidence Selector caused the most severe performance collapse, dropping Judge from 0.699 to 0.435 (-0.264) and Perfect Rate from 66% to 40%. Without explicit cross-view evidence re-ranking and deduplication, the generator is overwhelmed by semantically related but chronologically obsolete distractors.
- The Belief Track Eliminates Hallucinated Omniscience: While removing the Belief Track resulted in a modest overall Judge drop (-0.048), it caused Hidden Judge on unobserved state queries to collapse entirely from 0.278 to 0.000. Without explicit epistemic tracking of observer provenance and decay, agents exhibit 100% false confidence on unverified historical assumptions.
Highlights & Insights¶
- Ontological Decoupling of Events and States: ObsMem's core architectural insight is that immutable historical event streams and mutable physical environment states must be decoupled into separate memory primitives. This fundamentally prevents the classic error where an agent mistakes a past intermediate object location for its present whereabouts.
- Observer Provenance Grounding: By conditioning ingestion on observer visibility (\(r \in V_t\)) and tracking unobserved intervening transitions, ObsMem models what the robot is legitimately justified to know, instilling meta-cognitive awareness into embodied LLMs.
- Closing the Gap Between Memory and Actionable Execution: WorldLines moves beyond conversational question answering by establishing downstream embodied task planning as a primary benchmark metric, showing that structured state trails directly boost action precondition validity from ~56% to over 70%.
Limitations & Future Work¶
- Simulated Environment vs. Raw Sensory Perception: WorldLines traces currently rely on simulated environments with discrete entity identifiers, structured action schemas, and programmatic visibility annotations. Deploying ObsMem onto physical hardware will require integrating open-vocabulary 3D scene-graph perception, SLAM, and noisy multi-modal action recognition pipelines to extract these structured atoms from raw video and audio streams.
- Inference Latency Overhead: Due to multi-step query planning, multi-view retrieval, and LLM-based evidence selection, ObsMem incurs an average latency of 8.82 seconds per query (compared to 3.35s for A-mem). In real-time physical human-robot interaction, this latency must be curtailed through model distillation or lightweight specialized routing classifiers.
Related Work & Insights¶
- vs. Conversational Memory Benchmarks (LoCoMo, LongMemEval, RealMem): While pioneering long-term conversational recall and user profiling, these benchmarks remain text-bound. WorldLines introduces physical environment statefulness, device operations, and executable action constraints into long-horizon memory evaluation.
- vs. Short-Episode Embodied Benchmarks (ALFWorld, Habitat 3.0, PARTNR, BEHAVIOR-1K): Existing embodied benchmarks reset environment states at the end of short, isolated episodes. WorldLines introduces cross-day temporal state continuity, requiring agents to maintain evolving household threads across multiple sessions.
- vs. Generic Agent Memory Frameworks (Mem0, A-mem, Graphiti, HippoRAG2): Generic frameworks rely on flat text-vector rewriting or entity-centric graphs that create spurious associative bridges across unrelated events. ObsMem replaces flat memory with observer-grounded typed tracks (events, state trails, beliefs, commitments), yielding superior fine-grained event recall and state-consistent planning.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first benchmark uniting long-horizon state tracking, partial observability, and embodied physical planning; the four-track decoupled memory design is highly innovative.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 310 rigorous QA instances and 21 action-dense planning samples across 6 competitive baselines, complete with extensive ablations, diagnostic subsets, and human validation.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical formalizations, well-structured figures, and incisive empirical analysis.
- Value: ⭐⭐⭐⭐⭐ Provides an indispensable evaluation benchmark and reference architecture for advancing long-horizon autonomous domestic service robots.