Skip to content

MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Agentic Hierarchical Graph Memory

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Video Understanding
Keywords: Long Video Understanding, Decoupled Perception and Reasoning, Hierarchical Graph Memory, Agentic Retrieval, Observation-Reason-Action

TL;DR

MemDreamer decouples long-video understanding into streaming construction of a pure-text Hierarchical Graph Memory and tool-augmented agentic retrieval via an Observation-Reason-Action loop, establishing new SOTA across four benchmarks with a 40โ€“124ร— smaller reasoning context.

Background & Motivation

Long video understanding across hours-long horizons represents a core capability for advancing Vision-Language Models (VLMs) toward embodied intelligence and open-world interaction. However, prevalent paradigms tightly couple perception and reasoning into an end-to-end process, relying on ever-expanding context windows to ingest massive raw visual frame sequences. For a 2-hour video sampled at 1 FPS, standard visual tokenizers produce over 1.6M visual tokens even after spatial merging. This brute-force ingestion incurs severe token explosion, prohibitive memory footprints, and critical attention dilution. Furthermore, it exacerbates the "lost in the middle" phenomenon, burying sparse task-relevant reasoning cues beneath overwhelming perceptual noise.

Decoupling perception from reasoningโ€”where a lightweight perception model incrementally builds an external persistent memory while a downstream reasoning model retrieves evidence on demandโ€”offers a scalable path forward. Nonetheless, existing memory systems predominantly adopt flat chunk-based storage or similarity-based vector retrieval, which obliterates global narrative perspective and severs temporal-causal connections, degenerating retrieval into myopic, exhaustive trial-and-error. When humans comprehend long videos, they naturally deploy a coarse-to-fine hierarchical abstraction spanning narrative arcs down to atomic actions, while maintaining dense spatiotemporal and causal topological links across fine-grained entities and events.

Addressing this fundamental gap, this paper introduces structured topological memory combined with proactive agentic exploration. Core idea: decouple long video understanding into the streaming construction of a top-down three-tier Hierarchical Graph Memory and tool-augmented agentic retrieval over an Observation-Reason-Action loop, allowing the reasoning model to actively navigate a compact pure-text topological graph and directly unlocking agentic reasoning scaling for long multimodal comprehension.

Method

Overall Architecture

MemDreamer replaces end-to-end video ingestion with a decoupled two-stage paradigm: persistent memory construction and tool-augmented agentic retrieval. In the offline construction phase, an adaptive sliding window streams the video through a perception model to segment semantically self-contained Macro Events; the system then extracts fine-grained heterogeneous subgraphs downward (comprising entities, micro-events, and their topological relations) while aggregating events upward into Super Events and a Video Root, forming a unified pure-text three-tier Hierarchical Graph Memory \(G\).

In the online reasoning phase, a separate reasoning model acts as an autonomous agent operating exclusively over the pure-text memory \(G\) without revisiting raw pixels. Equipped with a dedicated 7-tool bank spanning hierarchical navigation, precise search, and graph traversal, the agent engages in a multi-step Observation-Reason-Action loop to selectively gather task-critical evidence and synthesize the final answer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    V["Raw Video Stream<br/>V (no external transcripts)"] --> A["Streaming Adaptive Segmentation<br/>Sliding window ฯ„=10min, bounding macro events e_i"]
    A --> MEM["Hierarchical Graph Memory Construction"]
    subgraph MEM["Pure-Text Hierarchical Graph Memory G"]
        direction TB
        B["Downward Subgraph Extraction<br/>Entities V^E and micro-events V^M form local subgraph g_i"]
        C["Upward Hierarchical Aggregation<br/>Macro Event โ†’ Super Event โ†’ Video Root"]
    end
    MEM --> G["Three-Tier Graph Memory G<br/>(Video Root / Super Events / Macro Event Subgraphs)"]
    Q["User Query Q"] --> AGENT["Agentic Observation-Reason-Action Loop<br/>Iterative agentic exploration"]
    G --> AGENT
    AGENT --> T["Multi-Dimensional Tool Bank<br/>Navigation / Precise Search / Graph Traversal"]
    T -->|Execute tool, return observation o_t| AGENT
    AGENT -->|Selective clue compression c_t| H["Trajectory Update<br/>H_t = H_{t-1} โˆช {(a_t, c_t)}"]
    H -->|Trigger can_answer termination| OUT["Final Answer A<br/>A = R(Q, C)"]

Key Designs

1. Streaming Adaptive Segmentation: eliminating arbitrary semantic cuts

Standard video chunking rigidly slices streams into fixed temporal windows (e.g., 30s intervals), frequently truncating coherent actions or mid-dialogue context. MemDreamer introduces streaming adaptive segmentation driven by semantic boundaries. At each iteration step \(k\), a sliding window \(W_k = [t_k^{start}, t_k^{start} + \tau]\) is maintained with maximum duration \(\tau = 10\text{min}\). The perception model \(P\) executes temporal grounding within \(W_k\) to identify self-contained narrative episodes, resolving a set of complete Macro Events \(\{e_1, e_2, \dots, e_N\}\). The concluding boundary of \(e_N\) directly anchors the start of the next window. This design guarantees semantic integrity for every Macro Event node while strictly capping the perception model's input within \(\tau\), eliminating long-context memory explosion during perception.

2. Bidirectional Graph Memory: downward heterogeneous subgraphs and upward semantic aggregation

Conventional LLM graph memories collapse interactions into static (subject, relation, object) triplets lacking temporal extent, n-ary participation, and causal grounding. MemDreamer defines a Video-centric Graph Ontology. For each Macro Event \(e_i\), a local spatiotemporal subgraph \(g_i = (V_i, E_i)\) is constructed with a partitioned node set \(V_i = V^E_i \cup V^M_i\), separating persistent entities \(V^E_i\) (person, object, location, group) from atomic micro-events \(V^M_i\) (explicit second-level time ranges, subject/object bindings, and physical actions). Heterogeneous edges \(E_i\) are categorized into three orthogonal types: - Spatial-attribute edges (\(V^E_i \leftrightarrow V^E_i\)): capture relative spatial layouts, coreference, and attribute bindings (e.g., LOCATED_IN, PART_OF, NEXT_TO); - Subject-object edges (\(V^E_i \leftrightarrow V^M_i\)): anchor entities to the actions they perform or receive (e.g., PERFORMS, RECEIVES, USES_TOOL, HAPPENS_IN); - Temporal-causal edges (\(V^M_i \rightarrow V^M_i\)): model chronological flow and explicit causality among micro-events (e.g., BEFORE, OVERLAP, CAUSES, PREVENTS).

Upward, the perception model performs semantic abstraction over all Macro Events guided by temporal contiguity, entity consistency (shared core participants), and goal consistency. Neighboring Macro Events (typically 3โ€“8) are clustered into Super Events (10โ€“20 per video) wired by progression edges (LEADS_TO, RESOLVES, CONTRASTS_WITH), converging at a single Video Root node capturing global themes and key entities.

3. Multi-Dimensional Tool Bank and Agentic Observation-Reason-Action Loop

To navigate this structured memory space, the reasoning model \(R\) is equipped with a 7-tool toolkit: - Hierarchical Navigation: GetSummary() retrieves the video-level root summary; GetSuperEvent(super_id) browses macro-narrative segments; GetMacroEvent(super_id/macro_ids) unpacks constituent macro events; GetSubgraph(macro_id) inspects dense fine-grained local subgraphs; - Precise Search: SearchNodes(query, top_k, node_types) performs vector-based semantic retrieval across entities and micro-events with 1-hop ego-graph context; SearchByTime(start_sec, end_sec) anchors macro events within specific intervals; - Graph Traversal: GetRelationGraph(node_id) traverses incoming/outgoing edges along spatial, temporal, or causal links.

During each round \(t\) of the Observation-Reason-Action loop, directly concatenating raw tool observations \(o_t\) would re-introduce token noise and attention dilution. MemDreamer introduces an information extraction operator: $\(c_t = R(o_t, Q)\)$ The agent distills only query-relevant factual clues \(c_t\) from raw observation \(o_t\), updating the compact trajectory \(H_t = H_{t-1} \cup \{(a_t, c_t)\}\). This maintains the active reasoning context strictly within 5.9Kโ€“6.3K tokensโ€”a 40โ€“124ร— reduction compared to full-context ingestionโ€”enabling uncompromised multi-step logical deduction.

A Worked Example: Tracing Ice Cream Origin in Zootopia

Consider the query: "Where does the ice cream for the pawpsicles Nick sells come from?" 1. Round 1 (Global Navigation): Initialized with the Video Root and Super Event summaries, the agent identifies SuperEvent 2 (00:16:47โ€“00:26:30, "Meter Maid Duty and the Fox Scam") as covering Nick Wilde's fraudulent schemes; 2. Round 2 (Precise Node Search): The agent executes SearchNodes(query="Nick buys ice cream at elephant parlor", top_k=5), directly hitting MacroEvent 2.2 (00:17:39โ€“00:23:43, "Pawpsicle Scam Discovery"); 3. Round 3 (Subgraph Expansion & Graph Traversal): Calling GetSubgraph(macro_id="MacroEvent2.2"), the agent inspects the local subgraph and observes micro-events "Nick enters Jumbeaux's Cafe" and "Nick buys Jumbo-Pop from Jerry", linked via subject-object edges to entity "Jumbo-Pop"; 4. Answer Emission: The agent extracts the concise clue "The ice cream comes from the Jumbo-Pop purchased at Jumbeaux's Cafe", determines that evidence is complete (can_answer=true), and outputs the final answer in just 3 interaction rounds over ~6K tokens.

Key Experimental Results

Main Results

MemDreamer was evaluated across four long-video benchmarks spanning 15 minutes to several hours: LVBench, LongVideoBench, Video-MME (long video without subtitles), and EgoSchema. Across all settings, MemDreamer consistently established new state-of-the-art performance.

Method Reasoning Model LVBench (Avg.) LongVideoBench (Avg.) Video-MME (Long w/o sub) EgoSchema (Val)
GPT-4o GPT-4o 48.9 66.7 65.3 70.4
Gemini-2.5-Pro Gemini-2.5-Pro 72.0 71.0 75.9 72.8
Gemini-3.1-Pro Gemini-3.1-Pro 78.2 78.6 80.3 76.4
Qwen3-VL-Thinking Qwen3-VL-235B-A22B-Thinking 63.6 71.4 71.2 75.9
VideoARM OpenAI-o3 79.7 78.0 81.2 76.2
DVD OpenAI-o3 74.2 71.6 67.3 76.6
MemDreamer (Ours) Qwen3-VL-235B-A22B-Thinking 78.9 (+15.3) 80.4 (+9.0) 80.3 (+9.1) 80.6 (+4.7)
MemDreamer (Ours) Gemini-2.5-Pro 77.8 (+5.8) 75.9 (+4.9) 82.6 (+6.7) 85.2 (+12.4)
MemDreamer (Ours) Gemini-3.1-Pro 85.5 (+7.3) 87.1 (+8.5) 86.1 (+5.8) 87.1 (+10.7)
Human Expert Human 94.4 - - -

On LVBench, MemDreamer with Gemini-3.1-Pro achieves a record score of 85.5, outperforming the native end-to-end Gemini-3.1-Pro baseline by +7.3 points and narrowing the gap with human experts (94.4) to just 8.9 points. When paired with the open-source Qwen3-VL-Thinking, MemDreamer delivers a massive +15.3 point gain (63.6 \(\rightarrow\) 78.9).

Ablation Study

The ablation on memory architecture and retrieval mechanisms on LVBench (with Gemini-3.1-Pro) isolates individual components:

Memory Architecture Config Hierarchy Graph LVBench Avg. Note
Flat-Chunk โœ— โœ— 71.0 Baseline flat memory without hierarchy or edges
Flat-Graph โœ— โœ“ 78.6 Adding spatiotemporal and causal edges (+7.6)
Hierarchical-Chunk โœ“ โœ— 80.6 Adding three-tier macro abstraction (+9.6)
Hierarchical-Graph (Ours) โœ“ โœ“ 85.5 Full model combining macro hierarchy and local graphs

In retrieval mechanism ablations: - Vanilla Embedding Similarity scores 66.6 (-18.9 points below full agentic retrieval); - Full Memory Context achieves 74.5 overall, but its reasoning score drops precipitously to 68.0, showing that uncompressed topological graphs cause severe attention dilution; - Agentic Search Only achieves 75.5; - Agentic Full Tools achieves the top score of 85.5.

Key Findings

  • Decoupling directly uncovers agentic reasoning scaling: In the coupled end-to-end regime, the Pearson correlation between models' AIME 2025 mathematical reasoning scores and LVBench performance is only \(R=0.702\) (\(p=0.052\), statistically insignificant). Under MemDreamer, this correlation scales to \(R=0.948\) (\(p<0.001\)), replicating on GPQA Diamond (\(R=0.926, p<0.001\)). This proves that decoupling removes perceptual noise barriers, allowing backbone reasoning capacity to transfer directly into long-video understanding.
  • Robustness against perceptual variance: When holding the reasoning engine constant (Gemini-3.1-Pro), swapping the perception model from Gemini-3.1-Pro to Gemini-2.5-Pro changes LVBench accuracy by only 0.4 points (85.5 \(\rightarrow\) 85.1); with Qwen3-VL, the difference is only 1.2 points (78.9 \(\rightarrow\) 77.7). Because the perception model only processes localized clips (\(\le 10\) min), performance differences across vision backbones are effectively smoothed out.
  • Self-terminating search loops: In the round budget sweep (\(T_{max} \in \{8, 10, 12, 15\}\)), the average number of rounds actually executed remains virtually flat (2.87 to 3.07), showing that the agent actively stops when sufficient clues are gathered rather than passively exhausting tool budgets.

Highlights & Insights

  • Decoupled perception vs. reasoning paradigm: Instead of endlessly scaling context windows or subsampling frames to fit token budgets, MemDreamer demonstrates that structured text memory allows a tiny 6K token reasoning context to decisively outperform 1M+ token brute-force multimodal ingestion.
  • Micro-events as first-class nodes: By promoting atomic actions into independent micro-event nodes rather than flat (s, r, o) triplets, the graph ontology cleanly decouples static spatial layouts from dynamic causal chains, providing explicit topological paths for multi-hop graph traversal.
  • Query-conditioned clue extraction: Suppressing raw tool outputs via \(c_t = R(o_t, Q)\) prevents multi-turn tool interaction history from re-inducing the "lost in the middle" problem inside the reasoning agent.

Limitations & Future Work

  • Visual-only modality: The current memory construction operates exclusively on visual frames, neglecting speech, dialogue audio, and ambient acoustic cues. For dialogue-heavy movies or lecture videos, auditory cues are lost, calling for future integration of audio-derived node types.
  • Offline pre-processing assumption: While segmentation operates via streaming sliding windows, the full video must be available offline before retrieval begins. True online streaming environments (e.g., live streaming, real-time surveillance, embodied agents) require continuous real-time graph updates, pruning, and concurrent query processing.
  • vs. VideoAgent / DVD: Earlier agentic video systems search over flat clip databases, frequently degenerating into repetitive trial-and-error; MemDreamer equips agents with a coarse-to-fine hierarchy and explicit causal edges for structured zooming and topological tracing.
  • vs. VideoARM / MM-Mem: Existing hierarchical memory frameworks maintain edge-less perceptual buffers and require revisiting raw video frames during inference; MemDreamer relies on a pure-text symbolic graph memory, decoupling inference entirely from pixel-level re-perception.

Rating

  • Novelty: โญโญโญโญโญ First framework to combine a three-tier narrative hierarchy with a heterogeneous entity-event causal graph for pure-text video memory, empirically demonstrating the direct transfer of agentic reasoning to long video understanding.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorously evaluated across four major long-video benchmarks with open- and closed-source backbones, complete with statistical significance testing, budget sweeps, and dual-model ablations.
  • Writing Quality: โญโญโญโญโญ Clear exposition, elegant formalization of the graph ontology, and highly intuitive conceptual narrative.
  • Value: โญโญโญโญโญ Establishes foundational design principles for long-horizon multimodal memory systems, bypassing brute-force context scaling.