MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Agentic Hierarchical Graph Memory¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Video Understanding
Keywords: Long Video Understanding, Decoupled Perception and Reasoning, Hierarchical Graph Memory, Agentic Retrieval, Observation-Reason-Action
TL;DR¶
MemDreamer decouples long-video understanding into streaming construction of a pure-text Hierarchical Graph Memory and tool-augmented agentic retrieval via an Observation-Reason-Action loop, establishing new SOTA across four benchmarks with a 40โ124ร smaller reasoning context.
Background & Motivation¶
Long video understanding across hours-long horizons represents a core capability for advancing Vision-Language Models (VLMs) toward embodied intelligence and open-world interaction. However, prevalent paradigms tightly couple perception and reasoning into an end-to-end process, relying on ever-expanding context windows to ingest massive raw visual frame sequences. For a 2-hour video sampled at 1 FPS, standard visual tokenizers produce over 1.6M visual tokens even after spatial merging. This brute-force ingestion incurs severe token explosion, prohibitive memory footprints, and critical attention dilution. Furthermore, it exacerbates the "lost in the middle" phenomenon, burying sparse task-relevant reasoning cues beneath overwhelming perceptual noise.
Decoupling perception from reasoningโwhere a lightweight perception model incrementally builds an external persistent memory while a downstream reasoning model retrieves evidence on demandโoffers a scalable path forward. Nonetheless, existing memory systems predominantly adopt flat chunk-based storage or similarity-based vector retrieval, which obliterates global narrative perspective and severs temporal-causal connections, degenerating retrieval into myopic, exhaustive trial-and-error. When humans comprehend long videos, they naturally deploy a coarse-to-fine hierarchical abstraction spanning narrative arcs down to atomic actions, while maintaining dense spatiotemporal and causal topological links across fine-grained entities and events.
Addressing this fundamental gap, this paper introduces structured topological memory combined with proactive agentic exploration. Core idea: decouple long video understanding into the streaming construction of a top-down three-tier Hierarchical Graph Memory and tool-augmented agentic retrieval over an Observation-Reason-Action loop, allowing the reasoning model to actively navigate a compact pure-text topological graph and directly unlocking agentic reasoning scaling for long multimodal comprehension.
Method¶
Overall Architecture¶
MemDreamer replaces end-to-end video ingestion with a decoupled two-stage paradigm: persistent memory construction and tool-augmented agentic retrieval. In the offline construction phase, an adaptive sliding window streams the video through a perception model to segment semantically self-contained Macro Events; the system then extracts fine-grained heterogeneous subgraphs downward (comprising entities, micro-events, and their topological relations) while aggregating events upward into Super Events and a Video Root, forming a unified pure-text three-tier Hierarchical Graph Memory \(G\).
In the online reasoning phase, a separate reasoning model acts as an autonomous agent operating exclusively over the pure-text memory \(G\) without revisiting raw pixels. Equipped with a dedicated 7-tool bank spanning hierarchical navigation, precise search, and graph traversal, the agent engages in a multi-step Observation-Reason-Action loop to selectively gather task-critical evidence and synthesize the final answer.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
V["Raw Video Stream<br/>V (no external transcripts)"] --> A["Streaming Adaptive Segmentation<br/>Sliding window ฯ=10min, bounding macro events e_i"]
A --> MEM["Hierarchical Graph Memory Construction"]
subgraph MEM["Pure-Text Hierarchical Graph Memory G"]
direction TB
B["Downward Subgraph Extraction<br/>Entities V^E and micro-events V^M form local subgraph g_i"]
C["Upward Hierarchical Aggregation<br/>Macro Event โ Super Event โ Video Root"]
end
MEM --> G["Three-Tier Graph Memory G<br/>(Video Root / Super Events / Macro Event Subgraphs)"]
Q["User Query Q"] --> AGENT["Agentic Observation-Reason-Action Loop<br/>Iterative agentic exploration"]
G --> AGENT
AGENT --> T["Multi-Dimensional Tool Bank<br/>Navigation / Precise Search / Graph Traversal"]
T -->|Execute tool, return observation o_t| AGENT
AGENT -->|Selective clue compression c_t| H["Trajectory Update<br/>H_t = H_{t-1} โช {(a_t, c_t)}"]
H -->|Trigger can_answer termination| OUT["Final Answer A<br/>A = R(Q, C)"]
Key Designs¶
1. Streaming Adaptive Segmentation: eliminating arbitrary semantic cuts
Standard video chunking rigidly slices streams into fixed temporal windows (e.g., 30s intervals), frequently truncating coherent actions or mid-dialogue context. MemDreamer introduces streaming adaptive segmentation driven by semantic boundaries. At each iteration step \(k\), a sliding window \(W_k = [t_k^{start}, t_k^{start} + \tau]\) is maintained with maximum duration \(\tau = 10\text{min}\). The perception model \(P\) executes temporal grounding within \(W_k\) to identify self-contained narrative episodes, resolving a set of complete Macro Events \(\{e_1, e_2, \dots, e_N\}\). The concluding boundary of \(e_N\) directly anchors the start of the next window. This design guarantees semantic integrity for every Macro Event node while strictly capping the perception model's input within \(\tau\), eliminating long-context memory explosion during perception.
2. Bidirectional Graph Memory: downward heterogeneous subgraphs and upward semantic aggregation
Conventional LLM graph memories collapse interactions into static (subject, relation, object) triplets lacking temporal extent, n-ary participation, and causal grounding. MemDreamer defines a Video-centric Graph Ontology. For each Macro Event \(e_i\), a local spatiotemporal subgraph \(g_i = (V_i, E_i)\) is constructed with a partitioned node set \(V_i = V^E_i \cup V^M_i\), separating persistent entities \(V^E_i\) (person, object, location, group) from atomic micro-events \(V^M_i\) (explicit second-level time ranges, subject/object bindings, and physical actions). Heterogeneous edges \(E_i\) are categorized into three orthogonal types:
- Spatial-attribute edges (\(V^E_i \leftrightarrow V^E_i\)): capture relative spatial layouts, coreference, and attribute bindings (e.g., LOCATED_IN, PART_OF, NEXT_TO);
- Subject-object edges (\(V^E_i \leftrightarrow V^M_i\)): anchor entities to the actions they perform or receive (e.g., PERFORMS, RECEIVES, USES_TOOL, HAPPENS_IN);
- Temporal-causal edges (\(V^M_i \rightarrow V^M_i\)): model chronological flow and explicit causality among micro-events (e.g., BEFORE, OVERLAP, CAUSES, PREVENTS).
Upward, the perception model performs semantic abstraction over all Macro Events guided by temporal contiguity, entity consistency (shared core participants), and goal consistency. Neighboring Macro Events (typically 3โ8) are clustered into Super Events (10โ20 per video) wired by progression edges (LEADS_TO, RESOLVES, CONTRASTS_WITH), converging at a single Video Root node capturing global themes and key entities.
3. Multi-Dimensional Tool Bank and Agentic Observation-Reason-Action Loop
To navigate this structured memory space, the reasoning model \(R\) is equipped with a 7-tool toolkit:
- Hierarchical Navigation: GetSummary() retrieves the video-level root summary; GetSuperEvent(super_id) browses macro-narrative segments; GetMacroEvent(super_id/macro_ids) unpacks constituent macro events; GetSubgraph(macro_id) inspects dense fine-grained local subgraphs;
- Precise Search: SearchNodes(query, top_k, node_types) performs vector-based semantic retrieval across entities and micro-events with 1-hop ego-graph context; SearchByTime(start_sec, end_sec) anchors macro events within specific intervals;
- Graph Traversal: GetRelationGraph(node_id) traverses incoming/outgoing edges along spatial, temporal, or causal links.
During each round \(t\) of the Observation-Reason-Action loop, directly concatenating raw tool observations \(o_t\) would re-introduce token noise and attention dilution. MemDreamer introduces an information extraction operator: $\(c_t = R(o_t, Q)\)$ The agent distills only query-relevant factual clues \(c_t\) from raw observation \(o_t\), updating the compact trajectory \(H_t = H_{t-1} \cup \{(a_t, c_t)\}\). This maintains the active reasoning context strictly within 5.9Kโ6.3K tokensโa 40โ124ร reduction compared to full-context ingestionโenabling uncompromised multi-step logical deduction.
A Worked Example: Tracing Ice Cream Origin in Zootopia¶
Consider the query: "Where does the ice cream for the pawpsicles Nick sells come from?"
1. Round 1 (Global Navigation): Initialized with the Video Root and Super Event summaries, the agent identifies SuperEvent 2 (00:16:47โ00:26:30, "Meter Maid Duty and the Fox Scam") as covering Nick Wilde's fraudulent schemes;
2. Round 2 (Precise Node Search): The agent executes SearchNodes(query="Nick buys ice cream at elephant parlor", top_k=5), directly hitting MacroEvent 2.2 (00:17:39โ00:23:43, "Pawpsicle Scam Discovery");
3. Round 3 (Subgraph Expansion & Graph Traversal): Calling GetSubgraph(macro_id="MacroEvent2.2"), the agent inspects the local subgraph and observes micro-events "Nick enters Jumbeaux's Cafe" and "Nick buys Jumbo-Pop from Jerry", linked via subject-object edges to entity "Jumbo-Pop";
4. Answer Emission: The agent extracts the concise clue "The ice cream comes from the Jumbo-Pop purchased at Jumbeaux's Cafe", determines that evidence is complete (can_answer=true), and outputs the final answer in just 3 interaction rounds over ~6K tokens.
Key Experimental Results¶
Main Results¶
MemDreamer was evaluated across four long-video benchmarks spanning 15 minutes to several hours: LVBench, LongVideoBench, Video-MME (long video without subtitles), and EgoSchema. Across all settings, MemDreamer consistently established new state-of-the-art performance.
| Method | Reasoning Model | LVBench (Avg.) | LongVideoBench (Avg.) | Video-MME (Long w/o sub) | EgoSchema (Val) |
|---|---|---|---|---|---|
| GPT-4o | GPT-4o | 48.9 | 66.7 | 65.3 | 70.4 |
| Gemini-2.5-Pro | Gemini-2.5-Pro | 72.0 | 71.0 | 75.9 | 72.8 |
| Gemini-3.1-Pro | Gemini-3.1-Pro | 78.2 | 78.6 | 80.3 | 76.4 |
| Qwen3-VL-Thinking | Qwen3-VL-235B-A22B-Thinking | 63.6 | 71.4 | 71.2 | 75.9 |
| VideoARM | OpenAI-o3 | 79.7 | 78.0 | 81.2 | 76.2 |
| DVD | OpenAI-o3 | 74.2 | 71.6 | 67.3 | 76.6 |
| MemDreamer (Ours) | Qwen3-VL-235B-A22B-Thinking | 78.9 (+15.3) | 80.4 (+9.0) | 80.3 (+9.1) | 80.6 (+4.7) |
| MemDreamer (Ours) | Gemini-2.5-Pro | 77.8 (+5.8) | 75.9 (+4.9) | 82.6 (+6.7) | 85.2 (+12.4) |
| MemDreamer (Ours) | Gemini-3.1-Pro | 85.5 (+7.3) | 87.1 (+8.5) | 86.1 (+5.8) | 87.1 (+10.7) |
| Human Expert | Human | 94.4 | - | - | - |
On LVBench, MemDreamer with Gemini-3.1-Pro achieves a record score of 85.5, outperforming the native end-to-end Gemini-3.1-Pro baseline by +7.3 points and narrowing the gap with human experts (94.4) to just 8.9 points. When paired with the open-source Qwen3-VL-Thinking, MemDreamer delivers a massive +15.3 point gain (63.6 \(\rightarrow\) 78.9).
Ablation Study¶
The ablation on memory architecture and retrieval mechanisms on LVBench (with Gemini-3.1-Pro) isolates individual components:
| Memory Architecture Config | Hierarchy | Graph | LVBench Avg. | Note |
|---|---|---|---|---|
| Flat-Chunk | โ | โ | 71.0 | Baseline flat memory without hierarchy or edges |
| Flat-Graph | โ | โ | 78.6 | Adding spatiotemporal and causal edges (+7.6) |
| Hierarchical-Chunk | โ | โ | 80.6 | Adding three-tier macro abstraction (+9.6) |
| Hierarchical-Graph (Ours) | โ | โ | 85.5 | Full model combining macro hierarchy and local graphs |
In retrieval mechanism ablations: - Vanilla Embedding Similarity scores 66.6 (-18.9 points below full agentic retrieval); - Full Memory Context achieves 74.5 overall, but its reasoning score drops precipitously to 68.0, showing that uncompressed topological graphs cause severe attention dilution; - Agentic Search Only achieves 75.5; - Agentic Full Tools achieves the top score of 85.5.
Key Findings¶
- Decoupling directly uncovers agentic reasoning scaling: In the coupled end-to-end regime, the Pearson correlation between models' AIME 2025 mathematical reasoning scores and LVBench performance is only \(R=0.702\) (\(p=0.052\), statistically insignificant). Under MemDreamer, this correlation scales to \(R=0.948\) (\(p<0.001\)), replicating on GPQA Diamond (\(R=0.926, p<0.001\)). This proves that decoupling removes perceptual noise barriers, allowing backbone reasoning capacity to transfer directly into long-video understanding.
- Robustness against perceptual variance: When holding the reasoning engine constant (Gemini-3.1-Pro), swapping the perception model from Gemini-3.1-Pro to Gemini-2.5-Pro changes LVBench accuracy by only 0.4 points (85.5 \(\rightarrow\) 85.1); with Qwen3-VL, the difference is only 1.2 points (78.9 \(\rightarrow\) 77.7). Because the perception model only processes localized clips (\(\le 10\) min), performance differences across vision backbones are effectively smoothed out.
- Self-terminating search loops: In the round budget sweep (\(T_{max} \in \{8, 10, 12, 15\}\)), the average number of rounds actually executed remains virtually flat (2.87 to 3.07), showing that the agent actively stops when sufficient clues are gathered rather than passively exhausting tool budgets.
Highlights & Insights¶
- Decoupled perception vs. reasoning paradigm: Instead of endlessly scaling context windows or subsampling frames to fit token budgets, MemDreamer demonstrates that structured text memory allows a tiny 6K token reasoning context to decisively outperform 1M+ token brute-force multimodal ingestion.
- Micro-events as first-class nodes: By promoting atomic actions into independent micro-event nodes rather than flat
(s, r, o)triplets, the graph ontology cleanly decouples static spatial layouts from dynamic causal chains, providing explicit topological paths for multi-hop graph traversal. - Query-conditioned clue extraction: Suppressing raw tool outputs via \(c_t = R(o_t, Q)\) prevents multi-turn tool interaction history from re-inducing the "lost in the middle" problem inside the reasoning agent.
Limitations & Future Work¶
- Visual-only modality: The current memory construction operates exclusively on visual frames, neglecting speech, dialogue audio, and ambient acoustic cues. For dialogue-heavy movies or lecture videos, auditory cues are lost, calling for future integration of audio-derived node types.
- Offline pre-processing assumption: While segmentation operates via streaming sliding windows, the full video must be available offline before retrieval begins. True online streaming environments (e.g., live streaming, real-time surveillance, embodied agents) require continuous real-time graph updates, pruning, and concurrent query processing.
Related Work & Insights¶
- vs. VideoAgent / DVD: Earlier agentic video systems search over flat clip databases, frequently degenerating into repetitive trial-and-error; MemDreamer equips agents with a coarse-to-fine hierarchy and explicit causal edges for structured zooming and topological tracing.
- vs. VideoARM / MM-Mem: Existing hierarchical memory frameworks maintain edge-less perceptual buffers and require revisiting raw video frames during inference; MemDreamer relies on a pure-text symbolic graph memory, decoupling inference entirely from pixel-level re-perception.
Rating¶
- Novelty: โญโญโญโญโญ First framework to combine a three-tier narrative hierarchy with a heterogeneous entity-event causal graph for pure-text video memory, empirically demonstrating the direct transfer of agentic reasoning to long video understanding.
- Experimental Thoroughness: โญโญโญโญโญ Rigorously evaluated across four major long-video benchmarks with open- and closed-source backbones, complete with statistical significance testing, budget sweeps, and dual-model ablations.
- Writing Quality: โญโญโญโญโญ Clear exposition, elegant formalization of the graph ontology, and highly intuitive conceptual narrative.
- Value: โญโญโญโญโญ Establishes foundational design principles for long-horizon multimodal memory systems, bypassing brute-force context scaling.