FindingDory: A Benchmark to Evaluate Memory in Embodied Agents¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Robotics & Embodied AI
Keywords: embodied memory, long-horizon reasoning, vision-language models, goal navigation, procedural evaluation
TL;DR¶
FindingDory uses 60 task templates in Habitat to turn object-rearrangement histories into memory-dependent navigation instructions, with hierarchical metrics showing that even GPT-4o reaches only 27.3% high-level success among frozen vision-language models (VLMs), while remembering the right target still does not guarantee efficient navigation.
Background & Motivation¶
For a household robot, memory involves more than locating an image of an apple in a recording: the robot must know whether the apple was moved, where it moved from and to, and where it should go now. Once interactions change object states, previously observed images can become outdated; finding an object that was not moved yesterday also requires distinguishing observation, proximity, and actual interaction. Long-video question answering tests some aspects of recall, but many questions can be answered from a few keyframes, and multiple-choice answers need not translate into executable spatial goals.
Conversely, evaluating only the final success of a robot placed in an unfamiliar room mixes exploration, visual recognition, memory, and control. A failure may mean the robot never observed the target rather than forgot it; success may come from searching again rather than using past experience. FindingDory therefore first uses a controlled oracle agent to generate rearrangement experiences, then gives the same histories to different evaluated models so that they share an information source and a dynamic environment.
The design uses simulator-provided object identities, states, and poses to associate natural-language tasks with automatically checkable goal conditions, reducing annotation costs while allowing more interactions to increase difficulty. It does not propose a new general-purpose memory network; it defines what must be remembered, how it must be used, and how failures should be diagnosed. Core Idea: use reusable dynamic interaction histories as a shared input, require agents to put memory into practice through navigation, and separately examine goal selection, spatial localization, and execution efficiency.
Method¶
Overall Architecture¶
FindingDory takes egocentric images, poses, and action records from prior experience together with an instruction that depends on that experience; the output is the outcome of executing the task in the environment after the experience ends. The protocol separates experience collection from evaluation: a scripted oracle first picks up, moves, and places objects, after which the evaluated agent uses the history to determine which objects or locations to visit. The paper's baselines use a high-level VLM to select historical goal frames and a low-level policy to navigate using the corresponding image or coordinates, but this frame-based interface is not a mandatory restriction on all future benchmark methods.
The environment uses Habitat 2.0 and HSSD, with 107/30 training/validation scenes and 1,478/100 experience episodes, producing 79,213/5,876 task instances. The number 60 refers to instruction templates, not the total number of validation questions; different entity bindings can generate multiple tasks from one experience. Training/validation scenes contain 839/247 object instances from 84/72 categories, with 17/16 receptacle categories. The Stretch embodiment uses 640 x 480 RGB-D observations and GPS+Compass; evaluation navigation actions are moving forward by 0.25 m, turning left or right by 10 degrees, and stopping. The robot also operates its arm during collection, so dynamic manipulation supplies the memory input; these results should not be interpreted as the evaluated VLM's end-to-end grasping success rate.
Key Designs¶
1. Two-phase dynamic experience: make decisions from a shared history
Collection is not an arbitrary exploration recording: the oracle moves objects between designated receptacles to create changes between previous and subsequent locations. The authors generate candidate experiences, perform physics checks on placements, and discard trajectories with unstable placements or failed oracle execution, reducing the risk of mistaking data-generation errors for memory failures. Section 3.1 reports histories of 400-3500 frames containing 2-11 pick-and-place interactions. During evaluation, models receive these records and cannot interpret qualifiers such as previously interacted with or picked up second from the current observation alone, making past experience part of the goal definition.
Separating collection from the evaluated policy controls differences in experience acquisition; it does not eliminate every perception or navigation challenge. For example, remembering that an apple moved to a table still requires identifying a usable historical goal view and approaching it in the current scene. More interactions and longer histories extend memory dependencies; the authors also inject noise into collection to test robustness to imperfect experience. These extensions change memory load and input reliability, respectively, and should not be conflated with a test of context-window length alone.
2. Templated tasks and state verification: turn memory requirements into checkable goals
Tasks span 11 categories across spatial, temporal, and multi-goal dimensions: object recall, interaction status, conditional interaction, object attributes, spatial relationships, room visitation, interaction order, time points, duration, and ordered or unordered revisitation. Finding any previously interacted object permits multiple valid answers but still calls for a sensible destination; revisiting every interacted object requires preserving the complete set, and a specified order additionally requires recovering event sequence. Attribute tasks refer to targets indirectly through color, material, shape, and related properties, testing whether memory retains visual details beyond category labels. This makes omissions in fixed textual summaries measurable instead of merely evaluating summary fluency.
Each template uses PDDL to describe entities, properties, and goal conditions: object categories, interaction order, and initial and final receptacles constrain entity bindings, while predicates correspond to functions that can be checked against simulator state. After sampling valid bindings, existential or universal quantifiers compose goal conditions into instantiated instructions and automatic validators, without requiring a separate answer for every room. Object attributes come from GPT-4o descriptions of multi-view images followed by manual review; five attribute templates appear only in validation, also testing generalization to templates absent from training. Automation comes from accessible simulator state, not another language model subjectively deciding whether the robot succeeded.
3. Hierarchical memory baselines: share a goal interface and expose information-compression costs
The direct-video baseline annotates historical frames with indices and timestamps, then asks a VLM to output an index that can serve as a navigation goal. Although the output is only a number, reasoning may span several parts of the history: the model might first determine which object was picked up after an event and then locate its final position, rather than retrieve the single frame most similar to the instruction. The text-memory baseline partitions the video into chunks, uses a VLM to summarize them, and lets a large language model (LLM) select goals from the summaries; 3D-Mem instead builds structured 3D memory from informative snapshots. These representations preserve different information, and FindingDory uses shared tasks to test whether lost temporal, attribute, or spatial cues affect behavior.
The low-level controller is either an OVRL-v2-based LSTM image-goal navigation policy or a policy that deterministically navigates to goal-frame coordinates using a global map. These alternatives test whether a visual goal can be recognized by the controller and whether the high-level target remains useful with stronger geometric support. However, if an uninteracted object was never approached closely enough, the history may contain no goal frame satisfying the navigation conditions; this is an interface limitation of the baseline, not evidence that the benchmark task is inherently unsolvable. The authors therefore compute a solvability upper bound using optimal historical frame selection and perfect teleportation, rather than treating an incomplete action interface as the definition of full task capability.
4. Hierarchical diagnostic metrics: distinguish recall, proximity, visibility, and efficient travel
High-Level Success Rate (HL-SR) checks whether selected goal frames correspond to goal states that satisfy the task; Low-Level Success Rate (LL-SR) checks the task outcome after actual navigation. HL-SPL and LL-SPL introduce success-weighted path length for high-level goal selection and low-level execution, respectively: agents must not only select valid entities but also avoid distant equivalent targets or unnecessarily long routes. For instructions permitting multiple valid entities, two models with equal success rates can have different SPL scores because they choose different destinations, so goal-index accuracy alone does not capture the full memory problem. Multi-goal tasks additionally require the correct target set and applicable ordering; the main baselines predict all goals in one prompt rather than reasoning again for each destination.
The authors also use DTG-SR to relax target visibility while retaining the distance condition, and SC-SR to ignore spatial proximity and check semantic target coverage. High SC-SR with low standard success can mean that the model sees a large table but selects a viewpoint too far away; small objects are less likely to remain clearly visible at such distances. This diagnostic distinction prevents recognition of a table from being confused with selection of an executable close-range goal. The supplied cache omits Appendix B.3, so this note states only the metric meanings supported by the main paper and does not reconstruct complete SPL formulas, aggregation rules, or category-specific success thresholds.
Loss & Training¶
The benchmark introduces no new loss function; the supervised fine-tuning (SFT) baseline trains the high-level VLM to predict goal frames from the task and history, supervising it with the full list of acceptable frames because multiple nearby views can be valid. At test time, a frame is selected from the predicted list and checked against valid targets, avoiding the assumption that one annotated frame is the only correct answer. The main paper does not establish the full optimizer configuration or each model's optimal frame budget; these details are in appendices absent from the supplied cache, so no additional hyperparameters are inferred here.
The cross-benchmark experiment separately adds 40,000 FindingDory samples to Video-R1-CoT-165k, constructs chain-of-thought supervision, and compares SFT and GRPO post-training performance on VSI-Bench. This tests whether simulated memory supervision transfers to real-world egocentric video reasoning, not whether it enables deployment on a real robot.
Key Experimental Results¶
Main Results¶
Table 1 summarizes FindingDory high-level results from Section 5 and Figure 2; higher HL-SR is better, and it must not be read as full robot-task LL-SR. The standard validation split contains 100 experience episodes and 5,876 task instances; video VLMs use their individually selected best frame budgets rather than strictly matched compute.
| Table 1: High-level baseline | HL-SR | Conditions and interpretation |
|---|---|---|
| GPT-4o | 27.3% | Frozen model with direct video input |
| Gemini-2.0-Flash | 25.7% | Frozen model with direct video input |
| GLM-4.1V-Thinking | 23.5% | Frozen model with direct video input |
| Text-memory baseline | 9.45% | Goal selection after video-chunk summarization |
| 3D-Mem | 6.9% | Only 95 validation episodes; memory construction capped at 6 hours per episode |
| Qwen SFT | Approximately 50% | Approximate overall description in Figure 2; trained with task supervision |
The 3D-Mem evaluation subset is not identical to that of the other results; Qwen SFT is also not a zero-shot comparison under the same training conditions as frozen models. The main text describes the average SFT improvement as 25% without clearly distinguishing relative percentage from percentage points in that sentence, so this note does not derive an exact overall success rate from it.
Ablation Study¶
Table 2 reproduces the training-data addition analysis from the paper's Table 3; all metrics are percentages on VSI-Bench, not FindingDory navigation success rates. The comparison tests adding the above 40,000 samples; because training data volume also increases, it is not a pure content ablation with fully controlled dataset size.
| Table 2: Training configuration | MCQ Acc. | Num. Acc. |
|---|---|---|
| Zero-Shot | 33.69 | 21.47 |
| Video-R1-Only SFT | 33.33 | 32.67 |
| Video-R1 + FindingDory SFT | 35.06 | 32.35 |
| Video-R1-Only GRPO | 33.57 | 33.9 |
| Video-R1 + FindingDory GRPO | 35.42 | 34.82 |
Adding FindingDory to SFT increases MCQ accuracy by 1.73 percentage points but reduces numerical-question accuracy by 0.32 percentage points; not every metric improves. Under GRPO, MCQ and numerical-question accuracy increase by 1.85 and 0.92 percentage points, respectively, a more precise description than claiming consistent gains in every setting.
Key Findings¶
- Figure 5 shows that more input frames generally provide little benefit to frozen VLMs and can hurt performance; the SFT model benefits from longer training videos, but exact plotted curve values are unavailable in the cached text.
- Frozen VLMs often achieve near-zero multi-goal success, versus approximately 20% for SFT and an attainable upper bound of approximately 99%; these figures concern multi-goal analysis, not the overall average.
- The Qwen2.5-VL-7B hierarchical experiment in Figure 3d triggers low-level navigation only when the high-level prediction is correct; ImageNav reaches 78% on its native validation tasks but degrades markedly on manipulation viewpoints, while the mapping policy still incurs an approximately 25% relative success-rate drop.
Highlights & Insights¶
- Dynamic experience requires object identity and location to be stored separately. Remembering an apple or retaining an old map is insufficient for recovering locations before and after rearrangement or the order of interactions.
- The goal-frame upper bound separates representational limitations from reasoning ability. Increasing VLM scale cannot repair an interface that lacks a suitable historical viewpoint.
- Relaxing distance and visibility produces different diagnoses. Similar metric decomposition can help other embodied tasks locate errors between semantic memory and executable spatial targets.
Limitations & Future Work¶
- The authors note that historical goal frames cannot express valid destinations for every task, and single-shot multi-goal output limits interactive correction. These are limitations of current baselines, not conclusions about every memory architecture.
- The authors report substantial degradation for zero-shot and SFT models under noisy experiences, making ideal oracle-log results an optimistic reference for complex real-world conditions rather than demonstrated deployment performance.
- This note's assessment: validation contains only 30 scenes and 100 experience episodes, while five attribute templates rely on a specific generation and manual-review process; broader object, language, and sensor-noise coverage remains necessary.
- This note's assessment: differing frame budgets, the different 3D-Mem subset, and increased cross-benchmark training volume limit single-factor attribution; future work could report equal-budget comparisons, per-event state updates, and closed-loop multi-goal replanning.
- Reading boundary: the supplied cache contains the complete main paper and references but not its cited appendices; appendix experiments, exact thresholds, optimizer settings, and frame budgets were not independently verified.
Related Work & Insights¶
- vs OpenEQA: both use embodied observations to evaluate reasoning, but FindingDory requires selecting and reaching targets from dynamic interaction histories rather than only producing language answers; the trade-off is that navigation interfaces also affect performance.
- vs MemoryMaze / MemoryGym: these study memory in controlled partially observable environments, whereas FindingDory adds realistic indoor visuals, semantic attributes, and natural-language constraints suited to testing VLM memory representations.
- vs ReMEmbR / 3D-Mem: text and 3D snapshots provide explicit memory compression, but event order and state changes may be lost; the results motivate evaluating event preservation rather than static scene coverage alone.
Rating¶
- Novelty: 4/5. Combines dynamic interaction histories, verifiable tasks, and hierarchical memory diagnosis into a clear evaluation problem.
- Experimental Thoroughness: 4/5. Covers several baseline families, context lengths, controllers, and transfer analyses, although comparison conditions are not always matched.
- Writing Quality: 4/5. Clearly distinguishes benchmark limitations from baseline limitations, while relying substantially on appendices for reproducibility details.
- Value: 4/5. Offers a useful diagnostic platform for embodied memory research but does not replace long-term evaluation on real robots.