HippoCamp: Benchmarking Contextual Agents on Personal Computers¶

Conference: ECCV2026
Paper: ECCV Paper
Dataset: MMMem-org/HippoCamp
Area: LLM Agent / Multimodal VLM
Keywords: personal file systems, contextual agents, user profiling, multimodal retrieval, evidence grounding

TL;DR¶

HippoCamp organizes heterogeneous personal-computer files into three isolated archetypal environments and evaluates factual retention and user profiling with 581 file-grounded questions; the strongest evaluated system achieves only 48.3% profiling accuracy, exposing a substantial gap between finding relevant files and reliably understanding a user.

Background & Motivation¶

A personal assistant answering a question about someone's weekly health routine cannot simply search for files containing the word "health." It must identify recurring calendar events, check activity logs to establish whether activities occurred, and connect evidence from photographs, voice memos, and emails. The same term may concern the account holder, a family member, or a pet, so semantic relevance does not establish that evidence concerns the correct person. Unlike web navigation, code generation, or single-document question answering, the answer is implicit in accumulated private material with irregular structure and heterogeneous modalities. Existing multimodal retrieval evaluations already cover images, tables, and documents, but often supply curated candidate sets or relatively bounded task contexts. That setup reduces the importance of directory navigation, temporal links, file versions, and cross-entity disambiguation encountered on real computers.

Expanding the context window does not directly resolve these problems: an entire device is unsuitable for a single input, and much of its content is irrelevant to the current question. Flat top-k semantic retrieval can instead confuse topically similar public reports with genuine personal records. Even after relevant files have been found, an agent must distinguish evidence sufficient for a conclusion from isolated events that do not establish stable habits. Existing memory systems often build user knowledge from dialogue histories, curated experiences, or narrow modalities, leaving messy file environments insufficiently tested. HippoCamp therefore introduces an evaluation setting that exposes these failure locations rather than another memory network. The evaluation must assess both whether the final answer is correct and whether it can be traced to real files.

The benchmark preserves native directories and metadata, constructs dense archetypal environments from contributor material, and uses human-authored, verifiable questions. This supports both localized fact finding and the aggregation of preferences, regularities, and constraints across time. The two tasks share a file environment but impose different evidence-integration requirements, helping diagnose systems that can search without understanding the user. Core Idea: operationalize personalization as evidence-grounded question answering within isolated personal file systems, measuring answer correctness and evidence-file retrieval separately to distinguish search, perception, and reasoning bottlenecks.

Method¶

Overall Architecture¶

HippoCamp takes a natural-language question and the complete file system of its associated archetypal profile as input, and expects an answer with supporting files as output. The benchmark contains 42.4 GB of data, over 2000 heterogeneous files, and 581 question-answer pairs. Text, images, documents, video, and audio form a shared search space rather than unrelated modality-specific test sets. Each question has a human-verified answer, evidence files, localized content pointers, and a reasoning trajectory with capability labels. These annotations support scoring and diagnosis; they are not reference solution paths directly supplied to the agent during testing. The evaluated capability is therefore discovering evidence from the question, interpreting its contents, and producing a traceable conclusion.

The data layer first constructs three archetypal profiles, and the annotation layer turns realistic needs into factual-retention and profiling tasks. The execution layer allows systems to explore files through their own retrieval or tool mechanisms while prohibiting external web retrieval and side-channel information. The scoring layer evaluates answers and evidence separately, then analyzes differences by task, profile, and capability label. The paper does not train a new unified model or introduce a dedicated loss function. Offline indexing for RAG is baseline preparation and should not be confused with training on evaluation answers. Here, "memory" primarily means recovering user context from longitudinal records, not an already demonstrated capacity for continual learning or online memory updating.

Key Designs¶

1. Archetypal file systems: preserve the structural difficulty of device-scale retrieval

The source pool comes from interviews with 100+ personal-device users, prioritizing material with stable routines, longitudinal traces, and cross-file evidence. Selected contributors' files are then aggregated into three coherent profiles by matching modality and file-type distributions, directory organization, temporal scope, and recurring workflows. Bei emphasizes a media-rich student and content-creation environment; Adam emphasizes legal document work; Victoria emphasizes financial analysis, scheduling, and administrative records. These environments are neither unprocessed disk copies from three real individuals nor a three-way demographic classification of computer users. They are diagnostic environments constructed to cover different structural challenges and cannot estimate success rates across the general population. The design tests whether an agent can handle a complex file ecosystem rather than supplying a fictional biography as a ready-made answer source.

Native directory hierarchies, irregular names, long-tail formats, temporal metadata, and redundant files are retained, making structure an informative but potentially misleading cue. A calendar and a photograph may corroborate each other or merely have similar names, requiring further inspection of their contents. The authors report opt-in consent, privacy filtering, consistent pseudonymization, and participant review before release. Limited FinanceBench and LegalBench-RAG material enriches professional document forms after identity sanitization and re-annotation. Thus, being grounded in real material does not mean that no external enrichment was used. Processing details are referred to Supplementary Section A, which is absent from the available main-paper cache and cannot be checked here at the implementation level.

2. Human-grounded trajectories: move from localized facts to longitudinal profiling

The 581 questions comprise 521 factual-retention questions and 60 profiling questions, accounting for 89.7% and 10.3%, respectively. Factual retention requires recovering verifiable information, such as finding maximum-flow notes and identifying the course and its duration. It need not be a single-file keyword lookup: answers may require understanding multiple formats and connecting several records. Profiling instead infers stable user-level patterns, scheduling constraints, preferences, retrospective accounts, or workflows from collections of facts. Planning an afternoon, for example, requires recognizing existing calendar commitments while incorporating historical routines and personal preferences rather than giving generic advice. Its central difficulty is whether evidence across time supports a coherent conclusion, not simply whether the answer involves more files.

Domain-aware annotators manually write questions and attach files, localized evidence, and step-wise rationales to each item. JSON trajectories can contain atomic pointers such as page indices, table cells, and text spans, specifying where a document supports a claim rather than merely naming the document. Search labels concern system navigation and semantic retrieval; perception labels cover file-system understanding, modality understanding, and grounding; reasoning labels cover inference, computation, summarization, and verification. Annotations follow a minimalist principle with explicit evidence links instead of retaining only an uncheckable final explanation. The 46.1K fine-grained annotations count evidence and trajectory fields, not additional independent questions. LLMs only suggest coverage directions from restricted metadata and seed examples; the main text explicitly reports that 0% of released QA/trajectory records are directly model-generated or accepted verbatim. Retained items still require human finalization, file verification, deduplication, and coverage across task families, modality combinations, and evidence-set sizes.

3. Isolated access and baseline regimes: compare how systems discover evidence

Each test instance exposes only its associated profile's file system, excluding other profiles, external retrieval, web access, and auxiliary side-channel metadata. Within that boundary, agents have full file-access permissions and may search, read, and reason through their native mechanisms. The task thus checks whether agents can use authorized local material rather than guess a user's identity from prior knowledge. Standard RAG and Self-RAG reformat and index the full corpus as required by their methods, then generate answers from retrieved candidates. ReAct and Search-R1 embed retrieval in multi-turn search and reasoning loops, revising exploration in response to new observations. Terminal agents use tools in a Dockerized Ubuntu replica of the file system, while hosted product-grade agents use their platform interfaces.

These systems represent different operating regimes, not an ablation that changes only the underlying model while holding every other component fixed. The main table uses a max-budget protocol: each system runs up to its predefined step, token, or platform budget without an additional shared wall-clock cutoff. Results therefore measure system performance under its tools and budget, not a ranking of model capability at equal computation. Three non-author human solvers follow the same answer-plus-evidence protocol but have no artificial wall-clock limit and serve only as a solvability reference. Claude-based systems are omitted from the main table because of severe file-system and long-document processing failures; an absent score does not imply a completed equivalent evaluation. Exact budgets, retry handling, and interface details are deferred to Supplementary Section D and cannot be confirmed from the available cache.

4. Separate answer and evidence scoring: distinguish correctness from supported correctness

Answer evaluation defaults to GPT-4o, which receives the question, reference answer, and system response and assesses factual alignment, reasoning soundness, and personalization. The judge produces a binary correctness decision and a 0–5 quality score; main-table Acc is the fraction judged correct, not a normalized quality score. The authors additionally calibrate with Claude Sonnet 4.5, DeepSeek V3.2, and human verification, reporting stable principal rankings and group-level trends. The main text does not provide the complete calibration tables, so this remains an author-reported result rather than grounds for inventing agreement rates or significance values. Evidence evaluation compares the submitted file set with the annotated evidence set and computes recall and file-level F1. Precision measures the fraction of submitted files belonging to the reference evidence, recall measures the fraction of reference evidence recovered, and F1 is their harmonic mean.

This dual evaluation distinguishes retrieving relevant files but interpreting them incorrectly from producing a plausible answer without submitting its supporting files. The latter suggests incomplete grounding and possibly reliance on parametric knowledge, but Acc exceeding F1 does not by itself establish a particular causal explanation. Likewise, file-level F1 does not prove that the crucial page, cell, or span within a retrieved file was understood correctly. Atomic annotations support finer diagnosis, while the main-table metric must still be interpreted at the file-set level. Table 2 Overall is a balanced average over the three profiles, not accuracy weighted by the number of questions across the entire benchmark. Table 3 instead aggregates by search, perception, and reasoning labels; its Overall differs from Table 2 and should not be mixed with it to calculate an alleged overall improvement.

Key Experimental Results¶

Main Results¶

The following values come from Table 2 on page 9; all four columns are percentages, and Overall is balanced across the three profiles. F1 denotes evidence-file-set F1 and Acc denotes LLM-judged answer accuracy; the human row is a solvability reference without an artificial time limit.

Method	Profiling F1	Profiling Acc	Factual Retention F1	Factual Retention Acc
Standard RAG	18.4	26.7	30.0	30.2
Self-RAG	15.2	10.0	31.9	27.5
ReAct (Qwen3-30B-A3B)	11.8	13.5	43.1	28.5
ReAct (Gemini-2.5-flash)	18.5	20.0	26.5	38.7
Search-R1	10.8	5.0	41.0	25.3
Terminal Agent (GPT-5.2)	11.1	30.0	24.6	48.2
ChatGPT Agent Mode	21.0	48.3	35.3	62.8
Human Solvers (3 non-author)	88.0	87.0	89.0	82.0

ChatGPT Agent Mode trails humans by 38.7 percentage points in profiling Acc and 19.2 percentage points in factual-retention Acc. It does not lead every metric: its factual-retention F1 of 35.3 is below ReAct (Qwen3)'s 43.1. The paper's broad description of "highest overall scores" therefore should not be extended into a claim of dominance in every F1/Acc column.

Ablation Study¶

The paper does not remove components of a new architecture; the following is the capability-group analysis from Table 3 on page 12. Only profiling Acc is included, aggregated by capability labels rather than measured as independent module success rates or as Table 2's profile-balanced average.

Method	Search Acc	Perception Acc	Reasoning Acc	Capability-Grouped Overall Acc
Standard RAG	26.2	13.8	25.5	21.8
ReAct (Qwen3-30B-A3B)	26.1	16.7	23.9	22.2
Search-R1	24.9	15.7	25.8	22.1
Terminal Agent (GPT-5.2)	46.3	27.3	44.1	39.2
ChatGPT Agent Mode	56.5	28.5	55.8	46.9

ChatGPT Agent Mode achieves 28.5% profiling perception Acc against 56.5% search Acc, a difference of 28.0 percentage points. This supports the difficulty of turning heterogeneous content into reliable evidence, but label-group comparisons alone do not establish a perception component's causal contribution.

Key Findings¶

In Table 2, ChatGPT Agent Mode reaches 90.3% factual-retention Acc on Adam but only 31.2% on Bei. The authors relate this difference to explicit document structure, entity boundaries, and media noise rather than simply declaring legal questions easier.
ReAct (Qwen3) records factual-retention F1/Acc of 43.1/28.5, whereas Terminal Agent (GPT-5.2) records 24.6/48.2 in Table 2. These opposite discrepancies show why retrieval quality and answer quality require separate measurement.
Pages 14–15 report that retrieval mismatch accounts for 40.7% of failed outputs in single-modality cases and 87.0% when at least three modalities are required. These are shares of error types among failures, not failure rates across all questions.

Highlights & Insights¶

Device-level file structure is part of the evaluation rather than merely a collection of semantic vectors. This exposes the role of directory depth, temporal traces, and attachment relationships in realistic retrieval.
Requiring both an answer and its evidence detects systems that hide missing support behind plausible language. For personal assistants, this is closer to practical reliability than response fluency alone.
Figure 5 on page 14 shows an entity-misattribution case in which genuine material leads to an answer about pet care rather than the user's health. Correct retrieval still requires binding the question's subject to the evidence's subject.
A transferable design implication is to narrow evidence to a minimally sufficient set before generating and verifying a conclusion. This is an author-proposed system direction, not a new component trained and validated in the paper.

Limitations & Future Work¶

Three aggregated profiles and only 60 profiling questions limit coverage of populations, occupations, and longitudinal behavior. Results should not be interpreted as overall real-world success rates for personal-computer assistants.
Tools, platform budgets, and interfaces are not identical, and humans are only a solvability reference. The main text also reports that the strongest hosted setting often takes 10–15 minutes per query, leaving cost and stability concerns beyond accuracy.
Both judges and reference evidence sets have limits: binary correctness depends on a judge, and alternative valid supporting files may not be fully represented in the reference set. The latter is a reader-identified evaluation risk of file-level metrics.
The available cache lacks supplementary material, preventing verification of complete budget, retry, judge-calibration, and privacy-review details. Dense figures such as Figure 2 also have damaged text extraction, so unconfirmed numbers are not reconstructed from them.
Page 13 describes the lower bound of profiling perception Acc as 13.2%, but Table 3 lists 5.7% for the Qwen3-VL terminal agent. This note does not repeat that range summary and bases its specific analysis on readable table rows.
Future work should test structure-aware search, entity-relation memory, and final evidence verification under unified budgets, and extend evaluation to changing files and revocable user authorization. These are research directions, not established experimental findings.

vs WebQA / MultimodalQA: these benchmarks emphasize retrieval and question answering over heterogeneous content; HippoCamp adds longitudinal personal context, file hierarchies, and entity attribution within one environment, at the cost of more constrained standardization and coverage.
vs M3DocRAG / MMDocRAG: document systems locate content within document collections, whereas personal computers also contain audiovisual material, calendars, and user-generated redundancy. Document relevance does not replace user-level evidence consistency.
vs PersonaAgent / Telemem: these systems study how to construct and use personal memory; HippoCamp primarily supplies an evaluation environment and does not claim a learning mechanism superior to those memory architectures.
vs ReAct / Search-R1: search-reasoning loops can improve evidence discovery, but Table 2 shows that such benefits do not automatically translate into profiling. Candidate-evidence filtering and entity binding deserve further investigation.
Resources and evidence: official paper and dataset links appear in the metadata; data and tasks are described in Section 3, pages 5–8, the protocol in Sections 4.1–4.3, pages 8–10, and failure analysis in Section 5, pages 11–15.

Rating¶

Novelty: 4/5. Combining longitudinal personal file ecosystems with localized evidence offers a more distinctive evaluation setting than simply scaling document question answering.
Experimental Thoroughness: 4/5. Multiple system families, a human reference, and capability analysis are included, but profile coverage is limited, budgets differ, and full reproduction details depend on supplementary material.
Writing Quality: 4/5. Task boundaries and representative failures are informative, but some range and best-metric summaries require cross-checking against the tables.
Value: 4/5. The benchmark gives local personal assistants concrete reliability targets, without replacing long-term deployment evaluation on real, changing devices.