EgoEverything: A Benchmark for Human Behavior–Inspired Long-Context Egocentric Video Understanding in AR Environment¶

Conference: ECCV 2026
Paper: Official page · PDF
Dataset: EgoEverything project page
Area: Video Understanding
Keywords: Egocentric video, gaze awareness, long-term memory, AR assistants, multi-agent annotation

TL;DR¶

EgoEverything samples question targets using real eye-tracking traces and combines delayed questioning, dual-agent evidence gathering, and human filtering to build an AR memory benchmark with over 100 hours of video and over 5,000 multiple-choice questions, where the best evaluated model, Gemini 1.5 Pro, achieves 63.1% accuracy, trailing human accuracy of 83.5% by 20.4 percentage points.

Background & Motivation¶

Long-context egocentric video understanding involves more than summarizing what happened in a recording. Someone wearing AR glasses may pass a restaurant and ask its name later, or finish packing and then ask where a particular garment ended up. An assistant must connect previously observed details, the activity surrounding them, and a subsequent question. Benchmarks such as EgoSchema and LongVideoBench have advanced long-context evaluation, but this paper addresses a more specific gap: whether questions concern things the wearer might have attended to, sound like everyday interaction, and arise before the activity has ended.

Seeing something cannot simply be equated with intending to ask about it. Gaze is a measurable behavioral signal, not an intention label; restricting questions to the fixation center would also exclude peripheral objects and incidental details. Meanwhile, manually watching long recordings, checking objects, and composing natural questions with distractors is expensive. Template generation is cheaper but can produce video-centric wording that does not resemble an actual assistant interaction. The paper therefore incorporates behavioral constraints into data production rather than introducing another question-answering network.

Core idea: use measured gaze as a weak prior over question targets while retaining peripheral objects, then have tool-using synthesis and validation agents repeatedly check naturally phrased questions with randomly delayed query times to approximate AR memory retrieval.

Method¶

Overall Architecture¶

The inputs are egocentric videos and synchronized gaze traces from Aria Everyday Activities (AEA) and Nymeria. The outputs are five-option questions with answers, evidence, and questioning timestamps. The pipeline performs Video Stream Summary and Clustering (VSSC), Gaze-Oriented Target Sampling (GOTS), and Question Generation and Manual Curation (QGMC). Below, QGMC is explained through separate evidence-refinement and quality-filtering designs to distinguish automated interaction from final acceptance.

Gaze primarily determines what to ask during offline dataset construction; it is not a prompt automatically attached to every final question. Gaze cropping, masking, and saliency-map prompting are separate evaluation input conditions, not mandatory components of every evaluated model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Video and synchronized gaze"] --> VSSC["Video Summary and Clustering"]
    VSSC --> GOTS["Gaze-Oriented Target Sampling"]
    GOTS --> Agents["Delayed Questions and<br/>Dual-Agent Evidence Gathering"]
    Agents -->|Review feedback, at most two rounds| Agents
    Agents --> Filter["Multi-Layer Quality Filtering"]
    Filter --> Output["Five-option questions and evidence"]

Key Designs¶

1. Video Summary and Clustering: build a navigable activity timeline first

AEA contains 143 daily-activity clips from 5 indoor environments, totaling approximately 7.3 hours. The source Nymeria resource contains approximately 300 hours across approximately 50 locations. The benchmark samples from both to obtain over 100 hours of footage; Nymeria's entire source duration must not be counted as the final benchmark size. Both sources have real gaze traces, but Nymeria additionally provides time-aligned activity narration, which AEA lacks. VSSC therefore primarily fills AEA's activity-timeline gap so downstream agents can locate relevant periods without repeatedly consuming the entire high-resolution recording.

A pretrained ResNet-50 extracts frame-level features, which are clustered with k-means. The authors compare \(K\in\{8,12,16\}\) and choose \(K=12\) for the best downstream question quality. Here, \(K\) controls summary granularity rather than specifying a fixed set of semantic action classes. To suppress label jitter, each frame takes the majority label within a window extending 5 frames in either direction. Consecutive identically labeled portions are merged, with a minimum segment duration of 2 seconds. A VLM produces activity summaries, and human inspection removes summary errors. Smaller values of \(K\) can merge distinct activities, while larger values produce fragmented, verbose summaries; the cache does not provide numerical quality scores for the three candidates.

2. Gaze-Oriented Target Sampling: favor attended objects without restricting questions to fixation centers

Within each activity segment, a VLM detects objects and provides labels and bounding boxes. Consecutive frames repeatedly detect the same item, so sampling directly from detections could let persistent objects dominate the questions. The pipeline crops objects, encodes them with a pretrained CLIP visual encoder, and uses lightweight re-identification (ReID) to consolidate highly similar detections into representative instances. This supports target diversity; it does not establish that the authors trained a complete, reliable cross-time identity tracker.

The pipeline then randomly selects a target frame and uses a Perception Sampler (PS) to choose one target object according to the Euclidean distance between its bounding-box centroid and the measured gaze point. PS has a two-dimensional Gaussian form centered on real gaze: more distant objects have lower selection probability, and \(\theta\) controls the decay range. Equation (1) is corrupted in the local text cache and retains only a formula label. Consequently, this note does not reconstruct its exponent or normalization constant, or assume that \(\theta\) denotes a standard deviation or variance. The prose explicitly specifies \(\theta=400\) pixels as a broad central sampling prior rather than precise intention localization; peripheral objects can still be selected.

This preserves the tendency to ask about attended objects while retaining peripheral details. AEA frames are \(1408\times1408\), whereas Nymeria frames are \(2016\times2208\), so a shared pixel parameter does not automatically correspond to a shared angular extent. Final questions need not expose gaze information, separating the use of eye tracking to construct a benchmark from its use to answer benchmark questions.

3. Delayed Questions and Dual-Agent Evidence Gathering: turn an object into a verifiable everyday memory question

Given the target-frame timestamp \(t_0\), the pipeline uniformly samples a later questioning time \(t_q\) within a valid range. The paper specifies a minimum recall interval of 3 minutes and denotes the video endpoint by \(T\). The recall interval is defined as follows:

\[ t_q\in[t_0+\Delta_{\min},T],\qquad \Delta_{\min}=3\text{ minutes},\qquad \Delta=t_q-t_0. \]

This separates the question from its evidence in time and avoids always asking at the end of a clip. It remains a controllable baseline for query timing rather than a distribution learned from real user queries. The cache also does not explain how target frames with insufficient remaining duration are handled.

The Synthesizer Agent (SA) receives the activity summary, target object, bounding box, and target frame, drafts a natural question, and actively gathers supporting evidence. Instead of ingesting the entire video, it invokes GetFrame for a high-resolution frame at a specified timestamp to inspect static details such as text or color, or GetSegment for a downsampled interval to verify dynamic actions such as placing or picking up an object. This separates inspecting an instant from establishing an event, preventing a static screenshot from being treated as proof of an action it cannot establish.

The Validator Agent (VA) receives the question, supporting evidence, and activity summary, but is not directly given the target-object and target-frame hints. It can use the same tools for its own inspection. It checks factual correctness, ambiguity, and clarity, adds at least one additional piece of evidence, and returns feedback to SA for revision. Withholding direct target hints helps expose whether the question itself identifies the intended object, rather than letting the reviewer silently fill in missing context. Questions that still fail after two review rounds are discarded rather than revised indefinitely.

4. Multi-Layer Quality Filtering: distinguish visual answerability from linguistic shortcuts

Agent approval is not sufficient for publication. Distance, occlusion, poor lighting, and viewpoint distortion can cause object misidentification, while spatial descriptions may become incorrect under viewpoint changes. Rule-based filtering removes unsuitable targets such as walls, ceilings, floors, and the wearer's body parts, along with questions that explicitly reference timestamps or mention a video. These constraints keep the final wording closer to ordinary assistant use.

Human reviewers then watch the corresponding video and choose among 5 options without seeing the proposed correct answer. They may repair minor defects but reject major ones; retained items must have agreement between the human selection and the pseudo answer. The 12 trained annotators reviewed approximately 21,600 questions over more than 400 hours. The human-stage adoption rate was approximately 70%, and the paper additionally reports approximately 50% acceptance for rule-based and blind filtering, without a complete stage-by-stage count ledger that can safely be multiplied together. Finally, an LLM guesses answers without viewing the video, and questions it answers correctly are removed. This weakens obvious textual cues but cannot guarantee that other models will not exploit option biases or common knowledge.

A Worked Example¶

The paper's pipeline figure uses a book on a coffee table. The target frame occurs at 32.5 seconds; SA locates the book from its bounding box and inspects a frame at 35 seconds to confirm its title before submitting the question and both evidence timestamps. VA separately inspects a frame at 34 seconds, finds the reference to the book insufficiently specific, and requests wording that identifies the coffee-table book. SA revises the question and resubmits it. These are evidence timestamps shown in the figure, not the final questioning time; tool interactions a few seconds apart must not be described as satisfying the 3-minute recall interval.

The example exposes two distinct quality requirements. Reading the title establishes answer correctness; specifying the coffee table establishes which object the question concerns. Without both, five-option accuracy may measure ambiguity or language-based guessing rather than long-term visual memory.

Loss & Training¶

The paper proposes no new answering model, loss function, or end-to-end training objective; its contribution is dataset construction and diagnostic evaluation. The prose describes SA as a pretrained VLM fine-tuned for tool invocation, but the cache supplies no specific training examples, objective, optimizer, or complete hyperparameter recipe. That description should not be expanded into an invented training procedure. Confirmed pipeline settings include \(K=12\), smoothing over 5 frames in each direction, a 2-second minimum segment, \(\theta=400\) pixels, a 3-minute minimum recall interval, and at most two review rounds.

Key Experimental Results¶

Main Results¶

The metric is five-option accuracy (%), with a uniform-random baseline of 20%. FR retains original resolution; AD uniformly downsamples the whole frame to approximately 10% of its original resolution; GC retains a square gaze-centered region covering approximately 10% of the original frame size; GM retains the complement of GC. This follows the paper's wording about 10% without inferring exact side lengths or token counts. VMP and AMEGO convert videos into structured memories for text-based reasoning. NA means not reported.

Model	FR	AD	GC	GM	VMP	AMEGO
VideoLLaMA3-7B	49.1	46.1	42.4	35.2	20.2	19.5
VideoLLaMA3-2B	46.5	44.9	40.5	34.5	21.3	19.4
Gemini 1.5 Pro	63.1	58.4	52.7	37.7	33.2	18.3
LongVA	34.6	31.6	28.9	22.2	NA	NA
LLaVA-Video	42.6	36.6	32.9	26.0	NA	NA

These values come from Table 1. Human average accuracy is 83.5%. Gemini 1.5 Pro performs best with FR but remains 20.4 percentage points below humans. Every model follows FR > AD > GC > GM: gaze-centered information matters, but peripheral context cannot be discarded indiscriminately. Actual token counts, compute, and hardware budgets are not normalized in this comparison, so it is not an equal-compute efficiency ranking.

Ablation Study¶

This is a benchmark paper. The following are reported input comparisons and factor analyses, not ablations that remove generation modules and retrain a model. The first analysis table combines Table 2 with the saliency-map prompting result in Section 4.1. Unreported conditions remain NA.

Model	Question and options only (%)	FR (%)	Saliency-map gaze prompting (%)
VideoLLaMA3-7B	22.9	49.1	51.19
Gemini 1.5 Pro	35.9	63.1	NA
LongVA	21.8	34.6	NA

Saliency-map prompting improves VideoLLaMA3-7B over FR by 2.09 percentage points, showing that gaze can provide additional guidance rather than only determine a crop. However, Gemini still scores 35.9% without video, substantially above the random baseline. Blind filtering reduces linguistic shortcuts but does not eliminate them.

The following values are selected from Table 3 and describe VideoLLaMA3-7B's object-distance and recall-interval analyses. They compare different question subsets and do not independently establish causal effects of distance or duration.

Factor	Subset or interval	Accuracy (%)
Object-centroid distance to gaze	0–100 pixels	54.2
Object-centroid distance to gaze	100–200 pixels	54.8
Object-centroid distance to gaze	200–300 pixels	51.8
Object-centroid distance to gaze	300–400 pixels	47.2
Object-centroid distance to gaze	400–500 pixels	42.6
Object-centroid distance to gaze	500–600 pixels	36.1
Object-centroid distance to gaze	600–700 pixels	31.4
Recall interval	Longest 10%	33.3
Recall interval	Shortest 10%	49.0

Key Findings¶

Peripheral details are harder: accuracy declines by 23.4 percentage points between the 100–200-pixel and 600–700-pixel groups. The 0–100 and 100–200 groups are not strictly monotonic, so it would be incorrect to claim a decrease at every distance step.
Longer recall is harder: the longest- and shortest-interval groups differ by 15.7 percentage points. The authors note that the input length does not exceed the model's supported range, suggesting redundant frames, distractor activities, and episode retrieval rather than merely an insufficient context window.
Small objects are harder: Table 3(b) reports accuracies of 44.3, 44.4, 44.5, 44.9, 49.2, 54.8, and 54.8% for area thresholds of 100, 500, 2k, 5k, 10k, 50k, and 100k square pixels, respectively. The cache does not clearly specify the threshold grouping convention, so these must not be rewritten as known disjoint area bins.
Structured memory does not automatically preserve evidence: the authors attribute poor VMP and AMEGO results to their focus on interacted objects, omission of activity-irrelevant items, and detail loss during image-to-text conversion. This suggests a memory-selection mismatch, not proof that all scene-graph approaches are ineffective.

Highlights & Insights¶

Put the attention distribution on the data side. The benchmark can shift which targets matter without requiring explicit gaze information in the question. This creates a useful distinction between behavior-oriented question selection and gaze-assisted inference.
Give validators fewer hints and more evidence obligations. VA is not directly given the target object and target frame and must gather additional evidence, helping expose ambiguous references. The transferable idea is evidence-based cross-checking with an information difference, not merely asking another model the same question.
Global coarse context and local detail should coexist. AD outperforming GC indicates that restricting input to the fixation region loses object relationships and event context. Future memory systems could preserve a low-resolution global stream while retrieving high-resolution local evidence on demand, although this paper neither implements nor validates that extension.

Limitations & Future Work¶

Behavioral realism remains approximate. Measured gaze is not actual question intent, and a fixed Gaussian range with uniformly delayed questions is not an empirical user-query distribution. Real query logs are needed to establish how much natural demand the sampling procedure covers.
Quality filtering remains model-dependent. Gemini's text-only accuracy of 35.9% shows that one blind-filtering model being wrong does not prove that every model needs video. Multi-model blind testing and option-bias analysis would provide stronger evidence.
Direct generation-component ablations are missing. The cache does not report question-quality comparisons without ReID, VA, or human review, or under different \(\theta\) values. Confidence intervals, detailed frame budgets, and hardware costs are also absent, limiting claims about independent component benefits.
Reproduction details are incomplete. Equation (1) is corrupted in the local extraction. Area-threshold grouping, numerical quality scores for clustering candidates, short-tail target-frame handling, and a complete filtering-count ledger are insufficient for exact reproduction; this note does not fill those gaps with invented details.
A streaming task design is not a deployment evaluation. Random query times are useful, but the main table lacks real-time latency, energy consumption, and online memory budgets, and does not fully specify how frames after the questioning time are excluded. Future work should define a causal input protocol and evaluate under wearable-device constraints.

Compared with EgoSchema, LongVideoBench, and HD-EPIC: the emphasis is on gaze-guided target distributions, natural wording, and questions issued during ongoing activities. The contribution is not simply longer video input, nor does it imply that every question in prior benchmarks is unnatural.
Compared with EgoGazeVQA, GazeVQA, and StreamGaze: the paper characterizes these as explicit gaze-assisted intent understanding, multiview collaborative question answering, and streaming temporal reasoning, respectively. EgoEverything mainly uses gaze for offline target selection without requiring its disclosure in natural questions. The saliency-map prompting comparison also shows that these directions can be combined.
Compared with AMEGO and VideoMindPalace: these methods compress long videos into structured object relationships, while this benchmark exposes how storing only interacted objects can omit background details that users ask about later. Preserving appearance evidence and inexpensive indices for non-interacted objects is a possible extension, but retrieval benefits and memory costs must be measured together.

Rating¶

Novelty: 4/5. Combining measured gaze, natural wording, and delayed queries creates a distinctive task design, but not a new reasoning architecture.
Experimental Thoroughness: 3/5. Models, input conditions, and difficulty factors are broadly covered, but generation-component ablations, statistical uncertainty, and strict deployment budgets are missing.
Writing Quality: 3/5. The pipeline and failure patterns are understandable, while parameter conventions and reproduction details need clarification; the local formula extraction is also damaged.
Value: 4/5. Useful for AR long-term memory, peripheral-detail preservation, and behavior-oriented evaluation, with representativeness of real queries still requiring validation.