EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding¶

Conference: ECCV 2026
arXiv: 2606.24422
Code: None
Area: Multimodal VLM
Keywords: Egocentric video understanding, streaming reasoning, VLM benchmarking, answerability evaluation, confidence calibration

TL;DR¶

EgoSAT is the first systematic benchmark for egocentric streaming interaction understanding. It unifies three types of temporal reasoning—past (retrospective), present (real-time), and future (prospective)—under a strict online protocol. Through answerability quantification and confidence diagnosis, it reveals that current VLMs are not only weak at temporal reasoning but also suffer from severe miscalibration problems (being "confidently wrong").

Background & Motivation¶

Wearable cameras and edge computing advancements, combined with VLM breakthroughs, are driving a new generation of AI systems capable of providing context-aware assistance to wearers through natural language. The core feature of such systems is streaming processing: an AI assistant must continuously perceive, understand, and remember video streams and user queries captured by the device to answer "what just happened" (retrospective reasoning), "what is being done" (online understanding), and "what will be done next" (prospective forecasting). However, these tasks have long been studied in isolation—video QA, online video narration, and activity prediction each have independent benchmarks and methods, and they have never been unified and evaluated under the same streaming framework.

Why has no one established a unified streaming benchmark before? There are two fundamental reasons: first, temporal annotations of egocentric videos are extremely expensive and sparse, and prior to Ego4D, there was a lack of large-scale, high-quality annotations for hand-object interactions; second, prior VLMs had almost no streaming reasoning capabilities (supporting only offline, full-frame inputs) until the recent emergence of streaming VLMs such as TimeChat-Online and Flash-VStream, which finally made online evaluation possible. Furthermore, a crucial but severely neglected issue in streaming scenarios is answerability: under partial observation conditions, certain queries are inherently unpredictable (e.g., there are multiple equally likely future branches), making it unfair to force the model to provide an answer. No prior benchmark has systematically modeled and evaluated this aspect.

The key enabling factors for EgoSAT are threefold: (1) Ego4D provides over 3,600 hours of dense hand-object interaction annotations covering 100+ scenarios; (2) the emergence of streaming VLMs makes benchmark evaluation under online protocols feasible; (3) the surprise and branchiness quantification metrics proposed by the authors offer operational mathematical definitions for "predictable vs. unpredictable". The core idea of this paper is: to unify the three types of temporal reasoning (retrospective, present, prospective) under the exact same streaming prefix constraint, and introduce answerability quantification + confidence diagnosis, upgrading the evaluation from pure end-point accuracy to uncertainty-aware evaluation of "whether the model knows it should not answer."

Method¶

Overall Architecture¶

The essence of EgoSAT is an "answerability-aware streaming VLM benchmark." It addresses three levels of problems: first, how to formalize the three temporal reasoning tasks in streaming interaction understanding; second, how to quantify "whether the model should answer"—i.e., inferring whether sufficient evidence is present from partial observations; third, how to diagnose whether the model's confidence aligns with its true answerability. The overall benchmark construction and evaluation follow the vertical pipeline shown in Figure 1.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Ego4D Video Sources<br/>3600+ Hours / 100+ Scenarios"] --> B["Interaction Segment Filtering<br/>Filter segments without visible interactions"]
    B --> C["QA Pair Construction<br/>Template-based + 4-option MCQ"]
    C --> D["Answerability Annotation<br/>Surprise + Branchiness"]
    D --> E["6 Streaming Task Definitions<br/>Present / Prospective / Retrospective"]
    E --> F["Online Protocol Evaluation<br/>Prefix truncation + KV cache control"]
    F --> G["Multi-dimensional Diagnosis<br/>Accuracy + Confidence calibration + Answerability stratification"]

Specifically, EgoSAT uses Ego4D as its video source. It first utilizes rejection tags to filter 1,997 video segments (165 hours in total, covering 56 scenarios) containing visible hand-object interactions. For each interaction segment, fixed templates are used to convert temporal annotations in Ego4D into multiple-choice questions (MCQs). Each MCQ contains 1 correct answer + 2 temporally-confused hard negatives + 1 semantically-distant absurd negative (sampled based on the furthest CLIP text embedding distance after removing high-frequency words from the training set), with the four options randomly shuffled. This yields approximately 4,800 high-quality QA pairs, uniformly distributed across 6 task types.

During evaluation, all models (including offline VLMs) must adhere to a strict online prefix constraint: at time \(t\), the model can only access observed frames \(X_{1:t}\) and is strictly prohibited from peeking into future frames. Models are required to answer across six tasks spanning Present (current narration + state switch), Prospective (short-horizon + multi-step forecasting), and Retrospective (short-horizon + multi-step retrieval). Moreover, EgoSAT labels samples from the perspective of answerability (surprise and branchiness) and records the token-level confidence of the model when choosing an answer \(Conf = \max_{k \in C} p(k)\) for confidence-answerability alignment diagnosis.

Key Designs¶

1. Unified Streaming Reasoning Formalization: Three Types of Temporal Reasoning Share the Same Prefix Constraint

A streaming video is modeled as a sequence of short segments \(X_{1:t} = \{x_1, \ldots, x_t\}\). Given a textual query \(Q\) and candidate options \(C\), the model \(\mathcal{F}_\theta\) outputs \(\hat{A} = \mathcal{F}_\theta(Q, X_{1:t}) \in C\). The difference between the three reasoning types lies solely in the target time steps pointed to by \(Q\):

Present modeling: \(Q\) targets the current segment \(x_t\). The model answers using only \(x_t\) and must determine whether an interaction is occurring (interaction presence).
Prospective modeling: \(Q\) targets a future segment \(x_{t+\tau}\;(\tau > 0)\). The model can only answer using \(X_{1:t}\) and must determine whether forecasting is feasible under current observations.
Retrospective modeling: \(Q\) targets a past segment \(x_{t-\tau} \in X_{1:t}\). The model must localize and retrieve the relevant historical context from accumulated observations.

This unified formulation naturally channels the previously isolated video QA, online narration, and activity prediction tasks into different temporal directions under the same framework, enabling cross-task capability diagnosis. Note that the authors intentionally use "prospective/retrospective" (rather than "anticipation/memory") to emphasize the streaming prefix constraint—the answer must be given under partial observations, and the model must judge by itself whether the timing of answering is appropriate.

2. Answerability Quantification: Dual Metrics of Surprise and Branchiness

The core dilemma of prospective modeling lies in the fact that under partial observations, some future events are inherently unpredictable (e.g., the verb "pick up" can be followed by multiple equally likely subsequent actions like "put down", "hand over", or "put in"). In such cases, low accuracy cannot be simply blamed on poor model capabilities. The authors propose two complementary answerability metrics to distinguish between "incapable models" and "inherently unsolvable questions."

Surprise quantifies the degree of abrupt changes between consecutive frames over a short horizon. For each query time \(t\), a context window \(A = [t-\tau, t)\) and a target window \(B = [t, t+h]\) (\(h\) is the short horizon) are defined. CLIP image features and CLIP text features (the text comes from action labels overlapping with \(A\)) are extracted to calculate visual similarity \(s_v = \cos(v_A, v_B)\) and semantic similarity \(s_t = \sum_i w_i \cos(t_{a_i}, t_B)\). Since \(s_v\) and \(s_t\) come from different modalities and have incomparable scales, the authors first apply empirical distribution transformations (probability integral transform + probit) to map both to the standard normal space before performing equal-weight fusion:

\[z_v = \Phi^{-1}(\hat{F}_v(s_v)), \quad z_t = \Phi^{-1}(\hat{F}_t(s_t)), \quad z = (z_v + z_t)/2, \quad \text{Surprise}(t) = -z\]

The negative sign ensures that lower similarity (stronger mismatch) corresponds to higher surprise. This definition is entirely parameter-free and reproducible, providing a concrete mathematical foundation for answerability stratification and subsequent confidence diagnosis.

Branchiness quantifies the diversity and semantic dispersion of subsequent interactions. Let \(O\) be the current interaction covering \(t\), and \(A\) be the earliest subsequent interaction within the short window \((t, t+h]\). By traversing the dataset, the conditional subsequent distribution \(p(A \mid O)\) is constructed, characterized by two components:

\[S(O) = -\sum_i p_i \log p_i \quad \text{(Shannon entropy, measuring the diversity and evenness of subsequent actions)}\]

\[Q(O) = \sum_i \sum_j p_i p_j \, d(A_i, A_j) \quad \text{(Rao's quadratic entropy, measuring the semantic dispersion among subsequent actions)}\]

where \(d(A_i, A_j) = 1 - \cos(e_i, e_j)\) represents the CLIP text embedding distance. Ultimately, \(\text{Branchiness}(O) = S(O) \cdot Q(O)\). High branchiness means there are multiple equally likely and semantically distinct futures, making predictions inherently unreliable. Surprise captures abrupt unpredictability, while Branchiness captures multimodal-divergent unpredictability, complementing each other. The authors validate the alignment of these automatic metrics with human judgment through manual verification (branchiness 78%, surprise 84%).

3. Confidence Diagnosis: Calibration Analysis at the Answer Distribution Level

Accuracy alone cannot resolve a key question: does the model "know" when it should not answer? Drawing on multiple-choice confidence analysis methods for LLMs, the authors extract self-contained confidence from the next-token distribution when the model selects an option letter (A/B/C/D). Let \(p_{\text{raw}}(k)\) be the probability mass of letter \(k\) (aggregating all of its single-token variations), and \(m = \sum_{k \in C} p_{\text{raw}}(k)\). The conditional probability is \(p(k) = p_{\text{raw}}(k)/m\), and the confidence and uncertainty are defined as:

\[\text{Conf} = \max_{k \in C} p(k), \quad \text{Ent} = -\sum_{k \in C} p(k) \log p(k)\]

This diagnostic framework supports analysis across three levels: (a) Predictability stratification—grouping short-horizon anticipation queries into predictable and unpredictable sets using surprise/branchiness, to test if the model assigns lower confidence to unpredictable queries; (b) Temporal distance calibration—in multi-step forecasting/retrieval, as lead/lag increases (\(\tau \rightarrow 2\tau \rightarrow 3\tau\)), theoretical answerability decreases, and the fitted confidence curve slope should be < 0; otherwise, the model's self-contained confidence fails to correctly track temporal difficulty; (c) Correct/incorrect confidence discrepancies—ideally, correct answers should have higher confidence while incorrect answers should have lower confidence; if the model suffers from "incorrect answers with higher confidence," it indicates a dangerously "confidently wrong" behavior.

4. ROI-Aware Streaming Visual Token Selection

To address the KV cache budget constraints of streaming VLMs, the authors introduce a simple yet effective improvement: building upon the Dynamic Token Drop (DTD) in TimeChat-Online, they incorporate egocentric priors—namely, hand and gaze regions. The intuition is: in generic egocentric interaction videos, critical information is highly concentrated in hand-object contacts and gaze points, whereas background tokens occupy a large portion of the cache budget with minimal contribution.

The implementation is completely training-free: standard MediaPipe Hand Landmarker is used offline to extract hand bounding boxes in each frame, and the GLC gaze estimator is used to extract gaze coordinates. These regions are defined as the Interaction ROI. During the token selection stage, visual tokens within the ROI are always preserved, while tokens outside the ROI are compressed using the original DTD—both share the same 6K token budget, with the 85% drop rate mainly applied to background regions. Although simple, this design directly tests an important hypothesis: whether egocentric priors can improve streaming interaction understanding without increasing the budget.

Concrete Example: Complete Workflow of Short-horizon Anticipation¶

Taking the "short-horizon anticipation" task as an example, here is the end-to-end workflow:

Video Source: Obtain an egocentric video of a kitchen scenario from Ego4D where the wearer is chopping vegetables (time \(t\)). Ego4D has already annotated the temporal boundaries of the vegetable chopping segment.
Query Timestamp Selection: Anchor \(t\) to a moment within the vegetable-chopping interaction segment. At \(\tau = 8\text{s}\) later, the wearer will "pick up a spatula."
Answerability Computation:
- Take \(A = [t-\tau, t)\) as the context window and \(B = [t, t+h]\) as the target window. Calculate \(s_v = \cos(v_{\text{chopping}}, v_{\text{spatula}})\) and \(s_t\) (based on the semantic text similarity between "chopping vegetables" and "picking up spatula"). Apply the probit transform to obtain the surprise score.
- Look up the subsequent distribution \(p(A \mid \text{chopping})\) in Ego4D and calculate branchiness. If the common subsequent actions include multiple semantically scattered actions like "picking up spatula," "washing hands," or "getting a plate," the branchiness is high; if the successor is almost always "chopping the next vegetable," the branchiness is low.
- Synthesize these values to classify the query as either "predictable" or "unpredictable."
MCQ Construction: The correct answer is "pick up a spatula." Hard negatives are sampled from neighboring interactions before or after the chopping segment (e.g., "pick up a kitchen knife," "turn on the faucet"), and absurd negatives are sampled from the cross-scenario candidate pool according to the furthest CLIP distance (e.g., "flip a page of a book"). The four options are randomly shuffled.
Model Evaluation: Input the video frame sequence \(X_{1:t}\) and the MCQ text into the model, requesting the output of the option letter. Record the accuracy and confidence \(Conf\).
Diagnosis: If the query is marked as predictable but the model's \(Conf < 0.5\) and the prediction is incorrect, it indicates a lack of temporal reasoning capacity in the model. If marked as unpredictable but the model's \(Conf > 0.9\) and the prediction is incorrect, it exposes a confidently wrong behavior—the model has no awareness of its own mistake.

Loss & Training¶

Beyond zero-shot evaluation, the authors perform light LoRA-based SFT on TimeChat-Online-7B and its ROI variants. SFT is divided into two stages:

MCQ SFT: Sample 1,200 samples from each of the 6 task training sets, mix them into a unified list, and train the model using next-token LM loss to produce structured MCQ answers (in the format of <ANS>, <VERB>, <NOUN>, <DESC>). The visual encoder is frozen.
State SFT: On top of the MCQ SFT checkpoint, 1,200 samples evaluating interaction visibility in Now Narration + 1,200 State Switch samples are added, supplemented by 120 samples from each MCQ task to maintain capabilities. The model is trained to output state formats like <STATE> INTERACTION </STATE> vs <STATE> NO_INTERACTION </STATE>.

Note: MCQ SFT and State SFT use different output schemas and cannot be trained together directly in a mixed fashion (otherwise, it leads to schema violations). This explains why the SFT model scored 0.00 in some state columns in Table 1; because the MCQ-first SFT training set contains only MCQ format data, the model cannot correctly output the pure state format required for state switching.

Key Experimental Results¶

Main Results¶

EgoSAT evaluates three groups of models: (1) offline commercial models (Gemini 2.5 Pro, Claude Sonnet 4); (2) offline open-source models (Qwen2.5-VL 72B/32B/7B, Video-LLaVA 7B); (3) online streaming models (TimeChat-Online-7B, ROI-TimeChat-Online, Flash-VStream, LLaVA-OV1.5) and their SFT variants. All offline models simulate online conditions via prefix truncation—retaining only the visual prefix \(X_{1:t}\) up to the query time \(t\) to block peeking into the future.

Table 1: EgoSAT Main Results (MCQ Accuracy)

Model	Now Narr. (MCQ Acc)	State Switch (Fg->bg)	State Switch (Bg->fg)	Short-horizon Anticipation	Multi-step Anticipation	Short-horizon Retrieval	Multi-step Retrieval
Human	73.13	60.63	63.75	70.63	63.13	76.25	73.75
Gemini 2.5 Pro	43.00	21.43	16.67	29.92	29.43	39.80	48.15
Claude Sonnet 4	38.43	11.59	12.00	32.20	29.98	40.46	39.64
Qwen2.5-VL-72B	38.86	3.95	8.64	43.06	26.00	51.87	43.46
Qwen2.5-VL-32B	39.91	2.63	7.41	37.24	25.23	50.86	40.74
TimeChat-Online-7B	33.33	0.00	0.00	29.72	26.01	37.84	31.63
ROI-TimeChat-Online	36.47	0.00	1.25	30.51	25.05	34.41	30.82
TimeChat-Online-7B (SFT)	45.59	0.00	0.00	61.63	42.79	43.90	45.43

Several key findings: (1) There is a massive gap between all models and human performance (63-76%) across all six tasks, showing that EgoSAT is far from being resolved; (2) commercial models lead on Present tasks but exhibit no obvious advantage on Prospective and Retrospective tasks—reflecting that although pre-training data can provide general visual common sense, it fails to make up for fundamental defects in streaming temporal reasoning; (3) online streaming models generally lag behind offline models of the same scale (Qwen2.5-VL-72B: 37.24 vs TimeChat-Online: 29.72 on Short Anticipation, and Qwen on Retrieval is 51.87 vs TimeChat 37.84), as KV cache compression causes information decay, which is especially aggravated in multi-step scenarios; (4) the ROI cache strategy brings a 3.14% improvement in Present tasks (36.47 vs 33.33) but degrades performance on Retrospective/Prospective tasks—possibly because these tasks require a broader background context, and the hand/gaze ROI might have discarded these cues; (5) SFT leads to significant improvements, particularly on Prospective tasks (29.72 → 61.63), showing that models have the capability to extract temporally-sensitive representations but simply lack instruction tuning to align with MCQ reasoning tasks.

Answerability Diagnosis Experiment¶

Table 2: Temporal Distance-Accuracy-Confidence Slope Analysis for Multi-step Prediction/Retrieval

Model	Anticipation Lead \(3\tau\)	Lead \(2\tau\)	Lead \(\tau\)	Conf Slope	Retrieval Lag \(\tau\)	Lag \(2\tau\)	Lag \(3\tau\)	Conf Slope
Qwen2.5-VL-72B	27.99	23.51	26.49	-0.76	47.78	42.22	40.37	-0.16
Qwen2.5-VL-32B	25.68	22.07	27.93	<0.01	42.96	40.37	38.89	<0.01
TimeChat-Online-7B	27.99	26.49	29.85	0.61	44.07	42.96	42.59	0.99
TimeChat-Online-7B (SFT)	42.91	42.91	42.54	<0.01	45.93	48.52	41.85	0.01

The ideal behavior is: as lead/lag increases (further away from the anchor event), accuracy and confidence decrease (conf slope < 0). However, the actual results are far from this: most models do not exhibit monotonic accuracy degradation, especially in Anticipation, where local visual similarities and intrinsic uncertainty often override the effect of temporal distance; confidence slopes are often inconsistent with or even opposite to the expected trend (0.61, 0.99), indicating that current VLMs' self-contained confidence does not reliably track factual answerability or temporal difficulty. SFT pulls the slope closer to 0, indicating that the model simply learns to maintain a constant confidence across a wider range instead of truly learning calibration.

Predictability stratification experiments on Short-horizon Anticipation reveal a more severe issue. After categorizing queries into predictable and unpredictable groups based on surprise/branchiness, models' accuracy is indeed higher on the predictable group (e.g., Qwen2.5-VL-72B: 37.56 vs 28.11), but the confidence gap between the two groups is tiny or even reversed—Qwen2.5-VL-72B actually shows higher confidence on the unpredictable group (73.00 vs 68.43). This shows that current VLMs can barely distinguish between predictable and inherently uncertain queries, assigning high confidence to unpredictable scenarios and frequently demonstrating "confidently wrong" behavior. While SFT suppresses the confidence of incorrect answers to some extent (TimeChat-Online SFT: conf_wrong drops from 48.68 to 43.62), it also dampens the confidence of correct answers (conf_correct rises marginally from 51.46 to 58.53); the main effect is suppressing overconfident errors rather than improving calibration precision.

Key Findings¶

Human upper bound is far from being reached: Humans significantly outperform the strongest models across all six tasks (e.g., Short-horizon Anticipation 70.63 vs Qwen2.5-VL-72B 43.06). However, humans also score lower in State Switch (60-64%) and Multi-step Anticipation (63.13%), confirming that interaction boundary sensitivity and insufficient observations are inherent challenges in streaming understanding.
The paradoxical effect of ROI caching is highly thought-provoking: Hand/gaze ROI improves Present tasks (+3.14%) but harms Retrospective and Prospective tasks—implying that background regions carry critical causal signals necessary for temporal dynamics (e.g., scene transitions implying upcoming interactions) that are independent of hand-object interactions. This challenges the intuition that "egocentric only requires looking at the hands."
The immense effect of SFT warrants scrutiny: The 30+ point gain on Prospective tasks may partially stem from the model learning to exploit the statistical biases of MCQ options—even though blind baselines show text-only is unreliable, the MCQ format itself restricts the decision space compared to open-ended generation.
State Switch reveals extreme asymmetry: Most models find it easier to detect "entering into interactions" (bg->fg) than "exiting interactions" (fg->bg), suggesting that visual cues indicating interaction onset (hands appearing, objects being touched) are more prominent than offset cues (hands leaving).
Memory proxy experiments show: Even when offline models are provided with longer, sparse history (keyframes + captions), the benefits are non-monotonic or yield no improvement. Simply "appending more past context" cannot solve streaming understanding—adaptive memory selection, compression, and retrieval mechanisms are required.

Highlights & Insights¶

The design of dual answerability metrics is highly ingenious: Surprise and Branchiness characterize unpredictability from two complementary dimensions (abrupt transitions and divergence). This elevates the evaluation framework from simple "correct/incorrect" to a two-dimensional framework of "should it answer + how well does it answer." The transformation of empirical distributions (PIT + probit) to map similarities across distinct modalities into a unified space is parameter-free, reproducible, and fully automated—serving as a direct paradigm reference for future benchmark designs.
Unearthing "confidently wrong" behavior exposes the vulnerability of VLM evaluation: Most VLM benchmarks look only at final accuracy. EgoSAT systematically reveals how widespread the "confidently wrong" behavior is in streaming egocentric scenarios. This essentially argues that VLM calibration is extremely fragile under partial observations and intrinsic temporal uncertainty. For safety-critical AI assistants (such as assisting visually impaired people in identifying hazards), this miscalibration is intolerable.
The double-edged sword of ROI caching offers a transferable lesson: Hand+gaze ROI is helpful for Present tasks but harmful for Retrospective/Prospective ones, revealing a general design principle: token budget allocation strategies must adjust dynamically according to the temporal direction of the task. This can be transferred to memory management designs for any streaming VLMs requiring temporal reasoning.
Limitations of SFT hint at the need for new training paradigms: SFT substantially boosts accuracy but hardly improves confidence calibration (with conf slopes remaining close to 0), indicating that supervised fine-tuning naturally fails to optimize uncertainty awareness. This provides a strong motivation for RL-based calibration training (which the paper also notes as future work) to serve as a general post-training calibration paradigm.

Limitations & Future Work¶

Limitations acknowledged by the authors: Currently, most evaluations still rely on offline models simulating streaming via prefix truncation. Models natively supporting streaming reasoning are scarce and mostly small-scale (7B) with weak baseline temporal reasoning capabilities. Answerability reasoning (especially calibrated confidence estimation) remains an open challenge, and SFT is far from sufficient.
Inherent limitations of the MCQ format: While MCQs facilitate deterministic evaluation and confidence extraction, the hard negatives in the four options still provide the model with temporal anchoring clues—which users do not provide in realistic streaming interactions. Open-ended experiments (Appendix) reveal that Gemini 2.5 Pro's verb-noun pair exact match rate is only around 4%, exposing a massive chasm between MCQ and actual performance.
Granularity of answerability annotations: Both surprise and branchiness are automatic metrics based on Ego4D annotations, with manual verification covering only a small portion of the samples (78%/84% alignment). For fine-grained action sequences (e.g., microscopic manipulations in cooking), these coarse-grained metrics may fail to sufficiently capture subtle variations in predictability.
Design flaws in the state switch task: State Switch requires models to identify "before-switch" and "after-switch" states in two separate queries, counting as a success only if both are correct. This joint constraint is overly strict, resulting in near-zero scores for almost all native streaming models (without SFT), failing to differentiate actual differences in model capabilities.
Directions for improvement: (1) Introduce RL-based training to enable joint calibration of answers and confidence, utilizing the Brier score or ECE as reward signals; (2) design adaptive memory selection mechanisms to dynamically balance hand/gaze ROIs with background contexts under token budget constraints rather than using static strategies; (3) upgrade open-ended evaluation from an auxiliary supplement to a first-class metric, leveraging LLM judges + structured extraction to replace rigid string matching.

vs. Existing Streaming VLM Benchmarks (StreamingBench, OVO-Bench, SVBench): These benchmarks focus on generic streaming video understanding, are not tailored for egocentric scenarios, and do not explicitly evaluate answerability. EgoSAT's uniqueness lies in simultaneously requiring egocentric interaction awareness, multi-level temporal reasoning, and answerability diagnosis.
vs. Egocentric VQA Benchmarks (EgoSchema, EgoTaskQA, QAEgo4D, EgoLifeQA): Though focused on egocentric, they evaluate models with offline full-frame observations without distinguishing predictable/unpredictable queries or conducting confidence calibration analysis. EgoSAT is the first to introduce strict online prefix constraints and answerability stratification in egocentric scenarios.
vs. Streaming VLM Systems (TimeChat-Online, Flash-VStream, StreamingVLM): These works focus on system designs (token compression, memory propagation, KV cache management). EgoSAT provides a unified benchmark and diagnostic tools for them, revealing that token budget management requires a dynamic balance between "preserving hand-object interaction evidence" and "preserving background temporal cues" rather than a one-size-fits-all compression.
vs. Confidence Calibration Research (LLM calibration literature): EgoSAT introduces LLM confidence calibration into egocentric video understanding for the first time, exposing that calibration fails even more severely under partial observations (as uncertainty stems not only from the model's internal knowledge gaps but also from insufficient observations and intrinsic future randomness). This opens up a new difficulty tier for calibration research.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The first egocentric benchmark to unify retrospective/present/prospective reasoning under streaming prefix constraints. The combination of answerability quantification and confidence diagnosis introduces entirely new evaluation dimensions.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 14+ models (commercial + open-source + streaming + SFT + ROI variants + humans) across 6 task types, supported by various diagnostic tables (accuracy/confidence slope/predictability stratification/memory proxy sensitivity/option analysis/blind baseline/open-ended). The appendix experiments are highly detailed.
Writing Quality: ⭐⭐⭐⭐ The task formalization and answerability definitions are exceptionally clear, and the experimental analyses are well-structured. However, the main table contains quite a few columns, and the narrative of some tables in the appendix (e.g., option shuffling validation) feels slightly loose.
Value: ⭐⭐⭐⭐⭐ Sets a new standard for key evaluative aspects of streaming egocentric VLMs. The issues unmasked by calibration diagnosis have broad implications for safety and reliability; the dual answerability metrics represent a prime reference paradigm for future benchmark designs.