EgoCogNav: Cognition-aware Human Egocentric Navigation

Conference: ECCV 2026
Paper: ECCV paper page
Area: Robotics & Embodied AI
Keywords: Egocentric navigation, perceived path uncertainty, trajectory prediction, head motion, cognition-conditioned decoding

TL;DR

EgoCogNav treats perceived path uncertainty as a supervised internal state that conditions joint trajectory and head-motion forecasting, reducing ADE/FDE by approximately 3.8%/5.0% against an adapted EgoCast on the CEN held-out test set while achieving a Spearman correlation of 0.788 for uncertainty prediction.

Background & Motivation

A person approaching a junction in an unfamiliar building does not necessarily continue at their current velocity: they may stop to read signs, turn their head to find clues, or return to an earlier junction. Conventional trajectory prediction primarily uses motion history and environmental constraints, explaining where someone can walk without necessarily capturing whether they know which direction to take. Third-person or bird's-eye observations can also expose information outside the pedestrian's actual field of view, obscuring the cognitive difficulties of navigating with partial observations.

Perceived path uncertainty here means difficulty choosing among possible actions. It is neither a neural network's confidence in its own output nor the statistical variance of sampled trajectories. Existing cognitive wayfinding models often encode signage, route-choice counts, and visibility as rules, whereas egocentric motion predictors can use video and sensors but lack continuous cognitive labels. The challenge is therefore not simply to provide another input: it is to make a person's hesitation a learnable state that influences motion forecasting.

The authors collect CEN by asking participants to continuously report uncertainty while navigating, then use that signal both to supervise the shared representation and to modulate decoding. Core Idea: estimate current perceived uncertainty from egocentric perception and motion history, then combine learned navigation patterns with uncertainty-conditioned decoding to capture the perception-cognition-motion relationships behind hesitation, scanning, and backtracking rather than motion inertia alone.

Method

Overall Architecture

The inputs are 3 seconds of egocentric video, body-frame motion increments, head rotations, two-dimensional gaze points, and the goal's distance and direction in the current body frame. All streams are synchronized at 10 Hz: 30 observed steps are used to predict 10 future steps over 1 second, alongside an estimate of current perceived uncertainty.

The pipeline comprises dual-stream temporal late fusion, gradient-coupled uncertainty estimation, navigation-pattern memory, and uncertainty-conditioned decoding. The fused state branches into cognitive-state estimation and memory retrieval; the uncertainty scalar does not directly query memory, and the branches meet at conditional decoding. Separate heads forecast body motion and head rotations. This is human behavior forecasting, not an already deployed closed-loop robot navigation policy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Video, motion, head pose,<br/>gaze, and navigation goal"] --> Fusion["Dual-stream temporal<br/>late fusion"]
    Fusion --> Uncertainty["Gradient-coupled<br/>uncertainty estimation"]
    Fusion --> Memory["Navigation-pattern memory"]
    Uncertainty ~~~ Memory
    Human["Human uncertainty labels<br/>Training supervision only"] -.-> Uncertainty
    Uncertainty -->|Predicted cognitive state| Decode["Uncertainty-conditioned<br/>decoding"]
    Memory -->|Memory-augmented features| Decode
    Decode --> Output["Future trajectory and head motion"]

The dashed edge denotes training supervision, while solid arrows denote inference-time data flow. Participants do not need to supply subjective ratings at inference time.

Key Designs

1. Dual-stream temporal late fusion: preserve modality-specific temporal patterns before forming a shared state

The perception stream resizes each frame to 224×224, extracts a 384-dimensional CLS descriptor using frozen DINOv2, projects it to 512 dimensions, and applies a 2-layer Transformer. The action stream combines body translation and heading increments, continuous 6D head rotations, normalized gaze coordinates, and goal features, projects them to the same width, and applies a 4-layer Transformer. Both streams use sinusoidal positional encoding. Encoding goal bearing with its sine and cosine avoids treating the angular wraparound as an abrupt change in direction.

Each stream is temporally mean-pooled, and the resulting vectors are concatenated into 1024 dimensions before a linear layer and layer normalization produce a shared 512-dimensional state. This late fusion lets visual changes and body movements develop their own temporal representations before they are combined. Mean pooling still compresses brief events, however. Furthermore, the comparison with the early-fusion baseline also changes the cognition-aware modules, so that comparison cannot attribute all gains specifically to late fusion.

2. Gradient-coupled uncertainty estimation: let human decision difficulty shape motion representations

A two-layer MLP and sigmoid map the shared state to current perceived uncertainty in [0,1]. Supervision comes from participants continuously reporting uncertainty with an Xbox controller, not from entropy computed by the model. Because the uncertainty head shares its encoder with the trajectory and head-pose tasks, its regression loss updates the same representation, encouraging features that explain both subjective difficulty and impending behavioral changes.

This distinction matters for the ablation study: adding uncertainty prediction can improve trajectory and head-motion forecasts even before conditional decoding is enabled, because auxiliary supervision already changes the shared representation. It does not establish that one scalar fully captures a person's mental state, nor does correlation with scanning or backtracking establish causality. The model estimates the current cognitive state; its future outputs are motion sequences, not a forecast of the entire future cognitive-state curve.

3. Navigation-pattern memory: add training-distribution patterns rather than remembered route episodes

With only 30 observed steps, the model cannot infer every relevant navigation behavior from the immediate clip. The authors introduce 16 learnable navigation-pattern vectors, each 256-dimensional. A projection of the shared state queries these patterns through cross-attention; the retrieved context is projected back to the shared feature space and injected through a residual connection. Zero initialization of the output projection prevents random memory features from disturbing the initial model, allowing their contribution to emerge during training.

This memory is a parameterized pattern bank updated during training, not an online record of the route a particular user has just traveled and not retrieval of actual episodes from a long-video database. Its query comes from the fused state, while predicted uncertainty controls later modulation rather than serving as the sole retrieval key. The bank can supply behavioral priors for similar situations, but it cannot guarantee knowledge of the previous decision point in the current environment, which helps explain the paper's backtracking failure case.

4. Uncertainty-conditioned decoding: change feature processing rather than merely add another output

After memory augmentation, a two-layer MLP with SiLU maps predicted uncertainty to channel-wise scale and shift parameters for adaptive layer normalization, forming uncertainty-conditioned decoding (UCD). Similar scene and motion features need not undergo the same decoding transformation when someone is confident about the route versus when they are hesitating. Memory provides reference patterns, and UCD determines how these patterns are processed under the current cognitive state.

The modulation network is zero-initialized, making the initial scale increment and shift zero, so there is initially no additional cognitive modulation. Strictly, layer normalization remains present, so this is not a numerical identity on the raw input. Two task heads then map the modulated features to 10 future steps of three-component body motion and 6D head rotation. The pipeline predicts one deterministic future; higher perceived uncertainty does not automatically produce multiple route hypotheses.

A Worked Example

Consider a pedestrian approaching an occluded junction after slowing down and looking left and right for signs over the past 3 seconds. This is an illustrative walkthrough, not an additional quantitative example reported by the paper.

The perception stream encodes occlusion and scene changes, while the action stream encodes deceleration, head turns, gaze, and the goal's relative direction. The fused state estimates current decision difficulty and separately retrieves relevant navigation priors from 16 patterns. UCD uses predicted uncertainty to modulate memory-augmented features before predicting displacement, heading, and head motion over the next 1 second.

During training, human ratings and actual future movements supervise this window jointly; testing uses only historical sensors and the goal. The model may learn hesitation or scanning near a junction, but if the correct response is to backtrack to an earlier branch, the parameterized pattern bank does not explicitly store that branch and can still produce an incorrect return route.

Loss & Training

The trajectory objective uses discounted L1 error that prioritizes near-future predictions, with discount parameter 0.98. A variance regularizer weighted by 0.3 matches per-coordinate temporal standard deviations between predicted and ground-truth motion sequences to discourage excessive smoothing. For head motion, predicted and reference 6D representations are converted into rotation matrices; the objective averages the L1 distance between the relative rotation and the identity matrix over future steps.

Uncertainty uses mean squared error against human ratings, and the three task losses receive equal weights. Based on the textual description, their combination can be written as:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{traj}}+\mathcal{L}_{\mathrm{head}}+\mathcal{L}_{U},\qquad \mathcal{L}_{U}=(\widehat{U}_t-U_t^{\mathrm{human}})^2. \]

Some symbols in cached Equations (1)–(8) are corrupted. The expression above only summarizes the clearly described loss combination; damaged cross-attention, discount exponents, and modulation equations are not reconstructed. Refer to the original typeset paper for the exact equations.

Training uses a single RTX 4090, AdamW, 300 epochs, cosine annealing, a maximum learning rate of 1×10⁻⁴, weight decay of 8×10⁻⁵, and batch size 64. DINOv2 remains frozen and its features are cached. The shared width is 512, the action/perception encoders have 4/2 layers, and memory has shape 16×256.

Key Experimental Results

Main Results

CEN contains approximately 6 hours from 17 participants across 42 sites, totaling 226k RGB frames. Outdoor capture uses Tobii Pro Glasses with GPS, while indoor capture uses Project Aria. Video, gaze, motion, and subjective ratings are aligned at 10 Hz. Additional annotations cover junctions, occlusion, crowds, spatial transitions, and behaviors including hesitation, wrong turns, backtracking, scanning, confirmation, and look-back.

The test set consists of held-out navigation recordings, with participants following the same waypoint sequence within each scenario. The paper characterizes this as generalization to held-out traversal instances, head and gaze movements, and local egocentric observations. It does not establish independent generalization to entirely new routes, sites, or participants.

ADE is the mean distance between predicted and actual positions over future steps, and FDE is the endpoint distance. Head is a rotation-matrix L1 error, not an angular error. MAE measures numerical error in perceived uncertainty, while Spearman ρ measures rank correlation. High-U denotes the top-20% high-uncertainty subset. The table preserves source precision without adding an unspecified displacement unit.

Table 1 excerpt, full test set. EgoCast* is adapted to these tasks using the same frozen DINOv2 features; these are not results from its original task.

Method	ADE ↓	FDE ↓	Head ↓	Uncertainty MAE ↓	ρ ↑
Const_Vel	0.1892	0.4257	0.0875	Not applicable	Not applicable
M_Transformer	0.1536	0.3213	0.0776	0.1247	0.683
EgoCast*	0.1092	0.2184	0.0712	0.1029	0.752
EgoCogNav	0.1051	0.2074	0.0698	0.0986	0.788

On High-U, EgoCast* obtains ADE/FDE of 0.1198/0.2369 versus EgoCogNav's 0.1155/0.2256. Full-test relative ADE/FDE reductions are approximately 3.8%/5.0%, not improvements of tens of percentage points.

Table 2 excerpt. ΔU is the difference in mean predicted uncertainty between segments with annotated navigation behaviors and neutral segments. It measures behavioral sensitivity, but alone does not establish calibration quality or a causal effect.

Method	All MAE ↓	All ρ ↑	High-U MAE ↓	High-U ρ ↑	ΔU ↑
EMU proxy	0.1887	0.081	0.1842	0.100	0.0122
PATH_U adapted	0.1857	0.195	0.1814	0.210	0.0212
EgoCogNav	0.0986	0.788	0.1017	0.636	0.0829

The EMU baseline linearly fits visual ambiguity and short-horizon behavioral variability to human labels. PATH_U uses linear regression over 5-dimensional features describing decision complexity and behavioral variability. The results favor multimodal learning over these particular adaptations, not a blanket rejection of cognitive theories or every PATH-U implementation.

Ablation Study

Table 3 excerpt. These are component combinations rather than a cumulative sequence: both "+ Memory" and "+ UCD" build on "+ U prediction."

Config	All ADE ↓	All FDE ↓	All Head ↓	High-U FDE ↓
Base module	0.1168	0.2443	0.0785	0.2630
+ U prediction	0.1121	0.2217	0.0721	0.2401
+ UCD, no memory	0.1096	0.2188	0.0712	0.2381
+ Memory, no UCD	0.1114	0.2193	0.0707	0.2384
Full model	0.1051	0.2074	0.0698	0.2256

Key Findings

• Cognitive supervision alone reduces FDE from 0.2443 to 0.2217, approximately 9.2%, and Head from 0.0785 to 0.0721, approximately 8.2%. Auxiliary supervision is itself an important source of gains.
• Combining UCD and memory achieves the lowest overall trajectory errors, supporting their complementary roles in providing patterns and cognitive modulation. The paper does not report repeated-run variance or significance tests.
• Table 4 gives uncertainty MAE of 0.0968 for "Motion+Goal+Video," lower than the full model's 0.0986, although the full model has higher ρ. Extra modalities do not improve every metric, and rank agreement is distinct from absolute numerical accuracy.
• In Table 5, look-back Head error is 0.1277 for the full model versus 0.1244 for "+ Memory." The complete architecture is not best on every behavioral subset.

Highlights & Insights

• Cognitive labels have two roles: auxiliary supervision first improves the representation, and internal conditioning subsequently changes decoding. Table 3 separates these effects instead of attributing every gain vaguely to cognition.
• Joint head and body forecasting is useful because people may scan or confirm before turning. Gaze adds information about attention beyond displacement inertia, but its value still needs per-metric evaluation.
• Distinguishing parameterized pattern memory from online episodic memory clarifies the capabilities demonstrated. The 16 learned vectors can encode recurring navigation patterns without replacing a route map or an individual's spatial memory.

Limitations & Future Work

• Reported failures include incorrect backtracking under occlusion and missed brief hesitation or look-back. Longer-horizon episodic memory and 3D/semantic maps could address the missing information more directly than simply enlarging the parameterized bank.
• The model outputs only one trajectory and head-motion sequence, although the same perceived uncertainty may accompany several plausible actions. Generative multi-hypothesis forecasting is proposed as future work, not demonstrated here.
• The dataset has 17 participants and approximately 6 hours, with fixed waypoint sequences. Strict participant-, route-, and site-held-out evaluation and multiple random seeds are needed to assess generalization and stability more strongly.
• Continuous controller ratings are useful but imperfect subjective measurements, and participants were instructed to seek cues when uncertain. Reporting burden and experimental instructions may affect natural behavior; this is a critical-reading consideration, not an effect quantified by the paper.
• Closed-loop assistive navigation and robot deployment benefits are not demonstrated, nor do intervention experiments establish that cognitive states cause particular actions. Environmental-experience analysis and predictive correlations are not safety guarantees.

Related Work & Insights

• vs EgoCast: the forecasting module is adapted to the three-task setup while keeping the visual backbone consistent. The comparison concerns forecasting architecture, not superiority on the original full-body pose task.
• vs EgoNav / LookOut: these approaches emphasize egocentric scenes, visual memory, or head-pose prediction. EgoCogNav adds continuous human cognitive labels and their role in conditional decoding rather than a more complete geometric map.
• vs EMU / PATH-U: cognitive theory and environment-feature-based wayfinding models help define the problem; this paper turns it into supervised multimodal egocentric learning. Its EMU/PATH_U comparisons use adapted proxies and should be interpreted accordingly.
• Research direction: under strictly held-out scenes and participants, compare auxiliary supervision alone, scalar conditioning, and actual route memory to test whether the gains persist beyond familiar waypoint sequences. This is a proposed study, not an established finding.

Project page: EgoCogNav. The cache provides this URL, but actual code and dataset availability has not been verified.

Rating

• Novelty: 4/5. Combining continuous subjective cognitive supervision with egocentric multitask forecasting is distinctive, while the components largely build on established architectures.
• Experimental Thoroughness: 3/5. Baselines and component, modality, and behavior analyses are included, but strict cross-domain splits and statistical stability evidence remain limited.
• Writing Quality: 4/5. Module motivations and failure cases are clear; some broad improvement statements need qualification by the non-monotonic metrics.
• Value: 4/5. The work provides a dataset and modeling entry point for cognition-aware navigation, while practical assistance benefits require closed-loop validation.