Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning¶

Conference: ECCV2026
arXiv: 2606.29334
Code: To be confirmed
Area: Human Understanding
Keywords: Gaze target estimation, hierarchical reasoning, object-level semantics, multi-scale fusion, geometric priors

TL;DR¶

This work reformulates gaze target estimation from pixel-level regression to a hierarchical reasoning problem. It first establishes candidate attention objects using object-level semantic representations, constructs a field-of-view (FOV) cone geometric prior using gaze direction to constrain the search space, and finally achieves precise localization via multi-scale residual fusion. This method achieves state-of-the-art (SOTA) performance on GazeFollow, VideoAttentionTarget, ChildPlay, and GOO-Real using only 7.1M parameters.

Background & Motivation¶

Gaze target estimation aims to predict the semantic object in a scene being observed by a gazer. It serves as a crucial task for understanding human visual attention and social interactions, with wide applications in autonomous driving, human-robot interaction, and social robotics. In recent years, substantial efforts have proceeded along two main lines. First, introducing auxiliary modalities such as depth, human pose, or 3D head information to construct multi-branch networks, which, while enhancing spatial understanding, continuously increases model complexity. Moreover, prediction performance heavily depends on the accuracy of upstream auxiliary models, limiting cross-scenario generalization. Second, adopting pre-trained vision foundation models (such as DINOv2/v3, ViT) as unified feature extractors, which offer parameter efficiency and simple architectures. However, prediction typically only utilizes features from the final layer through a shallow decoder, causing a large amount of rich spatial structure information within the hierarchical representations to be discarded.

These two types of methods share a fundamental modeling assumption, treating gaze prediction as a pixel-level regression problem that directly maps from global features to gaze heatmaps. However, human visual attention is naturally object-oriented: an observer first selects a specific semantic entity as the attention target, and then forms a precise fixation point within that entity's region. When a model lacks explicit modeling of object-level decision-making, predictions often become spatially diffuse or semantically unstable when faced with multiple competing target candidates in complex scenes. In other words, the core bottleneck of current methods is not that feature representation is insufficiently strong, but rather the lack of a structured target selection mechanism at the task-modeling level.

The key insight of this paper is that gaze target estimation is essentially a hierarchical reasoning process, which should be decomposed into "first determining 'which object to look at', and then deciding 'where on the object the gaze falls'". Based on this perspective, the authors reformulate the problem as a two-stage hierarchical reasoning task: first constructing a semantic hypothesis space of potential gaze targets in the scene, then introducing the observer's geometric constraints (head pose + eye position) to determine which spatial regions are visually reachable, and finally performing precise gaze localization within the candidate regions that satisfy both semantic and geometric consistency. Core Idea: Reformulate gaze target estimation from single-step pixel-level regression to "object-level discrimination \(\rightarrow\) region-level localization" hierarchical reasoning. This is co-operatively achieved within a unified single-modality framework by establishing candidate hypotheses via object semantic representation + constraining the search space with a field-of-view cone geometric prior + precisely localizing via multi-scale residual fusion.

Method¶

Overall Architecture¶

The proposed method is based on a frozen DINOv3 ViT-L/16 backbone and organizes its inference pipeline around three core mechanisms. First, object-level semantic representation explicitly associates scene features with discrete semantic entities. Second, the gaze direction is estimated from the head appearance and eye coordinates to construct a Field-of-View (FOV) cone geometric prior, constraining the search space to visually reachable regions. Finally, multi-scale residual fusion integrates low-level spatial details with high-level semantics, combining with the FOV prior to achieve precise localization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image + Head Region"] --> B["Object-Level Semantic Representation<br/>YOLO11x+SAM2 detection → Object tokens<br/>Cross-attention fusion with image tokens"]
    B --> C["Gaze Geometric Prior Construction<br/>Head features+Eye pos → MLP predicts gaze direction<br/>→ 120° FOV cone → Gaussian smoothing"]
    B --> D["Multi-Scale Semantic Localization<br/>DINOv3 hierarchical features → Deep-dominant<br/>+ Shallow residual compensation → FOV-guided decoding"]
    C --> D
    D --> E["Gaze Heatmap H<br/>+ In/Out Classification"]

Key Designs¶

1. Object-Level Semantic Representation: Explicitly establishing candidate gaze hypotheses during the feature encoding stage

Existing methods predict the gaze position directly from global scene features, lacking explicit modeling of potential gaze targets, which leads to unstable predictions in complex scenes with multiple competing targets or heavy background clutter. This paper introduces object-level semantic representation to address this pain point.

Specifically, the input image is first processed offline by YOLO11x and SAM2 to obtain \(N\) binary masks \(\mathbf{M}_i\) of candidate targets. The image is patchified into non-overlapping patches and projected into image tokens \(\mathbf{T}_{\text{img}}\). For each target area, masked features are encoded into object tokens \(\mathbf{T}^{(i)}_{\text{obj}}\) via fixed grid pooling (where each target is represented by \(n^2\) tokens). All object tokens and image tokens are concatenated into a unified sequence \(\mathbf{T}_{\text{fuse}} = [\mathbf{T}_{\text{img}}; \mathbf{T}^{(1)}_{\text{obj}}; \ldots; \mathbf{T}^{(N)}_{\text{obj}}]\), which is fed into a Transformer encoder for cross-token interactions. Through this interaction, semantic information of individual objects is propagated to the global visual representation, transforming pixel-based scene features into object-aware semantic representations and explicitly establishing candidate gaze hypotheses. Finally, an object-aware spatial response map \(\mathbf{O}\) is reconstructed from the fused representation and fused with the original image using channel concatenation and \(1\times1\) convolution to obtain the enhanced feature \(\mathbf{I}_{\text{enh}}\), providing a semantic foundation for subsequent stages.

2. Gaze Geometric Prior: Constructing field-of-view cone constraints using the observer's head pose

Scene semantics can suggest which objects might be looked at, but they are insufficient to uniquely determine the exact gaze point, as human attention is naturally constrained by head orientation and gaze direction, being restricted to the forward field-of-view cone. This work explicitly constructs such geometric priors to narrow down the inference space.

Given the head region image and the eye coordinates \((x_e, y_e)\), head appearance features \(\mathbf{f}_{\text{head}}\) are first extracted and fed alongside the eye coordinates into an MLP to predict a normalized 2D gaze direction vector \(\hat{\mathbf{g}}\). During training, the ground-truth target position is used to define the supervised direction \(\mathbf{g}^{gt}\) under a direction loss \(\mathcal{L}_{\text{dir}}=1-\hat{\mathbf{g}}\cdot\mathbf{g}^{gt}\). Based on the estimated gaze direction, the cosine similarity \(c_{(x,y)}\) between the unit vector \(\hat{\mathbf{v}}_{(x,y)}\) (from the eye coordinate to pixel \((x,y)\)) and \(\hat{\mathbf{g}}\) is calculated for each pixel \((x,y)\). Considering the limited field of view of human eyes, the FOV angle is set to \(\theta=120^\circ\) (determined via ablation studies), producing a cone-shaped response map \(\mathbf{C}_{(x,y)}\) (retaining values greater than \(\cos\theta\) and setting others to zero). After applying Gaussian smoothing to eliminate discontinuities along the boundary of the cone, it is combined with a learnable FOV embedding vector \(\mathbf{e}_{\text{fov}}\) via broadcasting multiplication to generate the geometrically-aware representation \(\mathbf{E}_{\text{fov}}\), which is injected into the subsequent localization stage. This prior acts as a spatial gate: only regions residing within the observer's FOV cone are allowed to generate high responses, considerably suppressing potential distractors that are out-of-field or geometrically unreachable.

3. Multi-Scale Semantic Localization: Deep-dominant + shallow-residual hierarchical fusion localization

After establishing target semantic hypotheses and geometric constraints, precise localization requires utilizing multi-scale spatial information in regions that are semantically coherent and geometrically feasible. This work adopts a deep-dominant residual fusion strategy: the enhanced feature \(\mathbf{I}_{\text{enh}}\) is fed into the frozen DINOv3 to extract features at different levels, where the deep semantic feature \(\mathbf{F}_{\text{deep}}\) serves as the primary representation, and mid-level and shallow features act as bounded residual compensations:

\[\mathbf{F}_{\text{fused}} = \mathbf{F}_{\text{deep}} + \alpha_{\text{mid}}\mathbf{F}_{\text{mid}} + \alpha_{\text{shallow}}\mathbf{F}_{\text{shallow}}\]

where \(\alpha_{\text{mid}}\) and \(\alpha_{\text{shallow}}\) are learnable fusion coefficients. The deep features dominate while shallow features purely serve as residual compensation because the deep features have already integrated object information via the object-level semantic representation. Shallow spatial details only need to perform fine-tuning within the deep semantic framework, preventing feature aliasing often caused by direct multi-scale concatenation. Subsequently, the FOV geometric prior \(\mathbf{E}_{\text{fov}}\) is merged with the fused representation to generate the geometrically-guided feature \(\mathbf{F}_{\text{guided}}\), which is processed by a lightweight decoding network to progressively restore spatial resolution, outputting the gaze probability heatmap \(\mathbf{H}\). The entire prediction is no longer merely a result of global pixel regression, but a joint outcome of object-level semantic hypotheses and observer-level geometric constraints.

Loss & Training¶

Training utilizes three joint losses for optimization: a pixel-level binary cross-entropy loss \(\mathcal{L}_{\text{hm}}\) for the gaze heatmap, where a 2D Gaussian kernel is applied at the ground-truth gaze point to construct the target heatmap; a gaze direction loss \(\mathcal{L}_{\text{dir}}\) (based on cosine similarity) to guarantee the accuracy of the FOV prior; and an auxiliary binary classification branch to determine whether the gaze target is inside or outside the frame (in/out-of-frame), corresponding to \(\mathcal{L}_{\text{inout}}\). The overall objective is formulated as: \(\mathcal{L} = \mathcal{L}_{\text{hm}} + \lambda_{\text{dir}}\mathcal{L}_{\text{dir}} + \lambda_{\text{inout}}\mathcal{L}_{\text{inout}}\), where \(\lambda_{\text{dir}}=0.5\) (determined via ablation), and \(\lambda_{\text{inout}}\) is a dataset-dependent binary hyperparameter (set to 1 when annotations are available, and 0 otherwise). The model is trained on a single RTX 5090 using the Adam optimizer with an initial learning rate of \(1\times10^{-3}\), an input resolution of \(512\times512\), and a heatmap output size of \(64\times64\).

Key Experimental Results¶

Main Results¶

Dataset	Metric	Ours	Prev. SOTA (Gaze-LLE)	Gain
GazeFollow	AUC ↑	0.961	0.953	+0.008
GazeFollow	Avg L2 ↓	55.6	58.3	-2.7
VideoAttentionTarget	AUC ↑	0.948	0.942	+0.006
ChildPlay	AUC ↑	0.987	0.977	+0.010
GOO-Real	AUC ↑	0.977	0.962	+0.015

All results are achieved with only 7.1M parameters (in contrast to Gaze-LLE which uses a 307M parameter DINOv2-giant, while the proposed method only employs ViT-L/16), demonstrating significant parameter efficiency improvements.

Ablation Study¶

Configuration	GazeFollow AUC	VideoAttentionTarget AUC	Description
Full model	0.961	0.948	Full model
w/o Object-level semantics	0.942	0.931	AUC drops significantly when removed
w/o FOV geometric prior	0.950	0.939	Performance decreases without geometric constraints
w/o Multi-scale fusion	0.953	0.941	Uses only a single layer of deep features
w/o All three components	0.925	0.912	Baseline (DINOv3 + simple decode only)

Key Findings¶

Object-level semantic representation contributes the most: Removing it causes a 1.9 AUC drop on GazeFollow, demonstrating that modeling gaze as discrete object selection is the core of the reasoning process, and pure pixel regression degenerates significantly in complex scenes.
FOV geometric prior effectively compresses the search space: Removing it leads to an approximate 1 AUC point decline. Qualitative results indicate that without this geometric constraint, heatmaps are more prone to dispersing toward distracting regions outside the gaze direction in multi-target scenes.
An FOV angle of 120° is optimal: Setting it too small (60°) segments reasonable gaze regions, while setting it too large (180°) approximates an unconstrained setup. The optimal value matches perfectly with the human visual field range.
Deep-dominant residual fusion outperforms direct concatenation: Ablation studies verify that the deep-dominant + shallow-residual design outperforms simple multi-scale concatenation by 0.3-0.5 AUC, showing stability over the fusion coefficient \(\alpha\) (performance variation <0.2 AUC within the range of 0.1-0.5).

Highlights & Insights¶

Problem Redefinition: From Pixel Regression to Object Reasoning. The most significant contribution of this work lies in shifting the modeling perspective of gaze estimation from "regressing a single point" to "reasoning about an object + locating a point". This paradigm shift is more enabling and inspiring than the specific engineering designs.
Geometric Prior without Extra Sensors. Relying exclusively on head appearance + eye coordinates, the model estimates gaze direction and constructs an FOV cone. This is accomplished entirely within single-modality RGB images. Compared with complex multi-branch and multi-modal methods, it requires fewer parameters and possesses stronger generalization.
Frozen Backbone + Lightweight Modules. The DINOv3 backbone remains frozen. All learnable modules (cross-attention fusion layer, MLP direction prediction, FOV embeddings, and decoding network) total only 7.1M parameters, validating the feasibility of the "frozen foundation model + lightweight head" paradigm on ViT-L/16 under a supervised setup.
Generality of Hierarchical Reasoning. This three-stage inference paradigm ("establishing semantic hypotheses \(\rightarrow\) filtering with physical constraints \(\rightarrow\) precise localization") can be transferred to other tasks requiring a combination of semantic understanding and spatial localization (such as proposal weeding in object detection and attention localization in vision-language navigation).
Outstanding Cross-Dataset Generalization. Without domain adaptation, the model achieves 0.977 AUC on GOO-Real with only lightweight fine-tuning. This demonstrates that combining object-level modeling and geometric priors effectively alleviates domain shift.

Limitations & Future Work¶

Misjudgments among spatially close objects with similar semantics. When multiple objects of similar appearance are placed in close spatial proximity, the more visually prominent instance might produce higher response, leading to incorrect target selection despite consistent geometric cues. Future work can explore adaptive object reasoning mechanisms (e.g., iterative attention contraction).
Misalignment between discrete annotations and continuous attention. In human-centric scenes, gaze targets often correspond to continuous regions (e.g., a person's face rather than a single pixel), yet the training supervision uses discrete point annotations. Consequently, the model tends to focus on semantically salient areas (such as faces) rather than exact annotated points. Future work may consider uncertainty-aware gaze modeling.
Multi-person scenarios are not yet covered. This method assumes a single observer in the scene; joint gaze reasoning in multi-person interactive scenarios is currently unsupported and could be expanded to multi-observer GNN interactive prediction.
Temporal information is underutilized. Although the model achieves good results on VAT using single frames, temporal gaze smoothness and attention transition trajectories in videos are not yet modeled. Incorporating temporal memory modules holds promise for further enhancing prediction stability.

vs Gaze-LLE (Ryan et al., 2025): Gaze-LLE first introduced a frozen DINOv2 backbone for gaze estimation, but utilized only the final layer of features for shallow decoding. Building on this, the proposed method incorporates three key designs—object semantic modeling, multi-scale fusion, and geometric priors—surpassing Gaze-LLE while employing a much smaller backbone (ViT-L vs ViT-g).
vs Multi-branch and Multi-modal Methods (Bao et al., ECCV 2022; Fang et al., 2021): These methods rely on extra modalities like depth, pose, or 3D head information, requiring synchronized acquisition during training and offline prediction from multiple upstream models before deployment. In contrast, the proposed method operates entirely in single-modality RGB, rendering it more lightweight with superior generalization.
vs Object-Aware Methods (Nieva et al., 2025; Jin et al., 2025): Prior works tried to incorporate object detection results as auxiliary guidance, yet only treated object information as supplementary feature additions without changing the underlying regression paradigm. In this paper, object-level semantic tokens directly participate in cross-attention fusion of feature encoding, fundamentally altering the decision-making process.

Rating¶

Novelty: ⭐⭐⭐⭐☆ The perspective of redefining the problem (regression \(\rightarrow\) hierarchical reasoning) is highly novel. While the individual components are not entirely unprecedented, their systematic integration and motivational cohesion are outstanding.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 4 benchmark tests + comprehensive ablations on components/multi-scale structures/backbones/FOV/hyperparameters + failure case analysis, forming a highly solid evaluation system.
Writing Quality: ⭐⭐⭐⭐⭐ The introduction establishes the problem definition and motivates the work extremely clearly. It articulates the necessity of redefining the task from a cognitive science perspective, and the methodological description flows coherently.
Value: ⭐⭐⭐⭐⭐ Outperforming a 307M model with just 7.1M parameters represents a parameter efficiency improvement of over 40x, which is significantly meaningful for deployment in resource-constrained environments. The paradigm shift in inference is bound to inspire future research.