Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/lingli1724/Egopoint
Area: Multimodal VLM
Keywords: egocentric vision, deictic gestures, visual grounding, spatial reasoning, visual chain-of-thought

TL;DR¶

The paper introduces EgoPoint-Ground with 15,338 images and SV-CoT, which separates deictic grounding into direction parsing, ray-based filtering, and semantic verification, achieving real-world EDG [email protected] of 0.818 versus Qwen3-VL-8B's 0.701, a gain of 11.7 percentage points.

Background & Motivation¶

Conventional referring expression comprehension usually assumes that language is sufficiently specific, distinguishing targets through color, position, and category. A smart-glasses wearer, however, is more likely to point and ask what something is than to verbalize information already conveyed through an action. What is missing is not a better object name but the correspondence between a deictic expression and a particular instance in the scene. Recognizing every object in an image therefore does not imply understanding which one the user intends. Language-centered datasets such as RefCOCO do not fully test this joint disambiguation through gesture and speech.

Finger direction may appear sufficient, but egocentric images contain occlusions, similar instances, and perspective compression. A pointing path can pass through several objects, making geometry alone ambiguous; a phrase such as "that boy" may likewise fail to distinguish people in a crowd. EgoPoint-Ground contains an average of 2.8 candidate objects per image, with at least 2 same-category objects in 63.7% of samples and moderate or severe target occlusion in 42.5%. These statistics turn the problem from simple gesture recognition into instance-level multimodal inference. The paper consequently needs both paired hand-target annotations and separate measurements of linguistic cues and hand-location prompts.

The authors build egocentric deictic data and impose an explicit reasoning order: inspect the hand, follow its direction, and verify the referent using language. This is not a newly trained hand-pose network but a more structured localization process for an existing multimodal large language model. The dataset, evaluation tasks, and zero-shot reasoning baseline form complementary contributions; the work should not be reduced to a prompting technique alone. Core Idea: use gestures to narrow spatial candidates before checking them against language, grounding natural underspecified references in a concrete target.

Method¶

Overall Architecture¶

The central EDG task takes an egocentric image, the pointing hand's 2D bounding box, and an underspecified linguistic referent, then outputs the target object's 2D bounding box. The hand box indicates which hand deserves attention, but it supplies neither the fingertip direction nor the target box directly. Although the dataset includes hand keypoints, the defined EDG input does not require passing their coordinates to the model. SV-CoT performs zero-shot inference with Qwen3-VL: it interprets hand orientation inside spatial anchors, simulates a pointing ray, and checks whether a target is semantically supported. If no candidate meets the semantic confidence requirement, the framework can return no target instead of forcing a selection. Dataset construction below is offline; the remaining nodes show inference data flow, without an additional parameter-training stage.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    DATA["Hybrid Data and<br/>Task Decomposition"] --> INPUT["Image, Hand Box,<br/>Linguistic Referent"]
    INPUT --> DIR["Spatial Anchors and<br/>Direction Parsing"]
    DIR --> RAY["Ray-Based<br/>Candidate Filtering"]
    RAY --> CHECK["Semantic Verification<br/>and Rejection"]
    INPUT -->|Image and language context| CHECK
    CHECK --> OUT["Target Box or No Target"]

Key Designs¶

1. Hybrid Data and Task Decomposition

Data acquisition combines smart-glasses captures, directed image generation, and edits that transform existing third-person images into egocentric pointing scenes. For real captures, collectors describe scenes verbally; a language model organizes the descriptions, and humans review the text and annotate hand and target boxes. Generated images supplement occlusions and lighting conditions that are difficult to capture consistently, but they still require human screening before becoming trusted annotations. Edited images particularly require correction when the generated finger does not point toward the intended target: visual plausibility alone does not guarantee valid referential supervision. A pose estimator supplies auxiliary hand keypoints for human review, while fine-grained captions support the construction of deictic question-answer pairs. The resulting annotations cover the whole image, the target, the hand, and their referential relationship rather than category labels alone.

Section 2.1 reports 6,035 real-captured, 5,976 synthetic, and 3,327 LLM-generated images, totaling 15,338. The mixed split combines sources and partitions them into training, validation, and test sets at 7:1:2; the domain-adaptive split reserves real captures for testing and synthetic/generated data for training. This is a dataset protocol, not evidence that SV-CoT was fine-tuned on those training images. EDG supplies both the hand box and language; D-REC removes the explicit hand box; POG removes the linguistic referent; D-VQA asks questions about the pointed-at object. Because D-REC still includes the image, "language-only" means no explicit hand-box prompt, not demonstrated removal or masking of the visible hand. These comparisons primarily diagnose input cues rather than individually ablating SV-CoT's internal stages.

2. Spatial Anchors and Direction Parsing

The hand box describes a continuous image region through position, width, and height, whereas a language model processes discrete tokens. The authors quantize each box coordinate into 1,000 spatial bins and inject the resulting discrete spatial anchors into the multimodal input. The purpose is not to encode the target answer but to turn "inspect this hand" into a spatial reference the model can use. The model then interprets the hand's appearance and orientation inside that region to infer a 2D pointing direction, converting local hand posture into a scene-level spatial prior. Consequently, a hand box cannot determine pointing direction by itself; the original image remains necessary. The extracted coordinate-quantization equation is corrupted, so this note retains the explicitly stated bin count and interface role without inventing a normalization formula.

Direction parsing is also the first vulnerable stage: occlusion or strong perspective distortion can send subsequent filtering in the wrong direction. The transferable idea is to require a spatial basis before object selection instead of directly producing a plausible-looking box. The direction remains a model-inferred variable, not a calibrated 3D pointing measurement. The paper describes the intermediate process as a reasoning chain of linguistic tokens and does not establish that an independent geometry solver executes it.

3. Ray-Based Candidate Filtering

After estimating direction, the framework starts a virtual ray at the hand-box center and retains candidate visual regions that intersect it. Section 4.1 gives the following core geometric relationship, with the hand center as the origin and the direction vector controlling its extension.

\[ L(t)=P_{\mathrm{hand}}+t\cdot\mathbf{v} \]

The ray is a spatial filter rather than the final answer: semantically similar objects elsewhere in the image should not receive the same priority as objects near the pointing path. Its intermediate output is a set of candidate regions, not a free-form description of the entire image. Several objects may remain along the same direction, which is precisely why linguistic verification follows. The paper does not adequately specify the detector producing candidates, numerical execution of ray intersections, or whether the entire operation is realized through textual model reasoning. Figure 6 should therefore be read as the structured inference design, without assuming an additional hard geometric filter whose implementation is fully documented.

4. Semantic Verification and Rejection

The final stage checks whether candidate regions satisfy the category, attributes, and semantic requirements of the question or referent. The paper describes a candidate-query semantic consistency score: select the highest-scoring candidate, but output its target only when that score reaches a threshold. Otherwise, return an empty result, consistent with the dataset's 2,171 negative samples. Rejection matters because a visible pointing hand does not guarantee that a determinate target exists within the image. Gesture supplies a spatial constraint and language supplies a semantic constraint; both should be satisfied.

The paper does not provide a concrete computable implementation of the consistency score, a threshold value, or complete handling of an empty candidate set. This note therefore does not describe the score as a validated class probability or rejection as a calibrated safety guarantee. The output contains the reasoning chain and final target box, making this a zero-shot inference structure rather than a new supervised loss. For D-VQA, the paper defines target-related question answering, but Table 3 has no SV-CoT row; localization scores cannot establish an improvement in its question answering.

A Worked Example¶

Consider an illustrative scene in which a user points into a crowd and asks what that boy is wearing, with other people and nearby objects also visible. Spatial anchors locate the user's hand, direction parsing estimates its orientation, and ray-based filtering excludes objects away from the pointing path. If both a person and a background object lie near that path, the semantic constraint "boy" helps reject the category-incompatible candidate. D-VQA subsequently answers the clothing question about the identified person; finding a person box does not guarantee a correct attribute answer. If none of the candidates matches the description, the framework permits rejection instead of treating the nearest object as the user's inevitable referent. This is a process illustration, not a reported step-by-step trace, and it introduces no fabricated candidate counts or confidence scores.

Loss & Training¶

Section 4.1 explicitly defines SV-CoT as a zero-shot inference framework on Qwen3-VL, without proposing a new training loss or optimizer configuration. The dataset provides training splits and rich supervision, but those resources should not be presented as fine-tuning performed by this method. The results label SV-CoT only as Qwen3-vl without specifying parameter size, so not every difference can be guaranteed to compare strictly matched backbone sizes. The main text also omits a complete prompt template, inference budget, sampling settings, and rejection threshold; reproduction still requires consulting the authors' implementation.

Key Experimental Results¶

Main Results¶

Localization uses [email protected], [email protected], and [email protected]: the fraction of samples whose predicted box has IoU with the ground-truth box above the corresponding threshold. The following selection comes from Table 2 on page 9; scores are proportions, Hybrid and Real-World are reported separately, and model names follow the source table.

Model	Hybrid [email protected]	Hybrid [email protected]	Real-World [email protected]	Real-World [email protected]
Qwen3-VL-4B	0.702	0.622	0.699	0.616
Qwen3-VL-8B	0.676	0.598	0.701	0.593
Qwen2.5-VL-7B	0.653	0.563	0.688	0.533
LLaVA-OneVision-4B	0.475	0.145	0.431	0.129
doubao-seed-2.0-lite	0.688	0.641	0.847	0.819
qwen3.6-flash	0.725	0.624	0.745	0.667
SV-CoT [Qwen3-vl]	0.756	0.718	0.818	0.727
Human Study	0.914	0.846	0.918	0.874

The real-world comparison of 0.818 against 0.701 gives 11.7 percentage points; the hybrid comparison of 0.756 against 0.702 gives 5.4 percentage points. Those open-source references are Qwen3-VL-8B and Qwen3-VL-4B, respectively, not the same baseline across both splits. Real-world doubao-seed-2.0-lite reaches 0.847, exceeding SV-CoT's 0.818, so the abstract's improvement should not be expanded into a claim of leading the entire table.

Ablation Study¶

The following selection is from Table 4 on page 13, comparing D-REC without the explicit hand-box prompt against POG without the linguistic referent. Unlike Table 2, the source table does not label these results as Hybrid or Real-World, so they are not assigned to either split or used to calculate cross-table paired drops.

Model	D-REC [email protected]	D-REC [email protected]	POG [email protected]	POG [email protected]
Qwen3-VL-4B	0.684	0.605	0.612	0.532
Qwen3-VL-8B	0.648	0.561	0.595	0.514
LLaVA-OneVision-8B	0.258	0.049	0.181	0.025
SV-CoT [Qwen3-vl]	0.750	0.694	0.639	0.556
Human Study	0.926	0.819	0.916	0.806

SV-CoT obtains POG [email protected] of 0.639 versus Qwen3-VL-4B's 0.612, a gain of 2.7 percentage points; D-REC increases from 0.684 to 0.750, a difference of 6.6 percentage points. These results support structured reasoning with restricted input cues, but cannot attribute the gains individually to ray filtering, quantization, or rejection.

Key Findings¶

The D-VQA evaluation in Table 3 on page 10 tests semantic understanding beyond localization; the selection below retains the highest-accuracy baseline, Qwen3-VL-8B, and human reference. Acc. denotes accuracy and M-F1 denotes macro F1, with macro-averaging reducing the dominance of class imbalance in evaluation.

Model	Hybrid Acc.	Hybrid M-F1	Real-World Acc.	Real-World M-F1
LLaVA-OneVision-8B	0.531	0.519	0.544	0.525
Qwen3-VL-8B	0.494	0.484	0.517	0.510
Human Study	0.811	0.631	0.877	0.756

The gap between real-world question-answering accuracy of 0.544 and the human score of 0.877 shows that identifying the referent and answering its attributes are not equivalent. Localization generally deteriorates at stricter IoU thresholds, demonstrating that finding roughly the correct region is insufficient for a tight and accurate target box.

Highlights & Insights¶

Joint annotation of the hand, target, and language relationship makes reference understanding independently measurable rather than an incidental aspect of object detection.
Spatial filtering and semantic verification serve different purposes: the former removes spatially irrelevant candidates, while the latter resolves category ambiguity along a shared path. This better matches natural interaction than merely increasing descriptive language.
Negative samples and explicit rejection avoid assuming that every pointing action has an answerable target. This is relevant to low-confidence smart-glasses interaction, although rejection quality still needs separate evaluation.

Limitations & Future Work¶

The authors explicitly restrict the benchmark to canonical index-finger pointing in static images, leaving complex gestures and continuous trajectories unexplored; severe occlusion and distant targets remain difficult.
Data collection involved 17 participants, and person accounts for 47.1%. These are distributional conditions for generalization assessment, not evidence that the total category count ensures coverage of all users and scenes.
Inference depends on a supplied hand box, while deployment must also detect the correct pointing hand; the reported results do not clearly isolate errors from that prerequisite.
Equations 2–4 are corrupted in the extracted text, and several implementation parameters are unspecified. This note only explains verifiable mechanisms rather than reconstructing damaged equations or inventing parameters.
The paper claims a localization gap exceeding 26% between baselines and humans, but this does not consistently match the best models and metrics in Table 2; the note relies on explicit table entries.
The ethics section requires researcher certification and formal licensing. A code link does not imply unrestricted dataset use, and release status was not checked online for this note.

vs RefCOCO / RefCOCO+ / RefCOCOg: Conventional REC localizes objects through language descriptions; this paper emphasizes complementary underspecified language and physical pointing, without replacing every descriptive grounding scenario.
vs YouRefIt: Earlier work already studies embodied reference understanding through language and gesture. This paper focuses more specifically on egocentric views and supplies hand-pose and deictic-QA annotations, so priority claims should remain limited to the authors' stated scope.
vs GroundingGPT / Ferret: A region-grounding interface does not guarantee association between hand orientation and a target. The implication is to evaluate spatial reference directly rather than infer interaction competence from generic grounding ability.
Future research direction: Add hand-box noise, multiple targets on the same ray, and answerability evaluation, separating direction, semantic-selection, and rejection errors. This is a suggestion from the note, not an experiment completed in the paper.

Rating¶

Novelty: 4/5. The egocentric deictic dataset and task suite offer clear value, while the reasoning framework itself is lightweight.
Experimental Thoroughness: 3/5. Multiple models, tasks, and real-world settings are covered, but internal ablations, rejection-specific metrics, and sufficient reproduction details are missing.
Writing Quality: 3/5. The problem and pipeline are clear, but some claims exceed table support, and corrupted equation extraction limits verification.
Value: 4/5. Useful for grounding references in smart-glasses and multimodal interaction research, but not sufficient evidence of deployment readiness.