Molmo-Point: Better Pointing for VLMs with Grounding Tokens¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://allenai.org/blog/molmopoint
Area: Multimodal VLM
Keywords: Visual Grounding, Vision-Language Models, Pointing, Grounding Tokens, Coarse-to-Fine Referencing
TL;DR¶
Addressing the learning inefficiency and high decoding overhead of text coordinates in vision-language models, MolmoPoint introduces a coarse-to-fine grounding mechanism via three specialized tokens (<PATCH>, <SUBPATCH>, <LOCATION>) that directly cross-attend to visual hidden features, setting new state-of-the-art benchmarks on PointBench, PixMo-Points, and ScreenSpotPro across image, GUI, and video tracking tasks.
Background & Motivation¶
Spatial grounding via pointing is a cornerstone capability for vision-language models (VLMs), powering robotic manipulation and navigation planning, graphical user interface (GUI) computer-use agents, and visual chain-of-thought counting. However, the vast majority of current VLMs point by generating coordinates as text output—either through raw coordinate number strings or discretized coordinate bin tokens—forcing the language backbone to memorize an artificial absolute coordinate grid within its textual vocabulary.
This text-coordinate paradigm suffers from three fundamental bottlenecks: first, coordinates are decoupled from visual representations, meaning that while the visual encoder and LLM hidden states have already formed rich semantic and spatial features for objects and parts, the model is compelled to translate these high-dimensional embeddings into arbitrary text digits; second, coordinate generation incurs significant token inflation, requiring up to 8 tokens per point (six digits plus whitespace and separators), which drastically slows down autoregressive decoding on dense point sets or long video sequences; third, coordinate systems generalize poorly across varying image resolutions, breaking down when subjected to inference-time aspect ratio shifts or unseen ultra-HD tiling crops.
This paper bypasses the detour of text-based coordinate generation by allowing the language model to reference visual tokens directly in the hidden feature space via cross-attention. Core idea: introduce hierarchical grounding tokens (<PATCH>, <SUBPATCH>, <LOCATION>) to cross-attend to LLM visual hidden states and raw ViT patches in a coarse-to-fine manner, combined with relative Rotary Position Embeddings (RoPE) and a no-more-points stopping token, enabling efficient and resolution-invariant spatial grounding.
Method¶
Overall Architecture¶
The pointing pipeline of MolmoPoint proceeds in three autoregressive, coarse-to-fine stages. When prompted to point, the model emits a <PATCH> token whose hidden state is projected into a query vector to score visual tokens across the image via scaled dot-product attention, selecting the coarse image patch containing the target concept. Next, it emits a <SUBPATCH> token that scores unpooled ViT patch representations within the selected coarse region to isolate a finer subpatch. Finally, a <LOCATION> token predicts the exact point within the subpatch using a lightweight linear layer over a 3×3 grid. Selected patch and subpatch embeddings are fed back into subsequent autoregressive steps, enabling the model to retain full spatial awareness of its previous predictions.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image / Video + Text Query"] --> B["Coarse Patch Selection<br/>PATCH token cross-attends to LLM visual states"]
B --> C["Fine Subpatch Refinement<br/>SUBPATCH token scores raw unpooled ViT features"]
C --> D["Grid Point Localization<br/>LOCATION token predicts 3x3 local subpatch grid"]
D --> E["Termination & Relative Position Modeling<br/>RoPE encodes offset + no-more-points stop class"]
E --> F["High-Precision Ordered Grounding Points"]Key Designs¶
1. Coarse Patch Selection: Direct Cross-Attention over LLM Visual Hidden States
To bridge the gap between semantic recognition and spatial coordinates, the vocabulary is augmented with a dedicated <PATCH> token. When generated, its hidden state \(h_p \in \mathbb{R}^{D \times 1}\) is normalized and linearly projected to form a query vector \(q_p = W_{pq}\text{Norm}(h_p)\) of dimension \(M\) (\(M=512\)). Concurrently, the LLM hidden states of all visual tokens \(H_i \in \mathbb{R}^{D \times I}\) are projected into keys \(K_p = W_{pk}\text{Norm}(H_i)\). The scaled dot-product yields an attention distribution over all candidate visual tokens:
$\(s_p = \frac{K_p^\top q_p}{\sqrt{M}}\)$
During inference, the top-scoring token \(p^* = \arg\max(s_p)\) is selected. Since each image token in Molmo2 pools 4 underlying ViT patches covering a \(28 \times 28\) pixel region, this step reliably locks down the coarse target region. When passed as an input token to the LLM in subsequent decoding, the embedding of the selected image token is added directly (\(q_p + E_i[:, p^*]\)), providing explicit grounding context to subsequent tokens.
2. Fine Subpatch Refinement: Scoring Unpooled ViT Features and Grid Localization
Because a \(28 \times 28\) pixel coarse patch is insufficient for tiny objects, dense GUI elements, or fine-grained video tracking, the model emits a <SUBPATCH> token immediately following <PATCH>. Its query vector \(q_s\) attends to the keys \(K_s\) formed from the unpooled ViT patch features \(U_s\) of the selected coarse patch, selecting the optimal \(14 \times 14\) pixel subpatch index \(s^*\). In the subsequent input step, the projected unpooled ViT feature \(W_{se} U_s[:, s^*]\) is incorporated into the token embedding.
To achieve sub-patch localization, the model emits a third <LOCATION> token. Its hidden state passes through a single linear classification layer that predicts one of 9 discrete positions arranged in a \(3 \times 3\) grid within the \(14 \times 14\) subpatch. This design achieves an effective spatial resolution of approximately 4.7 pixels. Crucially, this resolution is maintained regardless of input image dimensions or dynamic cropping setups, eliminating coordinate normalization drift on ultra-HD displays.
3. Relative Position Modeling & Termination: RoPE Relative Offsets and No-More-Points Class
When predicting multiple sequential points or tracking objects across video frames, models often suffer from duplicate pointing or runaway hallucinations that emit endless points. MolmoPoint introduces two targeted mechanisms:
First, it applies 2D Rotary Position Embeddings (RoPE) when computing <PATCH> attention scores:
$\(s_p = \frac{\text{Rot}(K_p, p_i)^\top \text{Rot}(q_p, p_q)}{\sqrt{M}}\)$
where \(p_i\) is the position of candidate image tokens and \(p_q\) is the position of the previously selected patch (or 0 for the first point). By sorting ground truth points in scanline order and masking out preceding tokens during both training and inference, the model strictly predicts ordered sequences without revisiting prior regions.
Second, a learnable static vector \(h_{done} \in \mathbb{R}^{M \times 1}\) is appended to the key matrix: \(K_p = [W_{pk}\text{Norm}(H_i);\, h_{done}]\). When no further target objects remain, \(q_p\) matches \(h_{done}\), halting generation and emitting the closing delimiter > to prevent over-generation.
A Worked Example¶
Consider the instruction: "Point to the right brake light of the left-most car":
1. The model ingests the high-resolution input and pre-caches the visual token hidden keys \(K_p\) and unpooled ViT keys \(K_s\);
2. The LLM generates the <PATCH> token; query \(q_p\) matches across visual tokens and selects token \(p^*\), locking onto the \(28 \times 28\) pixel region at the rear of the leftmost vehicle;
3. The embedding of patch \(p^*\) is fed forward, prompting the LLM to emit <SUBPATCH>; query \(q_s\) cross-attends over the 4 constituent ViT subpatches to select \(s^*\) (the \(14 \times 14\) pixel region containing the brake light);
4. The model emits <LOCATION>, classifying into index 5 in the local \(3 \times 3\) grid, which maps to absolute pixel coordinates via pre-indexed token coordinate offsets;
5. Generating the next <PATCH> token yields a top score on \(h_{done}\) (the no-more-points class), terminating generation with a closing delimiter. The entire sequence uses just 4 tokens instead of dozens of coordinate digits.
Loss & Training¶
During preprocessing, each ground-truth coordinate is mapped to a triple: (target image token index tp, ViT subpatch index ts, local grid index tl).
The training loss sums standard text autoregression loss with the cross-entropy losses from all three grounding stages:
$\(\mathcal{L} = \mathcal{L}_{\text{LLM}} + \mathcal{L}_p(s_p, t_p) + \mathcal{L}_s(s_s, t_s) + \mathcal{L}_{loc}(s_{loc}, t_l)\)$
where \(\mathcal{L}_p\), \(\mathcal{L}_s\), and \(\mathcal{L}_{loc}\) correspond to patch, subpatch, and grid location losses, respectively. The hidden dimension is set to \(M=512\). Pointing parameters use an independent learning rate matching the image-text connector parameters. During inference, caching the low-dimensional pointing keys adds minimal memory overhead, roughly equivalent to the KV cache of 1-2 transformer layers.
Key Experimental Results¶
Main Results¶
The authors evaluated MolmoPoint across three specialized variants: MolmoPoint-8B (general-purpose), MolmoPoint-Img-8B (fine-tuned on the new 36K screenshot dataset MolmoPoint-PointSyn), and MolmoPoint-Vid-4B (video-specialized). Benchmarks encompass natural image pointing (PointBench, PixMo-Points), GUI interaction (ScreenSpot-Pro, OSWorldG), and video tracking (Molmo2Track).
| Dataset | Domain | Metric | MolmoPoint-8B / Img-8B | Prev. Baseline (Molmo2-8B) | Gain |
|---|---|---|---|---|---|
| PointBench | Natural Image Pointing | Avg. Score | 70.7 | 68.7 | +2.0 |
| PixMo-Points | Multi-Target Pointing | F1 Score | 89.2 | 85.2 | +4.0 |
| ScreenSpot-Pro | High-Res GUI Grounding | Accuracy (Acc.) | 60.2 (64crops: 61.1) | 30.4 (FT baseline: 52.3) | +8.8 (vs. FT baseline) |
| OSWorldG | GUI Interaction Grounding | Accuracy (Acc.) | 70.0 | 54.1 (FT baseline: 66.1) | +3.9 (vs. FT baseline) |
| Molmo2Track | Video Object Tracking | J&F / HOTA | 62.5 / 60.0 | 56.2 / 57.5 | +6.3 / +2.5 |
| BURST-VC | Video Object Counting | Acc. / Close Acc. | 61.6 / 76.9 | 60.8 / 75.0 | +0.8 / +1.9 |
Ablation Study¶
Using an ablation pipeline fine-tuning a base captioning checkpoint on image and video pointing data for 6000 steps, the authors examined the contributions of RoPE relative encoding, the no-more-points token, and sequential point ordering:
| Config | PixMoPoint (F1) | PointBench (Avg.) | Molmo2-VC (Correct) | Molmo2-VC (Close) | Overcount Rate (Overcount ↓) | Note |
|---|---|---|---|---|---|---|
| Full Model (Molmo2-P-Ablation) | 85.2 | 67.8 | 58.0 | 36.6 | 3.6% | Baseline full ablation configuration |
| w/o rotary (remove relative RoPE) | 84.5 | 67.6 | 56.8 | 33.0 | 4.5% | Degraded spatial context, lower video counting |
| w/o no-more-points (remove stop class) | 84.7 | 66.6 | 52.3 | 32.8 | 10.3% | Hallucinates excessive points, overcounting nearly triples |
| w/o point sorting (randomized order) | 83.6 | 71.2 | 40.0 | 24.4 | 3.2% | Video tracking collapses; 18-point drop in video count |
Key Findings¶
- Substantial Sample Efficiency Advantage: With only 8,192 training examples, MolmoPoint outperforms text-coordinate baselines by over 20 points in F1 score on PixMo-Points, reaching near-saturation with a fraction of the full dataset.
- Robustness to Extreme Resolution Upscaling: When increasing inference crops to 64, text-coordinate models suffer massive degradation (-30 to -60 points on ScreenSpot-Pro) because absolute coordinates lose alignment with new tile layouts; MolmoPoint-Img-8B instead improves from 60.2% to 61.1%.
- Elimination of Degenerate Video Overcounting: Omitting the
no-more-pointsclass caused the video overcounting rate to jump from 3.6% to 10.3%, highlighting the importance of an explicit halting mechanism when selecting visual tokens.
Highlights & Insights¶
- From Text Coordinates to Token Referencing: Instead of forcing the LLM to serialize visual concepts into character strings, direct cross-attention to visual tokens leverages native representation alignment, delivering faster learning and superior spatial accuracy.
- Decoupled Multi-Scale Hierarchy: Structuring pointing into LLM patch selection, unpooled ViT patch refinement, and subpatch grid regression balances global receptive fields with fine-grained 4.7-pixel localization.
- Dramatic Token and Latency Reduction: Slashing per-point token count from 8 to 3 substantially reduces KV cache memory consumption and decoding latency, especially for dense video trajectories and GUI agents.
Limitations & Future Work¶
- Counting Inaccuracies on High-Frequency Repetitive Patterns: The model occasionally exhibits off-by-one errors when enumerating highly repetitive, densely packed objects such as texture grids.
- Restricted to Visual Modalities: The current grounding token formulation targets 2D image and 3D spatio-temporal video features, without extending to referencing textual tokens or temporal audio spans.
- Future Directions: Integrating reinforcement learning (RL) to refine spatial rewards, and combining grounding tokens with dense mask decoders to support unified point-box-mask multimodal outputs.
Related Work & Insights¶
- vs. Molmo / PixMo: Original Molmo relies on autoregressive string coordinate generation (e.g.,
[x, y]), incurring high token budgets and vulnerability to resolution shifts; MolmoPoint outperforms Molmo2 by 4.0 F1 points on PixMo-Points and 2.0 points on PointBench. - vs. PaDT / GUI-Actor: PaDT relies on external decoders for pixel grounding; GUI-Actor uses single-stage cross-attention without hierarchical refinement and targets only single GUI points. MolmoPoint offers end-to-end multi-scale refinement for images, GUIs, and videos.
- vs. Proprietary Models (Gemini 3 Pro / GPT-5): Without proprietary data or heavy RL tuning, open-weight MolmoPoint-8B achieves 70.7 on PointBench and 61.1 on ScreenSpotPro, matching or outperforming prominent proprietary frontier models.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Replaces coordinate text generation with a coarse-to-fine token referencing mechanism.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across natural images, GUI environments, video tracking, and general VLM QA benchmarks with detailed ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear architecture descriptions, explicit formulation of cross-attention scoring, and insightful qualitative analyses.
- Value: ⭐⭐⭐⭐⭐ Delivers an open, efficient, and robust foundation for future multimodal grounding and GUI agent research.