LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Visual Grounding, Object Detection, Vision-Language Models, Parallel Box Decoding, Multi-Token Prediction

TL;DR

LocateAnything introduces Parallel Box Decoding (PBD), framing 2D bounding boxes as atomic units predicted concurrently in a single forward pass, which—powered by 138M multi-domain training queries and an adaptive hybrid fallback mechanism—achieves over 2.5× higher throughput and superior high-IoU localization accuracy compared to state-of-the-art autoregressive VLMs.

Background & Motivation

Visual grounding and object detection serve as foundational perceptual capabilities that enable vision-language models (VLMs) to interface with physical environments, robotics, and graphical user interfaces. Under the conventional next-token prediction (NTP) autoregressive paradigm, modern multimodal models serialize 2D spatial coordinates into 1D discrete token streams—either spelled out as individual textual digits (e.g., serializing 1024 into "1", "0", "2", "4") or represented via quantized coordinate tokens (\(\langle x_1 \rangle \langle y_1 \rangle \langle x_2 \rangle \langle y_2 \rangle\)). However, this 1D sequence serialization conflicts fundamentally with the inherently coupled 2D spatial geometry of bounding boxes. Consequently, generating a single box requires 10 to 20 autoregressive forward passes, creating a severe decoding latency bottleneck that throttles real-time robotics and dense perception tasks.

While generic multi-token prediction (MTP) and masked diffusion language models have been explored to alleviate autoregressive latency, they remain predominantly structure-agnostic, relying on arbitrary token chunking or uniform masking. When applied to spatial coordinates, these unstructured chunks frequently slice across bounding box boundaries and object class transitions, forcing the network to learn erratic conditional distributions and spurious correlations. This structural misalignment not only degrades intra-box geometric consistency but also compounds error propagation, leading to syntactically corrupted outputs, severe coordinate drift, and spurious overlapping boxes in cluttered environments.

Resolving the tension between high-throughput parallel decoding and strict geometric fidelity requires moving beyond treating spatial coordinates as arbitrary text tokens. Core idea: propose Parallel Box Decoding (PBD) with an atomic block-based output formulation, binding bounding box coordinates into an indivisible single-step parallel prediction unit, supported by joint causal-NTP and bidirectional-MTP training alongside an on-demand hybrid fallback mechanism that eliminates autoregressive decoding bottlenecks while substantially improving fine-grained localization precision.

Method

Overall Architecture

LocateAnything builds upon a native-resolution multimodal vision-language architecture consisting of a Moon-ViT visual encoder, an MLP projection bottleneck, and a Qwen2.5 autoregressive language decoder. Given an input image \(\mathcal{I}\), the vision encoder extracts visual representations \(Z = \text{Encoder}(\mathcal{I})\) at native resolution to preserve delicate spatial cues essential for high-precision bounding box regression. The language decoder consumes the visual tokens alongside natural language instructions to generate a semi-autoregressive sequence of structured, box-aligned atomic prediction blocks.

The overall workflow maps multi-modal inputs through fine-grained visual encoding and block-wise sequence modeling to support varied tasks including multi-object referring, dense detection, GUI element grounding, and document layout analysis. During inference, the model prioritizes high-speed parallel block decoding while monitoring syntactic validity and spatial confidence, dynamically falling back to localized NTP re-decoding whenever ambiguities are detected.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Text Queries<br/>Native-resolution multimodal inputs"] --> B["Native-Resolution Visual Encoding & Projection<br/>Moon-ViT + MLP feature extraction Z"]
    B --> C["Atomic Block Output Formulation<br/>Fixed-length L=6 structured block sequence"]
    C --> D["Dual-Formulation Joint Training & Block-Causal Masking<br/>Causal NTP anchor + bidirectional intra-block MTP"]
    D --> E["On-Demand Inference Modes & Confidence Fallback<br/>Fast MTP execution with localized Slow NTP recovery"]
    E --> F["High-Throughput High-Precision Output<br/>Boxes, points, GUI targets, and layout segments"]

Key Designs

1. Atomic Block Output Formulation: encapsulating coupled 2D geometry into fixed-length atomic prediction blocks

Existing autoregressive grounding pipelines fracture spatial coordinates into isolated scalar tokens, discarding the strict inter-coordinate geometry binding \((x_1, y_1, x_2, y_2)\). LocateAnything discretizes continuous coordinates normalized to \([0, 1000]\) and reorganizes output predictions into a sequence of structured blocks \(B = (b_1, b_2, \dots, b_N)\). Each block \(b_i\) possesses a constant length \(L = 6\), neatly fitting a beginning structural tag, four quantized coordinates, and an ending tag (e.g., \(\langle\text{box}\rangle\langle x_1\rangle\langle y_1\rangle\langle x_2\rangle\langle y_2\rangle\langle/\text{box}\rangle\)), padded with \(\langle\text{null}\rangle\) tokens when unfulfilled. Four distinct functional blocks are defined: Semantic Blocks encoding entity category tags across chained units, Box Blocks containing full coordinate sets, Negative Blocks indicating target absence, and End Blocks signaling sequence termination. This formulation standardizes tensor dimensions, aligning MTP chunks exactly to physical instance boundaries.

2. Dual-Formulation Joint Training & Block-Causal Masking: unifying causal linguistic reasoning with concurrent intra-box spatial perception

Directly enforcing parallel decoding on a pretrained language model compromises its established causal reasoning pathways and induces distribution collapse. LocateAnything addresses this with a dual-formulation multi-stream concatenation \(x_{\text{all}} = x_{\text{vis}} \oplus x_{\text{q}} \oplus x_{\text{ntp}} \oplus x_{\text{blk}}\), jointly supervising the standard NTP sequence and the block-based MTP sequence under a unified shared context. A tailored attention mask governs information flow across three distinct behaviors: the shared context and NTP sequence follow strict causal attention with zero visibility into \(x_{\text{blk}}\) to prevent data leakage; across MTP blocks, causal dependencies are strictly maintained so active blocks attend to all historical verified prefixes (mitigating duplicate and omitted detections); within each active block, tokens share full bidirectional attention. By preserving the first token as an anchor and masking the remaining tokens with \([\text{mask}]\), the model is trained to predict all internal box tokens concurrently. The dual objective minimizes both cross-entropy losses simultaneously:

\[\mathcal{L} = \mathcal{L}_{\text{ntp}} + \mathcal{L}_{\text{blk}}\]

3. On-Demand Inference Modes & Confidence Fallback: balancing ultra-high decoding throughput against worst-case spatial ambiguity

In highly cluttered environments, parallel decoding can suffer from format irregularity (erratic interleaving of semantic and coordinate tokens at category transitions) and spatial ambiguity (outputting smoothed intermediate coordinates between dense, grid-aligned targets). LocateAnything establishes three on-demand decoding regimes: Fast Mode executing full parallel MTP for maximum throughput, Slow Mode generating tokens sequentially via NTP for absolute stability, and the default Hybrid Mode. In Hybrid Mode, the model continuously inspects output syntax and evaluates spatial ambiguity via two joint criteria:

\[P(\text{top-1}) < 0.7 \quad \text{and} \quad \max(\text{top-5}) - \min(\text{top-5}) > 80\]

When syntax violations or high spatial ambiguity are identified, the corrupted block is discarded, and the model reverts to the verified prefix to run localized NTP generation exclusively for the problematic unit. Once resolved, the system immediately switches back to parallel MTP for subsequent instances, preserving speed advantages while providing a rock-solid precision guarantee.

Loss & Training

The optimization roadmap proceeds across three distinct phases: 1. Base Visual-Language Pretraining: Foundational alignment on broad multimodal corpora focused purely on world knowledge, strictly omitting all detection and grounding data. 2. Stage-1 SFT on LocateAnything-Data: Incorporates 138M language queries, 12M unique images, and 785M annotated boxes to establish comprehensive localization capabilities across detection, GUI navigation, REC, OCR, and layout parsing. 3. Stage-2 Dense Scene Fine-Tuning: General data proportion is dialed down to 20% while heavily scaling up crowded, multi-instance imagery (e.g., MOT20Det, SKU110K). Coordinate sequences are deterministically ordered by Top-Left Distance to ensure monotonic spatial progression throughout autoregressive and semi-autoregressive decoding.

Key Experimental Results

Main Results

On standard object detection benchmarks (LVIS, COCO), dense object scenarios (Dense200, VisDrone), and specialized GUI and document tasks, LocateAnything-3B surpasses competing open-set detectors and same-sized specialist VLMs while delivering extraordinary throughput on a single NVIDIA H100 GPU (batch size = 1).

Dataset / Benchmark	Metric	LocateAnything-3B (Ours)	Rex-Omni-3B (SOTA VLM)	Grounding DINO-Swin-T	Gain / Comparative Advantage
LVIS (Zero-Shot)	Mean F1 / [email protected]	50.7 / 31.1	46.9 / 20.7	38.8 / 22.7	+3.8% Mean F1 gain; massive +10.4% surge on strict [email protected]
COCO (Zero-Shot)	Mean F1 / [email protected]	54.7 / 19.3	52.9 / 15.9	56.6 / 23.0	+1.8% Mean F1 improvement over leading 3B multimodal baseline
VisDrone (Dense Small)	Mean F1 / [email protected]	39.9 / 63.0	35.8 / 61.6	38.5 / 55.2	+4.1% Mean F1 over Rex-Omni, outperforming specialized detectors
Dense200 (Crowded Objects)	Mean F1 / [email protected]	58.7 / 18.5	58.3 / 10.3	33.1 / 19.7	[email protected] reaches 18.5 (nearly 1.8× that of Rex-Omni at 10.3)
ScreenSpot-Pro (GUI)	Average F1	60.3	36.8	-	Surpasses GUI-Owl-32B (58.0) and Qwen3-VL-30B-A3B (53.7)
DocLayNet (Layout)	Mean F1	76.8	70.7	81.1 (DocLayout-YOLO)	New multimodal SOTA; [email protected] improves from 28.4 to 35.8
Decoding Throughput	BPS (Boxes/Sec)	12.7 (Hybrid) / 16.9 (Fast)	5.0	-	Over 2.5× faster than Rex-Omni and >10× faster than Qwen3-VL (1.1)

Ablation Study

Ablations conducted strictly on the COCO dataset isolate the architectural benefits of PBD from large-scale data pretraining (all reported metrics represent averages across IoU thresholds for Recall, Precision, and F1).

Dimension & Configuration	Throughput (BPS)	Recall (R)	Precision (P)	Mean F1	Core Analysis & Key Insights
Coordinates: Textual Digits	1.3	45.7	52.3	49.1	Severe decoding latency; token fragmentation causes coordinate drift
Coordinates: Quantized Tokens	3.9	48.2	52.2	50.1	Faster than textual format but remains throttled by sequential steps
Coordinates: PBD (Slow NTP)	3.9	49.4	55.2	52.1	Box-aligned formulation improves representation quality even under NTP
MTP: Structure-Agnostic SDLM-B6	5.5	45.1	47.5	46.1	Arbitrary chunking induces spurious cross-boundary patterns, degrading F1
MTP: Block Diffusion Block Diff-B6	4.7	45.1	44.3	44.8	Unstructured spatial denoising fails to guarantee boundary precision
MTP: PBD (Fast Mode)	16.9	45.6	54.6	49.6	Box alignment unlocks 16.9 BPS throughput and beats SDLM-B6 by +3.5 F1
Losses: Only \(\mathcal{L}_{\text{ntp}}\) (Slow)	3.9	48.2	52.2	50.1	Baseline without MTP regularization reaches limited spatial ceiling
Losses: Only \(\mathcal{L}_{\text{blk}}\) (Fast)	16.7	45.6	49.0	47.2	Lacks autoregressive causal anchor, leading to semantic degradation
Joint Optimization: PBD (Hybrid)	13.2	48.7	54.8	51.6	Retains 13.2 BPS speed while boosting F1 to 51.6 via localized NTP fallback

Key Findings

• Box-aligned chunking is critical for MTP success in perception: Structure-agnostic MTP approaches (SDLM-B4, B6, B8) suffer from an adverse trade-off where increasing block sizes provides negligible throughput gains while degrading F1 from 46.5 down to 45.8. PBD eliminates this penalty by matching blocks directly to geometric entities, achieving 16.9 BPS and 49.6 F1.
• Substantial leaps under strict high-IoU thresholds: In rigorous [email protected] evaluations on LVIS (31.1 vs. 20.7) and Dense200 (18.5 vs. 10.3), LocateAnything nearly doubles baseline performance, proving that single-step box-level parallel generation enforces strict coordinate coherence and eliminates sequential drift.
• Superior throughput scaling across dense scenes: When predicted target counts scale from 20 to 300 instances, conventional NTP generation latency scales steeply upward, whereas PBD latency remains virtually constant, pushing throughput from 12 BPS to nearly 25 BPS and demonstrating a 2× to 6× empirical speedup.

Highlights & Insights

• Geometric-Token Parallelism: By decomposing output sequences into instance-aligned atomic blocks, LocateAnything introduces bidirectional attention inside each box while preserving causal attention across blocks, marrying non-autoregressive speed with autoregressive inter-object reasoning.
• Confidence-Aware Localized Fallback: Rather than discarding full generation sequences upon encountering failure cases, the hybrid inference strategy leverages top-1 probability and top-5 coordinate range thresholds to perform surgical NTP re-decoding on problematic blocks only.
• Massive Data Engine for Spatial Precision: The curated LocateAnything-Data corpus (138M queries, 785M annotated boxes) paired with two-stage crowded scene curriculum training provides broad zero-shot robustness across complex UI layouts, fine-grained REC, and dense aerial detection.

Limitations & Future Work

• Lack of Reinforcement Learning Post-Training: Currently, LocateAnything relies exclusively on supervised fine-tuning. Integrating reinforcement learning (e.g., GRPO with IoU-based reward modeling) represents an immediate avenue to refine the block decoding policy and reduce fallback rates in challenging long-tail cases.
• High Self-Attention Overhead for Ultra-High Resolutions: Although Moon-ViT preserves native resolution without downsampling artifacts, self-attention and KV-cache footprints remain demanding when processing gigapixel images containing thousands of microscopic targets.
• Generalization to 3D and Spatio-Temporal Bounding Volumes: The atomic block formulation is presently designed for 2D bounding boxes and points; extending parallel block decoding to 3D bounding boxes \((x,y,z,w,h,d,r)\) or video spatio-temporal tubes is a promising future trajectory.

Related Work & Insights

• vs. Rex-Omni: Rex-Omni relies on sequential quantized token generation and point guidance (5.0 BPS, 10.3 [email protected] on Dense200); LocateAnything adopts parallel atomic box decoding (12.7~16.9 BPS, 18.5 [email protected]), achieving superior high-precision localization and throughput.
• vs. Qwen-VL / DeepSeek-VL Series: General-purpose multimodal LLMs serialize coordinates into textual digits, incurring drastic latency penalties in crowded scenes (1.0~1.3 BPS); LocateAnything achieves up to 10× to 15× faster decoding via unified block-level MTP.
• vs. SDLM / Block Diffusion: Structure-agnostic semi-autoregressive models fragment token sequences arbitrarily across bounding box boundaries; LocateAnything demonstrates that aligning block units with physical object structures is essential to avoiding spurious correlations and error compounding.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First framework to align multi-token prediction directly with physical geometric bounding box units, overcoming the autoregressive latency wall in visual grounding.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across general detection, crowded scenes, GUI navigation, document parsing, and REC benchmarks with extensive architectural ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear conceptual motivation, precise methodological descriptions, and detailed architectural diagrams.
• Value: ⭐⭐⭐⭐⭐ Bridges the critical gap between powerful VLM reasoning and real-time inference throughput for embodied robotics and interactive computer-use agents.

Related Papers

• [ECCV 2026] Token-Based Affordance Grounding with Large Vision-Language Models
• [ECCV 2026] LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models
• [ICLR 2026] Grounding-IQA: Grounding Multimodal Language Models for Image Quality Assessment
• [ACL 2025] Scalable Vision Language Model Training via High Quality Data Curation
• [ECCV 2026] EGM: Efficient Visual Grounding Language Models