SpecEyes: Accelerating Agentic Multimodal LLM via Speculative Planning and Perception

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/MAC-AutoML/SpecEyes
Area: LLM Efficiency / Multimodal VLM
Keywords: Agentic MLLM, Speculative Reasoning, Cognitive Gating, Answer Separability, Heterogeneous Parallel Serving

TL;DR

SpecEyes lifts speculative acceleration from the token level to the agentic level by utilizing a lightweight non-agentic MLLM to draft tool-free answers, regulated by a scale-invariant cognitive gate based on top-\(K\) logit answer separability and served through a heterogeneous parallel funnel, delivering a 1.73x average speedup (up to 3.35x) and a 2.87% accuracy boost across V*, HR-Bench, and POPE.

Background & Motivation

Multimodal large language models (MLLMs) have undergone a significant paradigm shift from passive, single-pass visual perception toward dynamic, agentic interaction with the visual world. Modern agentic multimodal systems (such as DeepEyes, Thyme, OpenAI o3, and Gemini Agentic Vision) actively invoke external perception tools—including zooming, cropping, and OCR—within iterative perception-reasoning loops to progressively refine visual comprehension. This interactive loop has unlocked unprecedented capabilities in demanding tasks requiring fine-grained visual inspection, multi-step compositional reasoning, and active information acquisition.

However, the iterative tool-calling mechanism fundamentally introduces a severe efficiency bottleneck. Each user query triggers a Markovian cascade of tool invocations across an agentic depth \(D\), where each perception action strictly depends on the observation of the preceding step. This causal data dependency imposes a dual penalty on practical deployment: end-to-end latency explodes linearly with agentic depth \(D\), while system-level concurrency collapses under batched serving because queries with deep, heavy-tailed tool chains stall the entire batch on GPU hardware. Conventional token-level speculative decoding (e.g., SpecReason) and multimodal token pruning operate strictly within a fixed reasoning loop, unable to eliminate the repeated external tool invocations that dominate overall execution time.

Prior systems universally treat multi-step tool execution as a non-negotiable prerequisite, neglecting whether tool calling is genuinely required for every query. In practice, a substantial portion of queries can be accurately answered by lightweight models looking only at the original global image. The core idea is to lift speculative acceleration to the agentic level via a "think fast, think slow" paradigm, employing a stateless lightweight vision model for rapid tool-free speculation, governed by a calibration-free cognitive gate based on answer separability, while reserving the expensive stateful agentic loop strictly for ambiguous or complex queries.

Method

Overall Architecture

SpecEyes reorganizes agentic query processing into a four-phase speculative funnel combined with heterogeneous parallel serving. The system first evaluates tool necessity via a single token generated by the large model; tool-free candidates are concurrently drafted by a lightweight model; a top-\(K\) logit answer separability gate decides whether to accept the draft; and only low-confidence or tool-required queries fall back to the stateful multi-step agentic pipeline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query Batch (B queries) + Raw Image"] --> B["Phase I: Heuristic Tool-Use Judgment<br/>Large MLLM single-token decision g in {0, 1}"]
    B -->|g=1 tool required| F["Phase IV: Agentic Fallback<br/>Sequential multi-turn tool-calling loop"]
    B -->|g=0 tool-free candidate| C["Phase II: Speculative Prediction<br/>Stateless small MLLM parallel forward pass"]
    C --> D["Phase III: Cognitive Gating<br/>Top-K logit answer separability S_sep"]
    D -->|"S_sep >= tau (high confidence)"| E["Immediate Acceptance<br/>Bypass entire agentic tool loop"]
    D -->|"S_sep < tau (low confidence)"| F
    E --> G["Final Answer Output"]
    F --> G

Key Designs

1. Four-Phase Agentic Speculative Pipeline: Bypassing the Sequential Tool Dependency

The primary latency bottleneck of agentic systems stems from the strict causal dependency across state transitions \(s_{d+1} = f(s_d, t_d(s_d))\), forcing end-to-end latency \(L_{\text{agent}}(q) = \sum_{d=0}^{D(q)} (c_{\text{llm}} + c_{\text{tool}}(t_d))\). SpecEyes decouples this chain into four collaborative phases: - Phase I (Heuristic Tool-Use Judgment): The large agentic model \(\mathcal{M}_L\) executes a binary classification prompt to determine \(g(q, I) \in \{0, 1\}\). Because \(\mathcal{M}_L\) is inherently trained in tool-use policies, it acts as a highly reliable judge while generating only a single token with zero tool executions, incurring negligible latency \(c_J\). - Phase II (Speculative Prediction): Queries with \(g=0\) are routed to the lightweight model \(\mathcal{M}_S\) (e.g., Qwen3-VL-2B). Operating statelessly without tool dependencies, \(\mathcal{M}_S\) performs a standard parallel forward pass to generate speculative answer \(\hat{y}_S\) alongside token-level vocabulary logit distributions \(\{\boldsymbol{\ell}^{(n)}\}\). - Phase III (Cognitive Gating): The generated logits are assessed by the answer separability metric \(S_{\text{sep}}\). Answers with \(S_{\text{sep}} \ge \tau\) are accepted immediately, completely bypassing subsequent agentic stages. - Phase IV (Agentic Fallback): Rejected queries join the \(g=1\) pool and fall back to \(\mathcal{M}_L\) to run the full perception-reasoning loop. With screening ratio \(\beta\) and gate acceptance rate \(\alpha\), the expected per-query latency becomes: $\(\mathbb{E}[L_{\text{SpecEyes}}] = c_J + \beta c_S + (1 - \beta \alpha) L_{\text{agent}}\)$ Whenever \(\beta \alpha\) is high, the vast majority of requests bypass the expensive tool-calling loops, drastically slashing latency.

2. Cognitive Gating via Top-K Logit Answer Separability: Scale-Invariant Confidence Metric

Conventional confidence estimation relies on geometric mean of max-softmax probabilities \(S_{\text{log}}\), which suffers from softmax overconfidence, scale distortion, and dilution from high-probability formatting tokens.

To overcome these issues, SpecEyes introduces an uncalibrated, scale-invariant answer separability score \(S_{\text{sep}}^{(n)}\). For the \(n\)-th token with descending sorted logits \(\ell^{(n)}_{[1]} \ge \ell^{(n)}_{[2]} \ge \dots \ge \ell^{(n)}_{[|V|]}\), let \(\mu_K^{(n)}\) and \(\sigma_K^{(n)}\) be the mean and standard deviation of the top-\(K\) logits. The token-level separability is formulated as: $\(S_{\text{sep}}^{(n)} = \frac{\ell^{(n)}_{[1]} - \mu_K^{(n)}}{\sigma_K^{(n)} + \epsilon}\)$ The numerator computes the absolute decision margin of the top candidate over its immediate competitive neighborhood, while the denominator normalizes this margin by the local dispersion. Crucially, scaling the logit vector scales both numerator and denominator equally, rendering \(S_{\text{sep}}^{(n)}\) strictly scale-invariant and immune to temperature or logit calibration issues.

For answer-level aggregation, SpecEyes adopts a minimum pooling strategy: $\(S_{\text{sep}}^{\text{min}} = \min_{n \in [|\hat{y}_S|]} S_{\text{sep}}^{(n)}\)$ Grounding on risk theory (Proposition 1), the sequence-level error event is the union of token errors \(\mathcal{E} = \bigcup_n \mathcal{E}_n\). Assuming token error probability monotonically decreases with \(S_{\text{sep}}^{(n)}\), filtering by \(\min_n S_{\text{sep}}^{(n)}\) enforces a worst-case guard: if even a single token exhibits ambiguity, the query immediately falls back to \(\mathcal{M}_L\), thereby guaranteeing high verification precision.

3. Heterogeneous Parallel Funnel: Stateless Concurrency Masking Stateful Serialization

In real-world serving under continuous batching frameworks (e.g., vLLM), batch latency is dictated by the slowest query with the deepest tool chain. SpecEyes structures query processing into a heterogeneous funnel: - Across a batch of \(B\) queries, Phase I and Phase II execute purely stateless forward inferences. They are fully parallelizable across GPU tensor cores, incurring only a constant upfront cost \(c_J + c_S\). - Following cognitive gating, only the residual set \(R\) of size \(|R| = (1 - \beta \alpha) B\) enters the stateful, sequential agentic tool pipeline. By converting a fraction \(\beta \alpha\) of queries into single-turn stateless passes, the effective residual queue is dramatically shrunk, yielding an overall system throughput speedup of: $\(\frac{\Theta_{\text{SpecEyes}}}{\Theta_{\text{agent}}} \approx \frac{1}{1 - \beta \alpha}\)$ This design multiplies system throughput proportionally with speculative acceptance, eliminating concurrency collapse under production workloads.

Training & Deployment Strategy

SpecEyes is completely training-free (zero-shot). It requires no fine-tuning of the small model, the large model, or external auxiliary classifiers. Phase I relies on a static heuristic prompt, and Phase III operates directly on decoding logits. The threshold \(\tau\) is determined via coarse grid sampling on a 10% calibration subset, ensuring plug-and-play deployment across any agentic foundation model.

Key Experimental Results

Main Results

SpecEyes was evaluated on three comprehensive benchmarks: fine-grained visual perception V* Bench (Direct Attributes and Relative Position), high-resolution understanding HR-Bench (4K and 8K subsets), and visual hallucination probe POPE (Adversarial, Popular, and Random). The draft model \(M_S\) was Qwen3-VL-2B, paired with DeepEyes and Thyme as agentic backbones \(M_L\) (capped at 5 tool steps).

The following table summarizes the primary performance results from Table 1 of the paper (Accuracy Acc. % and wall-clock speedup Spd. relative to the agentic base model):

Backbone & Method	V* Attr (Acc / Spd)	V* Pos (Acc / Spd)	HR-4K (Acc / Spd)	HR-8K (Acc / Spd)	POPE Adv (Acc / Spd)	POPE Pop (Acc / Spd)	POPE Rand (Acc / Spd)	Overall Avg (Acc / Spd)
Qwen3-VL-2B (Draft-only baseline)	77.39% / 5.44x	82.89% / 5.31x	71.38% / 3.20x	68.00% / 2.90x	82.56% / 4.20x	83.80% / 3.78x	86.47% / 4.07x	78.93% / 4.13x
DeepEyes Base (w/ full tools)	90.43% / 1.00x	82.89% / 1.00x	75.85% / 1.00x	71.43% / 1.00x	78.43% / 1.00x	81.90% / 1.00x	88.83% / 1.00x	81.39% / 1.00x
DeepEyes (w/o tools)	80.87% / 4.08x	73.68% / 4.18x	75.25% / 2.71x	72.00% / 2.53x	46.90% / 3.78x	49.33% / 3.60x	48.20% / 3.81x	63.75% / 3.53x
SpecReason (Token-level speculative)	80.19% / 0.61x	73.91% / 0.38x	80.43% / 0.44x	72.54% / 0.42x	49.10% / 0.38x	51.55% / 0.38x	60.20% / 0.37x	66.85% / 0.43x
SpecEyes (log aggregation)	83.48% / 2.06x	88.16% / 2.05x	73.71% / 1.35x	69.67% / 1.28x	83.97% / 1.89x	86.70% / 1.95x	90.50% / 2.05x	82.31% / 1.80x
SpecEyes (mean aggregation)	78.26% / 2.89x	84.21% / 3.35x	71.62% / 1.88x	67.38% / 1.77x	85.13% / 2.06x	87.00% / 2.10x	90.13% / 2.14x	80.53% / 2.31x
SpecEyes (bottom-r aggregation)	83.48% / 2.13x	84.21% / 2.12x	75.22% / 1.20x	71.18% / 1.04x	85.13% / 2.08x	87.00% / 2.08x	90.13% / 2.11x	82.34% / 1.82x
SpecEyes (min aggregation, recommended)	90.43% / 1.53x	89.47% / 1.90x	75.85% / 1.13x	71.80% / 1.08x	85.13% / 2.13x	87.00% / 2.15x	90.13% / 2.19x	84.26% / 1.73x
Thyme Base (w/ full tools)	86.96% / 1.00x	82.89% / 1.00x	77.72% / 1.00x	72.43% / 1.00x	81.32% / 1.00x	84.53% / 1.00x	90.17% / 1.00x	82.29% / 1.00x
Thyme (w/o tools)	84.35% / 2.81x	76.32% / 2.56x	74.25% / 1.85x	69.88% / 1.97x	77.77% / 3.51x	78.17% / 3.32x	79.93% / 2.99x	77.24% / 2.72x
SpecReason (Token-level speculative)	89.57% / 0.48x	75.00% / 0.53x	80.01% / 0.52x	81.02% / 0.51x	84.62% / 0.46x	85.97% / 0.43x	90.27% / 0.46x	83.78% / 0.48x
SpecEyes (min aggregation, recommended)	87.83% / 1.32x	82.89% / 1.42x	78.47% / 1.01x	73.31% / 0.95x	85.87% / 1.77x	88.30% / 1.78x	91.27% / 1.70x	83.99% / 1.42x

Ablation Study: Draft Model Scaling

The authors further investigated larger draft model architectures in Table 2, evaluating Qwen3-VL-8B and Qwen2.5-VL-7B as alternative draft models \(M_S\):

Draft Model \(M_S\) Config	Metric	V* Attr	V* Pos	HR-4K	HR-8K	POPE Adv	POPE Pop	POPE Rand	Avg Total
\(M_S\) = Qwen3-VL-8B (Draft only)	Acc / Spd	81.74% / 4.23x	78.95% / 1.72x	77.50% / 2.46x	69.90% / 1.54x	84.47% / 2.48x	86.67% / 2.69x	91.33% / 3.06x	81.51% / 2.60x
+ DeepEyes Backbone (SpecEyes min)	Acc / Spd	92.17% / 1.47x	80.26% / 2.69x	78.49% / 1.05x	74.06% / 1.06x	84.23% / 1.75x	86.47% / 1.80x	89.70% / 1.85x	83.63% / 1.67x
+ Thyme Backbone (SpecEyes min)	Acc / Spd	90.43% / 1.28x	80.26% / 1.57x	78.38% / 0.96x	74.22% / 0.94x	85.52% / 1.60x	87.27% / 1.48x	90.80% / 1.56x	83.84% / 1.34x
\(M_S\) = Qwen2.5-VL-7B (Draft only)	Acc / Spd	79.13% / 3.95x	71.05% / 1.68x	74.88% / 2.75x	66.87% / 1.72x	79.66% / 2.68x	81.23% / 3.01x	88.79% / 3.55x	77.37% / 2.76x
+ DeepEyes Backbone (SpecEyes min)	Acc / Spd	90.43% / 0.92x	78.95% / 1.63x	75.97% / 1.02x	71.43% / 0.98x	80.60% / 1.20x	84.47% / 1.27x	89.77% / 1.23x	81.66% / 1.18x
+ Thyme Backbone (SpecEyes min)	Acc / Spd	87.83% / 0.93x	78.95% / 1.25x	77.72% / 0.85x	72.31% / 0.83x	82.82% / 1.06x	86.20% / 1.17x	91.10% / 1.19x	82.42% / 1.04x

Key Findings

1. Unintended Regularization Improves Accuracy: SpecEyes (min) improves average accuracy from 81.39% to 84.26% (+2.87%) on DeepEyes, with POPE Adversarial surging from 78.43% to 85.13% (+6.70%). Excessive visual tool invocations often induce cumulative hallucinations and over-fit to misleading local crops; bypassing tools for obvious queries preserves global consistency and prevents agentic distraction.
2. Token-Level Speculation Fails in Agentic Pipelines: Prior token-level baseline SpecReason causes severe slowdowns (0.37x-0.61x speedup, doubling runtime) because it incurs substantial draft-verification overhead (averaging 3.48 rounds and 414 tokens) while failing to eliminate repeated tool-calling latency. SpecEyes bypasses entire tool chains, achieving true macro acceleration.
3. Worst-Case Guarding is Essential for Robust Gating: Among all four aggregation strategies, \(S_{\text{sep}}^{\text{min}}\) achieves the highest matched-speed accuracy across all benchmarks. While mean aggregation achieves higher speedup (up to 3.35x), its accuracy drops to 80.53%. Minimum pooling guarantees that any single uncertain token triggers agentic fallback, eliminating false acceptances.
4. Diminishing Returns of Larger Draft Models: Although 8B and 7B draft models yield higher raw accuracy, their larger per-step forward latency \(c_S\) diminishes net speedup (average speedup on DeepEyes drops from 1.73x for 2B to 1.67x for 8B and 1.18x for 7B). A lightweight 2B model achieves the optimal Pareto frontier between drafting cost and verification accuracy.

Highlights & Insights

• Shifting the Optimization Frontier from Token to Agentic Level: While speculative decoding has traditionally focused on autoregressive token generation, this paper recognizes that sequential multi-step tool calls dominate latency in agentic systems. Bypassing an entire tool-use trajectory yields far greater returns than accelerating dozens of tokens.
• Label-Free and Scale-Invariant Cognitive Gating: By measuring the margin between the top logit and the mean/standard deviation of top-\(K\) competitors, \(S_{\text{sep}}\) provides a scale-invariant confidence boundary that naturally neutralizes softmax calibration artifacts without requiring any external training or ground truth annotations.
• Stateless Concurrency Masking Stateful Bottlenecks: Structuring early speculation as stateless batch forward passes allows continuous batching engines to maximize hardware occupancy, converting per-query speculative acceptance directly into multiplicative throughput gains.

Limitations & Future Work

• Binary Depth Limitation (\(D=0\)): SpecEyes currently operates only at \(D=0\) (purely tool-free speculation). On benchmarks like HR-Bench 8K where high-resolution inspection is indispensable, the tool-free screening ratio \(\beta\) and acceptance rate \(\alpha\) remain low, yielding modest speedups (0.95x-1.08x). A promising direction is multi-depth speculation (\(D \in \{1, 2, \dots, n\}\)), permitting bounded lightweight tool invocations before gating.
• Calibration Domain Transfer: While parameter-free, setting the operating threshold \(\tau\) and candidate size \(K\) still requires sampling a small fraction of queries from the target domain. Automated dynamic thresholding under open-ended distributions remains open for exploration.

Related Work & Insights

• vs Native Agentic MLLMs (DeepEyes, Thyme): Native agentic frameworks train models via reinforcement learning to iteratively invoke visual tools, pushing the frontier of complex visual reasoning at the cost of high sequential latency. SpecEyes is fully orthogonal and serves as an execution wrapper over these backbones, boosting serving speed without altering base model weights.
• vs Token-Level Speculation (SpecReason, RelayLLM): Token-level speculative decoding operates inside fixed reasoning trajectories, incurring substantial turn and token communication overhead that leads to net deceleration in agentic setups. SpecEyes bypasses entire trajectories, pioneering macro-level speculative routing.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First framework to formalize the agentic-level stateful sequential bottleneck and demonstrate speculative tool-chain bypass with scale-invariant cognitive gating.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across three distinct benchmarks, two representative agentic backbones, multiple draft model sizes, and detailed distribution KDE analyses.
• Writing Quality: ⭐⭐⭐⭐⭐ Flawless conceptual exposition connecting Markovian causal state transitions to concurrency collapse, backed by clean mathematical formulation and intuitive visualizations.
• Value: ⭐⭐⭐⭐⭐ Substantial practical value for accelerating agentic vision systems in cloud deployments under concurrent workloads.