Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Vision-Language Reasoning
Keywords: vision-language model, visual focusing, stateful decoding, continuous actions, evidence acquisition

TL;DR¶

FoveateR lets a vision-language model trigger continuous crop actions and incorporate high-resolution evidence within one autoregressive trajectory, improving multiple visual question-answering results with fewer visual tokens under low-resolution inputs without requiring human-annotated foveation trajectories.

Background & Motivation¶

When a vision-language model (VLM) processes documents, small text, or fine-grained attributes, failure may reflect missing input pixels rather than inadequate reasoning ability. Encoding the entire image at high resolution recovers detail but increases visual encoding and subsequent attention costs. Conversely, asking for a longer chain-of-thought (CoT) from a low-resolution overview cannot recover an unreadable table entry by itself. Visual focusing therefore starts with global context and retrieves local details, but the way new evidence enters reasoning still affects efficiency and reliability.

Multi-pass methods typically localize a region and then rerun an answering model on the crop, potentially starting a new decoding trajectory for every focusing step. Text-grounded methods can interleave observation and reasoning, but coordinates, tool arguments, or localization tokens must be generated through a discrete vocabulary, adding text overhead and formatting constraints. Rather than introducing another cropping heuristic, this paper embeds decisions about when and where to acquire evidence into ongoing decoding. That requires retaining previously observed information while accessing the original high-resolution image through geometric actions separate from language output.

FoveateR draws on the process of foveation: a coarse overview guides attention, while precise local observations enter an existing memory only when needed. Training must also prevent increasingly large crops from becoming an easy route to reward, since selective observation would otherwise collapse into full-image high-resolution input. Core Idea: share a persistent reasoning state, let the token policy decide when to look and a continuous region policy decide where, and learn economical evidence acquisition through task rewards and a conditional area constraint.

Method¶

Overall Architecture¶

The inputs are an original high-resolution image, its low-resolution global view, and a question; only the latter two initially enter the VLM. The environment retains the original image and encodes and returns only the regions requested by the model. The output remains a response structured with <think> and <answer> tags, while foveation actions execute during generation. The method comprises stateful decoding, continuous-action focusing, and task-driven two-stage learning; the first two perform inference, and the third trains them.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Low-resolution view + question"] --> Memory["Stateful decoding"]
    Memory -->|fov trigger| Focus["Continuous-action focusing"]
    Original["Original high-resolution image"] -->|crop and encode by box| Focus
    Focus -->|append evidence to existing context| Memory
    Memory -->|answer and terminate| Answer["Final answer"]
    Focus -.->|training only: sampled trajectories| Learn["Task-driven<br/>two-stage learning"]
    Learn -.->|training only: update both policies| Memory

A single decoding trajectory does not mean the image is encoded only once: each new crop still requires visual encoding, while the design avoids repeatedly starting a complete answering process. The dashed training edges do not imply online weight updates at inference time; deployment executes the learned token and region policies. The number of foveations depends on the question and acquired evidence rather than a fixed crop count imposed on every sample. If no foveation is triggered during generation, the system reduces to a conventional autoregressive VLM conditioned on the low-resolution image.

Key Designs¶

1. Stateful decoding: continue existing reasoning with newly acquired evidence

The paper models the process as a partially observable Markov decision process (POMDP): the high-resolution image is fixed, but the model does not observe all its details initially. The question is fully observed, whereas unqueried local image content remains unseen. The initial observation contains the overview and question; subsequent observations are either newly generated text tokens or freshly cropped visual evidence. The implementation uses the Transformer's KV-cache rather than a separately trained memory network. It retains global visual information, the text prefix, and previously acquired local evidence, so subsequent actions depend on the interaction history.

The memory update in the paper's Equation (5) is:

\[ M_t=M_{t-1}\oplus o_t. \]

Here $o_t$ is the current observation and $\oplus$ denotes appending to the history; this describes the input history represented by the cache, not raw pixels being concatenated directly into KV tensors. The decoder derives a hidden state from this history to summarize available information for both token generation and region selection. The paper interprets this state as a belief summary but does not construct a separate explicit probability distribution over beliefs. Once new evidence enters the same cache, the model can continue reasoning from earlier conclusions or request another region. Multiple observations therefore preserve state continuity without turning every crop into a new question-answering task.

2. Continuous-action focusing: dispatch geometric actions with a trigger token instead of coordinate sentences

The token policy usually predicts ordinary vocabulary tokens but can also select the special <fov> trigger. Only when this marker appears does a lightweight MLP foveation policy predict a continuous region action from the shared hidden state. The action uses $[x,y,w,h]$ box parameters, but these are not expressed by generating digits and delimiters sequentially. The environment crops the original image according to the box, encodes the crop into visual tokens, and inserts an evidence block delimited by <fov> and </fov> into the existing context. Subsequent reasoning reads the additional details until another action or the final answer is produced.

This separation assigns when to look to the token policy and where to look to the continuous action head, with communication through their shared hidden state. Non-linguistic actions therefore do not mean that the system adds no special tokens; they mean that the coordinates themselves bypass the text channel. The region head is not another autoregressive localization VLM and does not generate a separate localization response. Crop size determines the number of additional visual tokens, so the budget depends on queried area as well as action count. The main text does not fully specify the action distribution, out-of-bounds box handling, or positional encoding implementation, so no particular distribution or cache insertion detail should be assumed.

3. Task-driven two-stage learning: learn to acquire evidence before learning when it is worthwhile

Training a newly initialized region head directly from task reward is difficult, and the pretrained VLM does not understand <fov>, motivating coldstart supervised fine-tuning (SFT). A teacher VLM generates intermediate reasoning for the image and question, splits it into sentences, and localizes a corresponding region for each sentence. Each region is cropped and encoded before being interleaved with its reasoning sentence in the supervision sequence, teaching the student to receive evidence before generating the associated text. This avoids human-annotated foveation trajectories but still relies on teacher-generated localization and reasoning pseudo-labels; it is not fully unsupervised learning. Text cross-entropy covers only non-visual tokens and special markers; retrieved visual tokens are conditioning inputs rather than prediction targets. The region head receives box regression supervision at foveation trigger positions, bootstrapping both triggering behavior and spatial selection.

The second stage uses group-relative policy optimization (GRPO), comparing multiple sampled trajectories for the same question through their final rewards. The token policy uses answer accuracy plus format correctness, whereas the region policy uses answer accuracy alone because tag formatting does not directly assess a region. The main text defines the group-relative advantage as reward minus the group mean; no unreported standard-deviation normalization is added here. The student can consequently depart from the teacher's pattern of foveating before every reasoning sentence and choose no foveation, consecutive foveations, or alternating reasoning. Rewarding correctness alone would encourage larger crops, so the authors additionally penalize queried area on correct trajectories, leaving the term inactive on incorrect ones. This gate preserves the opportunity to find the answer before requiring successful policies to reduce unnecessary high-resolution observation.

A Worked Example¶

Figure 4 on page 13 presents a financial-document question asking for cash and cash equivalents for 2007, corresponding to Notes 1 and 14. The low-resolution overview provides the table layout but does not reliably expose the target cell. The first foveation retrieves the relevant row under current assets, and reasoning identifies the row after the evidence enters the existing cache. A second foveation reads the correspondence between year and amount, after which the model outputs $7,770. Both queries occur in one trajectory; the second does not discard the first observation and restart answering. The same figure includes a vehicle-category question with a prominent subject, answered as Motorbike without foveation. This illustrates that zero foveations are permitted, not that every question requires an additional crop.

Loss & Training¶

Training combines 438K Visual CoT samples, 321K RefCOCO/+/g samples, and 6K ScienceQA samples, totaling 765K image-question-answer triplets. The backbone is Qwen2.5-VL 3B or 7B; coldstart runs for 3 epochs, followed by 70k RL steps. Both stages use batch size 16, with learning rates of $3\times10^{-5}$ and $10^{-6}$, respectively. Although the simplified reward description uses binary correctness, the implementation section specifies semantic matching scores in $[0,1]$ for VQA, IoU of at least 0.5 for grounding correctness, and exact option matching for ScienceQA. The binary formulation should therefore be distinguished from task-specific scoring. The clearly specified region regularizer accumulates box area weighted by correctness, with $\lambda_{\mathrm{reg}}=0.2$ used in the main experiments. The cached rendering of Equations (12) through (16) loses mathematical content, so the exact regression norm, complete policy ratios, and clipping terms are not reconstructed; the explanation above follows the surrounding prose.

Key Experimental Results¶

Main Results¶

The following subset of Table 1 on page 11 uses $336\times336$ low-resolution inputs and the Visual CoT GPT-based evaluation protocol; higher scores are better. The visual-token column reports model visual-token usage as presented in that table; FoveateR reports means and standard deviations for its sample-adaptive budget.

Model	Human foveation-box supervision	Visual tokens	DocVQA	TextVQA	DUDE	SROIE	V7W
VisCoT 7B	Yes	1,152	47.6	77.5	38.6	47.0	55.8
FoveateR 3B	No	307.0 ± 156.0	74.0	83.0	58.2	75.4	56.1
FoveateR 7B	No	322.5 ± 162.7	83.3	86.2	62.7	83.2	61.3

DUDE, SROIE, and V7W use the paper's zero-shot setting; this does not claim that every column is evaluated without exposure to its training distribution. The 3B model exceeds VisCoT 7B by 26.4 DocVQA points, but this cross-model comparison changes the backbone and training procedure together, so the entire difference cannot be attributed to state continuity.

Ablation Study¶

The following subset of Table 3 on page 12 fixes the 7B model and $336\times336$ input, examining coldstart and area-regularization strength on DocVQA. $N_{\mathrm{fov}}$ is the number of additional visual tokens acquired by foveation, $\rho$ is the fraction of high-resolution image area revealed, and $T_{\mathrm{fov}}$ is the number of foveations; these are not equivalent to the main table's overall visual-token statistic.

Config	$\lambda_{\mathrm{reg}}$	$N_{\mathrm{fov}}$	$\rho$	$T_{\mathrm{fov}}$	DocVQA score
Low-resolution Qwen2.5-VL baseline	N/A	N/A	0%	N/A	55.1
Coldstart only	N/A	35.9	3%	2.3	64.2
Coldstart + RL	1.0	62.5	5%	2.2	61.5
Coldstart + RL	0.5	269.3	22%	2.3	81.5
Coldstart + RL	0.2	354.8	25%	2.3	83.3
Coldstart + RL	0.0	377.8	28%	2.4	74.8

The original-resolution Qwen2.5-VL reference scores 90.4, indicating that selective observation recovers substantial performance without completely eliminating document-detail loss. Changing the regularizer from 0.2 to 0.0 increases additional tokens from 354.8 to 377.8 but reduces the score from 83.3 to 74.8; more evidence is not automatically more useful. The authors interpret the decrease as possible distraction from unconstrained observation, but the table alone does not establish the specific source of that distraction.

Key Findings¶

Table 6 on page 14 measures V7W on a single H100, with latency from tokenization to final prediction and throughput defined as total processed tokens per second. The table below preserves the intermediate-reasoning condition so that speeds with different output lengths are not conflated.

Model	Intermediate reasoning	Foveated visual tokens	Text tokens	Throughput (tok/s)	VRAM (G, paper's unit)	Latency (s)
VisCoT 7B	No	576	0	58.9	25	9.78
FoveateR 3B	No	80.6	0	76.0	24	1.06
FoveateR 3B	Yes	76.6	121.9	75.8	25	2.62

The text-token column in Table 6 concerns intermediate textual overhead here; a value of 0 should not be interpreted as producing no final answer. Table 4 on page 13 also shows that removing intermediate coldstart rationales need not harm post-RL results: 3B DocVQA changes from 74.0 to 77.3, while OpenImages falls from 77.6 to 71.4. Acquiring the right observation can therefore matter more than additional narration, but the results do not establish that intermediate reasoning is unnecessary for every task.

Highlights & Insights¶

Evidence acquisition becomes part of the generation policy rather than an external fixed cropping procedure. Existing text and visual evidence jointly determine the next observation, fitting tasks that require progressive verification of details.
A continuous action head bypasses coordinate-text parsing while cooperating with language reasoning through the hidden state. The transferable design is shared state with separate output spaces, not an expanded localization vocabulary.
Applying the area constraint only to correct trajectories expresses a learn-to-succeed-before-economizing principle. Readers could explore this principle for other on-demand retrieval systems, but the reward gate would require fresh validation.

Limitations & Future Work¶

Efficiency depends on the task and output policy: Table 6 covers only V7W on a single H100, so its speedup ratios do not directly establish gains for other resolutions, batch sizes, or devices.
Major system comparisons change backbone size and training procedure together; the main text lacks an ablation that removes shared caching while holding other conditions constant, leaving its independent benefit unresolved.
No human foveation-box supervision does not mean no localization data: the training mixture still includes RefCOCO/+/g, and coldstart uses teacher-generated boxes.
The main paper leaves the action distribution and several implementation details to supplementary material, while this note relies on the main-text cache; these gaps prevent full training reproduction from the note alone.
Reader-proposed follow-ups include strict per-sample token limits, repeated-region penalties, and evidence compression, since current results primarily reflect average adaptive-budget gains and the KV-cache still grows with history.

Compared with VisCoT and UV-CoT: these represent multi-pass visual focusing, whereas FoveateR inserts evidence into the same decoding trajectory; the distinction concerns restarting reasoning as well as crop size.
Compared with text-grounded approaches such as VPT: localization tokens or coordinate strings occupy a discrete output space, whereas this method delegates geometry to a continuous policy while retaining a trigger marker for action dispatch.
Compared with token pruning / merging: compressing an encoded representation cannot recover details absent from the initial observation, making acquisition and compression potentially complementary; their combined benefit is a research direction, not a result established here.
Compared with early recurrent visual attention: integrating local glimpses through memory is not a new idea; this paper's contribution is implementing it within pretrained VLM interleaved decoding and task-driven training.

Rating¶

The following are subjective reader ratings, not assessments issued by the conference or authors. - Novelty: 4/5. The combination of shared decoding state and continuous geometric actions is clear, while foveation and active perception have established precedents. - Experimental Thoroughness: 4/5. The paper covers multiple tasks, budget ablations, and end-to-end timing, but does not fully isolate state mechanisms from backbone differences. - Writing Quality: 4/5. The POMDP explanation and examples aid understanding; exact implementation depends on supplementary material, and the cache separately suffers equation-extraction damage. - Value: 4/5. The approach is informative for document and detail-sensitive VQA under restricted visual budgets, with deployment gains requiring task- and hardware-specific measurement.