Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/ExplainableML/ug-framework
Area: Multimodal VLM
Keywords: uncertainty estimation, visual search, video frame sampling, temporal grounding, training-free inference

TL;DR

UG turns MLLM response entropy or affirmative confidence into a visual evidence selection signal for local search, video sampling, and temporal grounding without updating parameters, improving InternVL2.5-8B V*Bench accuracy from 71.7% to 91.1% while requiring additional candidate-scoring computation.

Background & Motivation

Multimodal large language models (MLLMs) can describe images and answer video questions yet often miss the details that determine the answer. Downsampling high-resolution images can remove crucial small-object information, while uniform sampling of long videos can leave the key action outside the context entirely. Temporal grounding adds another difficulty: recognizing an event does not necessarily enable a model to produce accurate start and end times. These failures share a question about which evidence deserves the limited visual context, rather than simply how much language reasoning is needed. Existing approaches compensate through generated crop boxes, external visual scoring models, or specialized temporal instruction tuning, introducing additional models, data, or task-specific mechanisms.

The paper asks whether the model itself can provide a query-dependent signal for selecting visual input. Output entropy has often been used to detect errors or hallucinations after generation; UG moves this signal earlier, using uncertainty induced by candidate visual inputs to decide where to look next. On V*Bench, the authors divide images into crops with side length equal to 1/6 of the shorter image dimension and observe that target-containing crops generally produce lower response entropy; Figure 1 shows distributions and spatial heatmaps. This supports the hypothesis that relevant visual evidence can reduce uncertainty, but it does not establish that greater model confidence always implies correctness. In particular, when asked whether an action occurs, a model can confidently answer that it does not, so entropy minimization alone may select clearly irrelevant segments.

UG retains the pretrained MLLM and reformulates the task as comparisons between candidate regions or temporal windows, followed by answering or localization using the selected evidence. Its unification concerns the scoring principle and two-stage workflow, not the removal of task differences between cropping, frame selection, and contiguous interval search. Core Idea: use query-conditioned response uncertainty to find useful visual evidence; select low-entropy inputs for open-ended questions, but select contiguous intervals supporting an affirmative response for event localization.

Method

Overall Architecture

The input is an image or video together with a question or event description; the output is an answer or an event's start and end times. The framework first scores candidate visual inputs and then makes the final decision; provisional answers generated during scoring are not directly used as final answers. Its three branches are Entropy-Guided Visual Search, Entropy-Guided Video Sampling, and Affirmative-Confidence Temporal Grounding, corresponding to UG-Search, UG-Sample, and UG-Ground. The first two reorganize visual context and invoke the model again; the third directly computes an interval from a confidence sequence rather than depending on a second free-form timestamp generation. The three middle nodes below are alternative task branches, not steps that must run consecutively.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Visual input + query"] -->|High-resolution image| B["Entropy-Guided Visual Search<br/>Candidate crops → lowest-entropy region"]
    A -->|Video question answering| C["Entropy-Guided Video Sampling<br/>Candidate frames → low-entropy frame set"]
    A -->|Event description| D["Affirmative-Confidence Temporal Grounding<br/>Window scores → maximum-sum interval"]
    B --> E["Answer again using selected crop"]
    C --> F["Order frames temporally and answer"]
    D --> G["Return start and end times"]

Uncertainty comes from token probability distributions during generation, not from an additional trained confidence head or a request for a textual confidence estimate. Deployment therefore requires access to output probabilities; an interface returning only text without token probabilities cannot directly reproduce the mechanism. Independent candidate scoring permits batching in principle, but this does not mean the reported implementation already performs fully parallel scoring.

Key Designs

1. Entropy-Guided Visual Search: identify the region worth inspecting through the response distribution

UG-Search produces candidate crops with a square sliding window and scores each against the same original question. During scoring, the model receives both the original image and the current crop: the former provides global context, while the latter preserves local detail, after which response entropy is computed. The score does not merely inspect the probability of the selected output word; it computes Shannon entropy over the entire vocabulary at each step and averages over response length. Let \(T\) denote generation length, \(N\) vocabulary size, and \(p_{i,j}\) the probability of vocabulary item \(j\) at step \(i\); the definition is:

\[ \mathcal{H}(v,q)=-\frac{1}{T}\sum_{i=1}^{T}\sum_{j=1}^{N}p_{i,j}\log p_{i,j}. \]

This is a normalized transcription of the textual definition in Section 3.2; equation layout is damaged in the supplied text extraction, so it is not a character-for-character reconstruction of the printed equation. Averaging prevents a longer response from accumulating greater total entropy solely because it contains more tokens, but does not guarantee perfect comparability between different response contents. A lower score indicates less hesitation under the current visual conditions, which the authors use as a proxy for evidence relevant to the question. After scoring all candidates, the method selects the single lowest-entropy crop and uses only that crop for final answering. This distinction matters: scoring has global context, but the final answer is not produced by feeding back the original image together with every candidate.

Visual search experiments use a crop side-length ratio of 1/6, while standard QA uses 1/2; the stride is half the crop side length. The motivating experiment explicitly defines 1/6 relative to the shorter image dimension; the experimental setup uses the shorter phrase "one-sixth of the image," so exact implementation details still require the code or appendix. Settings are fixed within each task group rather than tuned for individual benchmarks. To reduce repeated computation, the authors cache the original image's key-value states (KV cache) for reuse across crop-scoring passes. This saves repeated context processing but does not remove crop encoding or provisional response generation costs. Compared with asking an MLLM to generate a precise box first, UG uses candidate enumeration and comparison to avoid requiring direct coordinate prediction.

2. Entropy-Guided Video Sampling: reserve the frame budget for evidence that answers the current query

UG-Sample treats frames or short windows from a long video as candidates and scores them with the same query-conditioned average entropy. The default experiment uniformly samples 256 candidate frames, scores one frame at a time, and selects the 8 lowest-entropy frames for final question answering. The baseline uniformly selects 8 frames directly, so the final answering frame count is equal, but UG first examines a larger candidate pool. This is not a comparison with identical total inference computation; only the final-round frame budget is matched. Selected frames are sorted by their original timestamps before combining the context, allowing the model to use temporal order instead of mistaking confidence rank for event order. The intermediate representation is a list of frames and their query-conditioned entropies; final generation receives the filtered image sequence, not a natural-language explanation of these scores.

This strategy suits questions with sparse decisive evidence, such as how players celebrate a particular goal. It may be less suitable for counting an action across an entire video, because several low-entropy slots can be occupied by one especially clear occurrence. The authors therefore introduce UG-Sample+AKS, retaining AKS's segment-wise temporal coverage while replacing CLIP similarity with entropy-based scoring. This hybrid does not require additional training or a different answering model; it constrains which parts of the timeline should contribute evidence. It distinguishes a highly relevant individual frame from a set of frames sufficient to answer the question, which matters for counting and global summarization. Window-extension experiments also score multi-frame contexts and then select window-center frames as candidates; those configurations should not be conflated with the default single-frame setup.

3. Affirmative-Confidence Temporal Grounding: recover a contiguous segment from event-presence scores

UG-Ground must determine where an event occurs, rather than merely find a maximally certain response. With binary response entropy alone, background windows that confidently elicit "no event" also receive low scores and become confused with target windows. The authors instead use Binary Response Confidence (BRC), asking whether the described event is depicted in each video window. Following the explicit "difference between probabilities" definition in Section 3.2, the score subtracts the no-token probability from the yes-token probability in the first generated distribution:

\[ \operatorname{BRC}(v,q)=p_1(\mathrm{yes})-p_1(\mathrm{no}). \]

The extracted Equation (2) loses the intervening operator; this transcription follows the prose definition rather than interpreting the adjacent probabilities as multiplication. A positive score favors presence and a negative score favors absence; it is neither a renormalized probability over only yes/no nor a calibrated confidence estimate. Sliding-window evaluation turns model outputs into a signed temporal score sequence. Section 3.3 defines selection as a maximum-sum contiguous subarray problem; using \(s_t\) for the BRC score of window \(t\), its objective can be expressed as:

\[ (a^*,b^*)=\arg\max_{a\leq b}\sum_{t=a}^{b}s_t. \]

This notation explains the algorithm described in the paper; it is not an additional numbered equation from the authors. Contiguity produces an event interval rather than scattered high-scoring frames, while negative background scores generally discourage expansion into irrelevant content. Kadane's algorithm finds this interval in linear time; model forward passes remain the main expense, so the entire system should not be described as a cheap linear scan. The default uses a 15-frame window and stride 1; Charades-STA is sampled at 3 FPS and ActivityNet Captions at 1 FPS. The best subarray's start and end indices are mapped to timestamps instead of requiring a general-purpose MLLM to learn direct temporal coordinate generation. The contiguous-interval assumption also means that one maximum-sum segment may not express every instance when the same event occurs repeatedly with interruptions.

A Worked Example

Figure 2 illustrates grounding the event of a person sitting on a chair at temporal positions from 1.0 through 8.0 seconds. The corresponding BRC sequence is \([-0.77,-0.79,-0.69,0.78,0.91,0.93,-0.83,-0.92]\). The first three positions favor a negative answer, while 4.0 through 6.0 seconds provide contiguous positive support with sum \(0.78+0.91+0.93=2.62\). Adding any adjacent negative position reduces the sum, making 4.0 through 6.0 seconds the maximum-sum interval, as shown in the figure. These are illustrative framework values, not dataset-level performance, and do not establish second-level boundary accuracy under every window-to-time mapping rule.

Loss & Training

UG adds no training loss, updates no MLLM parameters, and requires no new supervised fine-tuning dataset for the three tasks. The backbone is still a pretrained and instruction-tuned MLLM; "training-free" refers specifically to adding UG. Scoring and answering can use the same model or different models, such as a smaller InternVL2.5 scorer selecting evidence for a fixed 8B answering model. Other efficiency options include external visual pre-filtering, larger temporal strides, original-image KV reuse, and candidate-level batching. These are deployment options, not components required by every default branch; external pre-filtering in particular can reintroduce the risk of missing small targets. The authors explicitly state that the current implementation scores sequentially because of resource and stability constraints, leaving full parallelization for future optimization.

Key Experimental Results

Main Results

The following table selects representative results from Table 1 on page 6, Table 4 on page 8, and Table 5 on page 9. QA scores are accuracy percentages and temporal grounding uses mIoU on a percentage scale; gains are absolute points, not relative percentages. Video QA selects 8 of 256 candidate frames, with no subtitles for Video-MME; grounding baselines are capped at 64 input frames, whereas UG uses the sliding-window setup described above.

Task / Dataset	Model	Metric	Baseline	UG	Gain	Source
Visual search / V*Bench	InternVL2.5-8B	Accuracy	71.7	91.1	+19.4	Table 1
Visual search / HR8K	InternVL2.5-8B	Accuracy	58.0	73.6	+15.6	Table 1
Long-video QA / Video-MME	Qwen2.5-VL-7B	Accuracy	53.7	59.8	+6.1	Table 4
Long-video QA / LongVideoBench	Qwen2.5-VL-7B	Accuracy	52.7	60.5	+7.8	Table 4
Temporal grounding / Charades-STA	LLaVA-OV-7B	mIoU	10.4	46.8	+36.4	Table 5
Temporal grounding / Charades-STA	InternVideo2.5-8B	mIoU	32.3	51.0	+18.7	Table 5
Temporal grounding / ActivityNet Captions	InternVideo2.5-8B	mIoU	14.0	38.5	+24.5	Table 5

Table 5 also reports recall at different IoU thresholds; gains at a lower threshold should not be treated as equivalent improvements in strict boundary localization. Results support effectiveness across backbones and tasks, but differing candidate budgets, baseline capabilities, and metrics prevent ranking task difficulty by the size of these gains.

Ablation Study

Table 6 on page 13 fixes the answering model to InternVL2.5-8B and changes only the scoring model size. Throughput is measured in examples per second, where higher is better; together with accuracy, it exposes the cost of additional scoring.

Scoring model	V*Bench accuracy	Search throughput	Video-MME accuracy	Sampling throughput
No scoring	71.7	1.961	57.8	0.476
InternVL2.5-1B	75.4	0.139	58.5	0.107
InternVL2.5-2B	79.1	0.097	59.0	0.097
InternVL2.5-4B	89.0	0.078	60.1	0.070
InternVL2.5-8B	91.1	0.060	60.6	0.054
InternVL2.5-26B	92.0	0.037	61.8	0.036

The 4B scorer trails 8B self-scoring by 2.1 points on V*Bench while providing 1.3 times its throughput; both remain substantially slower than the baseline without candidate scoring. A stronger scorer helps, but this observation alone does not prove that its probabilities are well calibrated.

The frame-selection comparison in Table 3 on page 7 uses LLaVA-OV-7B, with QA accuracy percentages for all three metrics. It analyzes complementarity between scoring and temporal coverage rather than removing a trained network module from an otherwise identical model.

Sampling method	Video-MME	MLVU	LongVideoBench
AKS	56.1	64.8	56.8
UG-Sample	58.6	62.0	59.5
UG-Sample+AKS	59.2	65.0	59.4

Key Findings

• Large gains concentrate on tasks prone to missing local evidence; smaller gains on standard QA do not establish equal improvement in every visual capability.
• Pure UG-Sample trails AKS on MLVU, which the authors attribute to the full-video coverage required by Action Count; hybrid sampling shows that entropy ranking cannot replace coverage constraints.
• Table 8b on page 14 increases the stride of a 15-frame window from 1 to 3: Charades-STA mIoU stays at 51.0 while throughput rises from 0.150 to 0.403, indicating redundancy in dense scoring.

Highlights & Insights

• Uncertainty becomes an active perception signal instead of a post-hoc warning. Candidate comparison can find useful evidence without first teaching the model to emit accurate coordinates.
• BRC retains the direction of support for event presence. It addresses the specific failure of low entropy rewarding both confident affirmation and confident rejection.
• Decoupling scoring from answering allows test-time compute to be allocated separately to evidence search and final generation. Table 6 makes the quality and cost of this allocation visible.

Limitations & Future Work

• The authors acknowledge substantial inference overhead from repeated forward passes; parallelizable candidates do not make total computation free, and the reported implementation is not fully parallel.
• Reader assessment: low entropy may still reflect language priors or incorrect confidence; the correlation in Figure 1 does not guarantee reliability across all distributions, questions, and crops.
• Reader assessment: a single region, a global low-entropy frame set, and a single contiguous interval may respectively miss cross-region relations, repeated actions, and separated event instances.
• Source boundary: the supplied full text includes the main paper and references but not the appendix mentioned in the text; it cannot establish complete prompts, entropy-scoring generation lengths, or every implementation edge case.
• Source inconsistency: LLaVA-OV-7B V*Bench UG-Search is 86.9 in Tables 1/2 but 81.2 in Table 7a; the supplied Section 4.5 does not sufficiently explain configuration differences, so these must not be merged into one setting.
• The setup paragraph in Section 4.3 names UG-Sample when describing the BRC sliding-window configuration, although the context is clearly UG-Ground; this note interprets the task rather than perpetuating the naming inconsistency.

Related Work & Insights

• Compared with ZoomEye, reference [30] in the paper: both are training-free; ZoomEye uses tree-based image exploration, while UG compares candidates through response entropy. Table 2 compares matched backbones but does not establish identical search budgets.
• Compared with AKS, reference [32] in the paper: segment coverage can complement UG's query-dependent scoring, as supported by the hybrid results in Table 3.
• Compared with TimeChat / VTimeLLM, references [28]/[11] in the paper: those approaches strengthen temporal expression through temporal training, whereas UG-Ground reformulates the task as window classification followed by deterministic interval search.
• Research direction: combining coverage, diversity, and low entropy in evidence selection could be tested for reducing missed occurrences in counting tasks. This is an idea motivated by the results, not an algorithm validated in this paper.

Rating

The following is the note author's subjective assessment, not a conference review score. - Novelty: 4/5; existing uncertainty signals are repurposed for active visual localization, with a task-specific rationale for directional BRC. - Experimental Thoroughness: 4/5; multiple tasks, backbones, and cost analyses are covered, but incorrect confidence and failure cases warrant more systematic study. - Writing Quality: 4/5; the shared principle and task instances are clear, although extracted equations, some configuration descriptions, and cross-table discrepancies require care. - Value: 4/5; a useful baseline for open models exposing output probabilities, provided that the additional scoring budget is made explicit.