ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models¶

Conference: ECCV 2026
arXiv: 2603.19466
Code: https://github.com/tdemin16/ProactiveBench
Area: Multimodal VLM
Keywords: Multimodal Large Language Models, Proactiveness, Benchmark, Ambiguous Queries, GRPO

TL;DR¶

This paper proposes ProactiveBench, the first benchmark to evaluate the "proactiveness" of Multimodal Large Language Models (MLLMs). By adapting 7 existing datasets into multi-turn interactive environments where "the initial frame is blurry and user intervention is required to find the correct answer," the authors evaluate 22 MLLMs. The results reveal that they rarely seek help proactively (either abstaining or hallucinating) and that proactiveness does not emerge with model scale. However, fine-tuning via GRPO with customized rewards can teach models to be proactive and generalize to unseen scenarios.

Background & Motivation¶

When faced with incomplete or ambiguous information, humans instinctively formulate hypotheses, actively seek clues, and refine their judgments—for instance, to identify an obscured object, one might ask someone nearby to move the obstacle. However, current Multimodal Large Language Models (MLLMs) are almost exclusively studied in "reactive" scenarios: given an image and a question, the model either directly outputs an answer or chooses to abstain when unable to answer. When the information in the image is completely insufficient (for example, asking "what is behind the blue block" when the block fully obscures the object behind it), reactive models only have two options—hallucinate an incorrect answer out of thin air, or simply say "I don't know." Neither behavior constitutes a good collaborator.

A more ideal behavior is "proactive": the model realizes the current visual information is insufficient, so it proactively suggests a simple intervention action to the user (e.g., "Could you move the block to the right?"), and answers after receiving the new visual clues. This task is challenging because the model itself cannot act in the physical world; it can only use natural language to tell the user what to do, leaving the execution and visual feedback to the user. This ability to "know what one does not know and know what help to request" has never been systematically measured. Existing benchmarks assume all questions are answerable, while the few works addressing active perception only perform fine-tuning on a single image or cover single-turn dialogues in a single scenario, failing to capture the complete picture of "multi-turn interactions where actions bring substantial changes (varying viewpoints, image quality, or timestamps)."

This work represents a classic "gap-filling" contribution: proactiveness has not been explored previously due to the lack of standardized evaluation protocols and the difficulty of annotating whether an image is informative for a question. The authors' key enabling insight is that many existing datasets naturally possess a sequential frame structure ranging "from most difficult to easiest to recognize" (e.g., obstacles being removed frame-by-frame, or sketches being added stroke-by-stroke). "Reusing and adapting" these datasets allows for low-cost, automated generation of proactive evaluation environments. Core Idea: Formalize MLLM proactiveness as a Markov Decision Process (MDP)—where the model chooses between "abstaining, guessing the category, or suggesting an intervention action to acquire more clues." A multi-turn environment adapted from 7 datasets is utilized to measure whether the model can step-by-step reach the correct answer through proactive suggestions. Furthermore, it is demonstrated that while this ability is not easily elicited through prompts or context, it can indeed be learned using reinforcement learning with tailored rewards.

Method¶

Overall Architecture¶

ProactiveBench is not a single model or method, but a comprehensive framework consisting of an "evaluation protocol + data construction pipeline + empirical analysis + learnability verification." It first defines "proactiveness": the model must either output the correct answer or suggest an action that makes the question answerable. The evaluation is conducted under two settings: Multiple-Choice Question Answering (MCQA), which supports multiple turns and controllability, serving as the primary vehicle for analysis; and Open-Ended Generation (OEG), which does not provide options and is closer to real-world scenarios, but is limited to a single turn because recommended actions may not be executable in the environment. On the data side, the authors select 7 datasets, each corresponding to one type of proactive scenario. Using rule-based automated annotation, they compile sequences ranging from "initial noisy/blurred frames \(\rightarrow\) intermediate frames \(\rightarrow\) reference frames (fully informative)." A filter is then applied to exclude samples where the correct answer can be guessed on the very first frame without requiring any proactiveness. Finally, evaluations on 22 MLLMs reveal a universal lack of proactiveness, and fine-tuning via GRPO is conducted to verify that proactiveness is learnable and generalizable.

The entire process can be divided into three stages: "Construction \(\rightarrow\) Evaluation \(\rightarrow\) Learning", as illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["7 Existing Datasets<br/>(ROD/VSOD/MVP-N etc.)"] --> B["MDP Formalization of Proactiveness<br/>State=Image+Available Actions, Action=Abstain/Guess/Proactive Suggestion"]
    B --> C["Seven-Scenario Data Construction<br/>Blurry-to-Reference Frame Sequences + Rule Annotation"]
    C --> D["First-Round Leakage Filtering<br/>Filter out samples guessable on the first frame"]
    D --> E["22 MLLM Evaluation<br/>MCQA(Multi-turn) + OEG(Single-turn)"]
    E -->|Findings: Universally not proactive, does not emerge with scale<br/>Prompts/History/Few-shot degrade performance| F["GRPO with Customized Reward Tuning<br/>Verify learnability + generalize to unseen scenarios"]

Key Designs¶

1. Formalizing Proactiveness as a Markov Decision Process: Making "Seeking Help" an Evaluable Action

The hardest part about proactiveness is that it is not a static QA task, but a sequence of "observation-decision-environment change" interactions. Without formalization, systematic measurement is impossible. Drawing on the perspective of "LLM as an agent," the authors model each sample as a finite MDP \((\mathcal{S}, \mathcal{A}, \pi_\theta, R)\): the state \(s_t=\{I_t, A_t\}\) consists of the current image \(I_t\) and the set of currently available actions \(A_t\). The policy \(\pi_\theta\) is the MLLM being evaluated, which samples an action \(a_t \sim \pi_\theta(\cdot\mid q, s_t)\) given the question \(q\) and state \(s_t\). The action set contains "abstain," "four candidate categories (with only one correct)," and several "proactive suggestions" (e.g., "move the obstacle to the left"). The key lies in the state transition rule: only selecting a proactive suggestion progresses the environment to \(s_{t+1}\) (replacing the image with a clearer one and offering a new set of actions). If the model chooses to abstain or selects an incorrect category, the evaluation immediately terminates as a "failure"; selecting the correct category terminates as a "success." The number of times a model can be proactive in each environment is limited (depending on the number of frames in the dataset); if it fails to answer correctly after exhausting these turns, it is marked as incorrect. The elegance of this design is that it puts "proactive help-seeking" and "answering" into the same action space for competition, thereby distinguishing whether a model genuinely uses proactive suggestions to approach the answer or is simply guessing blindly.

2. Adapting Seven-Scenario Datasets: Transforming Existing Data into Proactive Environments Using Rules

Constructing proactive data from scratch is costly and requires manual annotation to judge if a frame contains enough information. The authors instead reuse 7 existing datasets, each corresponding to a type of proactive behavior: ROD (moving obstacles to inspect blocked objects), VSOD (waiting/backplaying in videos to avoid temporal occlusions), MVP-N (rotating objects to find a more informative viewpoint), ImageNet-C (improving the quality of corrupted images), QuickDraw (asking the user to add drawing details), ChangeIt (requesting past/future frames to reveal key actions), and MS-COCO (moving the camera to shift viewpoint). Automatic annotation is feasible because ROD, MVP-N, QuickDraw, and ImageNet-C naturally order frames "from most difficult to easiest to recognize"—for instance, each ROD sample has 14 frames, with the middle frame heavily occluded and targets gradually revealed toward both ends; hence, the hardest frame can be directly taken as the initial input. For ChangeIt, the first frame is selected (usually uninformative), while for COCO, images containing a single bounding box are chosen to generate difficult crops with low IoU. Since VSOD lacks category labels, the authors identify celebrity names using Google Image Search and discard samples that fail. The final pool contains 108k images across 18k samples, covering 19 proactive behaviors. This "reuse and adapt" methodology allows a large-scale, diverse proactiveness benchmark to be implemented cost-effectively.

3. First-Round Leakage Filtering: Filtering Out Samples Answerable Without Proactiveness

One hidden risk is that even when presented with the hardest-to-recognize frames, models can sometimes guess correctly on the spot (e.g., in ImageNet-C, 55.3% of the samples are guessed correctly on average in the first frame). Allowing models to score on these samples without user intervention would bias the proactiveness estimation and introduce task-level difficulty imbalance. The authors' filtering rule is: a sample is discarded if it is correctly answered in the first round of MCQA by at least 25% of all tested MLLMs. This threshold balances "clean filtering" and "retaining a sufficient sample size." After filtering, the average first-round accuracy plummets from 32.5% to 6.4%, indicating that the remaining samples almost certainly require proactive suggestions to be answered correctly. The final benchmark is scaled down from the initial 17,909 samples to 7,557. This step is a prerequisite for "trajectory accuracy to truly measure proactiveness."

4. Metrics: Trajectory Accuracy + Proactive Suggestion Rate, Distinguishing "Effective Proactiveness" from "Blind Actions"

Since the answer-leaked samples are thoroughly filtered, the authors utilize trajectory accuracy (acc) to measure meaningful proactiveness—defined as the proportion of proactive sequences that eventually reach the correct answer. This metric rewards "approaching the correct state through proactive suggestions" rather than simply proposing more queries. They also report the proactive suggestion rate (psr), which is the average number of user interventions requested by the model. For OEG (single-turn), it is split into two metrics: the correct category selection rate (cc) and the proactive suggestion rate (ps) in the response. These are evaluated by an LLM-as-judge (Qwen3-8B), which allows paraphrasing (e.g., "change viewpoint") to count as valid. The authors emphasize that MCQA and OEG metrics cannot be directly compared—the former is the final accuracy of the trajectory, while the latter is the hit rate of the first-round response. The value of this metric set lies in its ability to expose "fake proactiveness": if valid proactive suggestions are replaced with random, invalid ones, the psr of truly proactive models drops drastically (e.g., an 86% relative decrease for LLaVA-OV-7B), whereas models that merely "dislike abstaining and prefer blind guessing" continue to choose those random suggestions, exposing their lack of genuine proactiveness.

Loss & Training¶

When verifying whether proactiveness can be learned, the authors employ reinforcement learning instead of supervised fine-tuning. This is because deciding "when to suggest and when to answer directly" requires dense annotations of frame information, which are generally unavailable. Specifically, GRPO is used: for each prompt-image pair, 8 outputs are sampled, and rewards are distributed based on rules—correct category \(r_c=1\); valid proactive suggestion \(r_p\in\{0.5, 0.75, 1.0\}\); and others \(r_w=0\). The intuition is that by setting the proactive suggestion reward lower than that of the correct category (\(r_p < r_c\)), the model learns to prioritize direct answering, falling back on proactive suggestions only when uncertain, thereby balancing efficiency (low psr) and accuracy. Training is conducted on only two scenarios, QuickDraw and COCO (covering both abstract and natural images, leaving the others for generalization testing), and is restricted to single-turn interactions. Both blurred and clear frames are sampled in training so that the model learns "which situations require proactiveness and which can be directly predicted."

Key Experimental Results¶

Main Results¶

Evaluating 22 open-source and closed-source MLLMs on ProactiveBench yields the core conclusion: models are universally lacking in proactiveness, and proactiveness does not scale monotonically with model size.

Setting	Model	Key Metrics	Value	Description
MCQA	GPT-4.1	avg. acc	30.9%	One of the best closed-source models, but achieves up to 94.4% on COCO, likely due to data contamination
MCQA	o4-mini	avg. acc	34.0%	Highest overall acc across open- and closed-source models
MCQA	InternVL3-1B	avg. acc	27.1%	Outperforms InternVL3-8B (12.7%); performance worsens with larger scale
MCQA	LLaVA-1.5-7B	avg. acc	24.8%	Older model outperforms the newer, larger LLaVA-OV-72B (13.0%)
MCQA	LLaVA-NeXT-Mistral-7B	avg. acc	4.5%	Worst performance, prone to abstaining
Reference setting	All-model average	avg. acc	79.8%	Answered correctly when given full reference frames, showing the gap stems from a lack of proactiveness rather than recognition capability

In the reference setting (directly feeding fully informative reference frames), the models average an accuracy of 79.8%, but this plummets by over 60 percentage points on the proactive-requiring ProactiveBench. The ROD scenario is particularly severe: 98.3% in the reference setting vs. only 8.2% on ProactiveBench. This directly proves that models "can understand" but "cannot proactively seek help."

Ablation Study¶

The authors systematically analyzed various methods to "elicit proactiveness," and almost all conclusions were negative:

Intervention Method	Impact on psr	Impact on acc	Conclusion
Add prompt (hint)	+1.9 (avg.)	25.8%, only +8.3%	Prompts elicit more suggestions, but acc does not exceed random choice, and models blindly select suggestions until exhausting steps in 16.0% of cases
Add dialogue history	0.5 → 1.8	Decrease of ~7%	Past proactive suggestions in the history bias the model to repeat them mindlessly, degrading performance
few-shot (ICL)	ROD +1.6 / MVP-N +0.5 (3-shot)	ROD drops, MVP-N stays flat	In-context examples introduce negative bias, prompting models to copy example categories or select suggestions frantically
Replace with random invalid suggestions	Truly proactive models drop significantly (-60%~-90%)	—	Exposes "fake proactiveness": those that do not drop are just "unwilling to abstain and guess randomly" rather than truly proactive

GRPO fine-tuning is the single positive outcome in the paper:

Model	Config	In-domain avg. acc	Out-of-domain avg. acc	Description
Qwen2.5-VL-3B	original	~7%	~11%	Before fine-tuning
Qwen2.5-VL-3B	\(r_p=0.75\)	46.2/49.4 (QD/COCO)	38.6	Outperforms all original models (including o4-mini at 34.0%), with CIT rising from 12.4% to 55.6%
Qwen2.5-VL-3B	\(r_p=1.0\)	—	15.1	Reward is too high; model excessively issues suggestions and rarely answers, causing acc to collapse
LLaVA-NeXT-Mistral-7B	\(r_p=0.5\)	43.6/57.3	40.4	Jumps from the worst performer (4.5%) to over 40%

Key Findings¶

Proactiveness is not an emergent property of model scale: InternVL3-1B outperforms 8B, and LLaVA-1.5-7B outperforms LLaVA-OV-72B. The choice of LLM base matters more than parameter size (e.g., LLaVA-NeXT using Vicuna reaches an acc of 19.3%, far exceeding the 4.5% of the Mistral-based version, which is more prone to abstaining).
"More proactive suggestions" does not equal "more proactive": after replacing valid suggestions with random invalid ones, the psr of truly proactive models plummets, whereas LLaVA-NeXT-Vicuna actually increases from 37% to 49% to choose those invalid suggestions—indicating it only "dislikes abstaining and prefers guessing blindly." Meaningful proactiveness occurs if and only if the model utilizes suggestions to reach a better state and improve acc.
The three tuning-free strategies (prompts, history, and few-shot) all make models "mindlessly issue suggestions until exhausting available steps." While psr increases, acc drops, suggesting that current MLLMs have not internalized the judgment of "when to be proactive."
After GRPO fine-tuning, proactiveness generalizes to unseen scenarios (trained only on QD+COCO, yet performance universally improves on ROD/VSOD/IN-C/CIT). However, a gap of approximately 35 percentage points remains compared to the reference setting (40.7% vs. 75.1%), indicating that a complete solution is still far off.
Reward design requires \(r_p < r_c\): once the proactive suggestion reward is set equal to the correct category reward (\(r_p=1.0\)), the model over-generates suggestions and rarely answers, leading to collapsed acc.

Highlights & Insights¶

The idea of "reusing and adapting existing datasets" is highly cost-effective: the main hurdle in proactiveness data is annotating whether a frame is informative. The authors find that ROD, MVP-N, QuickDraw, and ImageNet-C naturally order frames "from difficult to easy." Taking the hardest frame directly as the initial input completes the annotation, easily uniting 7 unrelated tasks into a standardized proactive environment.
Exposing "fake proactiveness" through "random invalid suggestion replacement" is a clever tactic: judging by psr alone can falsely suggest that some models are highly proactive. Checking whether the psr collapses after replacement distinguishes "models genuinely using suggestions to converge on the answer" from "blind guessers unwilling to abstain." This probe is transferable to any evaluation assessing whether an agent performs effective exploration.
Modeling proactiveness as an MDP allows "abstaining, guessing categories, and proactive suggestions" to compete within the same action space. Coupled with first-round leakage filtering, this keeps trajectory accuracy as a clean metric of proactiveness. This formula of "formalization + filtering + dual-metrics" can be repurposed to evaluate other "know-what-you-don't-know" capabilities.
A counterintuitive yet crucial finding: dialogue history and in-context learning are counterproductive here. Past proactive suggestions bias the model into mindless repetition, serving as a reminder that when building interactive agents, it should not be assumed that "more context is always better."

Limitations & Future Work¶

The authors admit that even after GRPO fine-tuning, a significant gap of ~35 percentage points remains compared to the reference setting. Proactiveness is far from resolved, representing only a promising starting point.
Closed-source models (such as GPT-4/o4-mini) score exceptionally high on COCO (about 3 times that of other models). The authors suspect training data contamination, but this is unverifiable due to closed weights, casting doubt on the reliability of closed-source benchmark scores.
OEG evaluation relies on LLM-as-judge. Although a reliable Qwen3-8B model is used, the judgment of "valid proactive suggestions" remains inherently subjective. Moreover, due to computing constraints, OEG is evaluated on only 100 samples per scenario, presenting limited coverage.
Training only utilizes two scenarios (QD+COCO) and is restricted to a single turn. While generalizability is demonstrated, the training setup is simplified, leaving performance under multi-turn training as an open question.
VSOD uses Google Image Search to find celebrity names for category annotations and discards failed cases, which may introduce selection bias. In OEG, "valid proactive suggestions" rely on flexible synonym matching by the judge, which lacks strict boundaries.

vs. Active Perception Works (ActiView / MLLMs know where to look): These works also study models actively seeking information, but they typically perform adjustment or exploration on a single image that is "inherently answerable." In contrast, this work evaluates models across 7 scenarios and multiple turns under settings where actions bring substantial changes (shifting viewpoint, image quality, timestamps), enabling a more comprehensive analysis of failure cases and "fake proactiveness."
vs. Right this way [36]: This work explores whether VLMs can guide visually impaired users to take better photos, but only covers a single proactive scenario, single-turn dialogue, and does not measure the effectiveness of the recommendations. In contrast, this paper covers 7 scenarios and multiple turns, explicitly measuring whether proactive suggestions truly lead to the correct answer.
vs. Classic Active Vision Works: While both share the spirit of an "active observer dynamically controlling perception policies," this paper utilizes diverse real-world images where the observer receives feedback from the MLLM via natural language, establishing human-AI collaboration. This aligns closer to human-AI cooperative tasks rather than pure robotic systems.
vs. Reliable VQA / Abstention Works: Abstention research encourages models to "shut up when unable to answer" to avoid mistakes. This paper points out that abstaining is passive and sub-optimal; the truly ideal behavior is proactive help-seeking. Additionally, experiments demonstrate that many models that "frequently initiate proactive actions" are actually just "prone to not abstaining," clearly distinguishing the two behaviors.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The first benchmark to systematically evaluate the "proactive help-seeking" capability in MLLMs, framing an unformalized capability as a measurable and trainable problem.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely comprehensive evaluation involving 22 models, 7 scenarios, and two settings (MCQA+OEG), supplemented by five analysis groups (hint, history, few-shot, random suggestions, and GRPO).
Writing Quality: ⭐⭐⭐⭐ Clear logic and highly informative charts. The MDP formalization and the "fake proactiveness" probe are particularly elegant, though some metrics (cc, ps, psr) require careful cross-referencing to fully digest.
Value: ⭐⭐⭐⭐⭐ Points the way forward for collaborative multimodal agents that "know when to seek help." The dataset is open-source, and proving that proactiveness is learnable and generalizable holds high value for future research.