ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/showlab/ReGRPO
Area: Multimodal VLM
Keywords: Multimodal agents, Tool-use learning, Reflection, Reinforcement learning, GRPO

TL;DR

Addressing the issue where multimodal vision–language agents collapse upon encountering intermediate tool failures and lack explicit recovery signals, ReGRPO constructs a structured reflective data engine and leverages Group Relative Policy Optimization (GRPO) to jointly optimize diagnostic reflection tokens and corrective actions within local trajectories, achieving state-of-the-art open-source performance on GTA and GAIA under zero-verifier single-path inference.

Background & Motivation

With the rapid evolution of vision–language models (VLMs), augmenting them with external tools such as web search, OCR engines, table parsers, PDF readers, and code execution environments allows agents to resolve complex multimodal, multi-step tasks. However, existing multimodal tool-using agents remain fragile when deployed in real-world scenarios. Predominant paradigms, exemplified by MAT-AGENT, rely heavily on supervised fine-tuning (SFT) over successful trajectories synthesized on simulated tasks. This reliance ties the controller tightly to ideal expert demonstrations. Once encountering out-of-distribution variations during inference—such as unfamiliar PDF layouts or degraded document scans—a single inaccurate tool call causes execution to derail without any learned mechanism for local recovery, triggering catastrophic error cascades.

Conversely, although self-exploratory reinforcement learning frameworks like SPORT introduce process-level feedback, standard RL in long-horizon tool execution depends primarily on sparse terminal scalar rewards. When an agent executes an erroneous crop, observes an empty OCR output, and consequently delivers an incorrect final answer, receiving a scalar reward of 0 fails to identify which intermediate step or argument caused the failure and offers no gradient on how to rectify it. Furthermore, inference-time reflection frameworks like Reflexion typically treat self-critique as a frozen prompting heuristic or rely on multi-trial episodic trial-and-error, lacking mechanisms to internalize structured reflection and corrective execution directly into the policy parameters.

Consequently, tool-augmented multimodal agents require a policy optimization framework where diagnostic reflection acts as an explicit learnable decision variable. Core idea: convert real tool execution failures into structured Reflection-of-Thought (RoT) triplets capturing error types, grounded visual/textual evidence, and fix plans, then jointly optimize reflection tokens and corrective actions within local trajectories using Group Relative Policy Optimization (GRPO) paired with an adaptive confidence trigger for zero-verifier single-path self-correction.

Method

Overall Architecture

The ReGRPO framework comprises three core stages: an offline Structured Reflective Data Engine, Reflection-Augmented Group Relative Policy Optimization (ReGRPO) training, and a Zero-Verifier Inference stage with deterministic single-path execution. First, the data engine perturbs expert trajectory actions, executes near-miss steps in a sandbox to capture genuine execution errors, and prompts a teacher model to produce standardized Reflection-of-Thought (RoT) triplets used for SFT warm-start initialization. Next, during policy optimization, the agent samples groups of local candidate trajectories per task context, optimizes both reflection tokens and corrective actions via group-relative advantages, and imposes a reflection token cost penalty to curb unnecessary verbosity. Finally, during inference, the policy executes single-path trajectories without calling external verifiers, dynamically triggering at most one local reflection-correction block when runtime errors, empty observations, or low action confidence occur.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Input (Query, Files)"] --> B["1. Structured Reflective Data Engine<br/>Inject near-miss action→Execute failure→Generate RoT triplet"]
    B --> C["SFT Warm Start<br/>Maximize likelihood of failure diagnosis & corrected action"]
    C --> D["2. Reflection-Augmented Policy Optimization<br/>Sample K local candidate trajectories & compute group advantages"]
    D --> E["Reflection Cost Penalty & Reward Computation<br/>Balance execution success against reflection token verbosity"]
    E --> F["3. Zero-Verifier Single-Path Inference<br/>Trigger dynamically via execution error / empty obs / low confidence"]
    F -->|Normal Execution| G["Standard Direct Action Output"]
    F -->|Failure Triggered| H["Single Local Reflection-Correction Block<br/>a(0)→o(0)→z→a(1)→o(1)"]
    H --> G

Key Designs

1. Structured Reflective Data Engine: Converting Execution Failures into Grounded Recovery Triplets Existing imitation tuning corpora consist exclusively of successful executions, leaving agents completely unprepared for tool failures, while casually discarding failures forfeits critical causal supervision. The Structured Reflective Data Engine begins with ground-truth single steps from expert trajectories (MM-Traj) and synthesizes realistic near-miss actions \(a_i^{\text{fail}}\) by perturbing tool selections or arguments (such as shifting visual bounding boxes, picking neighboring table columns, or invoking mismatched operators). Running these actions in a sandbox yields grounded failure observations \(o_i^{\text{fail}}\) (e.g., API runtime errors, empty OCR strings, or irrelevant image crops). Conditioned on this failure context, a teacher VLM (GPT-4o) synthesizes a strictly structured Reflection-of-Thought (RoT) triplet \(z_i = (\text{ErrorType}, \text{Evidence}, \text{FixPlan})\): ErrorType provides categorical diagnosis (e.g., ToolMismatch, ArgInvalid, GroundingDrift, InfoInsufficient); Evidence mandates explicit textual or visual references to prevent hallucinated rationales; and FixPlan specifies an actionable natural-language recovery strategy. Pairing this triplet with the original ground-truth action \(a_i^*\) produces complete reflective trajectories for SFT initialization.

2. Reflection-Augmented Policy Optimization: Joint GRPO Over Reflection Tokens and Corrective Actions SFT alone confines the model to curated patterns, whereas conventional PPO or DPO fails to treat reflection as an explicit decision variable, and sparse rewards fail to pinpoint errors in long tool trajectories. ReGRPO adopts Group Relative Policy Optimization (GRPO), sampling a group of \(K\) candidate local trajectories \(\{\tau_i^{(k)}\}_{k=1}^K\) for a given context \(x_i\). If the initial action \(a_i^{(0)}\) succeeds, the trajectory terminates early; if an error or low consistency is encountered, the agent generates reflection \(z_i\) and executes correction \(a_i^{(1)}\). The composite reward function balances execution success against reasoning verbosity: $\(R(\tau) = \lambda_{\text{exec}} \mathbf{1}_{\{\text{success}\}} - \eta C(\tau) + \lambda_{\text{val}} V(x_i, \tau)\)$ where \(C(\tau)\) is a reflection token cost penalty (zero for one-shot successes), and \(\eta\) forces the agent to reflect only when the expected recovery gain outweighs token overhead. Group-relative advantages \(A_i^{(k)} = R(\tau_i^{(k)}) - \bar{R}_i\) are computed against the group mean \(\bar{R}_i\) and applied directly to update trajectory likelihood \(\log \pi_\theta(\tau_i^{(k)} | x_i)\). Because the generation probability of structured reflection \(z_i\) is explicitly factorized inside the trajectory likelihood, advantages directly scale gradients on reflection tokens, training the model to produce concise and effective diagnostics.

3. Zero-Verifier Single-Path Inference: Lightweight Adaptive Confidence Gating Invoking external verifier models or executing multi-candidate beam searches at inference time introduces prohibitive latency and computational overhead. ReGRPO adopts a deterministic single-path execution scheme, opening at most one local reflection-correction block \(a_i^{(0)} \to o_i^{(0)} \to z_i \to a_i^{(1)} \to o_i^{(1)}\) only when explicit or silent failure evidence emerges. The gate trigger \(g_i\) combines hard environment signals with model confidence: $\(g_i = \mathbf{1}_{\{\text{ToolError}(\hat{o}_i) \lor \text{EmptyObs}(\hat{o}_i) \lor u_i < \kappa_i\}}\)$ where action confidence \(u_i = \exp(\frac{1}{|a_i^{(0)}|} \sum_j \log \pi_\theta(a_{i,j}^{(0)} | x_i, h_i))\). To avoid manual per-tool threshold tuning, the threshold \(\kappa_i = \frac{1}{\max(1, i-1)} \sum_{j=1}^{i-1} u_j\) updates dynamically online as the running average confidence of preceding steps. This design accurately detects silent failures—such as off-target crops where the tool executes without raising an API error—without incurring any auxiliary verifier calls.

A Worked Example

Figure 4 in the paper illustrates a representative GroundingDrift recovery trajectory: - User Query & Context: Given an image showing a smoothie glass beside a cutting board with sliced banana, knife, and granola, the user asks: "How many calories are in the smoothie shown in this image?" - Initial Action & Silent Failure: The agent executes \(a_i^{(0)}\) calling visualizer(q="fruits/utensils on the table"). The tool returns a valid description of the cutting board and utensils without throwing an error, but omits the smoothie. Because the target item is missing, model confidence drops below the adaptive threshold (\(u_i < \kappa_i\)), setting \(g_i = 1\). - Structured Reflection Generation: The model emits an explicit RoT triplet: ErrorType=GroundingDrift, Evidence="wooden cutting board with a sliced banana, a knife", and FixPlan="refocus from the table/cutting board to the smoothie glass and ask about its contents". - Correction & Successful Resolution: Guided by the fix plan, the agent invokes corrected action \(a_i^{(1)}\): visualizer(q="ingredients in this smoothie"), extracts the actual smoothie ingredients, and calculates an accurate calorie estimate. In contrast, a baseline agent without local repair continues with off-target items and outputs an incorrect answer.

Loss & Training

The framework uses a two-stage training strategy: first, an SFT warm-start maximizes the conditional log-likelihood of structured reflection and corrected actions: \(\mathcal{L}_{\text{SFT}} = -\mathbb{E}[\log P_\theta(z_i, a_i^* | x_i, a_i^{\text{fail}}, o_i^{\text{fail}})]\); second, the policy is optimized with LoRA (rank 32 on all self-attention Q, K, V projections) on Qwen2-VL-7B (freezing the vision encoder and token compressor) using ReGRPO across offline and self-explored rollout groups: $\(\mathcal{L}_{\text{ReGRPO}} = -\mathbb{E}_{x_i, \tau \sim \pi_\theta} \left[ \frac{1}{K} \sum_{k=1}^K A_i^{(k)} \log \pi_\theta(\tau_i^{(k)} | x_i) \right] + \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})\)$ Hyperparameters are set to \(\lambda_{\text{exec}} = 1.0\), reflection cost \(\eta = 0.1\), with default \(\lambda_{\text{val}} = 0\) (verifier-free RL). The optimizer is AdamW with a base learning rate of \(1.0 \times 10^{-6}\) and cosine decay.

Key Experimental Results

Main Results

Evaluated on the multimodal vision-centric GTA benchmark (229 tasks) and the document-centric long-horizon GAIA benchmark (446 tasks) under a standardized single-path, zero-verifier protocol:

Method	Controller	GTA ToolAcc	GTA CodeExec	GTA AnsAcc	GAIA L1	GAIA L2	GAIA L3	GAIA AnsAcc
Lego Agent	GPT-4	-	-	46.59	-	-	-	-
Lego Agent	GPT-4o	-	-	41.52	-	-	-	-
HF Agent	GPT-4o	63.41	95.12	57.05	47.17	31.40	11.54	33.40
HF Agent	Qwen2-VL-7B	44.85	65.19	42.31	16.98	8.14	0.00	9.70
T3-Agent (MAT)	MAT-Qwen2-VL-7B	64.63	84.32	53.85	26.42	15.12	3.84	16.97
SPORT Agent	Tuned-Qwen2-VL-7B	72.41	91.87	60.26	35.85	16.28	3.84	20.61
ReGRPO (Default, \(\lambda_{\text{val}}=0\))	MAT-Qwen2-VL-7B	76.35	93.77	67.66	39.02	18.71	4.89	23.35

Ablation Study

A controlled ablation isolating the contributions of training data, RL algorithms, and reflection formulations under identical Qwen2-VL-7B backbones and tool suites:

Config / Stage	SFT Data	RL Alg.	Explicit Reflect.	GTA ToolAcc	GTA CodeExec	GTA AnsAcc	GAIA AnsAcc	Note
MAT-AGENT Baseline	MM-Traj	-	-	64.63	84.32	53.85	16.97	SFT on successful traces only
+ RoT (SFT Warm Start)	RoT	-	✓	68.73	87.92	58.59	19.03	Adds RoT supervision (+4.74 GTA)
+ Optional Verifier Distill	RoT	-	✓	69.84	88.41	59.72	19.84	Metadata-derived subscore distillation, no RL
GRPO-only (w/o Reflection)	MM-Traj	GRPO	-	71.34	90.23	64.51	18.92	Action-only GRPO, no reflection variable
GRPO + Free-form Reflection	RoT	GRPO	✓	73.05	91.02	65.34	21.38	Unconstrained reflection text
ReGRPO Core (Default, \(\lambda_{\text{val}}=0\))	RoT	ReGRPO	✓	76.35	93.77	67.66	23.35	Structured reflection + joint RL
+ Optional Verifier Reward (\(\lambda_{\text{val}}>0\))	RoT	ReGRPO	✓	77.26	94.91	68.49	24.01	Offline metadata-derived deterministic reward

Key Findings

• Structured reflection is indispensable in policy optimization: When applying GRPO, ReGRPO core outperforms action-only GRPO by +3.15% AnsAcc on GTA and +4.43% on GAIA. Furthermore, structured triplets consistently beat free-form text reflection (+2.32% on GTA, +1.97% on GAIA), proving that strict (ErrorType, Evidence, FixPlan) schemas prevent rationalization drift and hallucinated excuses.
• SFT warm-start and RL exploration act complementarily: RoT SFT provides essential initialization for diagnostic syntax and common failure recovery (elevating GTA AnsAcc from 53.85% to 58.59%), while subsequent ReGRPO exploration unlocks robust recovery on out-of-distribution errors, further lifting accuracy to 67.66%.
• External verifiers provide additive, non-essential shaping: Setting \(\lambda_{\text{val}} = 0\) achieves top-ranking open-source performance without any verifier calls during training or inference. The optional rule-based verifier reward offers only modest gains (+0.83% on GTA), confirming that primary improvements stem from the reflective RL formulation and cost control.

Highlights & Insights

• Internalizing reflection as a learnable decision variable: Rather than treating self-correction as an inference-time verbal prompting heuristic (like Reflexion), ReGRPO factors reflection tokens into the trajectory likelihood under GRPO, directly optimizing diagnostic quality via relative advantage gradients.
• Single-path adaptive confidence gating: By pairing hard execution checks with an online adaptive action-confidence threshold, the agent catches silent failures without requiring external verifier models or multi-branch search trees.
• Generalizable near-miss synthetic failure pipeline: Injecting controlled perturbations into successful tool actions in a live sandbox provides a robust recipe for synthesizing rich failure-recovery data, which is readily transferable to coding and GUI automation agents.

Limitations & Future Work

• Author-admitted limitations: On extremely complex Level-3 GAIA tasks requiring extensive sequential deduction, a single reflection-correction cycle reaches only 4.89% accuracy, indicating that compound multi-step reasoning failures require deeper hierarchical planning.
• Independent observations: The confidence trigger relies on output token likelihoods; when a model produces an incorrect tool invocation with high overconfidence, the silent failure will bypass the gate. Furthermore, the RoT data synthesis depends offline on a proprietary teacher (GPT-4o).
• Future directions: Exploring multi-turn hierarchical reflection scheduling and bootstrapping synthetic RoT data directly from open-source models present promising avenues for investigation.

Related Work & Insights

• vs MAT-AGENT [5]: MAT-AGENT synthesized large-scale multimodal tool trajectories (MM-Traj) but restricted training to successful demonstrations; ReGRPO injects near-miss failures and optimizes recovery trajectories, substantially boosting error resilience.
• vs SPORT [9]: SPORT employed step-wise preference tuning via self-exploration but lacked structured diagnostic reasoning; ReGRPO incorporates explicit RoT triplets and GRPO trajectory optimization, achieving higher accuracy and superior interpretability.
• vs Reflexion [22]: Reflexion relies on test-time verbal critiques with multi-trial memory buffers; ReGRPO internalizes reflection into policy parameters, enabling single-path zero-verifier execution.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegantly integrates structured reflection triplets into GRPO as learnable decision variables for tool-using agents.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive comparisons across open and closed models on GTA and GAIA, accompanied by meticulous ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, crystal-clear conceptual diagrams, and lucid explanation of failure and recovery mechanisms.
• Value: ⭐⭐⭐⭐⭐ Offers a practical and highly reusable blueprint for overcoming imitation saturation and sparse-reward credit assignment in multimodal agents.