Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language Models

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/kawhiiiileo/KAWHI
Area: Multimodal VLM
Keywords: Large Vision-Language Models, Reinforcement Learning (RLVR), Visual Representation Alignment, Credit Assignment, Chart & Geometry Reasoning

TL;DR

To resolve the structural representation bottleneck in Large Vision-Language Models trained via Reinforcement Learning from Verifiable Rewards (RLVR) where dense tokenization mismatches sparse diagrammatic signals, this paper proposes KAWHI, a plug-and-play reward reweighting mechanism that extracts geometric key regions with a Structure-Guided Union-Find algorithm, performs asymmetric Q-K spatial alignment using vision-critical attention heads, and reallocates credit at the paragraph level, consistently improving multimodal reasoning benchmarks with only 3.6% training overhead.

Background & Motivation

Reinforcement Learning from Verifiable Rewards (RLVR), instantiated through online policy optimization algorithms such as Group Relative Policy Optimization (GRPO) and Group-level Surrogate Policy Optimization (GSPO), has catalyzed breakthrough reasoning capabilities in Large Language Models (LLMs) on unimodal mathematical and formal logic deduction tasks. This remarkable success has spurred active efforts to extend RLVR into the multimodal domain, aiming to bootstrap chain-of-thought (CoT) reasoning in Large Vision-Language Models (LVLMs) for complex diagram interpretation and geometric theorem proving. However, existing multimodal RL pipelines predominantly adhere to uniform sequence-level reward allocation borrowed directly from pure-text optimization, broadcasting an identical scalar advantage across all decoding steps. Simultaneously, the visual front-end standardly employs dense patch tokenization, implicitly presuming that homogeneous grid tokenization captures exhaustive, well-distributed visual semantics across arbitrary scenes. This assumption fundamentally fails in structured multimodal scenarios such as mathematical geometry, functional diagrams, and analytical charts, where task-critical visual evidence is inherently sparse and spatially confined to delicate strokes, numerical labels, and geometric intersections amidst vast expanses of semantically uninformative background.

This structural mismatch between dense visual token sequences and intrinsically sparse geometric information severely biases the reinforcement learning dynamics. During gradient descent, optimization signals are diffusely dissipated across background tokens, leaving the policy unable to ground its deduction on decisive visual evidence. Quantitative failure analysis conducted on Qwen2.5-VL-7B-Instruct on the MathVerse benchmark reveals that visual perception (VP) errors account for 48.9% of all model failures, far eclipsing rule application errors (11.0%) and pure calculation errors (17.3%). Qualitative inspection further demonstrates that misinterpreting subtle visual markers triggers cascading logical collapses along the entire reasoning trajectory. Prior efforts attempting fine-grained reward modeling fail to provide a satisfactory solution: methods like VPPO rely on indirect divergence-based token masking without establishing explicit geometric grounding, whereas methods like AT-RL mandate explicit extraction of full cross-modal attention weight matrices, fundamentally clashing with modern I/O-aware acceleration operators such as FlashAttention and hindering scalable distributed training.

The core insight of this work is that structured visual evidence can be captured non-invasively through classical geometric structure analysis, while the multi-head attention architecture already exhibits specialized functional differentiation that can be tapped without modifying internal attention kernels. Core idea: Introduce KAWHI (Key-Region Aligned Weighted Harmonic Incentive), a plug-and-play reward reweighting mechanism that adaptively clusters salient geometric regions via Structure-Guided Union-Find (SGUF), computes asymmetric cosine alignment between response Query probes and key visual Key representations over MME-ablated vision-critical heads, and performs temperature-smoothed credit reallocation over paragraph blocks (\n\n) to modulate policy advantages without disturbing FlashAttention compatibility.

Method

Overall Architecture

KAWHI is designed as an architectural-agnostic, drop-in reward reweighting module compatible with standard uniform-reward policy optimization frameworks (e.g., GRPO, GSPO, DAPO). The complete pipeline comprises three interconnected stages: first, on the visual input side, the Structure-Guided Union-Find (SGUF) algorithm computes local second-order structure tensors to partition the image grid into salient stroke/symbol regions and redundant background, generating explicit geometric spatial priors; second, following rollout generation, an auxiliary single forward pass over the final decoder hidden states reconstructs the Query and Key representations, evaluating asymmetric spatial alignment restricted to pre-identified vision-critical attention heads; third, token-level alignment scores are mean-pooled across paragraph boundaries (\n\n), normalized through temperature-controlled softmax smoothing, and applied multiplicatively to scale the group-standardized advantage values for policy gradient updates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>Geometry / Chart image & prompt"] --> B["Structure-Guided Union-Find (SGUF)<br/>CIELab structure tensor decomposition & dual-threshold clustering"]
    B --> C["Vision-Critical Head Selection & Q-K Alignment<br/>Global MME ablation + probe Query matching semantic Key"]
    C --> D["Semantic-Preserving Paragraph Weighting<br/>Double-newline segmentation + temperature-scaled credit allocation"]
    D --> E["Policy Optimization Objective Update<br/>Modulating advantage in GRPO / GSPO / DAPO"]

Key Designs

1. Structure-Guided Union-Find (SGUF): Geometry-aware spatial prior extraction via structure tensors

Structured visual inputs like geometric figures and analytical charts exhibit pronounced spatial sparsity and severe foreground-background imbalance, causing standard dense grid tokenization to inject excessive background redundancy into downstream policy gradients. SGUF resolves this by providing rotation- and illumination-invariant spatial priors without requiring trainable neural components. Given input image \(I\), the algorithm converts it to the CIELab perceptually uniform color space and isolates luminance \(L \in [0, 100]\). Following Gaussian smoothing, spatial gradients \(\nabla L = (\partial_x L, \partial_y L)^\top\) are computed via Sobel filtering. For each image patch \(\mathcal{P}_{i,j}\), the normalized second-order structure tensor is constructed:

\[\mathbf{S}_{i,j} = \frac{1}{|\mathcal{P}_{i,j}|} \sum_{(x,y) \in \mathcal{P}_{i,j}} (\nabla L)(\nabla L)^\top = \frac{1}{|\mathcal{P}_{i,j}|} \begin{pmatrix} \sum g_x^2 & \sum g_x g_y \\ \sum g_x g_y & \sum g_y^2 \end{pmatrix}\]

Eigendecomposition \(\mathbf{S}_{i,j} = R^\top \Lambda R\) yields eigenvalues \(\lambda_{\max} \ge \lambda_{\min} \ge 0\). Anisotropic stroke edges exhibit \(\lambda_{\max} \gg \lambda_{\min}\), whereas flat background satisfies \(\lambda_{\max} \approx \lambda_{\min} \approx 0\). Treating the patch grid as a 4-connected graph, SGUF groups patches using a conservative dual-threshold Union-Find criterion: neighboring nodes merge into component \(\mathcal{C}_k\) if their normalized eigenvalue Frobenius distance is below structural threshold \(\delta_s\) and luminance difference is below \(\delta_l\). The structural saliency of component \(\mathcal{C}_k\) is quantified by its average trace \(E(\mathcal{C}_k) = \frac{1}{|\mathcal{C}_k|} \sum_{\mathcal{P} \in \mathcal{C}_k} (\lambda_{\max} + \lambda_{\min})\), thresholded adaptively at \(\tau = \beta \cdot \text{median}(\{E(\mathcal{C}_k)\})\). All tokens in key regions are retained, while background components are sparsely sampled at rate \(1 - r_{\text{skip}}\), focusing computational resources squarely on high-information geometry.

2. Vision-Critical Attention Head Selection and Q-K Spatial Alignment: Asymmetric cross-modal semantic anchoring

Attention heads in large multimodal backbones display vast functional heterogeneity; averaging alignment across all heads introduces significant linguistic syntactic noise. Furthermore, extracting dense cross-modal attention maps directly disrupts optimized kernel execution in FlashAttention. KAWHI addresses both hurdles through functional decoupling: Query vectors act as information probes signaling textual retrieval demands, whereas Key vectors act as semantic identifiers for visual tokens. To locate the most discriminative heads, a global layer-wise head ablation is conducted across 14 subtasks in the MME benchmark; attention heads whose masking causes an aggregate score drop exceeding 100 points are designated as the vision-critical subset \(\mathcal{H}_{\text{critical}}\).

During RL rollout processing, rather than intercepting internal attention operations, an auxiliary single forward pass over the final decoder layer hidden states recovers the respective query and key vectors. For architectures utilizing Grouped Query Attention (GQA), key heads are repeated \(g\) times to align dimensions. For each generated response token \(t \in \mathcal{R}\) and SGUF-selected key visual token \(v \in \mathcal{S}\), the spatial alignment under critical heads is evaluated via cosine similarity:

\[s_{t,v}^{(h)} = \frac{\mathbf{q}_t^{(h)} \cdot \mathbf{k}_v^{(h)}}{\|\mathbf{q}_t^{(h)}\| \|\mathbf{k}_v^{(h)}\|}, \quad h \in \mathcal{H}_{\text{critical}}\]

Averaging across both key visual tokens \(\mathcal{S}\) and critical heads \(\mathcal{H}_{\text{critical}}\) produces the spatial attention score:

\[\alpha_t = \frac{1}{|\mathcal{S}| \cdot |\mathcal{H}_{\text{critical}}|} \sum_{h \in \mathcal{H}_{\text{critical}}} \sum_{v \in \mathcal{S}} s_{t,v}^{(h)}\]

This bounded score \(\alpha_t \in [-1, 1]\) directly quantifies how intensively token \(t\) retrieves grounded semantic evidence from salient visual regions.

3. Semantic-Preserving Paragraph Weighting: Structure-consistent credit allocation and advantage modulation

Assigning credit at excessively fine granularities (such as token or sentence levels) fragments coherent reasoning steps into isolated lexical tokens, amplifying variance and disrupting long-range deduction. KAWHI adopts a coarse-grained paragraph-level segmentation strategy using natural double-newline delimiters (\n\n) to preserve the semantic integrity of each CoT deduction step. For a sequence partitioned into \(M\) paragraphs \(\{P_j\}_{j=1}^M\), token-level spatial scores are first aggregated via mean pooling: \(\bar{\alpha}_j = \frac{1}{|S_j|} \sum_{t \in S_j} \alpha_t\).

To avoid reward starvation on non-visual deduction steps, paragraph scores are mapped into stable weights \(w_j \in [w_{\min}, w_{\max}]\) through temperature-scaled softmax and uniform smoothing:

\[\tilde{w}_j = (1 - \lambda) \frac{\exp(\bar{\alpha}_j / \tau)}{\sum_{k=1}^M \exp(\bar{\alpha}_k / \tau)} + \frac{\lambda}{M}, \quad w_j = w_{\min} + (w_{\max} - w_{\min}) \cdot \tilde{w}_j\]

Given the group-standardized scalar baseline advantage \(A_{g}\) computed from verifiable binary sequence rewards, the advantage for token \(t\) belonging to paragraph \(P_j\) is multiplicatively modulated:

\[\hat{A}_{g,t} = A_{g,t} \cdot w_j\]

This reweighting preferentially scales up policy gradients for paragraphs directly grounded in visual evidence extraction and rule invocation, while suppressing hallucinated or tangential steps without breaking deduction flow.

A Worked Example: Dynamic Weight Allocation on Geo3K Geometry Proof

In a geometric proof from Geo3K requiring the measure of arc \(m\widehat{BE}\) given congruent chords \(\overline{BE} \cong \overline{ED}\) and known arc \(m\widehat{ED} = 120^\circ\), the generated response is segmented by \n\n into six paragraphs: - para1 (Introductory framing): "To solve for \(m\widehat{BE}\), we need to understand the relationship between the arcs..." — generic restatement with low visual query specificity; receives baseline low weight. - para2 (Constraint injection): "Given: \(\overline{BE} \cong \overline{ED}\), \(m\widehat{ED} = 120^\circ\)" — extracts concrete chord labels and measures from the diagram; Query vectors strongly correlate with SGUF key regions, driving a sharp increase in weight. - para3 (Decisive rule application): "Since \(\overline{BE} \cong \overline{ED}\), the arcs \(\widehat{BE}\) and \(\widehat{ED}\) are congruent. This means \(m\widehat{BE} = m\widehat{ED}\)" — applies the core geometric theorem to bind visual entities; receives the highest allocated weight. - para4 (Redundant context): "We know that the total measure of the circle is \(360^\circ\)..." — factual but irrelevant to the immediate deduction; down-weighted by KAWHI. - para5 & para6 (Conclusion and boxed answer): "\(m\widehat{BE} = 120^\circ\)" and "\(\boxed{120}\)" — standard closing statements assigned smooth median baseline weights.

This behavior showcases that KAWHI selectively concentrates optimization pressure on the pivotal steps that bridge visual perception with logical rule binding.

Loss & Training

KAWHI seamlessly integrates into policy optimization algorithms utilizing group relative baselines (exemplified by GRPO). For multimodal prompt \(x = (I, q)\), the reference policy \(\pi_{\theta_{\text{old}}}\) samples \(G\) rollouts \(\{y_g\}_{g=1}^G\). The verifiable reward verifier assigns outcome rewards \(R_g \in \{0, 1\}\). Group-level advantage \(A_g = (R_g - \mu) / (\sigma + \epsilon)\) is modulated by paragraph weight \(w_j\) to form \(\hat{A}_{g,t} = A_{g,t} \cdot w_j\), entering the clipped surrogate loss:

\[\ell_{g,t}(\theta) = \min\left(r_{g,t}(\theta) \hat{A}_{g,t}, \, \text{clip}(r_{g,t}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{A}_{g,t}\right)\]

where importance ratio \(r_{g,t}(\theta) = \pi_\theta(y_{g,t} \mid x, y_{g,<t}) / \pi_{\theta_{\text{old}}}(y_{g,t} \mid x, y_{g,<t})\). All models are trained without any intermediate supervised fine-tuning (SFT) using the VERL distributed framework on 8 NVIDIA H200 GPUs. Training sets consist of Geo3K for mathematical geometry and a 20K subset of ChartQA for chart reasoning. Per-step training latency marginally shifts from 521.0s to 540.6s, demonstrating an overhead of merely 3.6%.

Key Experimental Results

Main Results

Evaluation spans Qwen2.5-VL-7B-Instruct and Qwen3-VL-4B-Instruct across four mathematical reasoning benchmarks (MathVista, MathVerse, MathVision, WeMath) under GRPO, DAPO, and GSPO. As detailed below, KAWHI consistently outperforms all base RL algorithms and fine-grained baselines:

Model & Method	MathVista	MathVerse	MathVision	WeMath	Average (Avg)	Gain vs Base RL
Qwen2.5-VL-7B-Instruct
Base	67.47	45.56	23.36	54.32	47.68	-
Fine-Grained Reward Baselines
Step-GRPO	66.87	44.32	24.76	55.32	47.82	+0.14
FT-RL	67.68	46.88	26.12	56.78	49.37	+1.69
VPPO	68.42	47.21	28.42	57.12	50.29	+2.61
Uniform Reward Baselines & Ours
GRPO	68.37	45.94	28.29	58.10	50.18	+2.50
GRPO + KAWHI (Ours)	69.30	47.56	29.28	57.84	51.00	+0.82 (+3.32 vs Base)
DAPO	69.20	47.75	28.21	57.98	50.79	+3.11
DAPO + KAWHI (Ours)	69.08	48.20	31.91	58.21	51.85	+1.06 (+4.17 vs Base)
GSPO	69.42	49.87	27.30	59.02	51.40	+3.72
GSPO + KAWHI (Ours)	69.37	50.10	32.57	59.71	52.94	+1.54 (+5.26 vs Base)
Qwen3-VL-4B-Instruct
Base	70.43	40.86	21.38	69.43	50.53	-
GRPO	71.07	41.12	22.16	70.31	51.17	+0.64
GRPO + KAWHI (Ours)	71.80	42.10	23.03	70.98	51.98	+0.81 (+1.45 vs Base)
DAPO	69.40	40.48	27.63	69.67	51.80	+1.27
DAPO + KAWHI (Ours)	71.21	41.36	28.43	71.23	53.06	+1.26 (+2.53 vs Base)
GSPO	70.50	41.62	32.89	71.55	54.14	+3.61
GSPO + KAWHI (Ours)	72.10	46.82	31.09	72.15	55.54	+1.40 (+5.01 vs Base)

On chart reasoning benchmarks under GRPO with Qwen2.5-VL-7B-Instruct, KAWHI provides consistent improvements across all five evaluated datasets: ChartXivDesc (+2.1%), ChartXivRea (+2.1%), ChartQA (+0.8%), ChartQA-Pro (+1.6%), and ChartMimic (+2.3%).

Ablation Study

Ablation experiments on Qwen2.5-VL-7B-Instruct with GRPO baseline evaluate five core component designs on MathVerse and MathVision:

Ablation Dimension	Configuration	MathVerse (%)	MathVision (%)	Average (%)	Drop vs Full KAWHI
Baseline & Full	GRPO baseline	45.94	28.29	37.11	-1.31
-	GRPO + KAWHI (Full Model)	47.56	29.28	38.42	-
① Region Selection	All vision tokens	46.58	28.45	37.52	-0.90
	Random selection	46.34	28.98	37.66	-0.76
	Inverse selection	45.74	27.82	36.78	-1.64
② Reward Metric	Key–Key symmetric matching	47.08	28.95	38.02	-0.40
③ Response Granularity	Sentence-level segmentation	47.11	26.64	36.88	-1.54
	Token-level segmentation	46.21	26.64	36.42	-2.00
④ Critical Visual Head	w/o Critical Head (All heads)	46.57	28.32	37.45	-0.97
⑤ Positional Encoding	Pre-RoPE states	47.08	28.62	37.85	-0.57

Key Findings

• Structured spatial selection prevents gradient pollution: Inverting region selection (Inverse selection) causes performance to degrade below vanilla GRPO (36.78% vs 37.11%), confirming that background regions inject harmful noise into policy gradients. Retaining all tokens avoids missing details but suffers from token dilution (-0.90% vs full).
• Semantic coherence dictates credit stability: Shifting from paragraph-level grouping to sentence-level (-1.54%) or token-level (-2.00%) segmentation precipitates acute drops (plunging by 2.64% on MathVision). Token-level reward splitting fractures semantic thought units and destabilizes long-horizon credit attribution.
• Asymmetric Q-K formulation matches causal reasoning: The Query-Key probe formulation beats symmetric Key-Key matching by +0.40% average accuracy, reflecting the true generative dynamic where text tokens actively query visual evidence.
• Universal plug-and-play modularity: Replacing SGUF with attention-based VisionZip (50% tokens) produces marginal drops of only 0.13% to 0.30% across math benchmarks, verifying the robust transferability of KAWHI's reweighting formulation while incurring only 3.6% additional runtime.

Highlights & Insights

• Geometric structure tensor priors for reinforcement learning: The method brings classical CIELab gradient structure tensors and graph Union-Find clustering into the neural RL paradigm, yielding zero-parameter, rotation- and illumination-invariant spatial priors that cleanly circumvent vision token redundancy.
• Hardware-friendly FlashAttention decoupling: Rather than intrusively extracting cross-modal attention maps during forward/backward passes, KAWHI performs a clean single forward pass over terminal hidden states to recover Query and Key vectors, fully preserving FlashAttention execution and multi-node training throughput.
• Generalizable paragraph-level credit allocation: The paradigm of aggregating credit over paragraph blocks (\n\n) smoothed via temperature-scaled softmax offers a versatile blueprint for step-level reinforcement learning across diverse multi-step reasoning and multi-turn agent tasks.

Limitations & Future Work

• Static offline head selection: Identification of vision-critical heads currently relies on a static global ablation over the MME benchmark, which cannot adapt dynamically to distinct visual domain distributions at inference or training time.
• Domain specificity of geometric edge priors: While SGUF excels at vector graphics, functional plots, and geometry problems, its trace-based gradient clustering may be less discriminative when applied to natural images dominated by rich, continuous textures.
• Future directions: Developing dynamic, input-conditioned sparse gating for visual heads during rollout, and integrating SGUF with end-to-end differentiable visual samplers for joint representation optimization.

Related Work & Insights

• vs VPPO: VPPO leverages indirect statistical masking and token divergence to estimate visual dependency, lacking explicit spatial structure; KAWHI establishes physically interpretable, geometry-grounded spatial priors via structure tensors.
• vs AT-RL: AT-RL requires extracting complete multimodal cross-attention matrices, fundamentally incompatible with FlashAttention; KAWHI recovers Q and K states via an auxiliary uncoupled pass, maintaining full FlashAttention compatibility with only 3.6% computational overhead.
• vs Step-GRPO / FT-RL: Prior step-level reward baselines operate exclusively within unimodal textual sequences or high-entropy tokens without grounding on visual features; KAWHI directly bridges visual representations with RLVR optimization.

Rating

• Novelty: ⭐⭐⭐⭐☆ Pinpoints the visual perception bottleneck in LVLM RLVR and introduces an elegant combination of classical geometric tensors with Q-K attention credit reweighting.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated across Qwen2.5/Qwen3 backbones, 3 policy optimization algorithms, 4 mathematical and 5 chart reasoning benchmarks, accompanied by comprehensive ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured motivation supported by diagnostic failure analysis, clear mathematical formulation, and transparent experimental reporting.
• Value: ⭐⭐⭐⭐⭐ Fully compatible with FlashAttention, plug-and-play across modern RL pipelines, providing tangible gains with minimal computational overhead (3.6%).

Related Papers

• [ECCV 2026] Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
• [NeurIPS 2025] Unified Reinforcement and Imitation Learning for Vision-Language Models
• [AAAI 2026] ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Models
• [ECCV 2026] EGM: Efficient Visual Grounding Language Models
• [ECCV 2026] Token-Based Affordance Grounding with Large Vision-Language Models