Decompose, Look, and Reason: Reinforced Latent Reasoning for VLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: VLM Reasoning
Keywords: Multimodal Reasoning, Latent Visual Reasoning, Reinforcement Learning, Premise Decomposition, Spherical Gaussian Policy
TL;DR¶
The authors introduce DLR, a reinforced latent reasoning framework that dynamically interleaves textual premise generation, premise-conditioned continuous visual latent extraction, and grounded rationale deduction, coupled with a Spherical Gaussian Latent Policy (SGLP) under GRPO that unlocks active latent exploration.
Background & Motivation¶
Vision-Language Models (VLMs) have achieved remarkable progress across visual question answering and multimodal understanding tasks. However, when confronted with complex visual reasoning problemsโsuch as fine-grained attribute identification, multi-step geometric deduction, and abstract spatial-relational reasoningโthey often suffer from verbose yet visually ungrounded speculation. Early text-only Multimodal Chain-of-Thought (MCoT) approaches abstract the input visual signals into a single static representation or initial textual descriptions, which inevitably causes critical visual details to degrade or vanish across extended textual reasoning trajectories. Interleaved MCoT and "thinking with images" paradigms attempt to alleviate this information loss by cropping local image patches, drawing auxiliary sketches, or invoking external vision tools. Nevertheless, tool-assisted image manipulation introduces heavy computational latency and remains constrained by predefined toolkits, while patch-based region-of-interest (ROI) embeddings inherently suffer from either over-including irrelevant localized context or under-including non-local, cross-patch global relations.
Moreover, while latent space reasoning offers an efficient alternative by directly projecting visual evidence into internal continuous embeddings without external tools, existing methods remain severely bottlenecked by their training paradigms. Prior latent visual reasoning approaches typically inject a single latent token once or rely on supervised fine-tuning (SFT) based on explicit ROI reconstruction. Constrained by teacher-forced log-likelihood objectives, SFT restricts the visual grounder to a deterministic feature extractor incapable of active exploration. Meanwhile, standard reinforcement learning algorithms such as GRPO are intrinsically designed for discrete token vocabularies and cannot be directly applied to continuous visual latents without a defined probability distribution, leaving the continuous latent manifold unexplorable.
This paper addresses these limitations by observing that human problem-solving naturally follows a cognitive cycle of decomposing sub-questions, inspecting targeted visual evidence, and deducing grounded rationales, while vision-language semantic manifolds are naturally governed by directional cosine geometry on the unit hypersphere rather than Euclidean magnitude. The core idea is to formalize multimodal reasoning as an interleaved trajectory of dynamic textual premise decomposition, premise-conditioned visual latent extraction, and grounded rationale deduction, and introduce a Spherical Gaussian Latent Policy (SGLP) to enable direct continuous latent exploration under GRPO.
Method¶
Overall Architecture¶
The DLR (Decompose, Look, and Reason) framework models multimodal reasoning over an input image \(I\) and query \(q\) as an iterative trajectory \(\tau = \{(p^{(t)}, z^{(t)}, r^{(t)})\}_{t=1}^N\) interleaving discrete text tokens and continuous visual latents. The architecture is driven by two learnable policies: a discrete VLM text policy \(P_\theta\) governing natural language generation, and a lightweight continuous latent visual grounder policy \(P_\phi\) (approximately 7M parameters). For each reasoning step, the VLM first generates a textual premise \(p^{(t)}\) tagged with <premise> delimiters to identify the specific visual sub-question. The visual grounder takes the premise's final hidden state \(s_t\) and the image features to extract \(L=32\) continuous latent tokens \(z^{(t)}\) tagged with <vis_thought>. Conditioned on these injected continuous latents, the VLM produces a grounded textual rationale \(r^{(t)}\) within <rationale> tags, iteratively progressing until generating the final answer \(a\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image & User Query"] --> B["Dynamic Premise Decomposition<br/>VLM generates premise & extracts hidden state"]
B --> C["Premise-Conditioned Latent Extraction<br/>Visual Grounder retrieves L continuous latents"]
C --> D["Spherical Gaussian Latent Exploration<br/>SGLP injects angular noise on hypersphere"]
D --> E["Latent-Grounded Rationale Deduction<br/>VLM deduces grounded text rationale & answer"]
E -->|Iterative loop| B
E -->|Reasoning complete| F["Final Output Answer"]
Key Designs¶
1. Dynamic Premise Decomposition: Context-Driven Identification of Visual Targets
Rather than indiscriminately attending to the full image at every step or mechanically inserting visual tokens at newline characters, DLR employs the discrete VLM policy \(P_\theta\) to dynamically parse the reasoning context and articulate sub-questions requiring empirical verification. The sub-question is explicitly enclosed within <premise> and </premise> special tokens. This design determines exactly what specific details need to be verified and when to look for them, eliminating redundant visual token overhead during purely linguistic or logical transitions. The hidden state \(s_t\) of the trailing </premise> token is subsequently extracted as a compact, task-focused conditioning vector for the visual grounder.
2. Premise-Conditioned Latent Extraction: Non-Local Semantic Representation
To overcome the spatial constraints of bounding boxes and patch embeddings, the visual grounder \(P_\phi\) utilizes a lightweight architecture comprising two cross-attention layers followed by a feed-forward network (totaling ~7M parameters). It introduces \(L=32\) learnable continuous latent queries \(z_0 \in \mathbb{R}^{L \times d}\). Conditioned on the premise hidden state \(s_t\), these queries attend across the full visual feature map \(V = \Phi_{\text{vis}}(I) \in \mathbb{R}^{m \times d}\) extracted by the frozen vision encoder. The resulting continuous latent matrix \(z^{(t)} \in \mathbb{R}^{L \times d}\) replaces reserved placeholder tokens (<VIS0> to <VIS31>) inside the reasoning chain. Because it attends globally rather than cropping explicit patches, it captures both localized fine-grained features and non-local abstract relationships (such as global color dominance or cross-region layout rules).
3. Spherical Gaussian Latent Exploration: Geometry-Aligned Continuous RL
Standard Gaussian noise \(\mathcal{N}(\mu, \sigma^2 \mathbf{I})\) defined in Euclidean space \(\mathbb{R}^d\) conflates vector magnitude variance with semantic variance, leading to catastrophic magnitude collapse when optimizing continuous embeddings. Recognizing that vision-language representations are inherently defined on the unit hypersphere \(\mathbb{S}^{d-1}\) (where cosine similarity dictates semantics), the authors propose the Spherical Gaussian Latent Policy (SGLP). The visual grounder predicts an \(\text{L}_2\)-normalized mean direction \(\mu_\phi \in \mathbb{S}^{d-1}\), injects isotropic exploration noise \(\epsilon \sim \mathcal{N}(0, \sigma^2 \mathbf{I})\), and explicitly re-projects the vector back onto the unit hypersphere:
This formulation guarantees that exploration operates purely along the angular manifold with fixed unit norm \(\|z\|_2 = 1\). The policy log-probability gradient simplifies cleanly to scaled directional cosine alignment: \(\nabla_\phi \log \pi_\phi(z|s_t) \approx \frac{1}{\sigma^2} z^\top \nabla_\phi \mu_\phi(s_t)\). When a sampled trajectory receives positive group advantage \(A > 0\), the gradient directly pulls the mean direction \(\mu_\phi\) along the hypersphere toward the advantageous latent sample \(z\), enabling effective reinforcement optimization for continuous latent variables.
4. Latent-Grounded Rationale Deduction: Dense Focus Rewards and Faithfulness Filtering
Conditioned on the continuous latents \(z^{(t)}\), the VLM generates detailed natural-language explanations enclosed in <rationale> and </rationale> tags. To stabilize latent policy exploration and prevent attention drift during reinforcement learning, a composite dense reward is formulated: \(r = R_{\text{outcome}} + \beta \cdot \mathbb{I}(R_{\text{outcome}} > 0) \cdot R_{\text{focus}}\). The outcome reward \(R_{\text{outcome}} \in \{0, 1\}\) measures exact-match answer correctness. The visual focus reward aligns the grounder's cross-attention map \(A_{\text{latent}}\) with an oracle attention map \(A_{\text{oracle}}\) derived from a frozen SigLIP-SO400M model: \(R_{\text{focus}} = \exp\left(-\lambda \cdot D_{\text{KL}}(A_{\text{oracle}} \parallel A_{\text{latent}})\right)\). Crucially, the indicator function \(\mathbb{I}(R_{\text{outcome}} > 0)\) suppresses the focus reward to zero whenever the final answer is incorrect, strictly preventing the model from reinforcing hallucinated attention patterns that do not yield factual correctness.
A Worked Example¶
Consider solving a 3ร3 grid puzzle determining the missing rotating circular pattern:
1. Decomposition 1: Unlike baseline models that generate over 15,000 tokens repetitively counting green slices, DLR first generates <premise>The image displays that each circle is divided into 9 equal segments, with some green and some white.</premise>.
2. Look 1 & Reason 1: The visual grounder outputs \(z^{(1)}\), and the model generates <rationale>The first two rows have specific patterns, and the third row's first two circles follow the same pattern as the first two rows.</rationale>.
3. Decomposition 2: The model dynamically formulates the next sub-problem: <premise>The key points mentioned are that the green segments are consistently placed in the same relative positions across the grid.</premise>.
4. Look 2 & Final Deduction: The grounder extracts relational angular alignments across the grid into \(z^{(2)}\), allowing the VLM to verify the relative offset and output <answer>D</answer> concisely without getting trapped in repetitive self-verification loops.
Loss & Training¶
The training pipeline consists of three sequential stages:
- Stage I (Pretraining): The base VLM is frozen, and the lightweight visual grounder is warmed up on 443k visual question-answering pairs from VQAv2 using a symmetric InfoNCE bidirectional contrastive loss \(L_{\text{pretrain}} = \frac{1}{2}(L_{v \to t} + L_{t \to v})\) with temperature \(\tau=0.07\) to establish cross-modal latent alignment.
- Stage II (Supervised Fine-Tuning): On a 200k subset of Vision-R1-cold annotated with DLR tags, LoRA (\(r=64, \alpha=128\), dropout 0.05) is applied to the VLM while optimizing both the VLM and visual grounder via autoregressive cross-entropy loss over discrete tokens to internalize the <premise>, <vis_thought>, and <rationale> structure.
- Stage III (Reinforcement Fine-Tuning): On the 39k ViRL dataset, Dr. GRPO is applied with group size \(G=4\) trajectories per query. The total objective optimizes both discrete text tokens and continuous latent variables via importance sampling: \(\mathcal{J}(\Theta) = \mathcal{J}_{\text{text}}(\theta) + \mathcal{J}_{\text{latent}}(\phi)\). Optimization uses AdamW with cosine learning rate decay (\(1 \times 10^{-6}\) for the VLM, \(1 \times 10^{-5}\) for the grounder).
Key Experimental Results¶
Main Results¶
Evaluated across fine-grained visual perception (V* Bench), visual mathematical reasoning (MathVista), multidisciplinary reasoning (MMMU-Pro), and core multimodal understanding (MMStar), DLR consistently outperforms open-source baselines as well as proprietary models like GPT-4o:
| Model | V* Overall | V* Attribute | V* Spatial | MathVista | MMMU-Pro | MMStar |
|---|---|---|---|---|---|---|
| GPT-4o (Proprietary ~200B) | 67.5* | 72.2* | 60.5* | 60.0* | 51.9* | - |
| Qwen3-VL-8B-Thinking (Backbone) | 79.6 | 81.7 | 76.3 | 79.2 | 60.9 | 72.3 |
| ICoT (Interleaved MCoT) | 79.6 | 81.7 | 76.3 | 77.1 | 61.2 | 67.0 |
| PixelReasoner (Thinking with Images) | 80.1 | 82.6 | 76.3 | 74.2 | 56.1 | 68.7 |
| LVR (Latent Visual Reasoning) | 82.2 | 82.6 | 81.6 | 77.3 | 60.5 | 72.7 |
| DLR (Ours) | 83.8 | 84.3 | 82.9 | 82.7 | 63.5 | 76.2 |
| Gain over Backbone | +4.2 | +2.6 | +6.6 | +3.5 | +2.6 | +3.9 |
Ablation Study¶
The ablation experiments quantify the contributions of individual training stages, reward designs, and latent policy optimization:
| Config | V* | MathVista | MMMU-Pro | MMStar | Note |
|---|---|---|---|---|---|
| DLR (Full model) | 83.8 | 82.7 | 63.5 | 76.2 | Full 3-stage pipeline with SGLP |
| w/o \(R_{\text{focus}}\) | 82.6 | 66.5 | 55.1 | 64.8 | Drop in alignment guidance drops math and reasoning |
| w/o \(\mathcal{J}_{\text{latent}}\) | 81.5 | 57.1 | 54.9 | 63.3 | Disabling latent RL causes 25.6% drop on MathVista |
| DLR-SFT | 80.5 | 65.1 | 54.6 | 64.6 | Deterministic SFT without RL active exploration |
| DLR-SFT w/o pretraining | 79.5 | 63.0 | 53.6 | 63.2 | Lacking Stage I warm-up hurts cross-modal grounding |
Quantitative faithfulness evaluation on V* Bench further confirms model grounding: - No mask (DLR): 83.8% accuracy. - Random mask: 77.7% accuracy (-6.1% drop). - DLR attention mask: 36.5% accuracy (-47.3% drop). This stark degradation demonstrates that the visual grounder's attention faithfully pinpoints image areas indispensable to final predictions.
Key Findings¶
- Latent policy optimization is essential for visual reasoning: Removing \(\mathcal{J}_{\text{latent}}\) causes MathVista performance to plummet from 82.7% to 57.1%, demonstrating that continuous angular exploration guided by SGLP overcomes the strict limitations of deterministic SFT.
- Internal latent reasoning preserves thinking capabilities: PixelReasoner requires external tool-calling fine-tuning, which inadvertently disrupts the native
<think>capabilities of reasoning backbones. DLR internalizes visual thoughts, maintaining seamless integration with generative reasoning. - Multi-step premise-conditioned latents surpass single coarse embeddings: Compared to LVR, which relies on a single coarse latent insertion, DLR's progressive premise-conditioned extraction yields large gains on complex multi-step reasoning tasks (+5.4% on MathVista).
Highlights & Insights¶
- Spherical Gaussian Latent Policy (SGLP): By restricting continuous policy actions to the unit hypersphere \(\mathbb{S}^{d-1}\), the method aligns reinforcement learning with the directional geometry of vision-language embeddings, eliminating magnitude collapse while yielding a clean cosine policy gradient.
- Factuality-Gated Dense Attention Reward: Incorporating the binary outcome reward as a gate for the continuous attention KL divergence prevents the reinforcement learning agent from learning specious attention alignments that fail to produce correct answers.
- Stepwise Interpretability Without Tool Overhead: DLR bridges the gap between opaque continuous latents and interpretable visual reasoning by explicitly surfacing textual premises and premise-conditioned attention maps alongside the thoughts.
Limitations & Future Work¶
- Static Image Scope: The framework is currently validated on vision-centric image benchmarks; extending dynamic premise-conditioned latent reasoning to video streams and embodied multimodal control remains an open challenge.
- Latent Transparency vs. Efficiency Trade-off: Although attention maps and premises provide stepwise interpretability, continuous vectors remain inherently less readable than human-drawn bounding boxes or verbal descriptions, warranting future exploration into interpretable continuous decoding.
Related Work & Insights¶
- vs ICoT (Interleaved MCoT): ICoT relies on explicit newline heuristics to insert cropped image patches, often bundling irrelevant context or failing when reasoning chains lack line breaks; DLR extracts premise-conditioned continuous representations globally.
- vs PixelReasoner (Thinking with Images): PixelReasoner utilizes external image editing tools that cause high computational latency and risk catastrophic forgetting of base thinking patterns; DLR executes purely internal latent operations with negligible parameter overhead (7M parameters, <0.09% of backbone).
- vs LVR (Latent Visual Reasoning): LVR introduces only a single static latent token supervised by ROI reconstruction; DLR enables dynamic multi-step premise decomposition and empowers continuous latents with active reinforcement exploration via SGLP.
Rating¶
- Novelty: โญโญโญโญโญ Formulates an elegant "Decompose-Look-Reason" loop and solves continuous latent reinforcement learning via the hyperspherical Gaussian policy.
- Experimental Thoroughness: โญโญโญโญโญ Evaluated across four comprehensive vision-centric benchmarks with meticulous ablations, quantitative faithfulness masking, and qualitative token trace comparisons.
- Writing Quality: โญโญโญโญโญ Highly structured narrative, rigorous mathematical derivations of the latent policy gradients, and clear intuitive justifications.
- Value: โญโญโญโญโญ Provides a foundational blueprint for continuous multimodal reasoning and latent-space reinforcement learning for next-generation VLMs.