HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization

Conference: ECCV 2026
arXiv: 2604.20328
Code: https://github.com/EthenCheng/HyLaR
Area: Multimodal VLM / LLM Reasoning
Keywords: Hybrid Latent Reasoning, Decoupled Policy Optimization, vMF Distribution, Visual Latent Representation, Hybrid Action Space RL

TL;DR

HyLaR achieves hybrid discrete-continuous reasoning by introducing control tokens to seamlessly switch between discrete text tokens and continuous visual latent vectors. For efficient reinforcement learning in hybrid action spaces, it proposes DePO (Decoupled Policy Optimization), which utilizes von Mises-Fisher (vMF) spherical modeling and decoupled trust-region clipping to address the geometric and variance mismatch issues of standard RL, significantly outperforming existing methods on fine-grained perception and general multimodal understanding benchmarks.

Background & Motivation

Background: CoT reasoning has significantly enhanced the complex problem-solving capabilities of Multimodal Large Language Models (MLLMs). However, most MLLMs suffer from an architectural bottleneck termed early semantic collapse: high-bandwidth continuous visual signals are forcibly compressed into discrete text tokens before entering the LLM, thereby losing fine-grained visual evidence that is difficult to verbalize.

Limitations of Prior Work: To alleviate this problem, existing studies have explored two main pathways. The first is the Think-with-Images paradigm (e.g., DeepEyes, Thyme), which re-perceives images via external tools but introduces tool-call errors, inference latency, and rigid operational bottlenecks. The second is latent-space reasoning (e.g., LVR, SkiLa, Monet), which shifts reasoning into a continuous latent space to preserve visual fidelity. However, the hybrid discrete-continuous action space of the latter poses new optimization challenges: discrete tokens and continuous latent vectors are inherently different in their probability densities and variance characteristics, which makes the unified trust-region clipping and Gaussian assumptions of standard RL algorithms (such as GRPO and DAPO) highly mismatched.

Key Challenge: Standard RL exhibits two major mismatches: (1) Variance mismatch: the density ratio (importance ratio) of continuous latent actions is highly sensitive to minor changes in policy parameters, which is significantly higher than that of discrete tokens; (2) Geometric mismatch: the RMSNorm layers of LLMs constrain the hidden states onto a high-dimensional hyperspherical manifold (directional semantics), whereas standard RL assumes a Gaussian distribution with Euclidean geometry, leading to extremely high sampling variance and poor performance in KL regularization.

Key Insight: Having identified these two key mismatches, the paper proposes a decoupled approach: designing separate trust-region clipping sizes for discrete tokens and continuous latent vectors, and modeling the continuous latent policy using a von Mises-Fisher (vMF) distribution (instead of a Gaussian distribution). This formulation simplifies the KL divergence into a closed-form cosine distance, entirely bypassing high-variance sampling.

Core Idea: HyLaR + DePO processes the differing variances of the hybrid actions via decoupled trust-region clipping, aligns with the LLM's native geometry via vMF spherical modeling, and achieves zero-variance exact regularization via closed-form KL. This renders reinforcement learning for latent reasoning stable, efficient, and geometrically precise.

Method

Overall Architecture

HyLaR extends the standard MLLM architecture to support a hybrid discrete-continuous action space. Special control tokens, <|canvas_start|> and <|canvas_end|>, are introduced to demarcate the visual reasoning process: in canvas mode, text generation is suspended, and the hidden states are recurrently fed back as inputs to the next step (bypassing the discrete vocabulary), establishing an internal visual working memory. This continuous recursion terminates when an end token is emitted or when the maximum canvas length budget is reached.

The entire training process is split into two phases: Stage I SFT Warm Start trains the model to alternate between text and latent steps, while Stage II DePO RL further optimizes the hybrid reasoning trajectories through reinforcement learning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["**1. Canvas Mode**<br/>Control tokens demarcate reasoning modalities<br/>Discrete text logic<br/>↕<br/>Continuous latent visual working memory"] --> B["**2. SFT Warm Start**<br/>Compress GT Canvas into 16 embeddings<br/>L_CE + λ·L_Canvas<br/>End-to-end joint optimization"]
    B --> C["**3. DePO Reinforcement Learning**<br/>Decoupled clipping: Text ϵ=0.2 vs Latent ϵ=0.05<br/>vMF spherical density ratio (cosine)<br/>Closed-form vMF KL regularization"]
    C --> D["Inference Deployment<br/>Canvas Compressor discarded<br/>0.96× latency, single forward pass"]

Key Designs

1. Canvas Mode: Control Tokens for Hybrid Discrete-Continuous Reasoning

This addresses the semantic collapse of standard text CoT and the external tool overhead of Think-with-Images. The core mechanism is the introduction of two control tokens, |<|canvas_start|> and |<|canvas_end|>, to define the boundaries of the visual reasoning session. In canvas mode, instead of generating discrete text tokens, the model feeds the hidden states of the previous step back as embeddings for the next step, enabling recurrent computation in the continuous latent space. This recursion acts as an internal visual working memory where the model dynamically zooms in or focuses on target regions within the latent space. This process is faster than external tool calls (dispensing with detokenization, re-encoding, and API calls) and is fully executed within a single forward pass. From a design motivation perspective, this mechanism retains the high fidelity of visual signals while avoiding the immense latency associated with pixel-level image generation.

2. Canvas Compression & Alignment: SFT Warm Start Training

The challenge lies in training the model to produce meaningful latent vectors in canvas mode without requiring explicit step-by-step supervision of the intermediate latent states. The authors employ a Canvas Compressor: during the SFT phase, a frozen SigLIP2 encoder is first used to extract \(P=729\) patch tokens from ground-truth intermediate canvas images, which are subsequently aggregated into \(N=16\) compact canvas embeddings via a learnable cross-attention compressor (\(L=2\) layers, \(N=16\) query tokens). During training, an MSE loss aligns the LLM's hidden state at the canvas position with the target embedding:

\[ \mathcal{L}_{\text{Canvas}} = \frac{1}{|\mathcal{S}|} \sum_{t \in \mathcal{S}} \| \mathbf{h}_t - \mathbf{e}_{t+1} \|_2^2 \]

where \(\mathbf{h}_t\) is the hidden state of the LLM at the canvas position \(t\), and \(\mathbf{e}_{t+1}\) represents the compressed target canvas embedding. A crucial detail is that the target embedding \(\mathbf{e}_{t+1}\) is not detached during backpropagation; gradients flow simultaneously into both the LLM backbone and the Canvas Compressor, actively aligning the visual features into the LLM's semantic space. The overall SFT loss is formulated as \(\mathcal{L}_{\text{SFT}} = \mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{Canvas}}\). The Canvas Compressor is discarded during inference, introducing zero architectural overhead.

3. DePO Decoupled Policy Optimization: vMF Spherical Modeling + Decoupled Clipping + Closed-Form KL

This is the core design of the paper, specifically targeting the optimization bottlenecks in hybrid action spaces for RL. Standard PPO/GRPO applies a uniform surrogate loss clipping range and sampled KL regularization across all action steps, which fails in hybrid spaces due to two primary reasons.

(a) Decoupled Trust-Region Clipping: The response is segmented into text positions \(\mathcal{Z}\) and latent positions \(\mathcal{S}\), applying dual-clipped PPO surrogates independently to each:

\[ \mathcal{L}_{\text{PPO}}(\theta) = \mathcal{L}_{\text{tok}}(\theta|\mathcal{Z}; \epsilon_l^{\text{tok}}, \epsilon_h^{\text{tok}}) + \alpha \mathcal{L}_{\text{lat}}(\theta|\mathcal{S}; \epsilon_l^{\text{lat}}, \epsilon_h^{\text{lat}}) \]

Text tokens utilize \(\epsilon_l^{\text{tok}}=0.2, \epsilon_h^{\text{tok}}=0.28\), while latent actions employ a significantly tighter \(\epsilon_l^{\text{lat}}=\epsilon_h^{\text{lat}}=0.05\) (\(\alpha=0.5\) controls the balance). This is because the continuous vMF density ratio is highly sensitive to minor policy changes (stemming from log-density properties in high-dimensional spaces and error accumulation during latent recursion), thereby demanding a much smaller trust region.

(b) vMF Spherical Modeling: For continuous latent positions, instead of a Gaussian, a von Mises-Fisher distribution is used to model the policy density:

\[ \pi_\theta(a_t|s_t) = C_D(\kappa) \exp\left(\kappa (\boldsymbol{\mu}_t^\theta)^\top \tilde{\mathbf{z}}_t\right), \quad \tilde{\mathbf{z}}_t \in \mathbb{S}^{D-1} \]

where \(\boldsymbol{\mu}_t^\theta\) is the \(\ell_2\)-normalized hidden state (a unit vector on the hypersphere), and \(\tilde{\mathbf{z}}_t\) is the rollout direction of the old policy (the mode of the old policy). The importance sampling ratio simplifies to \(\log r_t = \kappa(\cos(\boldsymbol{\mu}_t^\theta, \tilde{\mathbf{z}}_t) - 1)\), which is purely driven by angular deviations. This perfectly aligns with the directional geometry of the LLM hidden states under RMSNorm regularization.

(c) Closed-Form vMF KL Regularization: The KL divergence between two vMF distributions sharing the same concentration parameter \(\kappa\) can be simplified into a scaled cosine distance:

\[ \mathcal{L}_{\text{KL}}^{\text{lat}} = \frac{1}{|\mathcal{S}|} \sum_{t \in \mathcal{S}} \mathbf{W}_\kappa \cdot \left(1 - \cos(\boldsymbol{\mu}_t^\theta, \tilde{\mathbf{z}}_t)\right) \]

This provides a zero-variance, exact KL estimation, completely avoiding the immense noise that sampled KL encounters in high-dimensional spaces. In practice, the authors adopt an unnormalized relaxation (using inner products instead of cosines) to preserve magnitude information as an adaptive concentration factor, yielding an average performance boost of +2.87% across benchmarks.

An Illustrative Example

Take the identification of text on a distant billboard in a high-resolution image as an example. Traditional CoT: the model compresses the entire image into discrete tokens, struggle to attend to extremely small regions, and often hallucinates. HyLaR workflow: the model first generates a text reasoning framework (e.g., "I need to observe the billboard area in the image"), and then emits |<|canvas_start|> to enter canvas mode. Inside canvas mode, the hidden states undergo directional recursion in the latent space for 16 steps—analogous to performing visual "zoom in" and "focus" on the billboard area within the latent space. Once the model finishes its internal visual exploration, it emits |<|canvas_end|> to exit canvas mode, and resumes generating text based on the acquired fine-grained visual representations to output the final answer. The entire process takes place within a single forward pass, requiring no external tools, with a latency of only 0.96× compared to the baseline without canvas.

Loss & Training

SFT Phase: Jointly optimizes \(\mathcal{L}_{\text{SFT}} = \mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{Canvas}}\). Training is conducted for 1 epoch to prevent overfitting, with a learning rate of \(10^{-5}\), active batch size of 1 per GPU, and gradient accumulation over 16 steps. The Canvas Compressor utilizes \(L=2\) cross-attention layers with \(N=16\) query tokens, taking \(729\) patch tokens from the frozen SigLIP2 as input. The SFT dataset consists of approximately 96K samples filtered from Zebra-CoT.

DePO RL Phase: The total loss is formulated as \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{PPO}} + \beta_{\text{tok}} \mathcal{L}_{\text{KL}}^{\text{tok}} + \beta_{\text{lat}} \mathcal{L}_{\text{KL}}^{\text{lat}}\), where \(\beta_{\text{tok}}=0.01\), \(\beta_{\text{lat}}=0.005\), and \(\alpha=0.5\). The learning rate is decreased to \(10^{-6}\), with a rollout size \(G=8\), temperature 0.9, maximum response length 2048, and \(\kappa=0.01\). The RL dataset is composed of roughly 48.6K samples merged and deduplicated from DeepEyes, Thyme, and CodeDance. Dynamic rollout filtering is applied (discarding response groups whose group average accuracy lies outside the range \([0.1, 0.9]\)) to guarantee valid gradient signals. Training is executed on 8× NVIDIA H20 GPUs.

Key Experimental Results

Main Results

Dataset	Metric	Qwen2.5-VL-7B	SkiLa*	Monet*	HyLaR-SFT	HyLaR-7B(DePO)
V*	Overall	76.44	78.53	80.10	80.63	83.77
V*	Attribute	77.39	-	81.73	81.73	82.61
V*	Spatial	75.00	-	77.63	78.95	85.53
HRBench-4K	Overall	68.00	72.12	67.37	71.50	75.00
HRBench-8K	Overall	63.75	66.50	64.37	67.19	70.50

Analysis: HyLaR achieves comprehensive superiority across three ultra-high-resolution benchmarks. Compared to the baseline Qwen2.5-VL-7B, it achieves gains of 7.33% on V, 7.00% on HRBench-4K, and 6.75% on HRBench-8K. Among visual latent reasoning models, HyLaR substantially outperforms SkiLa and Monet (by 5.24% and 3.67% on V, respectively). The improvement in the Spatial dimension is the most significant (+10.53%), highlighting that latent-space reasoning is considerably better at preserving spatial relations compared to discretization alternatives.

Ablation Study

Method	V*	HRBench-4K	HRBench-8K	MMVP	Avg
Text-only SFT	69.11	65.88	61.00	70.20	66.55
HyLaR-SFT	80.63	71.50	67.19	71.00	72.58
+ GRPO	78.53	71.50	67.50	71.40	72.23
+ DAPO	79.06	71.75	68.00	70.20	72.25
+ VLPO (Monet)	80.10	72.84	68.00	69.67	72.65
+ DePO (Ours)	83.77	75.00	70.50	73.67	75.74

Analysis: Compared to standard RL algorithms (GRPO, DAPO, VLPO), DePO outperforms them by approximately 3 percentage points on average. Notably, both GRPO and DAPO on V* yield lower scores than the SFT baseline (80.63 \(\to\) 78.53 / 79.06), which confirms that unified clipping strategies are counterproductive in hybrid action spaces. The decoupled clipping and vMF modeling of DePO generate decisive improvements.

Key Findings

• SFT Latent Steps Suffer from 'Overthinking' Degradation: When the testing latent steps \(K_{\text{test}}\) far exceed the training steps \(K_{\text{train}}\), the performance of the SFT model degrades continuously; RL training effectively alleviates this issue, enabling the model to generalize to longer reasoning budgets.
• Decoupled Clipping Range is Highly Sensitive: The latent action clipping range of \(\epsilon^{\text{lat}}=0.05\) is optimal. Setting it to 0.2 (identical to text) drops V* performance to 80.10, indicating that unified clipping backfires in hybrid spaces.
• vMF Outperforms Gaussian: Switching from vMF to Gaussian degrades performance across all metrics. Under the Gaussian assumption, the model tends to maximize rewards by inflating embedding norms rather than learning superior semantics. The angular regularization inherent to vMF prevents this norm explosion.
• Remarkable Inference Efficiency: The operational latency and GPU memory usage of HyLaR are only 0.96× and 0.98× of Qwen2.5-VL-7B, respectively, whereas the tool-based method DeepEyes incurs 2.28× latency and 3.64× GPU memory usage.

Highlights & Insights

• The Elegance of the vMF-to-LLM Bridge: Recognizing that RMSNorm in LLMs constrains hidden states onto a hypersphere, the authors discard the ubiquitous Gaussian assumption in favor of the vMF distribution. This reduces the importance sampling ratio to simple cosine similarities and simplifies the KL divergence into a closed-form cosine distance, which boosts both training stability and computational efficiency.
• Decoupled Clipping is Intuitive and Practical: Instead of employing complex networks or adaptive mechanisms to modulate policy update steps across different action types, the authors simply apply two independent clipping hyperparameters. It is straightforward, highly tunable, and remarkably effective. This "identify variance \(\to\) divide and conquer" philosophy is highly instructive.
• Engineering Ingenuity of Unnormalized Relaxation: While maintaining the rigorous normalized derivation of vMF in theory, the implementation employs inner products instead of cosine similarities in practice, letting the hidden state's magnitude act as a dynamic concentration factor. This "theoretically elegant + engineeringly flexible" approach provides both academic rigor and practical utility.
• Canvas Compressor Discarded at Inference: The cross-attention compressor utilized during the SFT phrase (which assists by providing supervision signals) is completely discarded during RL and inference, resulting in zero architectural overhead. This design allows the model to "learn under supervision and thrive autonomously".

Limitations & Future Work

• Dynamic Determination of Canvas Steps: The current canvas step budget is a fixed maximum (set to 16 in experimental setups), with the model determining when to stop early. Enabling the model to adaptively allocate different numbers of latent steps according to task complexity remains an open avenue for exploration.
• Explainability of the Latent Space: Although attention maps verify that the canvas tokens indeed focus on critical image regions, what exactly is being 'computed' at each step of the latent-space recursion (e.g., internal visual search or feature alignment) remains an uninterpretable black box that lacks adequate visualization methodologies.
• Extension to Open-Domain Agentic Scenarios: Current experiments target single-turn, factual visual question answering. While the conclusion suggests extending this hybrid RL paradigm to open-ended agentic environments, this remains unverified.
• Large-Scale Scalability: While validation was conducted at the 3B and 7B scales, whether the vMF assumption and closed-form KL representation maintain their advantages on larger models (30B+) requires further empirical validation.

Related Work & Insights

• vs LVR: LVR aligns latent embeddings with auxiliary image features but mainly targets cropped regions and lacks global context. In contrast, HyLaR supports global-grained reasoning and optimizes the process via RL, going beyond simple SFT.
• vs SkiLa: SkiLa is a straightforward SFT-only latent reasoning scheme that trains via basic alignment. HyLaR builds upon this by adding DePO RL optimization (which SkiLa itself did not employ), lifting V* performance from 78.53 to 83.77.
• vs Monet: Monet designs a complex 3-stage SFT pipeline + RL. Though precise, it is prone to accumulating training biases. HyLaR achieves superior performance with a simple single-stage SFT + DePO, enjoying significant advantages in simplicity.
• vs Tool-Based Methods (DeepEyes/Thyme): Tool-based methods rely on external tool calls (re-perception/cropping), leading to 2.28× latency and 3.64× memory usage compared to the baseline. In contrast, HyLaR completes reasoning within a single forward pass, showing only 0.96× latency. HyLaR's vMF + decoupled clipping methodology is also transferable to other tasks requiring hybrid action spaces (e.g., low-level continuous control mixed with high-level discrete planning in embodied AI).
• Methodological Inspiration for RL for LLMs: DePO's philosophy of "identifying variance and geometric discrepancies across action spaces \(\to\) targeted dissection" can be adapted to other non-standard action spaces, such as hybrid generation schemes combining tool invocation and free text in tool-use agents.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introducing the vMF distribution to MLLM hybrid-space RL is highly original, with both the decoupled clipping design and closed-form KL derivation being highly novel.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers a wide range of multidimensional ablations, including high-resolution perception, general VQA, efficiency comparisons, two model scales (3B/7B), RL algorithm comparisons, distribution assumption alternatives, and hyperparameter sweeps, providing solid support for its findings.
• Writing Quality: ⭐⭐⭐⭐⭐ The motivation is exceptionally clear, with precise problem identification (variance mismatch + geometric mismatch), elegant equation derivations, and rigorously structured experimental designs.
• Value: ⭐⭐⭐⭐⭐ Resolves the most critical training bottleneck in the field of latent reasoning. The proposed method is simple, elegant, and state-of-the-art (substantially exceeding previous SOTA), and its methodological contributions to hybrid action space RL are highly transferable to multiple downstream domains.

Related Papers

• [ICLR 2026] MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization
• [ICLR 2026] Perception-Aware Policy Optimization for Multimodal Reasoning
• [CVPR 2026] Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization
• [CVPR 2026] CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
• [ICLR 2026] Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization