DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHF

Conference: AAAI 2026 arXiv: 2511.19097 Code: None Area: LLM Reasoning / RLHF / Reward Modeling Keywords: reasoning chain decoupling, parallel sub-step generation, cascaded DRPO, interpretable rewards, modular reasoning

TL;DR

DeCoRL transforms CoT reasoning from monolithic sequential processing into an "orchestra-style" modular parallel collaboration — nine specialized sub-models (parsing / semantic / entity / fact-checking / style / quality / computation / verification / integration) generate reasoning sub-steps in parallel, coordinated via dual reward attribution (local quality + contribution) and cascaded DRPO optimization, achieving 80.8% on RM-Bench (surpassing all baselines), a 3.8× inference speedup, and a 22.7% improvement in interpretability.

Background & Motivation

Background: CoT reasoning combined with RLHF is the dominant paradigm for enhancing complex reasoning in LLMs; methods such as DPO and GRPO are already widely adopted.

Limitations of Prior Work: Two fundamental limitations persist — (1) black-box rewards: existing RL methods provide holistic reward signals that cannot distinguish the contribution of individual reasoning steps, making error diagnosis extremely difficult; (2) sequential bottleneck: sequential decoding has time complexity \(O(n)\), which is impractical for complex reasoning tasks.

Key Challenge: A tension between coherence and modularity — end-to-end optimization preserves coherence but sacrifices modularity and efficiency, whereas naive modularization may compromise reasoning coherence.

Goal: Simultaneously achieve interpretability (step-level reward attribution) and efficiency (parallel generation) without sacrificing quality.

Key Insight: The reasoning process is analogized to a symphony orchestra — a solo pianist (monolithic model) is replaced by a team of specialist musicians (dedicated modules) guided by a conductor (reward coordination) and rehearsal (cascaded training). Each module focuses on one reasoning facet; outputs are generated in parallel and then integrated.

Core Idea: Dedicated sub-models generate reasoning sub-steps in parallel; dual reward attribution precisely evaluates each module; cascaded DRPO coordinates the optimization.

Method

Overall Architecture

A fixed ensemble of \(k=9\) specialized sub-models, each receiving a shared context \(\mathcal{C}\), generates its respective output \(O_i\) in parallel. A deterministic synthesis function \(\Phi\) integrates these outputs into a complete reasoning chain. A dual reward mechanism independently evaluates each module's local quality and its contribution to the overall output. Cascaded DRPO optimizes the parameters of each module.

Key Designs

1. Nine Specialized Reasoning Modules:

   • \(M_{\text{parse}}\): Structured decomposition of the problem
   • \(M_{\text{semantic}}\): Extraction of deep semantic information
   • \(M_{\text{entity}}\): Expansion of entity context and relations via a knowledge graph
   • \(M_{\text{factcheck}}\): Verification of factual consistency
   • \(M_{\text{style}}\): Analysis of style, tone, and lexical consistency
   • \(M_{\text{quality}}\): Assessment of diversity and creativity
   • \(M_{\text{compute}}\): Handling symbolic and numerical computation
   • \(M_{\text{verify}}\): Logical consistency checking
   • \(M_{\text{integrate}}\): Synthesis of all module outputs into the final response
   • All modules share the input context but maintain fully isolated states: \(\mathcal{H}_i^t \cap \mathcal{H}_j^t = \emptyset\); outputs conform to a standardized JSON interface.

2. Dual Reward Attribution Mechanism:

   • Local reward \(R_{\text{local}}^i = \text{RM}_\phi(O_i \| \mathcal{C})\): Evaluates the standalone output quality of module \(i\)
   • Contribution reward \(R_{\text{contrib}}^i = \text{RM}_\phi(O_{\text{full}}) - \text{RM}_\phi(O_{\text{full}}^{-i})\): Measures the marginal contribution of module \(i\) via counterfactual ablation (removing module \(i\))
   • Composite reward: \(R_i = \alpha \cdot R_{\text{local}}^i + \beta \cdot R_{\text{contrib}}^i\), where \(\alpha\) and \(\beta\) are adaptively balanced via learnable temperature parameters
   • Design Motivation: The local reward checks a module's intrinsic quality, while the contribution reward ensures the module provides genuine collective value — preventing modules that appear effective in isolation but contribute nothing to the overall output.

3. Cascaded DRPO (Differential Reinforcement Preference Optimization):

   • Function: Extends GRPO to multi-module systems, coordinating preference optimization across modules
   • Core loss: \(\mathcal{L}_{\text{DRPO}}(\theta_i) = -\mathbb{E}[\log \sigma(\gamma(\Delta R_i(O_w, O_l) - \eta \text{KL}(M_i(\cdot|\mathcal{C}) \| M_{\text{base}}(\cdot|\mathcal{C}))))]\)
   • The reward differential is decomposed into a local quality gap and a contribution utility gap
   • Cascaded training updates parameters in stages, preserving inter-module dependencies.

Key Experimental Results

Main Results — RM-Bench

Method	Chat	Math	Code	Safety	Avg
Skywork-Reward-8B	69.5	60.6	54.5	95.7	70.1
URM-LLaMA-8B	71.2	61.8	54.1	93.1	70.0
GPT-4o	-	-	-	-	~73*
DeCoRL-7B	68.1	68.3	55.9	94.0	71.6
DeCoRL-32B	76.8	81.6	67.9	95.5	80.8

RMB Benchmark

Method	Helpfulness	Harmlessness	Overall
GPT-4o	0.727	0.748	0.738
Claude-3.5-sonnet	0.772	0.641	0.706
DeCoRL-32B	0.741	0.774	0.757

Ablation Study

Configuration	RM-Bench	RMB	Latency	Interpretability
Full DeCoRL	80.8	0.757	316ms	84.0%
w/o contribution reward	76.1 (−4.7%)	0.721	316ms	51.9% (−32.1%)
Sequential execution	80.5 (−0.3%)	0.754	1172ms (+271%)	84.0%
Ad-hoc interface	74.3 (−8.0%)	0.698	316ms	63.7%

Key Findings

• DeCoRL-32B achieves 80.8% on RM-Bench — 10.7% above the strongest scalar reward model (70.1%) and surpassing GPT-4o.
• The Math domain shows the largest gain (+21.0% vs. Skywork), indicating that modularization is particularly effective for mathematical reasoning — the specialization of \(M_{\text{compute}}\) contributes significantly.
• 3.8× inference speedup (1202ms → 316ms), 72.4% reduction in energy consumption, and 68% throughput improvement.
• The contribution reward is critical for interpretability — removing it causes interpretability to drop from 84% to 51.9% (−32.1%) and performance to decline by 4.7%.
• Interface standardization is essential — using an ad-hoc interface format causes an 8% performance degradation.
• Sequential execution preserves nearly identical accuracy but incurs a 271% latency increase — confirming that parallelization does not compromise quality.

Highlights & Insights

• Paradigm shift in reasoning: Moving from "one model performs all reasoning" to "a team of specialized modules reasons collaboratively" — analogous to MoE, but operating at the reasoning-chain level rather than the token level.
• Counterfactual ablation for contribution rewards: Inspired by Shapley-value thinking — quantifying true contribution by measuring how much overall performance degrades when a module is removed; 89.3% of errors are correctly attributed to specific modules.
• Module scalability: Adding new modules (e.g., \(M_{\text{context}}\), \(M_{\text{ambiguity}}\)) requires only interface definition without retraining existing modules — yielding a 7.3% performance improvement with only an 18% latency increase.
• Engineering-oriented design: Standardized JSON interfaces, context isolation, and heterogeneous hardware deployment reflect an industry-grade engineering mindset.

Limitations & Future Work

• The division of labor among the nine modules is manually designed — automated learning of optimal module partitioning remains unexplored.
• Modules communicate only through shared context rather than direct interaction — fine-grained inter-module information exchange may be lost.
• The standardized JSON interface may limit output flexibility — potentially unfriendly to unstructured reasoning scenarios.
• Computing contribution rewards via counterfactual ablation requires \(k\) additional inference passes — overhead grows as the number of modules increases.
• Validation is limited to reward modeling; effectiveness on direct reasoning tasks (e.g., mathematical problem solving) remains to be confirmed.

Related Work & Insights

• vs. DPO/GRPO: DPO/GRPO optimize holistically; DeCoRL optimizes at the sub-step level — offering finer granularity and stronger interpretability.
• vs. Process Supervision (Math-Shepherd/OmegaPRM): Process supervision provides step-level feedback but still generates the full chain sequentially; DeCoRL achieves genuine parallel generation with independent evaluation.
• vs. MoE: MoE routes at the token level to different experts; DeCoRL assigns at the reasoning-step level to dedicated modules — modularization at a different granularity.
• Inspiration: The selective debate mechanism in iMAD could be integrated with DeCoRL — debate modules could be triggered only when a given module's output is uncertain.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Transforms reasoning from sequential processing to modular parallel collaboration — a paradigm-level innovation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Three benchmarks × multi-scale models × efficiency analysis × interpretability × ablation × scalability.
• Writing Quality: ⭐⭐⭐⭐ The symphony orchestra analogy is vivid; formalization is rigorous.
• Value: ⭐⭐⭐⭐⭐ Significant contributions to the efficiency, interpretability, and scalability of reasoning systems; well-suited for industrial deployment.

Related Papers

• [NeurIPS 2025] Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
• [AAAI 2026] On the Exponential Convergence for Offline RLHF with Pairwise Comparisons
• [AAAI 2026] GRAM-R²: Self-Training Generative Foundation Reward Models for Reward Reasoning
• [AAAI 2026] EPO: Diverse and Realistic Protein Ensemble Generation via Energy Preference Optimization
• [AAAI 2026] When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF