Consensus Group Relative Policy Optimization for Text Generation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Reinforcement Learning, GRPO, Minimum Bayes Risk Decoding, Reference-Free Distillation, Consensus Utility
TL;DR¶
Consensus Group Relative Policy Optimization (C-GRPO) distills computationally expensive inference-time Minimum Bayes Risk (MBR) decoding into a single-pass policy by formulating within-group consensus utility as group-relative advantages in GRPO, matching or exceeding multi-sample MBR performance without gold references or explicit preference labels.
Background & Motivation¶
In modern natural language generation and structured prediction, consensus-based decoding methods represent some of the most robust inference techniques. Among them, Minimum Bayes Risk (MBR) decoding operationalizes a sample-and-rerank paradigm: given an input prompt, the model generates multiple candidate hypotheses, evaluates each candidate under a task-specific utility metric against the entire candidate pool acting as pseudo-references, and finally selects the output that achieves the highest consensus score. Compared with traditional greedy decoding or mode-seeking maximum-a-posteriori (MAP) inference, MBR systematically circumvents pathological probability traps and hallucinated modes, yielding superior semantic fidelity across machine translation, text summarization, and constrained decision tasks.
Despite its performance benefits, consensus decoding imposes a prohibitive computational burden during inference. For each prompt, MBR requires generating \(G\) candidate sequences and executing \(O(G^2)\) pairwise utility evaluations. As foundational models scale up and production environments demand strict latency guarantees, this quadratic inference complexity creates an acute quality–latency bottleneck. Previous attempts to amortize this inference cost into training typically rely on teacher-generated outputs, gold reference texts, or curated preference datasets (e.g., using sequence-level SFT or Direct Preference Optimization). These approaches introduce heavy dataset construction overhead and are often constrained by the fidelity and prompt vulnerability of external reward models.
Consequently, the core dilemma lies in how to internalize the sample-and-rerank consensus decision rule into a single-pass generative policy without relying on gold references, teacher labels, or external reward models. The paper's key insight stems from the observation that consensus metrics are computed over a sampled candidate group and depend strictly on relative scores within that group. Core idea: formulate within-group consensus utility as group-relative advantages within Group Relative Policy Optimization (GRPO), amortizing MBR decoding into a single-pass policy in a purely reference-free, self-supervised manner.
Method¶
Overall Architecture¶
The Consensus Group Relative Policy Optimization (C-GRPO) pipeline comprises five interconnected stages: prompt sampling, on-policy candidate group generation, pairwise utility matrix computation, within-group advantage normalization, and policy gradient optimization. Given a prompt \(q\), the old policy \(\pi_{\theta_{\text{old}}}\) samples a group of \(G\) candidate completions \(\mathcal{G}=\{y_1, y_2, \dots, y_G\}\). Rather than querying an external reward model, C-GRPO evaluates the pairwise similarity between all sampled outputs using a bounded utility function \(u(\cdot, \cdot)\) (e.g., BLEURT), deriving an empirical consensus score \(\hat{u}(y_i|q)\) for each hypothesis. These consensus scores are normalized across the group to construct relative advantages \(A_i\), which directly drive the policy gradient update on \(\pi_\theta\). At test time, the trained model generates text in a single forward pass, completely eliminating sampling and reranking.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Prompt q ~ D"] --> B["Self-Consensus Group Utility Formulation<br/>Old Policy πθold Samples Group of G Candidates"]
B --> C["Within-Group Pairwise Utility Matrix<br/>Compute u(yi, yj) to Aggregate Consensus Score ˆui"]
C --> D["Group-Relative Advantage Normalization<br/>Standardize Consensus Scores via Mean and Variance"]
D --> E["Reference-Free Policy Gradient Update<br/>Train Single-Pass Policy πθ to Internalize Consensus"]
E --> F["Test-Time Single-Pass Generation (No Reranking)"]
Key Designs¶
1. Self-Consensus Group Utility Formulation: eliminating external reward models and gold references Standard reinforcement learning algorithms like GRPO or PPO depend on external scalar reward models \(R_\phi(q, y)\) or gold target supervision, both of which introduce vulnerability to reward hacking, judge prompt sensitivity, and evaluator bias. C-GRPO overcomes this by utilizing the candidate group \(\mathcal{G}\) itself as both the hypothesis set and the empirical reference distribution. For each candidate \(y_i\), its training reward is formulated as the mean consensus utility measured against all peer completions within the group: $\(\hat{u}(y_i | q) = \frac{1}{G} \sum_{j=1}^G u(y_i, y_j)\)$ where \(u(y_i, y_j) \in [0, 1]\) is a task-specific utility metric (such as BLEURT). Because the optimization signal is derived entirely from on-policy consensus, C-GRPO operates without gold references, teacher transcripts, or dedicated reward models, guiding the generative policy toward the highest-density semantic consensus region of the model's own distribution.
2. Group-Relative Advantage Normalization: converting MBR decision boundaries into RL policy updates Once consensus scores are evaluated, C-GRPO leverages GRPO's group-relative advantage structure to differentiate high-quality outputs from mediocre candidates. Given the group sample mean \(\mu_G = \frac{1}{G}\sum_{i=1}^G \hat{u}(y_i|q)\) and group sample standard deviation \(\sigma_G = \text{std}_G(\hat{u})\), the normalized advantage \(A_i\) is computed as: $\(A_i = \frac{\hat{u}(y_i | q) - \mu_G}{\sigma_G}\)$ In the C-Dr.GRPO variant, the denominator standard deviation \(\sigma_G\) is omitted to eliminate stochastic preconditioning noise, centering the advantage solely around the group mean. Crucially, subtracting the group mean provides an unbiased baseline because each candidate's peer mean \(u_{(-i)}\) is statistically independent of \(y_i\) under i.i.d. sampling. This assigns positive gradients to candidates exceeding group-level consensus and penalizes peripheral outliers.
3. Reference-Free Policy Gradient Update: theoretical directional alignment with expected utility Under an idealized sentence-level surrogate where the sequence \(y\) is treated as an indivisible action and candidate lengths are constant, the authors formally prove that the expected C-GRPO update aligns directionally with the true policy gradient of the underlying MBR objective \(\mathcal{L}_C(\theta) = \mathbb{E}_{q}[\mathbb{E}_{y\sim \pi_\theta}[u_{\text{model}}(y|q)]]\). Specifically, for group size \(G \ge 2\), under \(L\)-smoothness and bounded variance assumptions, there exists a positive scalar \(\alpha > 0\) such that: $\(\mathbb{E}_{\mathcal{D}}[\hat{g}_{\text{GRPO}}(\theta; q)] = \alpha \nabla_\theta \mathcal{L}_C(\theta)\)$ Furthermore, stochastic gradient ascent with learning rate \(\eta_t = 1/\sqrt{T}\) guarantees an ergodic non-asymptotic convergence rate of \(\mathcal{O}(1/\sqrt{T})\). This theoretical consistency confirms that C-GRPO shifts generative probability mass toward near-reference, high-expected-utility modes, allowing a single forward pass at inference time to replicate or exceed the quality of sample-and-rerank MBR.
Loss & Training¶
The training objective follows the clipped surrogate loss of GRPO: $\(\mathcal{J}_{\text{C-GRPO}}(\theta) = \mathbb{E}_{q\sim \mathcal{D}, \{y_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}}\left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \rho_i(\theta) A_i \right]\)$ where \(\rho_i(\theta) = \frac{\pi_\theta(y_i|q)}{\pi_{\theta_{\text{old}}}(y_i|q)}\) denotes the importance sampling ratio. Following DeepSeek-R1-Zero and Dr.GRPO practices, the KL divergence penalty is set to \(\beta = 0\), eliminating the need to maintain an active reference model in GPU memory. The model is trained using LoRA (rank=128, alpha=128) with AdamW (learning rate \(2\times 10^{-6}\), weight decay 0.1) on NVIDIA A100 (80GB) GPUs. For WMT machine translation, training runs for 2 epochs with candidate size \(G=32\) and per-device batch size 32. For XSum summarization, training runs for 1 epoch with \(G=16\) and batch size 16.
Key Experimental Results¶
Main Results¶
The authors evaluated C-GRPO on WMT24 machine translation across three language pairs (En \(\to\) {Ja, Zh, De}), XSum extreme summarization, and the JBBQ Japanese social-bias benchmark. Evaluation metrics include COMET (a reference-based neural translation metric unused during training), BLEURT, and ROUGE-Lsum.
Table 1: WMT24 Machine Translation Results (COMET and BLEURT across 3 Language Pairs)
| Base Model | Supervision Category | Method | En-Ja CMT | En-Zh CMT | En-De CMT | 6-Setting Avg. CMT |
|---|---|---|---|---|---|---|
| Llama-3-8B | Zero-shot Baseline | Base Model | 0.530 | 0.667 | 0.696 | 0.616 |
| Reference-based (‡) | BoN (Oracle, G=32) | 0.803 | 0.807 | 0.799 | 0.803 | |
| Reference-based (‡) | GRPO w/ reference | 0.823 | 0.802 | 0.778 | 0.801 | |
| Reference-based (‡) | SFT (Gold) | 0.805 | 0.793 | 0.781 | 0.793 | |
| Reference-free | MBR decoding (G=32) | 0.713 | 0.773 | 0.768 | 0.751 | |
| Reference-free | GRPO w/ Random | 0.516 | 0.720 | 0.692 | 0.643 | |
| Reference-free | GRPO w/ Self-Rewarding | 0.672 | 0.728 | 0.719 | 0.706 | |
| Proposed | C-GRPO (Ours) | 0.759 | 0.723 | 0.724 | 0.735 | |
| Proposed | C-Dr.GRPO (Ours) | 0.742 | 0.763 | 0.758 | 0.754 | |
| Mistral-7B | Zero-shot Baseline | Base Model | 0.567 | 0.576 | 0.657 | 0.600 |
| Reference-free | MBR decoding (G=32) | 0.632 | 0.672 | 0.707 | 0.670 | |
| Reference-free | GRPO w/ Self-Rewarding | 0.661 | 0.766 | 0.736 | 0.721 | |
| Proposed | C-GRPO (Ours) | 0.704 | 0.708 | 0.693 | 0.702 | |
| Proposed | C-Dr.GRPO (Ours) | 0.737 | 0.759 | 0.728 | 0.741 |
Table 2: XSum Extreme Summarization and JBBQ Social-Bias Evaluation
| Task & Benchmark | Base Model | Method | Target Metric Score | Absolute Gain over Base |
|---|---|---|---|---|
| XSum Summarization (ROUGE-Lsum) | Llama-3-8B | Base Model | 0.362 ± 0.002 | - |
| GRPO w/ Random | 0.329 ± 0.008 | -0.033 | ||
| MBR decoding (G=16) | 0.365 ± 0.004 | +0.003 | ||
| GRPO w/ Self-Rewarding | 0.254 ± 0.024 | -0.108 | ||
| SFT w/ MBR generations | 0.353 ± 0.009 | -0.009 | ||
| C-GRPO (Ours) | 0.410 ± 0.009 | +0.048 | ||
| C-Dr.GRPO (Ours) | 0.416 ± 0.006 | +0.054 | ||
| Mistral-7B | Base Model | 0.231 ± 0.002 | - | |
| MBR decoding (G=16) | 0.247 ± 0.002 | +0.016 | ||
| C-GRPO (Ours) | 0.247 ± 0.004 | +0.016 | ||
| C-Dr.GRPO (Ours) | 0.232 ± 0.003 | +0.001 | ||
| JBBQ Social-Bias (Accuracy %) | Mistral-7B | Base Model | 26.8% | - |
| MBR decoding | 31.0% | +4.2% | ||
| GRPO w/ Self-Rewarding | 27.4% | +0.6% | ||
| C-GRPO (Ours) | 34.4% | +7.6% | ||
| Llama-3-8B | Base Model | 42.6% | - | |
| MBR decoding | 48.0% | +5.4% | ||
| GRPO w/ Self-Rewarding | 46.0% | +3.4% | ||
| C-GRPO (Ours) | 49.0% | +6.4% |
Ablation Study & Empirical Analysis¶
An analysis of candidate group budget \(G \in \{4, 8, 16, 32\}\) on En-Ja translation reveals that inference-time MBR COMET improves monotonically with \(G\), requiring substantial compute. In contrast, C-GRPO trained with moderate budgets (\(G \ge 8\)) achieves near-saturated performance and outscores MBR decoding across all sample budgets on Llama-3-8B.
Across 13 instruction-tuned LLMs spanning scales from 0.5B to 8B (Llama-3/3.1/3.2, Mistral, Qwen2, and Qwen2.5), C-GRPO consistently improves ROUGE-Lsum on XSum (e.g., Qwen2.5-3B: +0.119, Qwen2.5-0.5B: +0.102, Llama-3.2-3B: +0.101). Degradation is strictly confined to the smallest models (Qwen2-0.5B at -0.082 and Qwen2-1.5B at -0.001), where base generation diversity and quality are insufficient.
Tracking training dynamics (Figures 4–7) shows that sampling variance contracts dramatically during early optimization (steps 1000–4000), while average BLEURT/COMET scores rise monotonically. Heatmap similarity between generated checkpoints and historical best outputs exceeds 0.8, demonstrating genuine mode-seeking convergence toward high-utility solution spaces.
Key Findings¶
- Surpassing finite-sample MBR bounds: On XSum with Llama-3-8B, inference-time MBR shows negligible improvement (0.362 \(\to\) 0.365) because reranking cannot escape the finite candidate pool. C-GRPO reshapes the generative distribution itself, boosting single-pass ROUGE-Lsum to 0.410 (paired sign-flip test \(p < 10^{-6}\), mean difference +0.0554), proving that policy updates synthesize high-consensus structures beyond discrete sampled candidates.
- Robustness against prompt sensitivity and social bias: Self-Rewarding baselines degrade severely under judge prompt modifications (dropping from 0.707 to 0.465 on MT under Unbabel prompts) and amplify social stereotypes on JBBQ. C-GRPO aggregates peer similarities via an objective metric, attaining 34.4% on Mistral and 49.0% on Llama on JBBQ without judge prompting artifacts.
Highlights & Insights¶
- Amortizing \(O(G^2)\) consensus inference into single-pass forward execution: C-GRPO elegantly repurposes GRPO's group normalization to turn peer consensus into policy gradient advantage, offering an attractive training-to-inference efficiency trade-off.
- Reference-free self-supervised RL framework: By eliminating gold targets, human pair curation, and auxiliary reward models, C-GRPO allows LLMs to iteratively self-align using only intrinsic peer consensus and an off-the-shelf semantic utility metric.
- Practical computational break-even profile: On a single NVIDIA A100 GPU, training C-GRPO takes approximately 5 hours, while MBR decoding consumes ~2 hours per 500 test queries, yielding a clear break-even threshold at roughly 1,300 inference requests.
Limitations & Future Work¶
- Dependency on utility metric fidelity: C-GRPO's optimization trajectory is bounded by the alignment of the chosen utility function (BLEURT) with human evaluation; metric blind spots will be reinforced during policy updates.
- Failure modes in sub-1.5B models: Extremely small models fail to maintain diverse, coherent candidates during early sampling, leading to collapsed policy updates.
- Lack of deterministic instance-level equivalence: Because C-GRPO optimizes expected utility over the policy distribution rather than executing an exact instance-level \(\arg\max\), it does not guarantee identical token sequences to test-time MBR.
Related Work & Insights¶
- vs. Sequence-Level MBR Distillation (Finkelstein & Freitag, 2024; Wang et al., 2024): Traditional distillation uses offline MBR outputs to fine-tune a student via SFT or DPO, which incurs massive offline generation storage and suffers from capacity constraints or likelihood degradation. C-GRPO performs online policy optimization directly on on-policy samples, providing smoother gradient landscapes.
- vs. Self-Rewarding Language Models (Yuan et al., 2024): Self-rewarding relies on LLM-as-a-judge prompting, which is vulnerable to hallucination, sycophancy, and prompt fragility. C-GRPO relies on mathematical consensus across candidate ensembles, achieving far greater stability in sensitive evaluation regimes like JBBQ.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Bridges the classical MBR consensus framework and modern GRPO group-relative RL, backed by rigorous theoretical alignment proofs.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across translation, summarization, and social bias over 13 open-source models, complete with variance convergence dynamics.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid mathematical formulations, coherent motivation, and meticulous empirical documentation.
- Value: ⭐⭐⭐⭐⭐ Provides an immediately deployable, reference-free industrial blueprint for serving MBR-grade text generation with zero test-time latency penalty.