Skip to content

Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/PuffYang/Tournament_GRPO
Area: LLM Reasoning
Keywords: Reinforcement Learning, GRPO, Tournament Rewards, Long-Form Generation, LLM-as-a-Judge

TL;DR

To resolve the calibration drift and reward saturation inherent in independent pointwise LLM scoring for open-ended long-form generation, Tournament-GRPO structures same-query rollouts into repeated multi-round elimination tournaments, deriving group-normalized relative rewards that substantially improve GRPO advantage estimation on Deep Research Bench.

Background & Motivation

Large language models are increasingly deployed for open-ended, information-seeking applications such as autonomous deep research, multi-page report synthesis, and complex open-domain analysis. Unlike standard question answering where outputs can be verified through deterministic execution, ground-truth match, or token-level F1 metrics, open-ended long-form generation synthesizes evidence from disparate web sources, adheres to multifaceted task-specific rubrics, and spans a broad space of valid candidate solutions. Consequently, post-training reinforcement learning in this non-verifiable regime has predominantly relied on LLM-as-a-judge evaluators that score each candidate rollout independently against defined rubrics.

However, this pointwise absolute-scoring paradigm exhibits three fundamental failure modes when guiding policy optimization. First, global score calibration across heterogeneous queries, multi-turn tool trajectories, and evolving policy distributions is notoriously fragile; the same numerical score assigned by an LLM judge rarely denotes identical semantic quality across training steps. Second, pointwise scoring produces weak discrimination among same-query rollouts; candidates frequently saturate easy stylistic rubric dimensions, compressing score variance and injecting extreme noise into group-based advantage estimation. Third, absolute scoring is acutely vulnerable to optimization-induced reward hacking, wherein the policy exploits superficial rubric cues while under-optimizing intricate reasoning, synthesis depth, and factual grounding.

Group-based reinforcement learning algorithms like GRPO rely intrinsically on relative advantage estimation among multiple rollouts sampled from the identical prompt. The training reward does not need to serve as an isolated, universally calibrated absolute score; instead, its paramount role is to reliably discern fine-grained superiority within the local query group. Core idea: replace pointwise absolute judge scoring with a group-wise tournament framework that pits same-query rollouts against each other in repeated multi-round elimination brackets, converting accumulated tournament victories into normalized relative rewards that naturally align with GRPO's group-relative advantage estimation.

Method

Overall Architecture

Tournament-GRPO constructs a relative reward generation pipeline tailored to the mathematical formulation of GRPO advantage estimation. Given a user query and a set of query-specific rubrics, the policy first samples a group of candidate trajectories. Rather than assessing them in isolation, Tournament-GRPO conducts an iterative tournament process repeated over multiple rounds. In each tournament, candidate responses are partitioned into smaller comparison groups where an LLM judge selects top winners according to rubric alignment. Winners accumulate victory points and advance to successive rounds until a termination threshold is reached. Finally, accumulated scores across all tournament repeats are normalized within the group via min-max scaling and combined with a binary format reward to update the policy via GRPO.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query and Task Rubrics<br/>Query q + Rubrics B"] --> B["Same-Query Rollout Sampling<br/>Policy samples K long-form trajectories"]
    B --> C["Multi-Round Group Tournament<br/>Random shuffling and rubric-guided winner selection"]
    C --> D["Cross-Repeat Point Accumulation<br/>M tournament repeats with advancement weights"]
    D --> E["Group-Wise Normalization & Advantage Alignment<br/>Min-max scaling fused with format reward into GRPO"]
    E --> F["Aligned Long-Form Reasoning Policy"]

Key Designs

1. Multi-Round Group Tournament: Eliminating absolute calibration drift by converting global scoring into localized discrete winner selection
To overcome the instability and variance compression of standalone scalar scoring, this design treats all \(K\) rollouts generated for a prompt as an active tournament candidate pool \(C^{(0)}\). At each round \(t\), the active candidates are first randomly shuffled to eliminate judge position bias and then partitioned into disjoint comparison subsets \(P_l^{(t)}\) of uniform size \(G\). An LLM judge \(J_{\text{tournament}}\) inspects the query, the rubrics, and the \(G\) responses simultaneously, making a discrete relative judgment: selecting exactly \(K_{\text{win}}\) winners (\(W_l^{(t)}\)) to advance while eliminating the remainder. This cycle continues until the total pool of surviving candidates satisfies \(\sum |W_l^{(t)}| \le K_{\text{final}}\). By framing evaluation as local, rubric-guided choice rather than unconstrained scalar regression, the judge operates in a format known to be far more consistent and robust against global metric drift.

2. Cross-Repeat Point Accumulation: Mitigating pairing variance while maintaining linear judge complexity
A single elimination tournament is susceptible to stochastic pairing noise, where two exceptionally strong rollouts might clash in early rounds and prematurely eliminate a runner-up. To ensure smooth, robust ranking, Tournament-GRPO repeats the complete multi-round tournament \(M\) times for each query group. Every candidate maintains an accumulated score \(s_i\) initialized to zero. When candidate \(i\) is selected as a winner in round \(t\), it accrues an advancement credit \(\omega_t\) (configured uniformly as \(\omega_t = 1\) in practice). Across \(M\) repeated permutations, high-quality responses consistently accumulate higher aggregate scores. Because candidates shrink geometrically across rounds at an approximate factor of \(K_{\text{win}} / G\), the cumulative judge-call overhead scales as \(O(MK)\). This achieves fine-grained preference distinction with linear scaling in candidate pool size \(K\), bypassing the quadratic \(O(K^2)\) bottleneck of exhaustive pairwise comparison.

3. Group-Wise Normalization & Advantage Alignment: Harmonizing relative victory metrics with GRPO advantage estimation
Raw tournament scores \(s_i\) represent discrete win counts whose scales vary across tournament settings and cannot be directly compared across different queries. To interface cleanly with GRPO, the scores are mapped to \([0, 1]\) using query-level min-max normalization: $\(r_i^{\text{tour}} = \frac{s_i - \min_j s_j}{\max_j s_j - \min_j s_j + \epsilon}\)$ This normalization strictly preserves the relative order and rank separation among same-query rollouts while standardizing the reward distribution. The final scalar reward combines this tournament reward with a binary format verification reward \(r_i^{\text{fmt}} \in \{0, 1\}\): \(r_i = r_i^{\text{tour}} + r_i^{\text{fmt}}\). When inserted into the GRPO advantage formulation: $\(A_i = \frac{r_i - \mu(r)}{\sigma(r) + \epsilon}\)$ the group-relative reward directly aligns with the group-relative advantage normalization, ensuring that policy gradients are driven strictly by intra-group response distinctions.

Loss & Training

The training pipeline is built upon the verl framework with Qwen2.5-7B-Instruct as the base policy and a self-deployed Qwen2.5-72B-Instruct as the reward judge. Rollouts are generated using vLLM in a ReAct framework equipped with google_search and browse_webpage tools, sampling \(K = 8\) trajectories per query with a maximum response limit of 18,000 tokens. The model is trained with AdamW at a learning rate of \(1\times 10^{-6}\), mini-batch size of 16, and entropy coefficient of 0. Tournament configurations explore binary brackets \((G=2, K_{\text{win}}=1, K_{\text{final}}=1)\) and larger-group brackets \((G=4, K_{\text{win}}=2, K_{\text{final}}=2)\) across repetition depths \(M \in \{1, 2, 3, 4, 8\}\).

Key Experimental Results

Main Results

On Deep Research Bench, models are trained on the open-ended RL partition of DR Tulu and evaluated across Readability, Instruction Following (Instruct.), Comprehensiveness (Compre.), Insightfulness (Insight), and Overall score using Qwen2.5-72B-Instruct. The table below presents performance at Epoch 1 (Step 600) and Epoch 2 (Step 1100).

Stage Method Readability Instruct. Compre. Insight Overall
Epoch 1 (Step 600) Zero-shot (Base Policy) 35.40 34.21 31.42 29.76 32.13
Epoch 1 (Step 600) GRPO implicit (Single Scalar) 41.41 40.23 37.85 36.88 38.60
Epoch 1 (Step 600) GRPO explicit (Multi-Rubric Sum) 41.75 43.43 43.04 42.61 42.79
Epoch 1 (Step 600) GDPO (Decoupled Normalization) 43.22 42.07 39.81 39.05 40.61
Epoch 1 (Step 600) Exhaustive Pairwise 47.01 47.25 46.36 45.22 46.25
Epoch 1 (Step 600) Tournament-GRPO (G=2, M=3) 52.09 52.71 52.32 52.04 52.28
Epoch 2 (Step 1100) GRPO implicit (Single Scalar) 33.09 32.97 32.14 31.17 32.14
Epoch 2 (Step 1100) GRPO explicit (Multi-Rubric Sum) 41.26 41.56 41.33 40.77 41.20
Epoch 2 (Step 1100) GDPO (Decoupled Normalization) 51.11 51.01 50.52 50.52 50.71
Epoch 2 (Step 1100) Exhaustive Pairwise 51.15 51.34 50.38 50.05 50.57
Epoch 2 (Step 1100) Tournament-GRPO (G=2, M=4) 53.69 54.92 55.63 55.16 55.04
Epoch 2 (Step 1100) Tournament-GRPO (G=4, M=8) 53.81 55.22 55.19 55.40 55.09

Efficiency and Ablation Study

The following table compares judge complexity, empirical judge invocations, training wall-clock time, and task performance across reward mechanisms at training Step 600 (under \(G=2, K_{\text{win}}=1, K_{\text{final}}=1\) for Tournament-GRPO).

Method Complexity Judge Calls/Query Judge Calls/Rollout Wall-Clock Time Overall Score
Pointwise Average \(O(K) \sim O(KR)\) 8 1.000 11.48 h 38.98
Exhaustive Pairwise \(O(K^2)\) 28 3.500 88.59 h 46.25
Tournament-GRPO (\(M=1\)) \(O(MK)\) 7 0.875 59.65 h 42.40
Tournament-GRPO (\(M=2\)) \(O(MK)\) 14 1.750 62.91 h 51.01
Tournament-GRPO (\(M=3\)) \(O(MK)\) 21 2.625 67.77 h 52.28
Tournament-GRPO (\(M=4\)) \(O(MK)\) 28 3.500 73.05 h 46.33

Key Findings

  • Substantial Gains Over Pointwise Baselines: Tournament-GRPO reaches an overall benchmark score of 55.09 at Step 1100, surpassing GRPO explicit (41.20) by 13.89 points and completely averting the late-training degradation observed in GRPO implicit (which collapsed to 32.14).
  • Superior Effectiveness-Efficiency Frontier: Compared to exhaustive pairwise comparison, Tournament-GRPO (\(M=3\)) reduces judge calls by 25% (21 vs. 28 per query), lowers wall-clock training time by 23.5% (67.77 h vs. 88.59 h), and simultaneously yields a +6.03 point improvement in overall generation quality.
  • Interplay of Tournament Structure and Training Stage: In early training (Epoch 1) when rollout variance is high, sharp binary brackets (\(G=2, K_{\text{win}}=1\)) filter substandard trajectories aggressively, peaking at \(M=3\) (52.28). In advanced training (Epoch 2) where candidate quality is more competitive, larger-group brackets (\(G=4, K_{\text{win}}=2, M=8\)) provide softer multi-winner signals that achieve the highest final ceiling (55.09).
  • Cross-Judge and Cross-Domain Robustness: When re-evaluated using an unseen Qwen3-30B-A3B evaluator, Tournament-GRPO maintains an overall score of 42.71 (vs. Pairwise 35.32 and Pointwise 30.49). Additionally, on the HealthBench-Professional medical long-form benchmark, Tournament-GRPO achieves 52.54, demonstrating consistent transfer across evaluators and domain distributions.

Highlights & Insights

  • Architectural Isomorphism with GRPO: Because GRPO relies on group-level mean-variance advantage normalization, relative tournament rankings provide an mathematically natural alignment. They circumvent gradient noise caused by drifting global baselines across queries.
  • Linear-Scaling Relative Comparison: By structuring candidate comparisons as geometric elimination trees, Tournament-GRPO preserves the cognitive benefits of comparative judgment while slashing evaluation complexity from \(O(K^2)\) to \(O(MK)\).
  • Dynamic Curriculum Insight: The empirical transition from binary elimination (early phases) to multi-winner group brackets (late phases) offers a valuable heuristic for curriculum reward scheduling in reinforcement learning.

Limitations & Future Work

  • Residual LLM Judge Biases: While relative tournaments insulate against absolute score drift, underlying judge biases—such as verbosity preferences or rhetorical formatting biases—can still influence winner selection.
  • Hyperparameter Sensitivity Across Phases: The optimal configuration of group size \(G\), winner count \(K_{\text{win}}\), and repeats \(M\) shifts dynamically throughout training, currently necessitating empirical selection rather than adaptive self-tuning.
  • Lack of Granular Token-Level Attribution: Tournament outcomes assign document-level scalar advantages, providing no direct localization of hallucinations, logical fallacies, or ungrounded assertions within a 10,000-word response. Combining tournament rewards with process reward models remains an open frontier.
  • vs. Standard GRPO / Dr.GRPO: GRPO was formulated for verifiable domains like mathematics or code; when applied to long-form generation via pointwise scoring, it suffers severe calibration and saturation issues. Tournament-GRPO retains the GRPO policy optimization mechanics while redesigning the reward engine into a tournament-based preference aggregator.
  • vs. ArenaRL / Pairwise Ranking: ArenaRL constructs win rates via quadratic pairwise enumeration (\(O(K^2)\)), incurring heavy computational costs. Tournament-GRPO replaces exhaustive matching with structured elimination trees, lowering computational overhead to \(O(MK)\) while outperforming pairwise baselines.
  • vs. GDPO: GDPO introduces decoupled normalization across rubric dimensions to stabilize multi-reward gradients, but still depends on pointwise scalar assessments. Tournament-GRPO addresses the vulnerability at its source by replacing scalar evaluation with relative comparative decisions.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Elegantly adapts tournament-style elimination to group-wise advantage estimation in open-ended LLM reinforcement learning.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations spanning training epochs, wall-clock efficiency, diverse backbones, unseen judges, and out-of-domain benchmarks.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition of motivation, explicit mathematical formulations, and thorough experimental documentation.
  • Value: ⭐⭐⭐⭐⭐ Offers a practical, highly scalable blueprint for non-verifiable reinforcement learning in autonomous research agents and long-form writing systems.