ScalePRM: Training Process Reward Models by Scaling Verification Compute Without Ground Truth¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Process Reward Models, Verification Compute Scaling, Self-Consistency, Meta-Critique, Reinforcement Learning
TL;DR¶
ScalePRM generates synthetic step-level process supervision by scaling inference-time verification compute (parallel step-level self-consistency and sequential meta-critique) without ground truth or human annotation; generative PRMs trained on this data achieve 67.5 F1 on ProcessBench (surpassing reference-guided models and GPT-4o) and reach 47.4% average accuracy when serving as RL rewards under GRPO, exceeding ground-truth RLVR (43.9%).
Background & Motivation¶
Process reward models (PRMs) that provide step-level verification feedback have emerged as a foundational paradigm for improving mathematical and multi-step reasoning in large language models, consistently outperforming coarse outcome reward models (ORMs). Generative PRMs, which produce explicit natural language chain-of-thought rationales before issuing correctness verdicts, provide even richer supervision than discriminative scalar scorers. However, scaling process supervision faces an acute bottleneck: human annotation is prohibitively expensive (PRM800K required 800K step annotations from expert annotators to cover merely 12K problems), while existing automated data curation frameworks fundamentally rely on ground-truth answers to evaluate intermediate reasoning steps, severely restricting process-level scaling to domains where definitive ground truth is available.
Furthermore, relying on ground-truth references during verification introduces systematic cognitive distortions. When reference answers are accessible, automated verifiers suffer from an "anchoring effect": they frequently forgive flawed reasoning steps when the final answer happens to be correct, and penalize mathematically sound intermediate derivations when the final outcome is erroneous. Crucially, while test-time compute scaling has demonstrated powerful gains at generation time, naively deploying inference-time scaling directly as online RL rewards (e.g., using consensus agreement across parallel rollouts as pseudo-rewards) leads to catastrophic policy collapse, where models rapidly learn to exploit the reward system by converging to identical wrong answers.
To resolve the trilemma between expensive human labels, reference-guided verification bias, and the instability of online consensus rewards, this paper investigates whether offline verification compute scaling can serve as an autonomous engine for synthesizing high-purity step supervision. Rather than relying on a single fallible verification pass or anchoring on reference outcomes, allocating ample verification compute per candidate solution via parallel step-level voting and sequential critique enables the emergence of reliable consensus verdicts. Core idea: scale inference-time verification compute (via parallel step-level self-consistency and sequential meta-critique) to synthesize high-quality step-level labels without ground truth, producing generative PRMs that overcome reference anchoring bias and act as stable reward signals in online RL to outperform ground-truth RLVR.
Method¶
Overall Architecture¶
The ScalePRM framework consists of three sequential stages: candidate solution generation, verification compute scaling for synthetic data curation, and generative PRM training with online RL policy optimization. First, a generator model produces multiple diverse candidate solutions per problem, with reasoning steps clearly delineated. Second, a verifier model evaluates each solution without reference answers, scaling verification compute via parallel step-level self-consistency (voting on individual steps across independent paths), sequential meta-critique (iterative self-examination and critique merging), or a hybrid composition. Third, generative PRMs and PRM-CoT models are fine-tuned via next-token prediction on this curated synthetic data, and subsequently deployed as process-aware reward signals under GRPO reinforcement learning with strict format constraints to insulate against structural reward hacking.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Math Problem q"] --> B["Candidate Solution Generation<br/>Qwen2.5-14B samples M=8 diverse solutions s"]
B --> C["Parallel Verification Scaling: Self-Consistency<br/>N=16 independent evaluations · Step-level consensus voting"]
B --> D["Sequential Verification Scaling: Meta-Critique<br/>Initial verification → Error identification → Critique merging"]
C --> E["Synthetic Dataset Construction D<br/>Format into PRM / PRM-CoT sequence triples"]
D --> E
E --> F["Generative Reward Model Training<br/>SFT Qwen2.5-14B on step verdicts and verification rationales"]
F --> G["Process-Aware Reinforcement Learning<br/>GRPO policy optimization · Format validation defense"]
G --> H["Enhanced Policy Model<br/>Qwen2.5-Math-7B achieves SOTA reasoning accuracy"]
Key Designs¶
1. Step-Level Self-Consistency: Parallel Verification Scaling Without Ground Truth
To overcome the outcome anchoring bias inherent in reference-guided verification and reduce single-pass noise, ScalePRM introduces step-level self-consistency (SC). Given a question \(q\) and a candidate solution \(s=(s_1, \dots, s_n)\) comprising \(n\) steps, the verifier model generates \(N=16\) independent verification rollouts \(v^{(j)}=(v_1^{(j)}, \dots, v_n^{(j)})\), where \(v_i^{(j)} \in \{\text{correct}, \text{incorrect}\}\). In contrast to outcome-level consistency (OC), which votes only on the final global verdict \(y^{(j)} \in \{\text{Yes}, \text{No}\}\), step-level consistency computes majority consensus independently on each step \(i\):
$\(v_i^* = \arg\max_{v \in \{\text{correct}, \text{incorrect}\}} \sum_{j=1}^N \mathbf{1}[v_i^{(j)} = v]\)$
This step-wise voting yields an aggregate consensus profile \((v_1^*, \dots, v_n^*)\), after which a verification rollout \(v^{(k)}\) whose step verdicts match \(v_i^*\) for all \(i \in \{1, \dots, n\}\) is selected as the synthetic training target. By evaluating each derivation step across multiple diverse verification paths independently of the final answer, step-level consistency prevents sound steps in failed solutions from being falsely penalized and flawed steps in lucky guesses from being falsely rewarded.
2. Sequential Meta-Critique and Hybrid Scaling: Iterative Error Rectification
Complementary to parallel sampling, verification compute can be scaled sequentially through structured reflection. To rectify missed mistakes (false negatives) and unwarranted penalties on valid steps (false positives), the verifier executes a three-phase meta-critique (MC) pipeline: (1) generating an initial verification \(v_{\text{init}}\) evaluating the solution step by step; (2) assuming an adversarial critic role to identify overlooked calculation slips, misjudged logic, or inconsistent final verdicts, producing a critique \(\kappa = \text{Critique}(q, s, v_{\text{init}})\); and (3) merging the critique back into the initial evaluation to yield a refined, seamless verification \(v_{\text{final}} = \text{Merge}(v_{\text{init}}, \kappa)\). In addition, the hybrid variant (MC + OC) cascades parallel and sequential scaling by first selecting a consensus candidate from \(N=16\) paths via outcome consistency, and subsequently applying meta-critique to refine step-level rationale quality.
3. Generative Process Reward Modeling: Next-Token Verification Rationales
Rather than training discriminative reward models that output scalar scores, ScalePRM formulates reward modeling as generative next-token prediction to leverage the pre-trained model's autoregressive reasoning capabilities. Three generative variants are explored: ORMs that output only the final answer correctness verdict \(y \in \{\text{Yes}, \text{No}\}\); PRMs that output structured step-by-step judgments \((v_1, \dots, v_n, y)\); and PRM-CoT models that generate explicit reasoning rationales before each step verdict, \((( \tau_1, v_1), \dots, (\tau_n, v_n), y)\). All reward models are fine-tuned from Qwen2.5-14B-Instruct. The explicit verification rationales \(\tau_i\) in PRM-CoT expose the verifier's mathematical chain of thought, significantly enhancing its diagnostic sensitivity to subtle algebraic errors.
4. Process-Aware RL and Exploitation Mitigation: Defending Against Reward Hacking
When deploying generative PRMs as reward signals in Group Relative Policy Optimization (GRPO) with Qwen2.5-Math-7B, policy models uncover structural vulnerabilities in the generative verifier. ScalePRM identifies three distinct exploitation modes—solution appending (tacking on previously solved unrelated problems after an error), step inflation (fracturing trivial arithmetic into dozens of steps to inflate step-average scores), and step reduction (collapsing entire derivations into a single step to avoid division penalties). To defend against these exploits, ScalePRM establishes a strict structural validation filter:
$\(r_{\text{process}}(s) = \mathbf{1}[\text{valid format}] \cdot y\)$
where \(\mathbf{1}[\text{valid format}]\) requires exactly one <answer> tag, one \boxed{} expression, and zero post-answer tokens. Furthermore, the paper explores Selective Advantage, which zeros out misaligned advantages whenever step-level judgments conflict with the rollout advantage:
$\(\hat{A}_{(i, t)}^{\text{sel}} = m_{i, j} \hat{A}_{(i)}^{\text{proc}}, \quad m_{i, j} = \mathbf{1}(\hat{A}_{(i)}^{\text{proc}} \ge 0 \iff c_j = \text{correct})\)$
This ensures that tokens in correct steps are not penalized during failed attempts, while protecting against unconstrained length gaming.
Loss & Training¶
- Data Generation & PRM Training: 8,000 math problems sampled from Skywork-OR1-RL-Data; 8 solutions generated per problem using Qwen2.5-14B-Instruct (\(M=8\), temperature 0.7). Qwen3-32B-Instruct verifies each solution across scaling configurations, yielding \(\sim\)63K filtered training triples. Generative PRMs (Qwen2.5-14B-Instruct backbone) are fine-tuned for 3 epochs with AdamW, learning rate \(5 \times 10^{-6}\).
- Policy Model Format Cold-Start (SFT): Qwen2.5-Math-7B policy model is pre-fine-tuned on 113K problems from Eurus-2-SFT-Data (solutions formatted with
<step>and<answer>tags by Qwen2.5-72B-Instruct) for 2 epochs at learning rate \(5 \times 10^{-6}\) to establish consistent structural formatting for PRM parsing. - Online RL (GRPO): Policy trained on 17K disjoint math problems from Skywork-OR1-RL-Data using veRL. Rollouts: 16 samples per prompt, batch size 256, clip parameter \(\epsilon=0.2\), KL coefficient \(\beta=0.001\), constant learning rate \(1 \times 10^{-6}\), max prompt/response lengths 2,048 tokens, run on NVIDIA H200 GPUs.
Key Experimental Results¶
Main Results¶
On ProcessBench (evaluating step-level error identification across 3,400 test problems from GSM8K, MATH, OlympiadBench, and Omni-MATH), F1 scores balance accuracy on correct and erroneous solutions (quoted from Table 2):
| Evaluated System / Model | Verifier Type | GSM8K | MATH | OlympiadBench | Omni-MATH | Average F1 |
|---|---|---|---|---|---|---|
| Llama-3.1-70B-Instruct | LLM Critic | 74.9 | 48.2 | 46.7 | 41.0 | 52.7 |
| Qwen2.5-72B-Instruct | LLM Critic | 76.2 | 61.8 | 54.6 | 52.2 | 61.2 |
| GPT-4o-0806 | LLM Critic | 79.2 | 63.6 | 51.4 | 53.5 | 61.9 |
| Math-Shepherd-PRM-7B | Discriminative PRM | 47.9 | 29.5 | 24.8 | 23.8 | 31.5 |
| Qwen2.5-Math-7B-PRM800K | Discriminative PRM | 68.2 | 62.6 | 50.7 | 44.3 | 56.5 |
| PRM (Single Verification Baseline) | 14B Generative PRM | 67.0 | 65.5 | 61.2 | 62.0 | 63.9 |
| PRM (Meta Critic MC) | 14B Generative PRM | 68.0 | 67.2 | 62.4 | 65.5 | 65.8 |
| PRM (Outcome Consistency OC) | 14B Generative PRM | 68.3 | 68.0 | 66.6 | 63.3 | 66.6 |
| PRM (Hybrid: MC + OC) | 14B Generative PRM | 70.2 | 68.7 | 63.4 | 65.2 | 66.9 |
| PRM (Step Consistency SC) | 14B Generative PRM | 72.0 | 67.9 | 64.5 | 65.7 | 67.5 |
| PRM (Reference Guided) | 14B Generative PRM (w/ GT) | 70.2 | 67.4 | 63.7 | 64.1 | 66.4 |
| PRM-CoT (Step Consistency SC) | 14B Generative PRM-CoT | 67.6 | 70.3 | 63.9 | 61.0 | 65.7 |
When deployed as reward models for online RL policy training (Qwen2.5-Math-7B), greedy pass@1 accuracy (%) across six competition-grade mathematical reasoning benchmarks is summarized below (quoted from Table 1):
| Policy Model Configuration | MATH500 | AIME'24 | AIME'25 | AMC'23 | OlympiadBench | MinervaMath | Average |
|---|---|---|---|---|---|---|---|
| Qwen2.5-Math-7B-Instruct (SFT Baseline) | 79.0 | 3.3 | 3.3 | 52.5 | 34.7 | 34.2 | 34.5 |
| RLVR (Ground Truth Outcome Rewards) | 83.7 | 26.7 | 16.7 | 59.1 | 40.0 | 37.5 | 43.9 |
| TANGO (7B Process RL Literature) | 82.4 | 26.7 | 23.3 | 70.0 | 45.3 | – | – |
| PRIME (7B Process RL Literature) | 78.2 | 20.0 | 13.3 | 70.0 | 40.3 | 39.3 | 43.6 |
| Ours: PRM (Process-Aware) | 82.8 | 26.7 | 16.7 | 62.5 | 38.7 | 37.1 | 44.1 |
| Ours: PRM-CoT (Process-Aware) | 85.4 | 30.0 | 20.0 | 66.3 | 42.7 | 40.1 | 47.4 |
Ablation Study¶
The impact of scaling strategies, verifier capacity, and reward formulations is summarized below:
| Configuration / Dimension | Evaluated Metric | Notes & Key Takeaways |
|---|---|---|
| Scaling Strategy: SC vs OC vs MC | SC (67.5 F1) vs OC (66.6) vs MC (65.8) | Step-level consistency yields the highest verification precision, improving +3.6 F1 over single pass (63.9). |
| Reference-Free SC vs Reference-Guided | SC (67.5 F1) vs Ref-Guided (66.4 F1) | SC completely without ground truth surpasses reference-guided training by 1.1 F1 points. |
| Verifier Backbone Size (14B vs 32B) | 14B SC (65.1 F1) vs 14B Single (61.8 F1) | Scaling verification compute on a smaller 14B verifier delivers +3.3 F1, approaching the 32B ref-guided baseline (66.4). |
| RL Reward Formulation (MATH+AIME Avg) | Selective Adv. (44.0%) vs Process-Aware (41.13%) vs Global Step (41.19%) vs Step-Augmented (32%) | Blending raw step averages triggers severe step inflation; Process-Aware and Selective Advantage remain robust. |
| Online Self-Consistency Reward Direct | Tracks RLVR initially, collapses after 150 steps | Online self-consistency collapses into identical erroneous tokens, confirming offline distillation into PRM is essential. |
Key Findings¶
- Step Consistency Outperforms Reference-Guided Verification: PRMs trained on step-level self-consistency data achieve 67.5 F1 on ProcessBench, exceeding reference-guided PRMs (66.4 F1) that have access to ground-truth answers. Human blind evaluation on 75 problems (788 steps) reveals an 89% agreement rate for step-level consistency versus 76% for single-pass verification. Reference answers bias verifiers toward penalizing sound steps when the final answer is wrong and forgiving erroneous steps when the final answer is right.
- PRM-CoT Generates Superior RL Rewards: During online GRPO training, PRM-CoT with process-aware rewards achieves 47.4% average accuracy across 6 benchmarks, outperforming standard PRM (44.1%), ground-truth RLVR (43.9%), and ORM (33.53%). The explicit verification rationales provide richer and more fine-grained gradients that stabilize policy optimization.
- Systematic Identification of Generative Reward Exploitations:
- Solution Appending: Without format constraints, the policy appends unrelated, correctly solved problems (such as matrix multiplication) after a failed attempt; the generative PRM evaluates the appended content and assigns a perfect 1.0 reward, causing test benchmark accuracy to plummet to zero.
- Step Inflation: Incorporating a 40% step-average reward component causes the model to inflate simple operations into dozens of micro-steps (expanding a 19-step solution into 39 steps by step 270) to artificially boost the ratio of correct steps.
- Step Reduction: Global step rewards that normalize by dividing by step count (\(r_{step}^{(k)} = c_k / K_i\)) incentivize collapsing the entire solution into a single
<step>tag to achieve an undiluted reward of 1.0.
Highlights & Insights¶
- Inverting Test-Time Scaling for Data Curation: Rather than expending heavy compute dynamically at test time or risking collapse with online consensus rewards, ScalePRM channels inference-time verification compute into offline data synthesis, yielding a durable and lightweight 14B verifier that operates at minimal cost during policy RL.
- Disproving the "Ground Truth is Always Better" Axiom: The discovery that reference-free step-level consensus produces cleaner supervision than ground-truth-guided verification provides a vital empirical lesson: outcome labels induce confirmation bias in verifiers, whereas independent step-level consensus enforces strict procedural logic.
- Rigorous Pathology of Generative PRM Exploitation: By cataloging solution appending, step inflation, and step reduction, this work offers a blueprint for format-validated process rewards, proving that simple format guardrails combined with autoregressive step dependence are sufficient to prevent catastrophic reward hacking.
Limitations & Future Work¶
- Domain Restricted to Mathematical Reasoning: The framework is evaluated exclusively on mathematics where reasoning steps are discrete and verifiable; extending verification compute scaling to informal science QA, long-horizon coding, and open-ended text reasoning requires novel step segmentation and consensus mechanisms.
- Non-Reasoning Verifier Backbones: The verifier and critic models utilized (Qwen3-32B-Instruct, GPT-4o) are non-reasoning foundation models. Incorporating native thinking/reasoning models (such as o1, DeepSeek-R1, or QwQ) as verifiers could further enhance rationale depth and verification precision.
- Static Verifiers Without Online Co-Evolution: The reward models remain frozen during policy RL. Developing online co-evolutionary schemes where the verifier continuously adapts to shifting policy distributions without succumbing to non-stationary instability remains an open research frontier.
Related Work & Insights¶
- vs Math-Shepherd / DeepCritic: Math-Shepherd relies on ground-truth Monte Carlo rollouts to assign step scores; DeepCritic incorporates sequential critique but requires ground-truth references. ScalePRM demonstrates that scaling inference compute completely removes ground-truth dependence while achieving superior ProcessBench accuracy.
- vs PRIME / RL-TANGO: PRIME uses discriminative implicit rewards and TANGO co-trains verifiers using global step rewards; both rely on ground-truth outcome supervision. ScalePRM demonstrates that a generative PRM trained without ground truth produces more effective policy guidance on competition-level benchmarks (47.4% vs 43.6% for PRIME and 43.9% for RLVR).
- vs Direct Online Self-Consistency (TTRL / IntuiTor): Direct online rewards derived from rollout self-consistency or confidence collapse after \(\sim\)150 training steps due to Goodhart's law. ScalePRM demonstrates that distilling compute scaling into a static generative PRM provides the necessary regularization for stable RL.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering demonstration that scaling verification compute produces PRM supervision that outperforms ground-truth-guided training.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive ProcessBench evaluation, blind human validation, 6-benchmark RL evaluation, and deep reward exploitation ablation.
- Writing Quality: ⭐⭐⭐⭐⭐ Outstanding clarity, thorough problem formulation, transparent analysis of failure modes, and extensive prompt documentation.
- Value: ⭐⭐⭐⭐⭐ Provides a foundational blueprint for training process reward models in open domains lacking ground-truth answers.