Skip to content

Beyond Leaderboards: A Frozen-Rubric Contract for Position-Invariant Winning-Argument Detection

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Winning-argument detection, Position invariance, Privileged self-distillation, Frozen rubric, Audit ledger

TL;DR

This work introduces TAPIR, an evaluation protocol that distils a frozen, survey-grounded persuasion rubric into an anti-symmetric dual-head architecture, achieving exact algebraic speaker-order invariance (WPI=1), calibrated probabilities under proper scoring rules, a 39.41× inference acceleration, and a +7.68 pp accuracy gain over the same-backbone MS-PS baseline.

Background & Motivation

As large language models transition from experimental prototypes into mission-critical evaluation infrastructure for content moderation, policy adjudication, and debate assessment, progress cannot be reduced to marginal accuracy gains on held-out test leaderboards. Reliable pairwise judging in online persuasion communities—such as predicting which argument earned an opinion-reversing delta on Reddit Change My View (CMV)—requires mathematical guarantees of system reliability. An evaluation judge must not have its verdicts dictated by arbitrary speaker presentation order, must produce calibrated probabilities under strictly proper scoring rules to support risk-controlled selective deployment, and must expose structured, inspectable diagnostics to human auditors without leaking test-time rubric prompts.

Prompt-engineered LLM judges fail fundamentally across all of these requirements. On the benchmark 807-pair winning-argument (WA) test split, an instruction-tuned Qwen3-14B zero-shot judge exhibits severe positional bias, switching its verdict on 35% of identical argument pairs when presentation order is inverted (yielding an empirical swap-consistency of only 0.65). State-of-the-art multi-strategy persuasion scoring (MS-PS) architectures attempt to mitigate this by prompting LLMs for strategy-level ratings, but they require an exorbitant 24 LLM calls per pair, feed raw rubric guidelines into test-time context windows, and rely on post-hoc natural language rationalizations that frequently uncouple from actual internal decision mechanisms.

When human-validated or survey-grounded rubrics exist, the central architectural question is what formal guarantees a judge consuming that rubric can be engineered to satisfy. Heuristic prompt fixes and vote ensembles merely suppress empirical variance without eliminating order dependence, all while multiplying operational compute costs. Core idea: Freeze survey-grounded evaluation rubrics as train-time-only privileged supervision, distil rubric knowledge via mode-split self-distillation to prevent test-time leakage, and enforce exact algebraic speaker-order invariance (WPI=1) through a parameter-shared anti-symmetric scalar head paired with a structured diagnostic ledger.

Method

Overall Architecture

The TAPIR (Theory-Aligned Persuasion-Invariant Rubric scoring) framework separates offline privileged training from online anti-symmetric inference. In the offline phase, the Dual-Artifact Frozen Protocol (DAFP) content-hashes a survey-derived rubric (\(P_{\text{trace}}\)) with SHA-256 and exposes it exclusively to a privileged teacher model. A student model receives only the operational input (\(P_{\text{ops}}\): opening post and candidate replies) and is trained via mode-split self-distillation, alternating online between ground-truth label supervision and teacher soft-target alignment. In the inference phase, the model executes a weight-shared dual forward pass over \((OP, r_A, r_B)\) and \((OP, r_B, r_A)\). An anti-symmetric scalar head computes exact difference logits, while an auxiliary head decodes a \(K\)-category strategy ledger; a tied-order wrapper around a gradient-boosted decision machine (GBM) routes the verdict while preserving exact invariance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Pair (OP, rA, rB)<br/>& Swapped Pair (OP, rB, rA)"] --> B["Dual-Artifact Frozen Protocol (DAFP)<br/>SHA-256 Hashed Rubric · Zero Test-Time Leakage"]
    B --> C["Mode-Split Privileged Self-Distillation<br/>Stepwise Bernoulli Draw · BCE vs JSD De-aliasing"]
    C --> D["Algebraic Anti-Symmetric Scalar Head<br/>z_head = phi_AB - phi_BA · Exact WPI=1 Guaranteed"]
    D --> E["Structured Audit Ledger Head<br/>K-Strategy Categorical Diagnostics · Error Flagging"]
    E --> F["Tied-Order Anti-Symmetric Wrapper<br/>R_anti Difference Transform · Preserves Contract"]

Key Designs

1. Dual-Artifact Frozen Protocol (DAFP): Zero Test-Time Rubric Leakage via Privileged Decoupling Standard LLM judges inject extensive scoring guidelines into test-time prompts, driving up latency and triggering erratic instruction following. Grounded in Vapnik's Learning Under Privileged Information (LUPI) paradigm, DAFP formally bifurcates the evaluation inputs into two sealed artifacts. The privileged artifact \(P_{\text{trace}}\) synthesizes 6 theoretical and operational blocks from computational persuasion surveys (Bozdag et al.), fixed and SHA-256 content-hashed before evaluation to serve as training-only guidance for the teacher. The operational artifact \(P_{\text{ops}}\) restricts runtime inference strictly to \((OP, r_A, r_B)\), completely barring rubric text from the deployment prompt. This architectural withholding prevents evaluation-time guideline leakage, bounds inference latency, and ensures that the model internalizes evaluative criteria into compact parameter representations.

2. Mode-Split Privileged Self-Distillation: Mitigating Within-Step Gradient Interference Directly optimizing a weighted sum of discrete ground-truth labels and continuous teacher strategy distributions induces severe gradient conflict. Empirical gradient diagnostics reveal that during early training, the binary cross-entropy gradient and the teacher Jensen-Shannon divergence gradient point in opposing directions (\(\cos(g_{\text{BCE}}, g_{\text{JSD}}) < 0\)) in 33% of optimization steps, causing destructive gradient cancellation. TAPIR resolves this by sampling an optimization mode dynamically at each step with probability 0.5: one mode optimizes standard BCE against empirical winning labels, while the alternate mode optimizes JSD against the teacher's Bernoulli win probability and categorical strategy distributions. Alternating steps preserves the directional integrity of both signals, yielding a +1.7 pp gain over compute-matched weighted-sum baselines.

3. Algebraic Anti-Symmetric Scalar Head: Mathematical Guarantee of Winning-Position Invariance (WPI=1) Positional bias is an endemic flaw of pairwise autoregressive evaluation. Rather than relying on approximate double-prompt ensembling, TAPIR feeds both candidate permutations \((OP, r_A, r_B)\) and \((OP, r_B, r_A)\) through a weight-shared LoRA encoder to extract order-conditional latent scalar projections \(\phi_{AB} = \phi(f_\theta(OP, r_A, r_B))\) and \(\phi_{BA} = \phi(f_\theta(OP, r_B, r_A))\). The final scalar verdict is defined as their elementary subtraction: $\(z_{\text{head}}(OP, r_A, r_B) = \phi_{AB} - \phi_{BA}\)$ By construction, \(z_{\text{head}}(OP, r_B, r_A) = -z_{\text{head}}(OP, r_A, r_B)\) holds algebraically for all parameters and inputs, ensuring that the empirical swap residual is identically zero (\(|\varepsilon_{\text{swap}}| = 0\)) across all 807 test pairs. Proposition 1 demonstrates via Jensen's inequality that under a swap-balanced data distribution and any convex margin loss, discarding the symmetric even component and projecting onto the swap-odd subspace satisfies \(R(s^-) \le R(s)\), establishing that exact structural invariance incurs zero theoretical surrogate risk penalty.

4. Structured Audit Ledger Head: Diagnostic Visibility and Error Triage Because free-form Chain-of-Thought justifications are susceptible to post-hoc rationalization unfaithful to actual model predictions, TAPIR outputs an inspectable \(K=6\) dimensional strategy ledger decoded from the shared verdict representation. Each cell predicts a 3-way distribution \(\hat{q}_k \in \Delta^2\) (A wins, B wins, or tie) across canonical persuasion dimensions. While not claiming faithfulness to internal neural circuits, the ledger functions as a calibrated operational diagnostic: when a candidate pair produces strategy conflict or low evidence-validity (\(<0.50\)), the odds of an erroneous verdict increase by 2.4×. Integrating ledger features raises error-detection AUROC from 0.63 to 0.71, providing a principled reject-to-human threshold for high-stakes deployment.

5. Tied-Order Anti-Symmetric Wrapper: Router Equivariance Preservation While the scalar head is mathematically anti-symmetric, the discrete categorical ledger head lacks inherent equivariance and empirically flips on only 58% of presentation swaps. To deploy a gradient-boosted decision machine (GBM) combining scalar logits and ledger statistics without forfeiting the WPI=1 guarantee, TAPIR wraps the router in a tied-order difference transform: $\(R_{\text{anti}}(F_{ab}, F_{ba}) = R(F_{ab}) - R(F_{ba})\)$ where \(F_{ab}\) and \(F_{ba}\) represent the feature vectors extracted from both presentation passes. The system predicts A if \(R_{\text{anti}} > 0\) and falls back to \(z_{\text{head}}\) whenever \(R_{\text{anti}} = 0\). This guarantees that the complete deployed router retains exact algebraic anti-symmetry on 807/807 test instances while capturing the additional +0.50 pp accuracy advantage enabled by ledger-guided routing.

Loss & Training

The pipeline is trained in three disciplined stages on Qwen3-14B (LoRA rank 16, alpha 32, AdamW, effective batch size 32): - Stage 0 (Privileged Teacher): Trained for 3 epochs on \((OP, r_A, r_B, P_{\text{trace}})\) to predict ground-truth labels and strategy categorical distributions; frozen upon convergence. - Stage 1 (Mode-Split Self-Distillation): The student consumes \(P_{\text{ops}}\) over 3 epochs, drawing label mode or distillation mode with equal probability at each step: $\(\mathcal{L}_{\text{stage1}} = \mathbb{E}\left[ \frac{1}{2}\mathcal{L}_{\text{BCE}}(\hat{p}_{Ai}, y_i) + \frac{1}{2} \left( \text{JSD}(\text{Bern}(\hat{p}_{Ai}) \parallel \text{Bern}(\tilde{p}_{Ai})) + \frac{1}{K}\sum_{k=1}^K \text{JSD}(\text{Cat}(\hat{q}_{ki}) \parallel \text{Cat}(\tilde{q}_{ki})) \right) \right]\)$ - Stage 2 (Clean Inner-Dev Retraining): Fine-tuned for 1 epoch using BCE exclusively on an inner-dev subset strictly scrubbed of SHA-256 collisions against test pairs, followed by post-hoc temperature calibration and 5-fold cross-validated GBM fitting.

Key Experimental Results

Main Results

On the canonical 807-pair Change My View winning-argument test split, TAPIR is compared paired against same-backbone implementations and zero-shot baselines:

System Protocol Acc (%) ↑ Brier ↓ ECE ↓ WPI Invariance Paired p vs. Re-run (McNemar)
Random (Empirical) Baseline 49.19 - - - -
Direct Zero-shot (Qwen3-14B) Single prompt 53.28 .248 .033 0.650 -
MS-PS-MLP Re-run (Qwen3-14B) 24 calls + MLP 58.12 .247 .059 - Reference baseline
TAPIR (Scalar head only) Dual-forward diff 65.30 .233 .102 1.000 (Algebraic) \(6.92 \times 10^{-4}\) (Sig.)
TAPIR (Deployed GBM Router) Full protocol 65.80 .233 .100 1.000 (Algebraic) \(\mathbf{3.41 \times 10^{-4}}\) (Sig.)
OpenAI-o3 + MS-PS-MLP (Closed) Unpaired context 64.53 - - - Unpaired reference

Ablation Study

Ablations on the WA test split (n=807) isolate the individual contributions of protocol components, decision channels, and training objectives:

Configuration / Variant Acc (%) Δ vs. Full TAPIR (pp) Paired p-value Architectural Note
TAPIR Deployed (Seed 42) 65.80 - - Complete three-stage system
No Distillation (BCE only) 60.35 -5.45 \(1.46 \times 10^{-3}\) Trained without privileged \(P_{\text{trace}}\)
Non-Anti-Symmetric Pairwise LoRA 59.80 -6.00 \(< 10^{-3}\) Standard concatenation without difference
Scalar-Only Distillation 64.10 -1.70 \(0.031\) Omission of \(K\)-strategy ledger distillation
Weighted-Sum Distillation 63.90 -1.90 \(0.018\) Simultaneous BCE+JSD without mode splitting
Post-hoc Anti-Symmetrized Prompt 57.60 -8.20 \(< 10^{-4}\) Operator applied to unadapted prompted judge
DeBERTaV3-large Cross-Encoder 58.40 -7.40 \(< 10^{-4}\) Compact backbone control against size scaling
Ledger Majority Channel Only 64.81 -0.99 - Independent discrete decision channel
Per-Pair Oracle (Diagnostic) 70.76 +4.96 - Theoretical headroom for head disagreement

Key Findings

  • Dominance of Privileged Distillation: Both the scalar head (65.30%) and the ledger channel (64.81%) substantially outperform the no-distillation baseline (60.35%), demonstrating that internalizing the frozen rubric—rather than head engineering—drives primary accuracy gains.
  • Inference Efficiency: Full deployed inference takes 0.846 seconds per pair on a single NVIDIA A800 GPU, yielding a 39.41× end-to-end speedup over the 33.33-second 24-call MS-PS baseline. Single-seed training amortizes after approximately 1,330 evaluated pairs.
  • Cross-Topic and Stratified Robustness: TAPIR achieves positive gains across all four topic domains, surviving Bonferroni correction on Gender/Minority Rights (+14.16 pp) and BH-FDR correction on Food & Culture (+12.05 pp), alongside consistent gains across argument length tertiles.

Highlights & Insights

  • Contract-Centric Evaluation Paradigm: Elevates NLP judging from benchmark hill-climbing to verifiable system contracts that enforce mathematical position invariance, strict proper calibration, zero runtime prompt overhead, and diagnostic transparency.
  • Algebraic Invariance over Heuristic Ensembling: Replaces inefficient double-prompt averaging with a weight-shared differential forward projection \(\phi_{AB} - \phi_{BA}\), guaranteeing zero swap residual without empirical failure modes.
  • Gradient De-aliasing via Mode Splitting: Documents that 33% of multi-task steps encounter gradient opposition between gold labels and teacher distributions, proving that online stochastic objective alternation cleanly bypasses optimization interference.

Limitations & Future Work

  • Rubric Maintenance vs. Semantic Redesign: The protocol relies on stable evaluation rubrics. Minor taxonomic paraphrasing incurs negligible degradation (-0.3 pp), but major shifts in persuasive values require re-executing offline distillation stages.
  • Shallow-Cue Susceptibility under Domain Shift: On the Anthropic/persuasion dataset, zero-shot transfer accuracy (60.74%) matches length-only heuristics due to strong dataset-level length bias; genuine semantic superiority is recovered only when evaluated on an adversarially balanced cue-neutral subset.
  • Diagnostic Non-Faithfulness: The audit ledger provides actionable operational indicators that boost error detection (AUROC 0.71), but it remains a pseudo-trace reflecting teacher behavior rather than a mechanically faithful readout of latent transformer activations.
  • vs. MS-PS (Labruna et al., 2026): MS-PS incurs prohibitive runtime latency (24 LLM calls/pair) and remains vulnerable to positional bias; TAPIR compresses all rubric dimensions into an internal representation, speeding up inference by 39.41× while guaranteeing WPI=1.
  • vs. Bavaresco et al. (2025): While prior work demonstrated that LLM-judge agreement with humans fluctuates unpredictably across tasks, TAPIR assumes an established rubric and addresses the complementary challenge: building verifiable algorithmic guarantees into the judge that executes it.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant synthesis of learning under privileged information and algebraic symmetry constraints for robust judging.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous multiple-testing corrections, paired bootstrap intervals, and exhaustive ablation baselines.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally disciplined framing that explicitly distinguishes contract preservation from leaderboard claims.
  • Value: ⭐⭐⭐⭐⭐ Provides an efficient, dependable, and reproducible blueprint for deploying production-grade LLM evaluation infrastructure.