Skip to content

Same Trajectory, Contradictory Rewards (RoboRMBench): Paraphrase Fragility in Vision Language Reward Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning / Robot Reward Models / Evaluation Benchmarks
Keywords: Vision Language Reward Models, Paraphrase Fragility, Robotic Reinforcement Learning, RoboRMBench, Score Crossing Rate

TL;DR

This paper introduces RoboRMBench, an evaluation benchmark measuring paraphrase robustness in vision-language robot reward models, revealing that general-purpose VLMs exhibit up to 60.7% failure-success score reversals under semantically equivalent goal rewrites for identical robot trajectories, while trajectory-grounded supervision and variance reduction training effectively alleviate this vulnerability.

Background & Motivation

Natural language has rapidly become a scalable and flexible interface for specifying complex robotic manipulation objectives. Recent research increasingly adopts vision-language models (VLMs) as learned reward functions, mapping a robot execution trajectory alongside a textual task description directly to a scalar estimate of task progress. Unlike coarse binary success detectors, continuous scalar progress scores provide graded feedback over partial task completions, reducing the reliance on laborious manual reward engineering or dense human demonstrations in reinforcement learning and policy improvement. In offline trajectory filtering or online exploration, reward predictions dictate which behaviors are reinforced and which candidate policies are deployed.

However, standard evaluation protocols for language-conditioned robotic reward models harbor a critical blind spot: virtually all benchmarks evaluate models using a single, canonical task instruction. In real-world environments, identical physical goals can be naturally formulated through diverse lexical, syntactic, and perspective variations. If a reward model is fragile to surface wording, holding the robot execution trajectory and visual evidence completely fixed while introducing a meaning-preserving paraphrase can trigger dramatic score fluctuations. In severe cases, the exact same physical robot behavior is flipped between outright failure and complete success based solely on the phrasing of the prompt.

This contradiction injects destructive noise into reinforcement learning loops, encouraging policy optimization toward prompt wording artifacts rather than the underlying physical intent. Crucially, this vulnerability cannot be resolved simply by expanding model parameters or enabling explicit chain-of-thought reasoning; in fact, larger models and reasoning variants often exhibit heightened sensitivity to surface text features. Core idea: Introduce RoboRMBench to systematically evaluate paraphrase robustness across 2,390 real-robot trajectories and 21,673 verified paraphrases, establishing Score Crossing Rate (SCR) and Flip Rate (FR) to measure category-level failure-success reversals, and proving that trajectory-grounded reward supervision and variance reduction training are essential for reliable embodied reward modeling.

Method

Overall Architecture

RoboRMBench evaluates whether vision-language reward models produce stable scalar progress predictions \(r_k = R_\theta(\tau, g_k)\) when evaluating a fixed robot trajectory \(\tau\) across a set of semantically equivalent goal descriptions \(\mathcal{G}(\tau) = \{g_1, \dots, g_K\}\). The systematic benchmark pipeline comprises four interconnected stages: (1) Balanced trajectory stratification over real-robot manipulation trajectories drawn from RoboRewardBench; (2) Multi-tiered paraphrase generation spanning three levels of increasing linguistic divergence (lexical substitution, syntactic restructuring, and action-goal perspective shift); (3) Multi-LLM ensemble filtering to conservatively eliminate candidates altering physical constraints or visual groundings; and (4) Comprehensive failure-success contradiction evaluation combining Score Crossing Rate (SCR), Flip Rate (FR), and Mean Error (ME) alongside variance reduction mitigation techniques.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real-Robot Trajectory Suite<br/>Fixed visual observation sequence τ"] --> B["Three-Tiered Paraphrase Generation<br/>Lexical / Syntactic / Perspective Shift"]
    B --> C["Conservative Semantic Filtering<br/>Multi-LLM consensus on physical invariants"]
    C --> D["Failure-Success Reversal Evaluation<br/>Reference-free SCR, FR, and Mean Error"]
    D --> E["Robustness Mitigation Strategies<br/>Inference aggregation & variance penalty"]

Key Designs

1. Three-Tiered Paraphrase Generation: Stratifying Lexical, Syntactic, and Perspective Shifts

Rather than relying on unconstrained text rewriting, the benchmark constructs paraphrases along three structured linguistic tiers with increasing divergence from the canonical instruction. The first tier, Lexical Substitution (LS), strictly preserves syntactic structure and clause order while swapping content words (nouns, verbs, adjectives) with valid synonyms (e.g., changing "Put" to "Place"). The second tier, Syntactic Restructuring (SR), preserves semantic entities but fundamentally rearranges sentence syntax, incorporating voice alternation, clause reordering, and nominalization (e.g., "You should put the blocks inside the black bowl"). The third tier, Action-Goal Perspective Shift (AGPS), reformulates imperative action sequences into declarative terminal state descriptions or vice versa (e.g., "The goal is for the blocks to end up inside the black bowl"). This progression isolates whether reward models degrade at surface-level wording or stumble over structural and perspective reformulations.

2. Conservative Semantic Filtering: Eliminating Subtle Physical Constraint Drifts

In physical robotic control, standard natural language synonyms frequently alter visual referents or operational constraints. For instance, substituting "orange block" with "amber block" can confuse visual object grounding; replacing "place" with "drop" alters force dynamics and motor primitives; and changing "near" to "next to" modifies spatial tolerance boundaries. To prevent false-positive contradictions caused by genuine semantic drift, RoboRMBench employs an ensemble of three independent frontier validators (gemini-3-flash, claude-sonnet-4-6, and deepseek-v3.2). A candidate paraphrase is discarded if any single model detects a discrepancy across eight embodied dimensions (object attributes, spatial relations, temporal dependencies, persistence constraints, and termination criteria). Validated against human annotations on 210 sampled instances, the ensemble achieved a 96% recall on semantically inequivalent rewrites. The final benchmark curates 2,390 trajectories and 21,673 verified paraphrased instructions.

3. Failure-Success Reversal Metrics: Reference-Free Category Crossing Detection

To capture consequential decision-level failures rather than harmless floating-point score perturbations, the benchmark defines categorical failure as scores in \(\{1, 2\}\) and categorical success as scores in \(\{4, 5\}\) on a 5-point progress scale (excluding intermediate score 3). The primary metric, Score Crossing Rate (SCR), measures the proportion of trajectories where the exact same robot behavior receives at least one failure prediction and at least one success prediction across its paraphrase set: $$ \text{SCR} = \frac{1}{|\mathcal{T}|} \sum_{\tau \in \mathcal{T}} \mathbb{I}\left( \min_{g \in \mathcal{G}(\tau)} R_\theta(\tau, g) \le 2 \;\land\; \max_{g \in \mathcal{G}(\tau)} R_\theta(\tau, g) \ge 4 \right) $$ Because SCR evaluates min-max crossings across \(\mathcal{G}(\tau)\), it is reference-free and independent of canonical wording choice. Complementary metrics include Flip Rate (FR), which measures the frequency of category flips relative to the original canonical instruction, and Mean Error (ME), measuring the absolute discrepancy between the ensemble-averaged predicted score \(\mu_\theta(\tau) = \frac{1}{K}\sum_{k=1}^K R_\theta(\tau, g_k)\) and human ground-truth progress labels.

4. Variance Reduction Training and Paraphrase Aggregation: Enforcing Invariance

To mitigate paraphrase fragility, the paper investigates inference-time marginalization and training-time regularization. At inference time, Paraphrase Aggregation generates five semantically equivalent prompt variants for any query instruction and averages predicted scalar rewards, marginalizing out localized wording idiosyncrasies. To eliminate the multi-query API latency overhead at deployment, Variance Reduction Training augments the standard supervised prediction loss \(\mathcal{L}_{\text{pred}}\) with an explicit variance penalty across predictions on equivalent paraphrases: $$ \mathcal{L} = \mathcal{L}{\text{pred}} + \lambda \operatorname{Var} \left[ R_\theta(\tau, g_k) \right] $$ This loss penalizes dispersion over the semantic equivalence manifold, training the network to anchor rewards directly onto invariant visual evidence during single-pass forward inference.}(\tau)

Key Experimental Results

Main Results

The benchmark evaluated 18 vision-language models spanning proprietary frontier APIs, open-source generalist VLMs, and specialized robotic reward models across all three paraphrase tiers:

Model Category Evaluated Model LS: SCR ↓ / FR ↓ / ME ↓ SR: SCR ↓ / FR ↓ / ME ↓ AGPS: SCR ↓ / FR ↓ / ME ↓
Proprietary Models GPT-4o 0.339 / 0.101 / 1.199 0.329 / 0.110 / 1.260 0.379 / 0.141 / 1.285
GPT-5.1 0.153 / 0.041 / 1.036 0.204 / 0.073 / 1.024 0.300 / 0.101 / 1.015
Gemini2.5-flash-lite 0.483 / 0.139 / 1.027 0.563 / 0.217 / 0.951 0.607 / 0.264 / 0.970
Gemini2.5-flash 0.593 / 0.188 / 1.011 0.521 / 0.176 / 0.864 0.521 / 0.184 / 0.854
Gemini3-flash 0.271 / 0.096 / 0.681 0.329 / 0.104 / 0.744 0.314 / 0.136 / 0.817
Claude-haiku-4.5 0.475 / 0.196 / 1.102 0.515 / 0.166 / 1.027 0.493 / 0.199 / 1.001
Claude-sonnet-4.6 0.051 / 0.013 / 0.999 0.144 / 0.051 / 0.938 0.164 / 0.047 / 0.970
Open-Source General Qwen3-vl-8B 0.153 / 0.044 / 0.893 0.192 / 0.074 / 0.981 0.257 / 0.096 / 1.060
Qwen3-vl-32B 0.136 / 0.064 / 0.904 0.150 / 0.075 / 0.838 0.193 / 0.096 / 0.832
Qwen3-vl-235B-22A 0.203 / 0.093 / 0.921 0.228 / 0.132 / 0.892 0.257 / 0.134 / 0.906
Gemma3-12B 0.119 / 0.041 / 1.080 0.162 / 0.059 / 1.085 0.207 / 0.055 / 1.131
Gemma3-27B 0.178 / 0.053 / 1.371 0.293 / 0.098 / 1.216 0.321 / 0.138 / 1.124
Llama4-scout 0.483 / 0.150 / 1.178 0.473 / 0.152 / 1.175 0.557 / 0.216 / 1.109
Llama4-maverick 0.271 / 0.104 / 1.105 0.335 / 0.093 / 1.229 0.357 / 0.112 / 1.245
Dedicated Reward Models RR-4B 0.034 / 0.009 / 0.323 0.072 / 0.020 / 0.375 0.086 / 0.045 / 0.456
RR-8B 0.034 / 0.009 / 0.255 0.102 / 0.025 / 0.336 0.114 / 0.042 / 0.453

Ablation and Downstream Policy Selection

The study analyzed the efficacy of variance reduction training and verified how SCR directly impacts downstream policy decisions in offline Best-of-N (\(N=20\)) selection and pairwise preference ranking under AGPS:

Evaluation Setting Model / Configuration SCR ↓ ME ↓ Selected Score (SS) ↑ Regret ↓ Pair Accuracy ↑
Variance Reduction Qwen3-VL-4B (Baseline) 0.200 (AGPS) 1.020 (AGPS) – – –
Qwen3-VL-4B (+Variance Reduction) 0.057 (AGPS) 0.372 (AGPS) – – –
Offline Best-of-N Random Selection – – 3.158 1.398 0.500
(\(N=20\), AGPS) Llama4-Scout 0.557 1.109 3.434 1.121 0.614
Gemini2.5-Flash-Lite 0.607 0.970 3.222 1.333 –
GPT-5.1 0.300 1.015 3.737 0.818 0.716
Gemini3-Flash 0.314 0.817 3.687 0.869 0.720
Claude-Sonnet-4.6 0.164 0.970 3.980 0.576 0.713
RoboReward-4B (RR-4B) 0.086 0.456 3.889 0.667 0.824
RoboReward-8B (RR-8B) 0.114 0.453 3.960 0.596 0.855
Theoretical Oracle – – 4.556 0.000 1.000

Key Findings

  • General-Purpose VLMs Suffer Severe Category Inversion: Under Action-Goal Perspective Shift (AGPS), models such as Gemini2.5-flash-lite and Llama4-scout yield SCR scores exceeding 55–60%, indicating that more than half of all evaluated trajectories flip between failure and success under equivalent goal rephrasings.
  • Model Scaling Fails to Alleviate Fragility: Within model families (e.g., Qwen3-vl and Gemma3), increasing parameter scale from 4B to 27B or 235B consistently increases SCR rather than decreasing it, demonstrating that broader pretraining amplifies textual bias rather than resolving grounding fragility.
  • Reasoning Enhancements Do Not Systematically Fix Fragility: Enabling chain-of-thought reasoning in Qwen3-vl-235B and Gemini3-flash elevates SCR (increasing vulnerability to linguistic nuance), while yielding divergent effects across GPT-5.1 and Claude-sonnet-4.6.
  • Trajectory Supervision Is Indispensable: Dedicated reward models (RR-4B and RR-8B) trained directly on physical trajectory-reward data maintain SCR below 0.12 across all tiers, outperforming closed-source models orders of magnitude larger.
  • Paraphrase Robustness Directly Drives Downstream Policy Quality: In offline Best-of-N trajectory selection, SCR exhibits a strong positive correlation with downstream policy regret (Pearson \(r = 0.882\), Spearman \(\rho = 0.856\)) and negative correlation with pairwise preference ranking (\(r = -0.893\)), proving that paraphrase stability is a prerequisite for reliable policy optimization.

Highlights & Insights

  • Unveiling a Foundational Evaluation Blind Spot: Exposes the vulnerability of evaluating robotic reward models with single-wording prompts, demonstrating that linguistic style bias can completely invalidate reward feedback.
  • Embodied Semantic Verification: Distinguishes superficial natural language fluency from strict robotic action invariants, ensuring benchmark integrity by filtering out subtle changes in physical affordances, grasping verbs, and threshold metrics.
  • Zero-Latency Invariance Regularization: Proves that adding a simple reward variance penalty over equivalent instructions during training eliminates 71.5% of score crossings on Qwen3-VL-4B without adding any inference-time computational overhead.

Limitations & Future Work

  • Monolingual Focus: All instructions in the benchmark are currently restricted to English, leaving multilingual robustness and cross-lingual code-switching unmeasured.
  • Episode-End Evaluation Scope: The current experimental setup targets terminal episode progress scoring; step-level dense reward shaping and continuous video comparisons remain open for future exploration.
  • Robotic Exploration Integration: Future research could incorporate online paraphrase augmentation into closed-loop robotic RL rollouts to study policy stability under dynamic verbal guidance.
  • vs RoboRewardBench (Lee et al., 2026): RoboRewardBench provides multi-embodiment real-robot trajectories with human progress labels under fixed prompts; RoboRMBench extends this foundational substrate with three-tiered validated paraphrases and failure-reversal metrics to evaluate prompt-invariance.
  • vs RewardBench 2 (Malik et al., 2026) & VLRMBench (Ruan et al., 2025): While existing LLM and VLM reward benchmarks assess preference alignment and hallucination on general text/images, RoboRMBench ties evaluation directly to physical robot execution dynamics, spatial constraints, and downstream RL trajectory regret.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the first rigorous evaluation protocol and metrics for paraphrase fragility in embodied reward models.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 18 models, 2,390 trajectories, 21,673 verified rewrites, and downstream policy selection tasks.
  • Writing Quality: ⭐⭐⭐⭐⭐ Outstanding clarity, precise empirical formulation, and well-grounded analyses.
  • Value: ⭐⭐⭐⭐⭐ Crucial empirical warning and methodological guidance for robotic learning relying on foundation models for reward specification.