Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis¶

Conference: ECCV2026
Paper: ECCV Official Page · Paper PDF
Area: Alignment & RLHF; Multimodal VLM
Keywords: omni-modal reward model, evaluation rubrics, preference synthesis, dual-teacher consensus, GRPO

TL;DR¶

Omni-RRM synthesizes preference data with five-criterion justifications through dual-teacher consensus, then uses SFT→GRPO to learn pairwise response discrimination across images, video, and audio, raising the 7B backbone's five-benchmark average accuracy from 60.2% to 70.4% and improving response selection without updating the generator's parameters.

Background & Motivation¶

A multimodal model's ability to answer questions does not imply that it can determine which of two answers is more trustworthy. An image response may misidentify an attribute, a video response may reverse an event sequence, and an audio response may contradict the actual sound; fluent wording can conceal all of these errors. A reward model must distinguish such differences to provide useful signals for post-training or inference-time reranking. Existing reward data and models, however, emphasize vision, while reducing a complex comparison to a single score obscures whether the problem concerns facts, reasoning, or instruction following.

Free-form chain-of-thought or critiques provide more text, but do not ensure that the same criteria are checked every time or that images, video, and audio share a comparable evaluation interface. Meanwhile, human annotation of individual criteria makes expansion into underrepresented modalities expensive. The paper does not aim to train another larger multimodal generator. It connects reliable preference construction with an auditable judge: candidate answers should differ in quality, but the stronger generator cannot simply be assumed correct; explanations need structure, but that structure should do more than specify an output template.

The authors therefore adopt a fixed five-criterion rubric, requiring heterogeneous teachers to agree on the winner, scores, and reasoning, and carrying the same rubric through supervised fine-tuning and reinforcement learning. Core Idea: replace isolated preference labels with criterion-wise comparative records, and use that structure to constrain both data synthesis and reward-model optimization.

Method¶

Overall Architecture¶

Omni-RRM receives a multimodal context and two candidate text responses, and generates a structured comparative record rather than a scalar alone. The record contains two overall scores on a 0–10 scale, an A/B/E preference label, comparative justifications for five criteria, and a redundant verdict field for robust parsing; E denotes a tie. Here, omni-modal refers to support for text, image, video, and audio conditions, whereas the reported synthesized training data cover images, video, and audio.

Training data pass through capability-gap candidate pairing and dual-teacher five-criterion consensus to form Omni-Preference; progressive rubric training then produces Omni-RRM. At deployment, the data-synthesis teachers are no longer queried. The trained reward model compares new candidates directly, and repeated comparisons can implement Best-of-N selection.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Images, video, audio<br/>and user query"] --> B["Capability-Gap<br/>Candidate Pairing"]
    B --> C["Dual-Teacher<br/>Five-Criterion Consensus"]
    C -->|Training supervision: preferences and rationales| D["Progressive Rubric Training<br/>SFT → GRPO"]
    D --> E["Omni-RRM"]
    F["Inference: new context<br/>and two candidate responses"] --> E
    E --> G["Structured comparative record<br/>or Best-of-N winner"]

Key Designs¶

1. Capability-Gap Candidate Pairing: increase distinguishable response differences through generator capability gaps

Repeated sampling from one model can produce two equally correct or equally vague responses, leading teachers to call ties and leaving fewer clear preferences under a fixed annotation budget. Instead, the authors pair strong and weak generators on the same context: Qwen2.5-VL-7B is paired with Qwen2.5-VL-3B or LLaVA-1.5-7B for images, Qwen2.5-VL-7B with 3B for video, and combinations involving R1-AQA-7B, Qwen2.5-Omni-7B, and weaker audio models for audio. This stage broadens quality differences without treating model size as ground truth. The stronger model can still be wrong, so final labels are assigned in the next stage.

The sources are RLAIF-V for images; ActivityNet, Charades, Ego4D, NextQA, and YouCook2 for video; and Clotho-AQA for audio. Paper Table 1 reports 41.0k final pairs: 17.0k image, 12.2k video, and 11.8k audio pairs. Automatically generating preference annotations should not be confused with eliminating human involvement from all underlying resources. The pipeline reuses existing datasets, so its automation primarily concerns candidate generation, preference filtering, and structured rationale supervision.

2. Dual-Teacher Five-Criterion Consensus: constrain verdicts through scores and specific evidence

GPT-4o-mini and Gemini-2.0-Flash independently provide scores, a preference label, and five-criterion justifications for each pair. A pair is retained only when both teachers produce the same non-tie verdict and their respective score rankings and rationales agree with it. This handles quality reversals more carefully than always treating the stronger model's output as chosen, and offers more inspectable information than agreement on a letter alone. Consensus remains a filtering mechanism rather than unbiased ground truth, however: blind spots shared by both teachers can survive into the dataset.

The five criteria are fluency and coherence, relevance, accuracy and completeness, reasoning quality, and safety and ethical alignment. The interface is fixed, but its evidence changes with the modality: images require checking object attributes, spatial relations, and OCR details; video additionally requires temporal ordering and event consistency; audio requires acoustic events, spoken content, and agreement with the response. The method therefore maps different evidence into a common set of evaluation questions rather than imposing identical observations on every modality. Each criterion receives comparative reasoning, while the two numerical scores are overall response scores, not five separate pairs of criterion-level scores.

The authors define difficulty using the difference between the two reconciled overall scores:

\[ \Delta = |s_A - s_B|, \qquad \Delta < 2 \Rightarrow \mathrm{Hard}. \]

A small margin indicates similar response quality, making discrimination more sensitive to label noise. The paper emphasizes improving this boundary, but this does not establish that training uses only Hard pairs; the main text does not specify a separate hard-example resampling ratio sufficiently clearly to confirm one.

3. Progressive Rubric Training: learn valid judgments before refining subtle discrimination

The first stage performs SFT on structured teacher records, teaching the model to produce comparative rationales and verdicts under a shared interface. The second stage uses GRPO to sample multiple complete judgments for the same input, compare their rewards, and update the reward model itself. Reinforcement learning here optimizes how the judge compares two answers, not the downstream response generator. SFT provides a stable output format so that RL exploration need not spend most of its effort repairing JSON or missing fields.

GRPO uses three reward components: a format term checks strict JSON/schema validity and required fields; a preference term checks the correct winner and consistency between scores and verdict; a rubric term encourages substantive criterion-wise explanations through justification-consistency heuristics and consistency checks. For example, an A verdict requires A's score to be higher, whereas a tie requires equal scores. Explanations consequently participate in training rather than serving only as display text. These heuristic rewards alone do not prove that a rationale faithfully reflects the model's internal decision, which is distinct from making a decision easier to inspect.

A Worked Example¶

The following is a teaching example constructed from the mechanism, not an experimental sample from the paper. Suppose a video shows someone opening a box and then taking out a cup. Candidate A describes this correctly, while candidate B reverses the actions. Both answers are fluent and relevant; the important differences should concern accuracy and reasoning quality rather than automatically favoring the longer answer.

Regardless of whether A came from the strong or weak generator, both candidates are sent to the teachers. Suppose both teachers prefer A and provide consistent score rankings and rationales, with reconciled overall scores of A=8 and B=7. The pair can be retained, and its margin of 1 makes it Hard. If either teacher returns a tie or the teachers disagree on the winner, it is excluded. The precise reconciliation of teacher scores requires the supplementary material; this example should not be interpreted as the authors' aggregation algorithm.

SFT learns the complete judgment, and GRPO compares multiple generated judgments for the same input against format, preference, and rationale requirements. At deployment, the model directly receives a new video and candidates without consulting the teachers again. With 10 candidates, the paper uses a fixed-bracket single-elimination tournament requiring 9 pairwise comparisons to select a response, rather than treating pairwise overall scores as absolute quality values comparable across contexts.

Loss & Training¶

SFT applies standard autoregressive negative log-likelihood to the structured target sequence and uses LoRA to reduce fine-tuning cost. GRPO normalizes rewards within each group to compute relative advantages, applies a clipped policy-gradient objective, and constrains updates with a KL penalty against the SFT reference policy. The backbones are Qwen2.5-Omni-3B/7B, with both SFT-only and SFT+RL results reported.

PDF text extraction has corrupted Equations (2) and (3) in the supplied full text. This note therefore explains the objectives from the intact prose instead of reconstructing purportedly exact author equations. The main text places full GRPO updates, reward rules, and weights in Supplementary A.1, and synthesis, training, and evaluation budgets in A.2. Those supplements are absent from the supplied cache, so it cannot establish the learning rate, LoRA rank, group size, GPU time, or specific reward weights.

Key Experimental Results¶

Main Results¶

The table reproduces selected preference accuracies from paper Table 2 (%, higher is better), measuring agreement with benchmark preference labels rather than response-generation accuracy. VL-Reward uses the Reasoning split, and ShareGPT-Video uses test pairs filtered from DPO data for large human-score margins. Audio-HH converts HH-RLHF prompts into speech while retaining its responses and human preferences. TA2T is an external audio-conditioned benchmark, and its main-table results include ties, so they should not be interpreted as purely binary classification.

Model	VL-Reward	MM-RewardBench	ShareGPT-Video	Audio-HH	TA2T (with ties)	Five-Benchmark Overall
Qwen2.5-Omni-7B	57.8	57.5	66.3	62.4	56.9	60.2
Omni-RewardModel-BT	60.4	58.4	63.7	61.3	60.5	60.9
Gemini-2.0-Flash	73.4	62.8	74.6	60.1	59.9	66.2
Gemini-2.5-Pro	79.6	63.3	78.8	66.5	64.9	70.6
Omni-RRM-7B, SFT	60.4	61.0	70.5	62.8	58.5	62.6
Omni-RRM-7B, SFT+RL	67.1	72.9	80.2	66.8	65.0	70.4

Overall is the unweighted mean of five columns. The full 7B model improves over its backbone by 10.2 percentage points, with a reported relative gain of 17.0%; it improves over SFT-only by 7.8 percentage points. Its average trails Gemini-2.5-Pro by 0.2 percentage points, but VL-Reward remains 12.5 percentage points lower, so this is not evidence of superiority on every modality or subtask.

Ablation Study¶

The following results come from paper Table 3, fixing the size at 7B and comparing rationale supervision and rubric rewards under the same SFT+RL budget. TA2T is excluded: Overall averages four benchmarks and must not be directly subtracted from the 70.4 in the main table.

Config	VL-Reward	MM-RewardBench	ShareGPT-Video	Audio-HH	Four-Benchmark Overall
Without five-criterion rationale supervision	64.0	60.6	68.7	64.3	64.4
Without the rubric reward term in GRPO	65.2	66.6	75.4	65.0	68.1
Full model	67.1	72.9	80.2	66.8	71.8

The full model exceeds the no-rationale variant by 7.4 percentage points and the no-rubric-RL variant by 3.7 percentage points. The former supports retaining structured rationales during training; the latter shows that learning to output rationales alone is insufficient and that related RL constraints also contribute. Neither ablation establishes that every generated explanation is faithful.

Key Findings¶

GRPO provides a substantial video gain: the 7B model rises from 70.5% after SFT to 80.2%, a 9.7-percentage-point increase. This is more substantive than simply demonstrating output-format compliance.
Paper §4.4 fixes Qwen2.5-Omni-7B as the generator. At N=10, Omni-RRM Best-of-N improves over majority voting from 54.5% to 56.4% on MMMU, 52.4% to 54.1% on Video-MME, and 77.7% to 80.3% on AVQA in its audio-only setting. These are downstream question-answering accuracies, and the extra 9 reward-model comparisons mean the gains are not compute-matched.
Paper §4.5 varies the modality mixture while holding the 3B backbone, SFT+GRPO hyperparameters, and total update/pair budget constant. Joint training reaches 65.1% on Audio-HH versus 63.8% for audio-only training. This supports cross-modal benefits in the tested setting, not the absence of negative transfer under arbitrary data mixtures.

Highlights & Insights¶

Candidate diversity and preference truth are separated: capability gaps produce different answers, while teacher consensus determines the winner. This avoids directly encoding a stronger source as the correct answer.
The same rubric spans annotation, SFT, and RL, making criterion-wise explanations more than an inference-time prompt. The four-benchmark ablation separately tests rationale supervision and rubric rewards rather than reporting only aggregate gains.
A shared rubric need not erase modality differences. Instantiating accuracy through visual grounding, temporal consistency, or acoustic evidence is a particularly useful design to preserve when adapting this approach.

Limitations & Future Work¶

The authors explicitly describe Audio-HH as a controlled TTS-based proxy that does not cover real noise or speaker variability. Adding TA2T reduces dependence on one test source but does not replace realistic audio-robustness evaluation.
The authors acknowledge additional reward-model decoding latency from structured rationales; detailed latency and budgets are in the unavailable supplements. The cache is insufficient to establish gains per unit cost or reproduce the exact reward heuristics.
Note author's assessment: strict consensus and tie exclusion discard examples with the strongest teacher disagreement, potentially biasing training toward preferences on which teachers readily agree. Further tests should examine disagreement retention and calibration between tie-free training preferences and tie-inclusive TA2T evaluation.
Note author's assessment: readable rationales are not necessarily causally faithful, and a fixed five-criterion rubric may omit task-specific requirements. Response-order swaps, irrelevant wording perturbations, and evidence masking could test whether the model relies on the evidence its explanations claim to use.

vs Omni-Reward: both target omni-modal reward modeling. This paper emphasizes the combination of fixed five-criterion records, teacher-consensus synthesis, and rubric-aware GRPO rather than treating audio support alone as its complete novelty.
vs LLaVA-Critic / UnifiedReward-Think: critiques or chain-of-thought can explain judgments; Omni-RRM additionally fixes the evaluation criteria and incorporates rationale constraints into training. The trade-offs are longer outputs and dependence on rule design.
vs text-based rubric methods: this work extends rubric supervision to visual, temporal, and acoustic evidence. The transferable principle is a common evaluation interface with modality-specific evidence, not simply reusing a text-evaluation prompt.
Resources: the paper explicitly supplies the author project page as an entry point for data, code, and weights; this note has not checked current download availability online.

Rating¶

Novelty: 4/5. The contribution integrates rubric-grounded data, omni-modal judgments, and training objectives rather than inventing SFT or GRPO separately.
Experimental Thoroughness: 4/5. Multimodal preferences, budget-matched ablations, and downstream selection are covered, but the available main text lacks complete cost and reward-implementation details.
Writing Quality: 4/5. The pipeline is clear, with care needed around different averaging scopes and the reliance on supplementary details.
Value: 4/5. A useful approach to auditable cross-modal reward models, with deployment still requiring assessment of latency, teacher bias, and rationale faithfulness.