Same Pool, Different Answer: Stable Best-of-N Selection for Vision-Language Models

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/PengfeiZheng-721/DSPA
Area: Multimodal VLM
Keywords: Vision-Language Models, Best-of-N Sampling, Test-time Selection, Reproducibility, Hallucination Mitigation

TL;DR

Addressing the issue where small scoring perturbations frequently flip the winning answer in likelihood-based Best-of-N selection on a fixed candidate pool, this paper proposes DSPA, which pairs a frozen full-context reference anchored selector with training-time shortcut filters, cutting the perturbation flip-rate from 44.3% to 24.9% and POPE object hallucination from 9.6% to 6.2%.

Background & Motivation

In multimodal large model inference, Best-of-N (BoN) selection is widely used across rejection sampling alignment, code generation, and complex reasoning pipelines: a system samples \(N\) candidate responses for a given input query, evaluates each candidate with a scoring function, and returns the highest-scoring response as the final output. Intuitively, when the candidate pool and compute budget are held strictly fixed, the selected winning answer should be reproducible and stable. In practice, however, this baseline expectation fails surprisingly often. On the very same candidate pool and input, minor engineering variations in scoring—such as a \(\pm 0.05\) softmax temperature calibration shift, slight adjustments to score-time top-\(p\) probability truncation, or accidentally leaving dropout noise enabled during evaluation—cause the standard likelihood-based selector to flip its winning answer on 41%–47% of test items.

This instability is deeply rooted in how likelihood-based scoring is structured. First, when scoring functions sum token-level log-probabilities across full sequences, response length exerts a mechanical accumulation effect: an accurate, concise answer followed by 20 uninformative, rambling tokens can see its summed score surpass the clean version, allowing length rather than quality to dictate the ranking. Second, fine-tuning or post-training causes the policy's log-probability scale to shift, compressing score gaps among top candidates so that minor perturbations easily invert their order. While previous literature extensively explored "reward hacking" when candidate pools scale up, it largely overlooked the more fundamental failure: even on small (e.g., \(N=8\)), completely frozen pools, likelihood scoring alone cannot agree with itself.

To resolve this conflict, this paper decouples selection stability from candidate pool quality instead of altering generation sampling. It separates test-time scoring invariance from training-time preference hygiene. Core idea: score candidates at test time using a token-averaged log-ratio against a frozen, full-context reference model to eliminate scale drift, while enforcing length, prompt-copying, and hallucination filters during preference training to purge shortcuts from the pool.

Method

Overall Architecture

The proposed DSPA (Dual Stability & Preference Alignment) framework tackles stability and factuality across two distinct phases: test-time selection and training-time data curation. At test time, given a multimodal input \(x\) consisting of an image and prompt, a candidate pool \(\mathcal{Y}(x)\) of fixed size \(N\) is generated by the policy model. Next, the anchored selector evaluates each candidate in a single forward pass under both the active policy model and a pre-committed, frozen reference baseline, computing per-token log-probability ratios and averaging them across the sequence to pick the candidate with the highest normalized score. At training time, the framework applies strict length-ratio constraints, verbatim prompt-copying exclusion, and object-existence hallucination filters to clean the preference dataset before policy fine-tuning.

To evaluate and replicate scoring vulnerability without introducing confounding computational costs, the authors introduce the SRP (Same-pool/Same-budget Replay Protocol) audit benchmark. By replaying only the scoring pass on identical frozen pools under controlled temperature shifts, probability truncations, and dropout noise, SRP measures the flip-rate and winner agreement rate. The end-to-end workflow is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input x<br/>(Image + Text Prompt)"] --> B["Fixed Pool Generation<br/>Sample N=8 candidates Y(x)"]
    B --> C["Anchored Log-Ratio Scoring<br/>Policy vs frozen full-context ref"]
    C --> D["Per-Token Mean Normalization<br/>Remove sequence length accumulation"]
    D --> E["Single-Pass Top-1 Output<br/>Return argmax normalized score"]
    F["Preference Pair Construction"] --> G["Multidimensional Shortcut Filtering<br/>Constrain length/drop copy/fix halluc."]
    G --> H["LoRA-DPO Fine-Tuning<br/>Produce aligned policy model"]
    H -.-> C
    B -.-> I["SRP Stability Audit Replay<br/>Evaluate under controlled noise"]
    D -.-> I

Key Designs

1. Anchored log-ratio scoring: establishing an invariant full-context reference to eliminate scale drift

Standard BoN selection relies directly on the policy's raw log-likelihood \(\log \pi_\theta(y \mid x)\). After RL or DPO fine-tuning, the policy's probability landscape and output scale fluctuate, narrowing the decision boundary between top candidates. The proposed anchored selector introduces a reference model \(\pi_\text{ref}\) that is frozen and hash-identified prior to evaluation (typically the matched pre-fine-tuning base checkpoint). Crucially, \(\pi_\text{ref}\) must receive the complete multimodal context \(x\) (image, prompt, and prior tokens) rather than serving as an unconditional text prior like pointwise mutual information (PMI). At each position \(t\), the log-ratio is evaluated: $\(\Delta \ell_t(y \mid x) = \log \frac{\pi_\theta(y_t \mid y_{<t}, x)}{\pi_\text{ref}(y_t \mid y_{<t}, x)}\)$ This ratio quantifies how much the fine-tuned policy prefers token \(y_t\) over the base reference. Because \(\pi_\text{ref}\) is frozen, it acts as a constant, non-drifting baseline scale for all candidates, expanding the margin between superior and inferior candidates and shielding the ranking from logit scale shifts.

2. Per-token mean normalization: smoothing length accumulation to prevent verbose suffixes from hijacking ranks

Summing token-level scores across the sequence grants uninformative verbosity an additive advantage. If a concise 20-token answer has a mean log-ratio of 0.15 (sum = 3.0), appending 20 redundant tokens with a weak positive ratio of 0.02 pushes the sum to 3.4, allowing the padded candidate to win. To enforce length invariance, DSPA computes the token-level average: $\(r_\text{norm}(y \mid x) = \frac{1}{|y|} \sum_{t=1}^{|y|} \log \frac{\pi_\theta(y_t \mid y_{<t}, x)}{\pi_\text{ref}(y_t \mid y_{<t}, x)}\)$ Under this formulation, appending 20 low-value tokens dilutes the mean score from 0.15 to 0.085, preserving the victory of the concise answer. Selection requires only a single \(O(N)\) pass: \(\operatorname{argmax}_{y \in \mathcal{Y}(x)} r_\text{norm}(y \mid x)\), avoiding pairwise quadratic comparisons like Minimum Bayes Risk (MBR).

3. Multidimensional shortcut filtering: preventing preference learning from absorbing spurious pool artifacts

Even a perfectly stable selector cannot fix an inherently corrupted candidate pool. During preference fine-tuning, vision-language models exploit three common shortcuts that degrade pool factuality. First, prompt copying: models often copy large chunks of the input instruction in captioning tasks; candidates with a contiguous verbatim overlap exceeding \(K=12\) tokens are discarded. Second, answer length: human and model judges exhibit length bias; preference pairs are strictly restricted to length ratios within \([0.8, 1.2]\). Third, object-existence hallucination: in binary existence questions ("Is there a [object]?"), pairs where the preferred candidate falsely asserts existence against ground-truth labels are purged. After filtering, candidates are segmented into sub-sentences, scored by LLaVA-NeXT-34B, and assembled into preference pairs for LoRA-DPO training.

4. SRP audit protocol: benchmarking replay stability under fixed pools and budgets

To assess scoring vulnerability in isolation from generation overhead, the authors establish the Same-pool/Same-budget Replay Protocol (SRP). SRP controls four variables: (1) identical frozen candidate pools \(\mathcal{Y}(x)\); (2) identical compute budgets (single scoring pass per candidate, \(O(N)\)); (3) pre-specified perturbation families: temperature shifts \(\tau_0=0.7 \pm 0.05\), score-time top-\(p\) truncation \(p_0=0.9 \pm 0.05\), and dropout noise with fixed seeds; and (4) pre-committed reference checkpoint hashes. SRP measures flip-rate (percentage of items where at least one replay alters the winner) and agreement with the original winner, isolating the variance inherent to the scoring rule.

A Worked Example

Consider the query: "Is there a dog on the couch?" The policy generates two candidates: - Candidate A (concise, 20 tokens): "No, there is only a cat sitting on the couch." (Policy-to-reference mean log-ratio is 0.15; sum is 3.0). - Candidate B (verbose, 40 tokens): "No, there is only a cat sitting on the couch, and the room has bright lighting with a wooden coffee table in front of it." (Tokens 1–20 have mean ratio 0.15, while tokens 21–40 have mean ratio 0.02; sum rises to 3.4, but mean drops to 0.085).

Under traditional sequence summation, Candidate B scores 3.4, beating Candidate A's 3.0. When scoring temperature shifts by \(\pm 0.05\), Candidate B's tail logits re-normalize unpredictably, causing its total sum to oscillate between 2.9 and 3.5, flipping the winner back and forth. Under DSPA's token-averaged anchored scoring, Candidate A scores 0.15 while Candidate B scores 0.085. Candidate A holds a wide margin of 0.065 that cannot be overturned by small perturbations, guaranteeing output consistency.

Loss & Training

During the training stage, the policy is fine-tuned using LoRA under the Direct Preference Optimization objective (LoRA-DPO). Given filtered preference pairs \((x, y_w, y_l)\), the optimization minimizes the standard DPO loss: $\(\mathcal{L}_\text{DPO}(\theta; \pi_\text{ref}) = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)}\right)\right]\)$ During inference, both the policy \(\pi_\theta\) and reference \(\pi_\text{ref}\) evaluate candidate tokens in parallel via teacher-forcing. While scoring-stage forward time increases by approximately 1.8×, autoregressive decoding dominates total runtime, meaning the end-to-end wall-clock overhead is only about +5%–10%.

Key Experimental Results

Main Results

Experiments were conducted on LLaVA-1.5 7B across Greedy decoding (\(N=1\)), standard length-normalized BoN, an \(\epsilon\)-margin tie-break baseline (selecting the shorter response when the top-two score gap is \(<0.02\)), and the separately aligned RLAIF-V 7B model. All multi-sample methods evaluated identical frozen candidate pools with \(N=8\).

Table 1: Task utility on standard benchmarks and hallucination rate (LLaVA-1.5 7B, N=8)

Selector	VQAv2 ↑	OK-VQA ↑	TextVQA ↑	MME ↑	MMMU ↑	POPE Halluc. Rate ↓
Greedy (N=1)	78.5	59.2	58.1	1510.5	35.1	10.5%
BoN + length-norm	79.4	60.8	59.6	1532.8	36.4	9.6%
BoN + \(\epsilon\)-margin tie-break	79.1	60.5	59.3	1528.4	36.2	9.8%
BoN + Anchored Selector (Ours)	79.7	61.2	60.1	1541.0	36.8	6.2%

On identical candidates, switching only the selection rule reduces POPE object hallucination from 9.6% to 6.2% (a 35% relative reduction) without compromising general VQA capabilities.

Table 2: Stability under SRP scoring perturbations (n=500, average across perturbation families)

Model & Selection Strategy	Flip-rate ↓	Agreement with Orig. Winner ↑
BoN + length-norm (LLaVA-1.5 7B)	44.3%	57.2%
BoN + \(\epsilon\)-margin tie-break	37.1%	63.2%
RLAIF-V 7B (length-norm)	35.1%	65.6%
Anchored Selector (Ours DSPA)	24.9%	77.5%

The anchored selector lowers the average flip-rate from 44.3% to 24.9% (a 19.4 percentage point improvement) and increases agreement with the original winner by over 20 percentage points.

Ablation Study

Table 3: Training-time filter ablation (using anchored selector, n=200)

Setting	Object Halluc. Rate (Judge-est.) ↓	Overall Halluc. Rate (Judge-est.) ↓	AMBER Acc. ↑	Pairwise Win Rate ↑	Human Pref. (Wins/200) ↑
DSPA Full (Filters + Anchored Selector)	6.2%	29.7%	81.7	48.2%	192
RLAIF-V 7B (Anchored Selector)	9.6%	43.9%	80.1	46.8%	–
All Three Primary Filters Off	23.6%	36.6%	77.5	45.8%	182

Disabling the three primary training filters causes judge-estimated object hallucination to quadruple from 6.2% to 23.6%, demonstrating that training filters are critical for pool factuality.

Table 4: Selector constraint ablation (fixed training filters, n=200, Dropout family)

Selector Variant	Flip-rate ↓	Agreement with Orig. Winner ↑
Seq-level log-ratio (no per-token avg)	40.8%	58.6%
Per-token, self-anchored (PMI; context-free denominator)	35.4%	64.9%
Per-token, co-moving reference	33.6%	66.2%
Anchored Selector (Fixed ref + per-token avg, ours)	24.1%	78.4%

Key Findings

• Decoupling stability levers: Table 4 demonstrates that discarding per-token averaging returns flip-rate to 40.8%. Using an image-blind denominator (PMI) or a co-moving reference keeps the flip-rate above 33%. Both the frozen reference and per-token averaging are strictly necessary to reach 24.1%.
• Benign phrasing flips dominate residual instability: Manual annotation of 500 residual flip cases from the anchored selector (Cohen's \(\kappa=0.82\)) reveals that 71% of flips are semantically equivalent near-ties (e.g., "The cat is on the couch" vs. "There is a cat sitting on the couch"). Only 29% involve factual divergence, yielding an estimated factual change probability of approximately 7.2%.
• Model scale and cross-architecture generalization: On LLaVA-13B, the SRP flip-rate drops from 38.5% to 22.3%. When transferred to Qwen2.5-VL-7B-Instruct without architecture-specific re-tuning, judge-estimated object hallucination plunges from 38.4% to 8.7%.

Highlights & Insights

• Spotlighting reproducibility in test-time scaling: The work exposes a critical operational blind spot—that likelihood-based Best-of-N selection fails to agree with itself on identical candidate pools under negligible hyperparameter shifts.
• Orthogonal separation of stability and factuality: By proving that test-time anchored selection drives ranking stability while training-time filtering governs pool quality, the paper establishes a clear mental model for VLM test-time compute.
• Practical plug-and-play efficiency: Requiring only teacher-forced parallel scoring with +5%–10% end-to-end latency overhead, DSPA provides an immediate 35% hallucination reduction without separate reward model training.

Limitations & Future Work

• Scope of perturbation families: SRP audits stability across temperature shifts, top-\(p\) truncation, and dropout, but does not cover numeric precision changes (e.g., FP16 vs. BF16 or INT8 quantization) or hardware operator non-determinism.
• Cross-family reference calibration: When using a reference checkpoint from a different model family, score margins compress, leading to wider confidence intervals for near-tie boundaries.
• Future directions: Integrating anchored log-ratio scoring into multi-step Chain-of-Thought reasoning steps, and designing semantic equivalence clustering at the selection boundary to eliminate the remaining 71% of phrasing flips.

Related Work & Insights

• vs. Reward Hacking & Regularized BoN (Gao et al., Jinnai et al.): Prior work examined policy exploitation of proxy reward models as \(N\) grows large; this paper investigates the instability of the scoring function itself when \(N\) and the pool are completely fixed.
• vs. Contrastive Decoding & MMI (Li et al., Chuang et al.): While contrastive methods use model probability ratios to steer token decoding, DSPA repurposes the log-ratio as an offline sequence selector and proves that maintaining full multimodal context in the reference denominator is vital for selection stability.
• vs. Rule-based Verifiers & LLM Judges (RLVR, Lightman et al.): Deterministic rule verifiers or heavy LLM judges either suffer from narrow domain applicability or prohibitive inference overhead; DSPA achieves robust selection purely within the likelihood framework at minimal added cost.

Rating

• Novelty: ⭐⭐⭐⭐☆ Pinpoints an overlooked reproducibility vulnerability in BoN selection and formulates an elegant anchored ratio solution with the SRP protocol.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 7B/13B models, cross-family transfer to Qwen2.5-VL, meticulous ablations, and double-annotator semantic flip audits.
• Writing Quality: ⭐⭐⭐⭐⭐ Crisp narrative structure, precise terminology, transparent boundary definitions, and strong methodological clarity.
• Value: ⭐⭐⭐⭐⭐ Highly valuable for practitioners deploying rejection sampling, Best-of-N selection, and inference-time scaling systems in production.