TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs¶

Conference: ECCV2026
Paper: ECCV Paper
Project: TARS
Area: Hallucination Mitigation / Multimodal VLM
Keywords: direct preference optimization, visually agnostic tokens, adaptive perturbation, spectral alignment, data efficiency

TL;DR¶

TARS perturbs prompt tokens with weak image relevance during training and combines DPO with spectral preference alignment, reducing LLaVA-v1.5-7B's AMBER hallucination rate from standard DPO's 26.4% to 13.2% using only 4.8k preference samples while improving descriptive coverage.

Background & Motivation¶

Multimodal large language models can produce fluent descriptions while mistaking common linguistic associations for visual facts. Mentioning a bookshelf in an office may be a reasonable association, but asserting that a cat rests on the sofa can exceed the visual evidence. Preference optimization uses preferred and rejected answers to the same image and question to increase the probability of factual answers and suppress hallucinated ones. Direct preference optimization (DPO) does not require a separately trained explicit reward model, making it suitable for post-training with a small preference dataset. However, labeling an answer as better does not ensure that the model learns its visual basis; common wording, object co-occurrence, and style can also distinguish the two answers.

This problem reflects a mismatch between static supervision and changing inputs: training responses remain fixed, whereas real questions vary in wording and image-text associations. If training always preserves the same prompt cues, the model can reduce preference loss without learning to rely on the image when those cues change. Adding paraphrases expands the input distribution but does not necessarily expose the fragile positions on which the current model relies. Arbitrary word deletion or replacement is also risky because changing an object, negation, or relation can invalidate the original preference label. The proposal is therefore not that more noise is always better, but that weakly visually related tokens can be modified while retaining the semantic target of preference learning.

TARS first uses CLIP to measure each text token's relevance to the image, then adaptively masks or substitutes low-relevance positions. The trainable model must still prefer the correct answer under these changes, while the reference model retains the unperturbed input to provide a more stable representation reference. Spectral regularization additionally constrains hidden representations, aiming to preserve global semantics instead of forcing position-by-position recovery of pre-perturbation features. Core Idea: make preference learning continue to rely on visual evidence under constrained textual perturbations, and stabilize that alignment through spectral preferences instead of memorizing the linguistic patterns of fixed training pairs.

Method¶

Overall Architecture¶

A training instance contains an image, a text question, a preferred answer, and a rejected answer; the response pair and preference label come from existing data. Rather than primarily generating additional complete response pairs, TARS modifies some question tokens when processing an instance. The pipeline comprises visually agnostic token selection, constrained token perturbation, and spectral preference alignment combined with output-probability supervision. The perturbed question and original image enter the trainable policy, while the unperturbed question enters the frozen reference model. The policy computes conditional probabilities and hidden states for both candidate responses; the reference supplies original-input representations for the spectral branch. Training produces updated model parameters; ordinary inference only takes an image and question and generates an answer autoregressively, without running this training branch online.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Image, question,<br/>preferred and rejected answers"] --> SELECT["Visually Agnostic<br/>Token Selection"]
    SELECT --> PERTURB["Constrained<br/>Token Perturbation"]
    PERTURB --> POLICY["Policy model<br/>probabilities and hidden states"]
    INPUT -->|Unperturbed input| REF["Frozen reference model<br/>original hidden states"]
    POLICY --> ALIGN["Spectral Preference<br/>Alignment"]
    REF --> ALIGN
    POLICY -->|DPO loss| UPDATE["Joint loss<br/>update policy model"]
    ALIGN --> UPDATE

Key Designs¶

1. Visually Agnostic Token Selection: constrain which positions may change

The image passes through CLIP's visual encoder and question tokens through its text encoder, after which cross-modal cosine similarities are computed. A low similarity means that the token lacks direct visual association under this proxy, not that it is linguistically meaningless. Selecting these positions aims to avoid changing the main visual content supporting the answer while weakening superficial language cues available to the model. For example, common connective expressions may not carry image-object information, but this does not imply that all prepositions or abstract words are safe to delete. The selection is a correlation-based heuristic, not a direct measurement of each word's causal role.

The number selected is not a fixed fraction either: the authors define \(P\) as negated similarity scores and use the gap between its largest and second-largest values as \(\Delta P\). The count relation in Equation (9) is given below; the accompanying prose then selects the \(N_t\) positions with the lowest visual relevance:

\[ N_t=\left\lfloor\frac{\omega}{\Delta P}\right\rfloor. \]

A larger gap produces fewer perturbations, whereas a smaller gap broadens the variation. Here, confidence is an interpretation of the ranking gap, not a calibrated probability of correctness or the average uncertainty over all tokens. The mechanism controls both perturbation location and breadth, unlike random selection or removing the same number of words from every question. The extracted Top notation is damaged, so the interpretation follows the explicit lowest-similarity description rather than inventing a sorting operator. The available main text does not specify handling for a zero gap, counts exceeding question length, or a minimum of one selected position; reproduction requires the supplement or code to resolve these cases.

2. Constrained Token Perturbation: approximate inner maximization through local transformations

After selection, Mask replaces the relevant tokens with [MASK], while Replace uses synonym substitution; all other positions remain unchanged. The image and response pair are fixed, so training requires the model to preserve the original answer ranking when the prompt changes. These are training alternatives, not inference-time reranking of multiple answers or pruning of visual tokens. The paper frames them as a MinMax problem: the inner process creates difficult inputs, and the outer process optimizes preference consistency on those inputs. This perspective emphasizes resistance to input variation rather than merely increasing the number of training examples.

The objective and its practical solution must nevertheless be distinguished: Equation (7) describes a quantity named Sim as token deviation and approximates maximization through local transformations. The available main text does not provide a complete search procedure that evaluates preference loss for each candidate and selects the actual worst perturbation. Worst-case language should therefore be understood as the authors' optimization objective and approximation rationale, not as evidence that masking or synonym substitution exactly solves the inner adversarial problem. Likewise, low CLIP relevance constrains semantic disruption but does not mathematically guarantee preservation of negation, quantities, or relations. The method is best understood as targeted perturbation training, not an adversarial algorithm with a verified semantic-preservation certificate.

3. Spectral Preference Alignment: impose relative preferences on hidden states too

Token perturbations alone change policy hidden states while response supervision remains fixed, potentially introducing further instability. The authors avoid requiring position-wise equality between perturbed and unperturbed hidden vectors, which could force recovery of the local linguistic patterns the model previously relied on. They pair policy hidden states for a perturbed question and candidate answer with reference hidden states for the original question and the same candidate answer. Both preferred and rejected answers participate, so this is not simply a one-way feature reconstruction loss on the correct response. The main text applies the Fast Fourier Transform along the token axis and describes a spectral summary involving the real part and a norm. A DPO-like term then compares policy/reference spectral ratios for the preferred and rejected answers, with a temperature-scaled preference objective encouraging favorable relative changes.

The authors argue that low-frequency components better reflect global semantics, whereas high-frequency components contain more position-dependent lexical detail. A local token change affects multiple frequency components, motivating a spectral view that tolerates some positional variation while preserving global structure. This explains the choice of spectral regularization over direct position-wise alignment, but switching to FFT does not universally guarantee greater robustness. Frequency selection, normalization, and aggregation affect that property; the Fourier transform itself should not be confused with active low-pass filtering. The main text does not clearly describe an explicit low-frequency truncation module, so none is added here. Equations (10) and (11) contain extraction damage: scalar aggregation across frequencies and numerical stabilization of log ratios cannot be fully recovered, so the exact spectral loss is not guessed.

A Worked Example¶

Consider the office question in Figure 2, which asks whether a cat or another animal is present; the illustrated DPO response invents a cat. During training, TARS would first score question tokens against the image and use the margin to determine selection breadth, rather than predefining that the word cat must change. Mask hides only selected low-relevance positions, retaining the original image and response pair while the model learns their probability ordering. Meanwhile, policy hidden states under the perturbed question and reference states under the original question enter the spectral preference branch. One training pair thus supplies both a signal about which answer should be more probable and a signal about stabilizing relative preferences in hidden representations. Inference still uses the original question; Figure 2 shows TARS correctly denying the presence of animals, but this is a qualitative case rather than a published token-by-token selection trace.

Loss & Training¶

Combining Equations (6) and (12), the central training intent can be compactly expressed as follows without redefining the damaged spectral subterm:

\[ \min_{\theta}\max_{\varphi\in\Phi(\mathcal A)}\mathbb E_{(x,q,y_w,y_r)\sim\mathcal D}\left[\mathcal L_{\mathrm{DPO}}(x,\varphi(q),y_w,y_r)+\lambda\mathcal L_{\mathrm{freq}}(x,q,\varphi(q),y_w,y_r)\right]. \]

Here \(\Phi(\mathcal A)\) permits changes only at selected positions, while \(y_w\) and \(y_r\) denote preferred and rejected responses. The DPO branch optimizes response log-likelihood ratios relative to the reference policy under perturbed conditions; the spectral branch additionally uses unperturbed reference hidden states. The practical inner step uses the local-transform approximation described above, and the outer step updates the policy while the reference remains frozen. The main experiments randomly sample 4.8k instances from RLHF-V-Dataset and use LLaVA-v1.5-7B and LLaVA-v1.5-13B backbones. The authors report 8 NVIDIA A100 80GB GPUs, the CHiP-DPO training strategy, and settings \(\alpha=1\), \(\beta=1\), \(\omega=0.1\), and \(\lambda=0.1\). Evaluation uses greedy decoding with temperature 0; the training-dynamics figure separately analyzes 20 epochs, which should not be assumed to specify the training duration of every main experiment. No expert feedback means that this method does not additionally rely on expert-model feedback generation, not that the public preference data have no human annotation provenance.

Key Experimental Results¶

Main Results¶

The following subset of Table 1 (page 9) retains representative AMBER and MMHal metrics, using the original numerical scales. AMBER Cover measures descriptive coverage, Hal-Rate measures hallucination incidence, and Cog is a cognition-related hallucination metric rather than general reasoning accuracy; MMHal Score is judged by GPT-4V.

Model / Method	AMBER Cover ↑	AMBER Hal-Rate ↓	AMBER Cog ↓	MMHal Score ↑
LLaVA-v1.5-7B	51.7	35.4	4.2	2.02
7B + DPO	56.6	26.4	2.5	2.19
7B + CHiP-DPO	57.3	19.9	1.0	2.32
7B + OPA-DPO	47.4	12.5	0.9	2.78
7B + TARS (Mask)	59.6	13.2	0.4	2.48
7B + TARS (Replace)	59.3	14.9	0.7	2.54
13B + DPO	56.7	24.3	2.2	2.48
13B + TARS (Mask)	59.8	12.5	0.6	2.89
GPT-4o (reference model)	60.9	17.6	0.8	3.87

Relative to DPO, 7B Mask reduces hallucination by 13.2 percentage points, a 50% relative reduction, while increasing Cover by 3.0 percentage points. However, 7B OPA-DPO still has a lower hallucination rate of 12.5% versus TARS's 13.2%, albeit with lower coverage; TARS is not best on every metric. The 13B TARS model outperforms GPT-4o on AMBER hallucination rate but scores 2.89 versus 3.87 on MMHal, so conclusions must remain specific to the task and metric. The original table combines reference foundation models, re-tested checkpoints, and reproduced methods; it is not a controlled comparison with identical training costs for all models.

Ablation Study¶

The following subset of Table 2 (page 10) uses LLaVA-v1.5-7B; TP denotes token perturbation, CAS cross-modal relevance selection, and SPA spectral preference alignment. OBJHal CRs and CRi are response-level and object-level hallucination proportions, respectively; lower is better.

Config	AMBER Cover ↑	AMBER Hal-Rate ↓	AMBER Cog ↓	OBJHal CRs ↓	OBJHal CRi ↓
TARS	59.6	13.2	0.4	12.0	3.2
w/o TP	56.6	26.4	2.5	14.0	5.0
w/o CAS	55.9	17.7	1.3	12.7	3.5
w/o SPA	58.3	15.1	0.7	12.5	3.7
w/o CAS&SPA	55.1	18.5	1.5	12.6	3.8

Removing TP returns performance to the tabulated DPO level, making perturbation an important foundation of the system, but this ablation cannot independently attribute every gain to TP. With the other components retained, removing CAS increases hallucination by 4.5 percentage points and removing SPA by 1.9 percentage points, supporting contributions from targeted selection and the spectral term. Component interactions mean these differences cannot be added as a decomposition of total gains, nor do they replace a statistical significance test.

Key Findings¶

Table 3 (page 13) additionally compares data augmentation on LLaVA-7B; the subset below retains AMBER metrics for direct comparison. The authors' 5× denotes added expansion: the text reports 28.8k examples including the original data, not a total of only 5 times the original size.

Method	AMBER Cover ↑	AMBER Hal-Rate ↓	AMBER Cog ↓
DPO + Paraphrasing	57.9	24.8	2.3
DPO + LLM Aug. (1×)	58.8	22.9	2.0
DPO + LLM Aug. (5×)	59.3	16.0	1.2
TARS	59.6	13.2	0.4
TARS + LLM Aug. (5×)	60.8	12.3	0.2

TARS with 4.8k examples achieves a hallucination rate 2.8 percentage points below the augmented DPO baseline with 28.8k examples, but this is not a claim under equal total training compute. Combining augmentation with TARS further reduces hallucination from 13.2% to 12.3%, indicating complementary roles for perturbation robustness and data diversity. Figure 5 shows faster gains at smaller data scales and saturation beyond 3.6k; no unlisted exact values are inferred from the curves here.

Highlights & Insights¶

The most transferable idea is to constrain which positions may vary before learning preferences that should remain invariant. This extends hallucination mitigation from answer labels to how the model uses prompt cues.
Representation supervision also compares preferred and rejected answers instead of copying correct-answer features position by position. Hidden-state and output-probability supervision are thereby organized around the same preference relation.
Reporting coverage alongside hallucination is important. Saying less can lower some hallucination metrics, whereas the improvement over DPO here also increases coverage.

Limitations & Future Work¶

The cache contains only the main paper and references, not the repeatedly cited supplement. Mask/Replace details, hidden-layer selection, spectral aggregation, and several hyperparameter boundary cases remain incompletely verified.
Low CLIP relevance does not imply causal irrelevance: weakly visually related negation or relation expressions can still determine question meaning. Stronger semantic-preservation checks are a reasonable future direction, not an implemented component of this paper.
The authors interpret the gains as causal alignment, but attention maps and hidden-state visualizations alone do not establish causal identification. Controlled input interventions and measurements of semantic preservation and error transfer would provide more direct evidence.
The main text focuses on two LLaVA sizes, while Muffin-13B results are referred to the unavailable supplement. Effectiveness across architectures, video, long contexts, and out-of-domain images should not be inferred from these results.
Greater frequency-domain stability under local changes depends on the specific statistic and constraint. FFT is an invertible linear transform, and an equivalent full-spectrum Euclidean norm does not automatically remove perturbations, making the actual spectral aggregation operation important to verify.
The tables do not provide multi-seed error bars or a complete time/memory cost comparison. Data efficiency should not be rewritten as a matching reduction in compute.

DPO (original reference 53) learns preferences from relative response likelihoods; TARS retains this foundation while changing input conditions and adding hidden-state preference constraints.
CHiP (original reference 24) emphasizes cross-modal hierarchical preference optimization; TARS reuses its training strategy and contributes constrained input perturbations and spectral regularization rather than rebuilding the multimodal architecture.
OPA-DPO (original reference 75) emphasizes on-policy preference data; TARS shows that optimizing a fixed dataset differently can also deliver substantial gains, not that data quality is unimportant.
Research direction: combine visual relevance with explicit semantic-preservation checks, then compare exact loss-driven perturbations against the current approximation to disentangle token selection, difficult-example construction, and spectral statistics.

Rating¶

Novelty: 4/5. Constrained token perturbations and spectral preferences form a distinctive combination, although the presentation of inner worst-case optimization remains approximate.
Experimental Thoroughness: 4/5. Multiple hallucination benchmarks, component ablations, and augmentation comparisons are included, but statistical stability and full efficiency comparisons are limited.
Writing Quality: 3/5. The motivation and main tables are clear, while spectral mathematics and implementation details depend on the supplement; extraction damage further limits reproducibility.
Value: 4/5. The work offers transferable ideas for small-scale preference post-training, with semantic preservation and target-domain generalization still requiring deployment-specific verification.