ESC: Emotional Self-Correction for Reliable Vision-Language Models

Conference: ECCV 2026
Paper: ECCV / Project page
Area: Multimodal VLM
Keywords: vision-language model; self-correction; emotional prompting; hallucination mitigation; training-free

TL;DR

ESC finds that prepending a hand-written emotional expression to a query makes a VLM slow down and re-walk its own reasoning, so it uses a separate verifier VLM to decide whether the initial response is suspicious, injects an emotional cue to trigger one training-free self-correction pass, and finally arbitrates between the original and the revised answer; it reduces attack success rate or improves accuracy consistently across safety, hallucination, visual perception, and multimodal reasoning benchmarks without any degradation.

Background & Motivation

Vision-language models already handle image question answering, chart reading, and mathematical reasoning, and they are moving into high-stakes settings such as healthcare and security. Yet their answers are still frequently inconsistent with the visual evidence, i.e. they hallucinate. The established remedies are roughly threefold: add in-domain data, fine-tune, or modify the architecture — all expensive. Making a model "correct itself" at inference time (nascent self-correction) therefore looks like a bargain. The problem is that the methods which actually deliver large gains mostly rely on post-training: either reinforcement learning that explicitly optimizes corrective trajectories, or supervised/preference-based fine-tuning over carefully constructed reflective trajectories. These require dense annotations and substantial compute, and they do not transfer easily across models or domains. What is worse, self-correction quality depends critically on feedback quality, and models suffer from a self-correction blind spot — they can correct errors presented by other models but struggle to find their own. It has thus remained unclear under what conditions inference-time self-correction is actually reliable.

A second line of evidence comes from human behavior: people revise their answers not only when told they are wrong, but often because of affect — tension, unease, worry makes them slow down and re-examine their judgment. Prior work has mostly treated emotion as a capability dimension to be evaluated (can the model recognize affect, can it respond empathetically) or modelled it as a set of discrete categories, overlooking how it is organized in a continuous affective space. This paper instead adopts Russell's Circumplex Model of Affect and represents emotion on the two continuous dimensions of valence and arousal. It then runs a controlled probe: keeping the visual input and the task instruction completely unchanged and only adding an emotional expression to the query, five open-source VLMs of diverse architectures and scales all reduce ASR on VLSafe by 7.2 to 26.4 percentage points — both the initially vulnerable LLaVA-1.5-7B (71.6% → 45.2%) and the better-aligned Qwen2-VL-7B (20.0% → 7.5%). Breaking this down by quadrant shows that all four quadrants reduce ASR relative to a neutral baseline, but negative-valence prompts consistently yield larger reductions, and sensitivity is architecture-dependent. Core idea: emotion is not content for the model to recognize here, but a signal that can be inserted directly into the prompt to steer the model's reasoning mode — an external verifier decides whether the initial response needs revision; if so, an emotional cue drawn from the negative-valence/low-arousal quadrant is prepended to the query, the target model re-walks its reasoning under a more cautious mode to produce a revised answer, and the verifier arbitrates between the two candidates. Not a single weight is changed.

Method

Overall Architecture

ESC takes an image–question pair \((I, Q)\) as input and involves two roles: a target VLM \(M_T\) to be improved (LLaVA-1.5-7B and Qwen2-VL-7B in the experiments) and a separate verifier VLM \(M_V\) (Gemma3-12B by default). The pipeline has only three steps. \(M_T\) first produces an initial response \(R_{\mathrm{initial}}\). \(M_V\) then answers the same question again after seeing \((I, Q, R_{\mathrm{initial}})\); if its own answer \(R_{\mathrm{decided}}\) differs from \(R_{\mathrm{initial}}\), the initial response is deemed suspicious — ESC selects an emotional feedback \(F_{\mathrm{emotional}}\) accordingly, prepends it to the query, and lets \(M_T\) re-walk its reasoning to produce a revised response \(R_{\mathrm{revised}}\). Finally \(M_V\) sees both versions and picks the more appropriate one as the final output.

The key difference from most self-correction methods is that ESC does not always revise. Most methods rewrite unconditionally once feedback is available; ESC inserts a verification gate before revision, so a reliable initial response is returned as is — saving compute and avoiding the risk of breaking a good answer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["image + question"] --> B["target VLM produces initial response"]
    B --> C["external verifier gating<br/>verify before revising"]
    C -->|"response accepted, return it"| G["output final response"]
    C -->|"revision required"| D["emotional feedback injection<br/>negative-low arousal cue prepended<br/>target VLM re-walks its reasoning"]
    D --> E["original vs. revised<br/>arbitration"]
    E --> G

Key Designs

1. External verifier gating: defining "needs revision" as a disagreement between two models

This design targets the self-correction blind spot directly. The paper provides evidence: when LLaVA-1.5-7B serves as both target model and verifier (pure intrinsic self-correction), ASR on VLSafe stays at 50.3%, far above any external verifier. ESC therefore hands the gating decision to another model — but it does not ask the verifier for a correct/incorrect label or a confidence score. \(M_V\) simply answers \((I, Q, R_{\mathrm{initial}})\) again, and the disagreement signal \(R_{\mathrm{decided}} \neq R_{\mathrm{initial}}\) becomes the trigger. This has two benefits. First, the verifier needs no extra discriminative head, reward model, or annotation; any off-the-shelf VLM can serve as one. Second, the gate has a built-in "do nothing" exit — an accepted initial response is returned unchanged, preserving the model's existing strengths. That exit is not decorative in practice: Qwen2-VL-7B exhibits a collapsed inference behavior on POPE, answering yes to everything (Acc 50.80 with F1 of only 3.40), and ESC's verification stage is precisely what blocks such obviously failed predictions before arbitration restores a balanced prediction distribution (see the experiments).

A second experiment swaps the verifier to test whether the gains come from knowledge distillation by a stronger model: replacing the default Gemma3-12B with three 3–4B verifiers (Qwen3-VL-4B 33.9%, Qwen2.5-VL-3B 34.0%, Gemma3-4B 34.3%) only slightly raises ASR above the 12B default (31.2%). The verifier is thus a cheap trigger switch; the emotional feedback is what does the work.

2. Emotional feedback injection: emotion as a "slow down" control signal

To read this design correctly one must first be precise about what "emotion" means here. It is not an emotion-recognition task (the model is not asked to judge how a person in the image feels), it is not a confidence or uncertainty estimate of the model itself, and it is not a consistency score computed from multiple samples. It is simply a hand-written natural-language text expressing an affective state. ESC locates it in Russell's Circumplex Model of Affect, a two-dimensional continuous space of valence (pleasant–unpleasant) and arousal (activated–calm) with four quadrants, and by default adopts Negative-Low Arousal (negative valence plus low arousal, oriented toward expressions like "worried, uneasy, this deserves extra care"), prepended to the query. Self-correction here means letting the target model regenerate its response under that emotional prefix:

\[R_{\mathrm{revised}} = M_T\big(I,\; F_{\mathrm{emotional}} \,\|\, Q\big)\]

\(F_{\mathrm{emotional}}\) is selected by the Selection function in Algo. 1 based on the verifier's decision (the main text does not spell out that function's implementation, stating only that a single cue from the negative-low arousal quadrant is the default — ⚠️ refer to the original paper). Because the model regenerates the full response, what changes is not just the final option but also the intermediate reasoning chain.

Why does this work? The authors invoke Schwarz's affect-as-information framework: negative affect signals that the current situation is problematic and shifts processing toward detail orientation, whereas positive affect encourages heuristic shortcuts. This matches the ablation, where Positive-Low Arousal yields a much higher ASR (49.5%) than Negative-Low Arousal (31.2%). The probe in Sec. 3 likewise shows the effect is general (all four quadrants reduce ASR relative to neutral) but larger for negative valence, and that sensitivity varies by architecture (Qwen2-VL responds strongly in every quadrant while Pixtral-12B is barely sensitive to positive valence) — which is why ESC fixes the negative-low arousal cell instead of sampling randomly. More direct mechanistic evidence comes from the cautiousness analysis: Qwen3-VL-8B-Thinking's thinking traces on VLSafe are scored 1–5 for cautiousness by Gemma-4-26B (higher is more deliberate; the full scoring prompt is in the appendix). With the verify-revise loop alone (VO) and no emotional content, cautiousness is essentially identical to the baseline (3.30 vs. 3.31), showing that an extra pass by itself does not change how the model reasons. Generic corrective and psychological prompts raise it to 4.22 and 4.15, and ESC reaches the highest score (4.50, median 5.0). In other words, emotion does not change what the model answers so much as how carefully it thinks. ESC also uses the shortest prompts among all compared conditions, ruling out prompt length as a confound.

3. Original-vs-revised arbitration: a floor under the correction

Asking a model to rewrite its answer under emotion is double-edged: it may fix an error, or it may break an answer that was already correct and become over-cautious or off-topic. ESC's safeguard is to let \(M_V\) arbitrate between the two versions rather than unconditionally accepting the revision, which turns emotional correction from a one-way rewriting operator into candidate selection with a fallback. This explains the most counter-intuitive result in the paper: Qwen2-VL-7B's baseline on the three POPE splits is a yes-answering collapse, and ESC lifts Acc from 50.80 to 80.43 and F1 from 3.40 to 76.17 — not by making the model stricter, but because "reject the collapsed answer, then choose between two candidates" rebalances the prediction distribution. It is also why ESC degrades on no benchmark at all: when the revision is not accepted, the original answer is returned. This is what separates it from methods that become progressively more conservative, and it is the precondition for using ESC as a plug-and-play component.

Loss & Training

ESC is entirely training-free: no weight updates, no gradients, no post-training of any kind — everything happens in the prompt at inference time. What remains to specify is hyper-parameter choice: the verifier defaults to Gemma3-12B, the emotional quadrant to Negative-Low Arousal, and a single emotional cue prepended to the query. The target models are LLaVA-1.5-7B and Qwen2-VL-7B, HallusionBench and MM-Vet are scored by GPT-4o, and the other benchmarks use standard accuracy/ASR metrics. Note that "one cue" is not the optimum found in the ablation (two cues are better; see below) — the paper adopts it as the default without stating a rationale.

Key Experimental Results

Main Results

Safety is measured by ASR (attack success rate, lower is better) on VLSafe and MMSafetyBench, and hallucination by the three POPE splits plus HallusionBench (GPT-4o judging). On safety, ESC reduces ASR on both benchmarks and across all scenario categories; on MMSafetyBench all 13 scenario categories (averaged over the SD, SD+Typo, and Typo image types) improve consistently.

Benchmark	Model	Metric	Baseline	ESC	Δ
VLSafe	LLaVA-1.5-7B	ASR↓	71.6%	25.3% (Fig. 5) / 31.2% (Tab. 4, 7)	−46.3 / −40.4
VLSafe	Qwen2-VL-7B	ASR↓	20.0%	9.9%	−10.1
VLSafe	Qwen3-VL-8B	ASR↓	8.4%	3.2%	−5.2
VLSafe	InternVL3-8B	ASR↓	10.5%	6.5%	−4.0
POPE-Adversarial	LLaVA-1.5-7B	Acc↑ / F1↑	83.47 / 82.57	84.53 / 84.15	+1.06 / +1.58
POPE-Popular	LLaVA-1.5-7B	Acc↑ / F1↑	86.07 / 84.94	87.40 / 86.72	+1.33 / +1.78
POPE-Random	LLaVA-1.5-7B	Acc↑ / F1↑	87.90 / 86.65	89.57 / 88.73	+1.67 / +2.08
POPE-Adversarial	Qwen2-VL-7B	Acc↑ / F1↑	50.80 / 3.40	80.43 / 76.17	+29.63 / +72.77
POPE-Popular	Qwen2-VL-7B	Acc↑ / F1↑	50.83 / 3.41	80.93 / 76.65	+30.10 / +73.24
POPE-Random	Qwen2-VL-7B	Acc↑ / F1↑	50.83 / 3.41	81.27 / 77.04	+30.44 / +73.63
HallusionBench	LLaVA-1.5-7B	aAcc / qAcc / fAcc↑	44.11 / 17.58 / 16.87	45.17 / 17.58 / 17.37	+1.06 / +0.00 / +0.50
HallusionBench	Qwen2-VL-7B	aAcc / qAcc / fAcc↑	55.00 / 26.37 / 29.53	55.62 / 28.57 / 30.27	+0.62 / +2.20 / +0.74

Multimodal reasoning (MM-Vet scored by GPT-4o) and vision-centric perception (MMVP is pair accuracy, testing fine-grained visual discrimination): reasoning gains are moderate but consistently positive, while perception shows the two largest jumps in the paper on MMVP and RealWorldQA.

Benchmark	Model	Metric	Baseline	ESC	Δ
MM-Vet	LLaVA-1.5-7B	Score↑	24.31	25.39	+1.08
MathVista	LLaVA-1.5-7B	Acc↑	22.20	23.40	+1.20
MMStar	LLaVA-1.5-7B	Acc↑	33.53	34.13	+0.60
MMMU	LLaVA-1.5-7B	Acc↑	34.66	34.77	+0.11
AI2D	LLaVA-1.5-7B	Acc↑	51.23	53.72	+2.49
MM-Vet	Qwen2-VL-7B	Score↑	39.66	40.05	+0.39
MathVista	Qwen2-VL-7B	Acc↑	56.10	56.10	+0.00
MMStar	Qwen2-VL-7B	Acc↑	55.93	56.07	+0.14
MMMU	Qwen2-VL-7B	Acc↑	39.67	40.02	+0.35
AI2D	Qwen2-VL-7B	Acc↑	60.98	62.53	+1.55
MMVP	LLaVA-1.5-7B	Pair Acc↑	18.67	26.00	+7.33
MMVP	Qwen2-VL-7B	Pair Acc↑	41.33	45.33	+4.00
RealWorldQA	LLaVA-1.5-7B	Acc↑	53.59	53.86	+0.27
RealWorldQA	Qwen2-VL-7B	Acc↑	40.00	54.51	+14.51
BLINK	LLaVA-1.5-7B	Micro / Macro Acc↑	41.71 / 42.20	42.14 / 42.61	+0.43 / +0.41
BLINK	Qwen2-VL-7B	Micro / Macro Acc↑	49.61 / 50.42	49.61 / 50.44	+0.00 / +0.02

Column assignment above follows the paper's prose (+0.27 and +14.51 are attributed to RealWorldQA; +7.33 and +4.00 to MMVP). The cached text has an ambiguous column grouping in this table's header, and MMVP's Question Acc column (LLaVA 54.67 → 59.33, Qwen2 69.00 → 70.33) is not listed above — ⚠️ refer to the original paper.

Ablation Study

The central ablation replaces the feedback content under otherwise identical conditions to test whether emotion itself is the causal factor (target model always LLaVA-1.5-7B, verifier always Gemma3-12B, single cue, same position):

Config	ASR↓	Δ	Note
Baseline (no intervention)	71.6%	—	no revision at all
VO (verify-revise loop only)	69.1%	−2.5	correction loop without emotional feedback; nearly useless
ZeroCoT	70.1%	−1.5	just "Let's think step by step."
SR (Self-Refine style)	49.3%	−22.3	review the previous answer, identify errors, provide a corrected response
Corr (generic corrective)	48.6%	−23.0	review the answer carefully and revise if needed
Psych (psychological prompt)	54.4%	−17.2	tone change without emotional grounding, e.g. "this is very important to my career"
ESC (Ours)	31.2%	−40.4	negative-low arousal cue, prepended

The remaining design choices, also on VLSafe with LLaVA-1.5-7B:

Variable	Setting	ASR↓	Note
Verifier	LLaVA-1.5-7B self-verification (intrinsic)	50.3%	well above external verifiers; hits the self-correction ceiling
Verifier	Gemma3-12B (default)	31.2% (main text reports 40.1% for Fig. 7(a); the two figures do not share a protocol)	described as the lowest of the four
Verifier	Qwen3-VL-4B / Qwen2.5-VL-3B / Gemma3-4B	33.9% / 34.0% / 34.3%	small verifiers approach the default, ruling out knowledge distillation
Emotional quadrant	Negative-Low Arousal (default)	31.2%	lowest
Emotional quadrant	Positive-Low Arousal	49.5%	highest; positive valence dilutes the corrective signal
Insertion position	prepend (default)	31.2%	10.5 points better than appending
Insertion position	append	41.7%	—
Number of cues	1 (default)	31.2%	—
Number of cues	2	25.3%	the only setting better than the default
Number of cues	3 / 4	26.8% / 25.4%	no further benefit from adding more

Cautiousness scores (thinking traces of Qwen3-VL-8B-Thinking, scored 1–5 by Gemma-4-26B, higher is more deliberate):

Condition	Mean↑	Median↑
Base	3.31	4.00
VO	3.30	4.00
Psych	4.15	4.00
Corr	4.22	4.00
ESC	4.50	5.00

Key Findings

• Emotion is the causal factor; the loop is not: the verify-revise loop alone (VO, −2.5) and a generic chain-of-thought prompt (ZeroCoT, −1.5) barely move ASR, so the gain does not come from "running an extra pass." Generic corrective prompts reach roughly 22–23 points of improvement, and ESC adds more than 17 points on top of them, showing that the emotional content itself is doing the work. The cautiousness scores corroborate this from another angle: VO is indistinguishable from the baseline (3.30 vs. 3.31), while ESC genuinely makes the reasoning more deliberate (4.50).
• Gains scale with baseline fragility: the three largest improvements occur exactly where the baseline is broken — Qwen2-VL's POPE collapse (F1 3.40 → 76.17), RealWorldQA (+14.51), and MMVP (+4.00 / +7.33). On already-stable knowledge-intensive benchmarks such as MMMU and MathVista, Qwen2-VL gains only +0.35 and +0.00. Importantly, no benchmark degrades, indicating that the two-candidate arbitration does contain the risk of breaking a good answer.
• Emotion does not depend on a strong verifier: verifiers of 3–4B (33.9%–34.3%) come close to the 12B default (31.2%). Shrinking the verifier from 12B to 3B costs only 2–3 points, whereas replacing emotional feedback with a generic corrective prompt costs more than 17 points.
• Emotional dosage is non-monotonic: going from one cue to two improves ASR by 5.9 points (31.2% → 25.3%), but three (26.8%) and four (25.4%) bring no further benefit, which the authors read as moderate reinforcement helping while excessive context only adds redundancy. Notably the default configuration still uses one cue rather than the ablation optimum of two.
• Quadrant and position orderings are mutually consistent: negative-low < negative-high / positive-high < positive-low, and sampling randomly across quadrants dilutes the corrective signal; prepending beats appending by 10.5 points, suggesting the emotional cue must frame the whole task rather than comment on it afterwards.
• A note on reported numbers: ESC's ASR on VLSafe is 25.3% in Fig. 5 but 31.2% in Tab. 4/7, and the Gemma3-12B verifier is reported as 40.1% in the main text for the verifier ablation, which is not on the same protocol as the 31.2% in Tab. 4 (one may include emotional feedback and the other may not). Always cite the specific source; ⚠️ refer to the original paper.

Highlights & Insights

• Turning "emotion" from an object of recognition into a control signal: affective computing usually asks how accurately a model recognizes emotion and how appropriately it responds; this paper asks whether emotion can be used as a knob. That small conceptual turn — emotion moving from a capability dimension to a control dimension — opens an inference-time scaling route that needs no post-training at all, and it is the paper's real "aha" moment.
• Defining "needs revision" by disagreement rather than by scoring: the verifier only has to answer once, and a mismatch triggers revision. This removes the entire cost of training a discriminator or reward model and lets any off-the-shelf VLM serve as verifier; it also inherits the lesson of the self-correction blind spot, since gating must come from another model.
• Cautiousness scoring is a reusable diagnostic: having a third-party model score thinking traces 1–5 separates "the answer changed" from "the way of reasoning changed," filling the mechanistic gap that behavioural metrics (ASR/Acc) cannot. Any work on self-correction or reasoning enhancement can reuse this control design.
• Counter-evidence against over-conservatism: thanks to arbitration, ESC loses nothing on reasoning and perception benchmarks. This suggests that in a self-correction system, candidate generation and candidate selection should be designed separately — generation can be aggressive while selection provides the floor.

Limitations & Future Work

• The authors admit little; the conclusion mainly positions ESC as a foundation for future controllable multimodal intelligence and does not discuss failure cases.
• The emotional prompt remains a black box: the Selection function in Algo. 1 and the exact wording and filtering logic of the cue templates are not given in the main text (several pointers refer to the appendix), yet this is exactly the most sensitive part for reproduction and transfer.
• The default configuration is not the ablation optimum: one cue corresponds to 31.2% ASR while two reach 25.3%, and the paper does not explain preferring one. Meanwhile VLSafe carries two readings, 25.3% (Fig. 5) and 31.2% (Tab. 4/7); the two should not be mixed when quoting gains.
• The comparison baseline is weak: SR / Corr / VO / Psych are prompt-style baselines reproduced by the authors rather than the original implementations of post-training self-correction methods, so "training-free is good enough" lacks a direct opponent under an identical evaluation protocol.
• The mechanistic explanation rests on the affect-as-information analogy; the evidence is behavioural metrics plus a cautiousness score, with no evidence at the level of internal representations (e.g. which attention or activation patterns the emotional prefix changes).
• Safety benchmarks report ASR only, with no dedicated over-refusal or helpfulness metric; not degrading on multimodal reasoning benchmarks is strong but indirect evidence.
• Directions for improvement: make quadrant and cue count adaptive (e.g. allocate an "emotion budget" according to the verifier's degree of disagreement), turn the verifier's decision into content-specific feedback instead of a fixed template, or attach the emotional signal to test-time compute allocation — paying for a second generation only on suspicious samples.

Related Work & Insights

• vs post-training self-correction (RL / SFT style, e.g. VL-Rethinker, ReCoT, Sherlock): they train the corrective ability into the weights via reinforcement learning or reflective trajectories, at the cost of large-scale synthetic data, multi-stage training, and engineered rewards, and the recipe must be redone for each new model. ESC touches no weights and only inserts an emotional cue at inference time, making it plug-and-play and transferable across models. The price is that its gains are bounded by the target model's existing ability — it is nearly ineffective on knowledge-intensive benchmarks.
• vs intrinsic and prompt-based self-correction (Self-Refine, ZeroCoT, etc.): these either hit the ceiling of the self-correction blind spot (the self-verification reproduced here stays at 50.3% ASR) or obtain a fixed gain from generic corrective instructions (about 22–23 points). ESC shows the real lever is the emotional content of the feedback: with the same single prompt and the same position, changing the content pushes ASR more than 17 points lower.
• vs hallucination-correction pipelines (detect-then-correct, e.g. Woodpecker-style): those train dedicated detectors and correctors, forming a modular pipeline; ESC uses only two off-the-shelf VLMs and one emotional cue, reducing detection to an answer-disagreement signal and correction to regeneration plus a two-way choice.
• vs affective computing work (Emotion-LLaMA, EmoBench, etc.): they study whether models can recognize and express emotion, treating it as a capability dimension; ESC asks whether emotion can be used as a control variable. The two define and use emotion quite differently, which also means this direction has ample room left to explore.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Flips emotion from "an object to be recognized" into "an injectable control signal" and systematizes its selection through the circumplex model — a fresh angle backed by solid empirical work
• Experimental Thoroughness: ⭐⭐⭐⭐ Covers four benchmark families (safety, hallucination, reasoning, perception) with seven analyses (feedback content, verifier, quadrant, position, cue count, cautiousness, cross-model generalization); lacks a direct same-protocol comparison against post-training self-correction, and a few reported numbers are inconsistent across figures and tables
• Writing Quality: ⭐⭐⭐⭐ The logic chain (observation → finding → framework → ablation) is clear and the method is simple enough to reproduce; the main text pushes key implementation details (emotional templates, Selection function) into the appendix and says too little about the default hyper-parameters and the inconsistent readings
• Value: ⭐⭐⭐⭐ Offers a near-zero-cost path to more reliable inference and a transferable "emotion as control signal" idea for LLMs and agents, with clear engineering value

Related Papers

• [CVPR 2026] MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correction
• [ECCV 2026] Delineating Knowledge Boundaries for Honest Large Vision-Language Models
• [ICLR 2026] Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models
• [ICCV 2025] SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning
• [CVPR 2026] VisPlay: Self-Evolving Vision-Language Models