Safe Responses Matter: Output-Aware Safety Guardrail Mitigate Over-Refusal in MLLMs

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/kunzhan/OutGuard
Area: Multimodal VLM / AI Safety
Keywords: output-aware guardrail, over-refusal, hidden states, multiple instance learning, contrastive learning

TL;DR

Instead of blocking a query merely because its input appears sensitive, OutGuard trains hidden-state classifiers on actual-response safety labels and combines multi-instance contrastive learning with cross-layer voting, reducing in-distribution ARSP to 0.0749 and 0.0634 on two MLLMs while retaining missed risks, distribution-shift degradation and substantial refusal rates on difficult benign queries.

Background & Motivation

Multimodal large language models (MLLMs) typically already possess some safety alignment when processing images and text. A risky request may elicit an unsafe answer, but it may instead produce a refusal or lawful guidance and alternative suggestions. Input-side guardrails commonly judge only whether the image-text request is dangerous and terminate generation on detection, collapsing these different outcomes into the same refusal decision. Although HiddenDetector's refusal direction, QGuard's auxiliary safety questions and LoD's layer-wise detectors use different signals, they can inherit the same input-label definition. Over-refusal is therefore not simply inaccurate classification: correctly identifying a risky input can still suppress a response that would have been safe and useful.

This distinction is particularly important for legitimate requests containing sensitive words, such as terminating a Python process or discussing photographic shooting. The paper pairs safe XSTest text prompts with generated images to examine such superficially suspicious but benign multimodal requests. Another outcome worth preserving is a model's refusal or admonition in response to a risky request; a guardrail should not treat successful internal safety handling as a failure. However, adding a large output-side judge requires extra calls and waiting, and checking a fully generated answer is not the same deployment problem as intercepting it early. The authors consequently seek signals of eventual response safety in the original model's hidden representations without changing its parameters.

The intervention is not to relax every safety threshold, but to change the supervision target: only examples that actually yield harmful answers belong to the class requiring interception. Training data must consequently include the target model's own responses rather than simply reuse the dataset's harmful/benign input annotations. Moreover, only a few passages may determine a response's risk, so assigning the sequence label to every token introduces noise. Core Idea: define the safety boundary through actual responses, learn from each layer's token hidden states as a multiple-instance bag, and aggregate only validated layers to preserve safe responses while blocking harmful outputs.

Method

Overall Architecture

OutGuard trains on layer-wise hidden states produced when the target MLLM processes image-text requests, supervised by the text that this model actually generates. The authors train an independent multi-instance contrastive learning (MICL) classifier for every layer, select layers using validation data, and deploy a vote among the retained classifiers. The backbone remains frozen: the guardrail is not a response-rewriting model and does not rewrite safe guidance into a uniform refusal.

The process comprises Output Safety Supervision, Gated Multiple-Instance Aggregation, Intra-Bag Discrimination and Reliable Layer Selection. These four steps are offline preparation; deployment reuses the selected classifiers and existing backbone hidden states without querying external judges. Dashed arrows in the diagram indicate supervision or trained-parameter transfer, not retraining on each user request.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Offline image-text requests<br/>Backbone responses and hidden states"] --> Labels["Output Safety Supervision"]
    Source --> Pool["Gated Multiple-Instance<br/>Aggregation"]
    Labels -.->|Response-level labels| Pool
    Pool --> Learn["Intra-Bag Discrimination"]
    Learn --> Select["Reliable Layer Selection"]
    Select -.->|Selected classifiers| Vote["Online layer-wise predictions<br/>and majority voting"]
    Live["Online request<br/>Backbone hidden states"] --> Vote
    Vote -->|Majority predicts safe| Allow["Release original response"]
    Vote -->|Otherwise| Block["Block output"]

Here, output awareness primarily describes the prediction target and the source of supervision; it does not mean feeding the complete output text into another large model. However, the method defines a bag using the hidden states of all tokens in a generated response, while the abstract emphasizes deciding before generation finishes. The supplied main text does not specify the online prefix length, the first decision point, whether pending text is buffered, or how a later transition to unsafe content is handled. The evidence therefore establishes a hidden-state framework for output-safety prediction, not a demonstrated guarantee of interception before the first token.

Key Designs

1. Output Safety Supervision: separate risky inputs from harmful responses

The authors distinguish four groups: ordinary responses to benign inputs, refusals after risky inputs, admonitions after risky inputs, and genuinely harmful responses. The first three are safe positive examples; only the last group consists of unsafe negative examples. Calling a refusal or admonition safe does not authorize executing its input; it means that the deliverable text does not carry out the risky behavior requested. This directly corrects the target mismatch in input-side labels and shows that OutGuard still permits the model to refuse inappropriate requests.

Five judges from the GPT, Gemini, Grok, Qwen and DeepSeek families score or assess responses, with majority voting determining the training labels. A response's safety label is assigned to its complete hidden-state bag at every layer, teaching the classifier how this model responds rather than only which topic the input concerns. These labels depend on the backbone's behavior: a different model, alignment version or substantially changed decoding procedure can produce a different target for the same input. The t-SNE visualization in Figure 5 motivates this distinction: benign and risky inputs are easily separated, but the desired task further separates safe and harmful responses within risky-input examples.

2. Gated Multiple-Instance Aggregation: extract salient semantics from variable-length responses

At layer \(l\), the output hidden states form a bag \(H_l=\{h_l^1,\ldots,h_l^N\}\), and each token representation is an instance. The bag carries one response-level label, without requiring every instance to receive the same token-level risk annotation. This accommodates sequences whose harmfulness is triggered by local passages and autoregressive responses with varying length \(N\). Simply averaging all tokens could dilute decisive passages with neutral introductions and formatting text.

MICL passes instances through two learnable projections, applies tanh and sigmoid respectively, and multiplies the results element-wise to form gated features. A learnable vector maps these features to attention scores, which are normalized into token weights. The weights aggregate the original hidden representations into a fixed-size bag vector, and a linear classification head predicts response safety. The gate does not hard-match sensitive vocabulary; it learns which instances deserve attention within the target model's internal semantic space. Nevertheless, attention represents learned correlation rather than human-validated token-level explanations of harm, so high-weight positions should not automatically be treated as reliable evidence.

3. Intra-Bag Discrimination: concentrate on important instances instead of distributing attention uniformly

With response-level classification loss alone, attention could spread over many irrelevant tokens that happen to fit the training data. The authors therefore introduce attention-entropy regularization to encourage sparsity, allowing fewer instances to carry more of the decision signal. They also select the top-k% instances by attention as a salient set and use the remaining low-attention instances as an intra-bag comparison set. This selection comes from the current model, not additional manual span annotations, so the formulation remains weakly supervised.

The intra-bag contrastive objective brings high-attention instances closer together and separates them from low-attention instances. The paper implements this through an InfoNCE-style objective using cosine similarity and a temperature parameter that controls the influence of similarity differences. Contrastive positives here are high-attention instances from the same bag, not the positive examples in response-safety classification; the two meanings should not be conflated. The objective strengthens consistency among salient semantics within a bag rather than directly optimizing the safety-class margin between different responses. This also introduces a possible cost: if early attention is misled by sensitive words, contrastive learning may reinforce the wrong focus, motivating evaluation on difficult benign queries.

4. Reliable Layer Selection: validate individual layers before combining their judgments

OutGuard neither assumes that the final layer is best nor averages predictions from every layer indiscriminately. After independently training each MICL classifier, the authors select layers using validation Safety-Score, retaining only those scoring strictly above 0.9. This score lies in \([0,1]\) and considers thresholded risk for both safe and unsafe examples, with higher values preferred. The paper uses a danger function to turn the relation between a prediction and a threshold into risk, then combines class-wise averages; the score should not simply be called accuracy. Operators and fractions in cached equations (8) and (9) are corrupted, so this note preserves the intended definition and selection threshold without guessing the exact implementation.

During inference, each selected classifier makes a thresholded safety decision, and the original response is released only if more than half predict safe. Under this strict-majority rule, a tie does not qualify for release; the main text does not clearly define deployment fallback when no layer is selected. The Safety-Score cutoff of 0.9 is a layer-selection criterion, not necessarily the final harmful-probability threshold. Table 7 reports 18 selected layers for LLaVA 1.6 and 30 for Qwen 3.5, so the claimed lightweight overhead must be accumulated over selected layers.

A Worked Example

Consider the benign XSTest request about terminating a Python process, with an auxiliary image semantically matched to the text. If the backbone produces ordinary process-management guidance, output supervision labels the response safe rather than marking it harmful because it contains a sensitive verb. Layer-wise MICL classifiers weight the response's token representations, retained classifiers predict safety, and majority voting determines whether the response is delivered. If the model itself refuses, OutGuard may also consider that refusal safe and release it; it does not actively repair the backbone's own over-refusal. Reducing refusals introduced by a guardrail and making the base model more willing to answer are therefore different goals. This example explains a benign scenario through the paper's mechanism; it is not an additional measured success-rate experiment.

Loss & Training

The classifier jointly optimizes response-level binary cross-entropy, attention-entropy regularization and an intra-bag contrastive loss. The following equation restates only the weighted combination explicitly described in the prose, using clear notation rather than reconstructing corrupted component formulas:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{cls}}+\lambda_{\mathrm{ent}}\mathcal{L}_{\mathrm{ent}}+\lambda_{\mathrm{con}}\mathcal{L}_{\mathrm{con}}. \]

Optimization updates only MICL classifiers, with early stopping on validation loss; the MLLM backbone does not participate in this guardrail training. The pool comprises 2,694 image-text pairs from GQA, AdvBench and SafeBench, with 2,000 allocated to the training process, 20% of that allocation reserved for validation, and the remainder used for testing. The reference configuration in Table 8 includes \(\lambda_{\mathrm{con}}=0.10\) and \(\tau=0.50\); its top-k (%) row reports 0.30, but the unit convention between that value and the prose's percentage notation is insufficiently clear, so no conversion is assumed. Some attention-dimension rows in that table are merged in extraction, and the main text does not fully specify reproduction settings such as optimizer and learning rate; no missing hyperparameters are invented here. The five judge versions are gpt-4o-mini, gemini-2.5-flash, grok-4-1-fast, qwen-flash and deepseek-chat. The authors state that an appendix supplies judge prompts and label-reliability evidence, but the provided text ends after the references, preventing verification of those appendix claims.

Key Experimental Results

Main Results

The evaluated backbones are LLaVA-v1.6-Vicuna-7B and Qwen3.5-9B; the out-of-distribution (OOD) data combine MM-Vet and MM-SafetyBench. LoD-PP retrains LoD on the same training set and output labels as OutGuard, making it an important control for separating label changes from classifier changes. The following selection from Tables 2 and 3 (page 12) retains F1, the missed-risk metric URUP and the over-refusal metric ARSP; values are proportions, not percentages. The paper expands URUP as Unsafe Response rate to Unsafe Prompt and ARSP as Abstention Response rate to Safe Prompt, while also describing the latter using benign outputs. Because these input-conditioned and output-conditioned descriptions are not fully aligned, this note retains the reported values without claiming to have established their precise counting denominators.

Setting	Backbone	Method	F1 ↑	URUP ↓	ARSP ↓
In-distribution	LLaVA 1.6	QGuard	0.6667	0.0000	1.0000
In-distribution	LLaVA 1.6	LoD	0.7604	0.0029	0.6254
In-distribution	LLaVA 1.6	LoD-PP	0.8943	0.0948	0.1124
In-distribution	LLaVA 1.6	OutGuard	0.9335	0.0550	0.0749
In-distribution	Qwen 3.5	LoD-PP	0.9377	0.0029	0.1297
In-distribution	Qwen 3.5	OutGuard	0.9635	0.0115	0.0634
OOD	LLaVA 1.6	LoD-PP	0.8140	0.1223	0.2788
OOD	LLaVA 1.6	OutGuard	0.8953	0.0213	0.2077
OOD	Qwen 3.5	LoD-PP	0.8186	0.0037	0.4378
OOD	Qwen 3.5	OutGuard	0.8735	0.0260	0.2560

On the Qwen in-distribution test, OutGuard reduces ARSP by 0.0663 relative to LoD-PP, but URUP increases from 0.0029 to 0.0115. The prose claim of outperforming LoD-PP on every metric is therefore not fully consistent with the tables; a more accurate conclusion is substantially better over-refusal and overall classification, not uniformly better missed-risk metrics.

Ablation Study

The following test-set results come from Tables 9 and 10 (page 15); their original three-decimal precision is preserved rather than replaced with Table 2's precision. The two comparisons change supervision labels and feature space respectively, with the complete configuration serving as the reference for both.

Config	AUPRC ↑	AUROC ↑	F1 ↑	URUP ↓	ARSP ↓
Input-label variant, Table 9	0.750	0.693	0.760	0.003	0.625
OutGuard, Table 9	0.973	0.973	0.934	0.055	0.075
Logit features, Table 10	0.634	0.706	0.729	0.074	0.615
Hidden states, Table 10	0.973	0.973	0.934	0.055	0.075

Switching from input labels to output labels reduces ARSP from 0.625 to 0.075, a decrease of 0.550, while URUP increases from 0.003 to 0.055. This confirms that the target change affects over-refusal, but it should not be described as an across-the-board improvement without safety costs. The prose discussing Table 10 gives hidden-state ARSP as 0.074, whereas the table reports 0.075; this note follows the table and explicitly preserves the discrepancy. Figure 8 (page 14) reports that removing gating, contrastive learning or sparsity regularization leaves no classifier above the layer-selection cutoff. The authors then assign URUP+ARSP the worst value of 2; this is not a measurement showing both error rates equal 1, so the bars should not be interpreted as ordinary measured error rates.

Key Findings

The difficult benign-query evaluation uses safe XSTest prompts with images generated by SD 3.5 Medium, as reported in Table 5 (page 13):

Backbone	HiddenDetector	QGuard	LoD	LoD-PP	ProGuard	OutGuard
LLaVA 1.6	0.7400	1.0000	0.8840	0.7080	0.7760	0.5840
Qwen 3.5	1.0000	1.0000	1.0000	1.0000	0.7760	0.6320

OutGuard is best in this comparison, but 0.5840 and 0.6320 still mean that many benign requests are refused; over-refusal has not been solved. Table 4 (page 13) reports F1=1.0000, URUP=0.0000 and ARSP=0.0000 for Qwen under Visual Adversarial Examples; this holds for that attack-test configuration, not as a universal safety guarantee. Table 6 (page 14) reports average decision times of 0.208 seconds for OutGuard, 0.003 seconds for LoD and 3.154 seconds for QGuard, showing that avoiding external judges does not mean zero overhead. Table 7 (page 14), evaluated on an RTX 3090 in FP16, reports additional memory of 1.37GB for LLaVA and 2.24GB for Qwen, with 37.01M per-layer classifier size.

Highlights & Insights

• The most valuable change is the supervision target, not simply classifier capacity. The improvement from LoD to LoD-PP also supports the importance of label selection itself.
• Multiple instance learning acknowledges that a harmful passage does not make every token harmful. This better matches the local semantic structure of generated text than copying a risk label to every position.
• Validation-based layer selection allows different backbones to rely on different internal representations. The cost is model-dependent layer count and memory, so reporting only one classification head's size is insufficient.

Limitations & Future Work

• The authors acknowledge reduced OOD classification performance, with ARSP increasing to 0.2077 and 0.2560 in Table 3. Broader training coverage is a reasonable motivation, but the paper does not prove that it would resolve generalization.
• The authors report missed implicit risks and false refusals on benign inputs containing sensitive words. The difficult-query results likewise show that hidden-state probes do not eliminate contextual-understanding bias.
• Reader assessment: the temporal relation between complete-response hidden states and pre-generation decisions needs clarification. Reproduction should specify prefix length, streaming policy and safety-decision delay.
• Reader assessment: leaving backbone parameters unchanged does not preserve all utility, because the guardrail may still block correct answers. Evaluation should separate task accuracy, native backbone refusals and additional guardrail refusals.
• Evidence boundary: corrupted equations, an unavailable appendix and missing confidence intervals limit exact reproduction and reliability assessment. Further work needs independent human auditing, complete threshold definitions and adaptive-attack evaluation.

Related Work & Insights

• vs HiddenDetector / QGuard: the former uses a refusal direction and the latter auxiliary safety questions; OutGuard learns labels for actual target-model responses and needs no auxiliary question-answer calls during deployment.
• vs LoD / LoD-PP: both also use layer-wise hidden states, while LoD-PP additionally controls training data and labels. It is the more informative baseline for assessing whether MICL adds value beyond relabeling.
• vs ProGuard: the paper compares it as an independent post-output safety classifier; OutGuard instead uses internal target-model representations to reduce external calls, but cannot directly operate through closed interfaces that expose no hidden states.
• Research direction: model the appropriateness of backbone refusals separately from response harmfulness, then study prefix-level risk calibration. This follows from the limitations and is not an experiment already completed by the paper.

Rating

• Novelty: 4/5. Output labels, multiple-instance aggregation and layer selection form a targeted combination, although the individual components are not entirely new.
• Experimental Thoroughness: 3/5. Two backbones, OOD data, attacks and difficult benign queries are covered, but judge reliability and online timing need more evidence.
• Writing Quality: 3/5. The motivation and architecture are clear, but some blanket superiority claims, table-prose discrepancies and deployment timing are insufficiently precise.
• Value: 4/5. Useful for joint safety-utility evaluation, and a reminder to distinguish input risk, actual output risk and native model refusals.