SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Ed-Bg/SaFeR-Steer-full
Area: LLM Reasoning
Keywords: Multimodal Safety, Multi-Turn Alignment, Synthetic Bootstrapping, Feedback Dynamics, Group Relative Policy Optimization
TL;DR¶
To combat safety decay and jailbreak vulnerabilities in multi-turn multimodal conversations, SAFER-STEER establishes a staged alignment framework combining synthetic bootstrapping for defensive cold-start with tutor-in-the-loop GRPO guided by trajectory-consistent summative rewards, achieving state-of-the-art bidirectional transfer across static and dynamic safety regimes.
Background & Motivation¶
Multimodal large language models (MLLMs) are increasingly deployed in real-world multi-turn conversational systems where visual and textual context accumulates over extended interaction horizons. However, this deployment setting exposes serious safety decay vulnerabilities. Adversaries rarely trigger blatant violations in isolated single-turn queries; instead, they distribute unsafe intent across benign-looking turns, manipulate conversational momentum, and leverage subtle visual elements or prompt injections embedded in images to circumvent safety boundaries. Despite these emerging risks, existing multimodal safety alignment paradigms—exemplified by methods like TIS, SPA-VL, and SafeRLHF-V—remain heavily tethered to single-turn VQA benchmarks and static preference pairs, cultivating localized reflexes that degrade precipitously when exposed to dynamic interactive dialogues.
The core tension underlying this widespread failure is the static-to-dynamic distribution mismatch between offline demonstration curation and adaptive on-policy interaction. In practical deployments, human red-teamers and adversarial agents do not execute immutable scripted sequences; they actively inspect the model's intermediate concessions, tonality, and compliance thresholds, dynamically adjusting tactics via progressive escalation (such as crescendo strategies, sunk-cost hooks, or conversational bait-and-switch). Existing multi-turn datasets generated via offline decomposition lack this feedback loop, resulting in RL rollouts that cannot target the policy's real-time vulnerabilities. Furthermore, conversational safety is an inherently trajectory-level property: a single catastrophic leak at turn eight invalidates an otherwise impeccably safe interaction. Conventional reward modeling that relies on step-wise independence or simple arithmetic averaging severely dilutes worst-case failures, permitting reward hacking and unaligned policies.
This paper tackles these challenges by reframing multimodal multi-turn safety alignment into a closed-loop co-evolutionary game where the model is directly optimized against adaptive, on-policy conversational adversaries. Core idea: SAFER-STEER resolves the static-to-dynamic generalization gap through a progressive three-stage pipeline that decomposes single-turn queries into multi-turn seed trajectories for dual-agent synthetic bootstrapping, and then couples a feedback-driven Safety Tutor with Trajectory-Consistent Summative Reward (TCSR) to train the student policy under adaptive multi-turn GRPO.
Method¶
Overall Architecture¶
The SAFER-STEER framework orchestrates three progressive stages to align multimodal assistants against multi-turn contextual attacks. In Stage I (Intent Decomposition and Reconstruction), static single-turn vision-language inputs are systematically expanded into three disjoint multi-turn seed pools: benign inquiries, obfuscated-risk prompts, and verified strong red-team trajectories. In Stage II (Synthetic Bootstrapping), a red-blue dual-agent simulation environment rolls out complete dialogues from these seeds; trajectories meeting stringent turn-level safety and trajectory-level helpfulness thresholds are retained to compile the STEER-SFT dataset, providing a supervised defensive foundation. In Stage III (Tutor-in-the-Loop Agentic RL with Feedback Dynamics), an active Safety Tutor interacts on-policy with the student policy during Group Relative Policy Optimization (GRPO). The tutor continuously evaluates responses across safety, usefulness, and faithfulness, dynamically evolves the next adversarial challenge based on the student's defensive stance, and aggregates feedback using Trajectory-Consistent Summative Reward (TCSR) to drive robust policy updates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Single-Turn Multimodal Inputs<br/>Image and query seed collection"] --> B["Stage I: Intent Decomposition & Reconstruction<br/>Benign, obfuscated, and red-team multi-turn expansion"]
B --> C["Stage II: Synthetic Bootstrapping SFT<br/>Red-blue simulation and strict dual-threshold filtering"]
C --> D["Stage III: Tutor-in-the-Loop Feedback Dynamics<br/>Tutor diagnoses response and generates adaptive next attack"]
D --> E["Trajectory-Consistent Summative Reward<br/>Joint historical minimum and trajectory average aggregation"]
E --> F["Student Policy GRPO Update<br/>Group-relative advantage computation and KL regularization"]
Key Designs¶
1. Intent Decomposition & Reconstruction: Multi-Turn Adversarial Dispersal of Visual Intent Isolated single-turn safety seeds fail to simulate the gradual compliance-building typical of long-horizon jailbreaks. To address this, the pipeline deploys a probing operator that transforms single-turn inputs \((x_i, q_i)\) into benign variants \(\tilde{q}^{\text{ben}}_i\) and obfuscated-risk variants \(\tilde{q}^{\text{obf}}_i\), which a high-capacity generator \(G_\psi\) expands into multi-turn user sequences \(U^{\text{ben}}_i\) and \(U^{\text{obf}}_i\). To construct severe adversarial vectors, the system integrates a taxonomy \(\mathcal{A}\) of 8 distinct psychological jailbreak strategies (including roleplay, educational bypasses, fiction writing, and emotional urgency) combined with stochastic visual perturbations (Gaussian noise at \(\sigma=0.03\) and high-contrast typographic OCR injection). Only dialogues that successfully compromise a fixed target policy \(\pi_{\theta_0}\) are retained: $\(S_{\text{rt}} = \{(x_i, U^{\text{rt}}_{i, z}) : z \in \mathcal{A}, \mathbb{I}_{\text{succ}}(x_i, U^{\text{rt}}_{i, z}; \pi_{\theta_0}) = 1\}\)$ This ensures that the initial multi-turn seed pool authentically mirrors covert, escalating attack trajectories rather than synthetic concatenations.
2. Synthetic Bootstrapping SFT: Dual-Agent Rollouts with Dual-Threshold Filtering Directly initializing reinforcement learning on unaligned base models leads to exploration collapse due to the vast dialogue space and lack of defensive reflexes. Stage II resolves this initialization bottleneck by pairing a red user agent \(\pi_{\theta_\rho}\) (gpt-4o-mini) that contextually escalates template turns with a blue expert agent \(\pi_{\theta_\beta}\) (Gemini-3-Pro) instructed to provide firm, non-preachy refusals alongside educational alternatives. To prevent toxic leakage from entering the fine-tuning mixture, candidate dialogues are audited with GPT-5.2 under strict whole-dialogue gating: $\(\mathbb{I}_{\text{keep}}(D) = \mathbb{I}\left(\min_{t \in [T]} s^{\text{safe}}_t \ge \tau_{\text{safe}}\right) \cdot \mathbb{I}\left(\frac{1}{T} \sum_{t=1}^T s^{\text{help}}_t \ge \tau_{\text{help}}\right)\)$ The resulting 12,934 high-fidelity trajectories form the STEER-SFT corpus, training the student model via teacher-forced likelihood to establish a robust baseline policy before on-policy exposure.
3. Tutor-in-the-Loop Feedback Dynamics: Adaptive Vulnerability-Targeting Attacks Static multi-turn datasets cannot challenge the specific runtime weaknesses that emerge as a model aligns. Stage III instantiates an on-policy training-time Safety Tutor \(R_\phi\) powered by Qwen3-VL-32B that concurrently evaluates the student's output \(a_t\) across safety (\(s^{\text{safe}}_t\)), usefulness (\(s^{\text{use}}_t\)), and visual faithfulness (\(s^{\text{faith}}_t\)), and dynamically formulates the subsequent user attack \(u_{t+1}\). The tutor executes targeted state transitions: if the student provides an unhelpful flat refusal (Standard Refusal), the tutor applies a Persistence strategy challenging its rigidity; if the student offers an educational alternative (Constructive Refusal), the tutor deploys a Bait & Switch tactic attempting to pivot the alternative back toward the hazard; and if the student partially complies, it executes a Deepen probe to extract operational details. This feedback dynamic guarantees that the rollout distribution continuously probes the student's evolving frontier of vulnerability.
4. Trajectory-Consistent Summative Reward: Weakest-Link Sensitivity with Global Coherence Standard reinforcement learning objectives aggregate turn rewards via linear discounting, allowing strong early performance to mask catastrophic late-turn failures. To enforce trajectory-level consistency, SAFER-STEER first maps the tutor's evaluation into a scalar turn reward \(r_t = \text{Norm}(w_{\text{safe}} s^{\text{safe}}_t + w_{\text{use}} s^{\text{use}}_t + w_{\text{faith}} s^{\text{faith}}_t) \in [0, 1]\), and then constructs the Trajectory-Consistent Summative Reward (TCSR): $\(R^{(k)}_{\text{TCSR}} = \lambda \min_{1 \le t \le T^{(k)}} r^{(k)}_t + (1 - \lambda) \frac{1}{T^{(k)}} \sum_{t=1}^{T^{(k)}} r^{(k)}_t, \quad \lambda \in [0, 1]\)$ The historical minimum term \(\min_t r^{(k)}_t\) acts as a strict bottleneck filter, ensuring that a single safety breach or severe hallucination penalizes the entire rollout advantage, while the trajectory mean term rewards sustained helpfulness and conversational engagement across the dialogue.
Loss & Training¶
In Stage II SFT, the student policy is fine-tuned with standard autoregressive likelihood loss \(\mathcal{L}_{\text{SFT}}(\theta) = -\mathbb{E}_{(h_t, a^*_t) \sim \mathcal{D}_{\text{SFT}}} [\log \pi_\theta(a^*_t | h_t)]\) for 2 epochs on 8\(\times\) NVIDIA A800 (80GB) GPUs using LLaMA-Factory, with learning rate \(1 \times 10^{-5}\), batch size 16, and sequence length 32,768.
In Stage III on-policy RL, verl implements Group Relative Policy Optimization (GRPO). For each multi-turn seed context, \(K=5\) trajectories are sampled via tutor rollouts. The baseline return \(\bar{R}_{\text{TCSR}} = \frac{1}{K}\sum_{k=1}^K R^{(k)}_{\text{TCSR}}\) and group-relative advantages \(A^{(k)} = R^{(k)}_{\text{TCSR}} - \bar{R}_{\text{TCSR}}\) guide the policy update over all rollout tokens, constrained by a reference model \(\pi_{\text{ref}}\): $\(\mathcal{L}_{\text{GRPO}}(\theta) = -\mathbb{E}\left[ \frac{1}{K}\sum_{k=1}^K A^{(k)} \sum_{t=1}^{T^{(k)}} \log \pi_\theta(a^{(k)}_t | h^{(k)}_t) \right] + \beta \mathbb{E}\left[ \sum_t \text{KL}(\pi_\theta(\cdot | h_t) \parallel \pi_{\text{ref}}(\cdot | h_t)) \right]\)$ Hyperparameters include \(\beta = 0.1\), global batch size 64, mini-batch size 2, AdamW optimizer with learning rate \(1 \times 10^{-5}\) and weight decay \(10^{-2}\) in BF16 precision.
Key Experimental Results¶
Main Results¶
Evaluation spans five established static benchmarks (Beavertails-V, MM-SafetyBench, SPA-VL, VLGuard, VLSBench) alongside the comprehensive multi-turn benchmark STEER-BENCH (comprising 3,227 dialogues across five diverse subsets). Evaluations are conducted using GPT-5-nano as an automated judge, which achieves an 89% pass agreement and 87% quadratic weighted kappa with human consensus.
Table 1: Static Single-Turn Safety and Helpfulness Evaluation (Safety↑ / Helpfulness↑)
| Model & Alignment Method | Beavertails | MM-Safety | SPA-VL | VLGuard | VLSBench | Average (Avg.) |
|---|---|---|---|---|---|---|
| Qwen2.5VL-3B (Base) | 35.31 / 41.93 | 36.10 / 25.20 | 48.68 / 53.20 | 66.20 / 56.30 | 55.22 / 52.66 | 48.30 / 45.86 |
| + TIS | 79.15 / 73.73 | 72.79 / 61.52 | 72.45 / 61.32 | 74.30 / 71.00 | 67.31 / 47.39 | 73.20 / 62.99 |
| + SPA-VL | 80.81 / 76.36 | 72.45 / 57.85 | 76.04 / 69.62 | 77.11 / 68.67 | 61.25 / 44.07 | 73.53 / 63.31 |
| + SaFeR-VLM | 76.44 / 81.19 | 74.52 / 61.79 | 68.87 / 73.21 | 75.10 / 75.30 | 65.15 / 54.26 | 72.02 / 69.15 |
| + SAFER-STEER (Ours 3B) | 88.89 / 85.97 | 81.08 / 63.56 | 83.40 / 77.03 | 79.62 / 73.25 | 76.20 / 54.05 | 81.84 / 70.77 |
| Qwen2.5VL-7B (Base) | 55.93 / 58.98 | 20.94 / 37.72 | 69.43 / 73.20 | 71.90 / 72.37 | 62.86 / 59.31 | 56.21 / 60.32 |
| + TIS | 85.59 / 77.80 | 79.11 / 70.00 | 85.66 / 78.30 | 80.10 / 76.70 | 82.11 / 68.27 | 82.51 / 74.21 |
| + SPA-VL | 80.85 / 77.76 | 68.94 / 51.64 | 83.40 / 76.42 | 82.60 / 74.51 | 69.89 / 55.80 | 77.14 / 67.23 |
| + SaFeR-VLM | 76.78 / 75.93 | 72.08 / 62.89 | 71.13 / 79.81 | 79.88 / 77.88 | 72.54 / 61.92 | 74.48 / 71.69 |
| + SAFER-STEER (Ours 7B) | 92.03 / 86.83 | 88.71 / 73.42 | 85.80 / 79.17 | 88.48 / 81.82 | 84.41 / 65.74 | 87.89 / 77.40 |
Table 2: Dynamic Multi-Turn Safety Evaluation on STEER-BENCH (Safety↑ / Helpfulness↑)
| Model & Alignment Method | STEER-Beaver | STEER-MMSafe | STEER-VLS | STEER-SPA | STEER-DyS | Average (Avg.) |
|---|---|---|---|---|---|---|
| Qwen2.5VL-3B (Base) | 21.79 / 42.88 | 4.92 / 15.91 | 11.76 / 34.12 | 14.89 / 23.28 | 9.40 / 19.44 | 12.55 / 27.13 |
| + TIS | 23.84 / 26.16 | 33.24 / 38.71 | 24.48 / 31.34 | 28.24 / 22.14 | 24.92 / 16.82 | 26.94 / 27.03 |
| + SPA-VL | 34.90 / 53.35 | 23.63 / 40.32 | 17.94 / 36.18 | 25.86 / 33.84 | 16.74 / 19.69 | 23.81 / 36.68 |
| + SaFeR-VLM | 26.23 / 37.32 | 32.23 / 55.92 | 29.91 / 53.08 | 41.13 / 52.08 | 32.42 / 35.17 | 32.38 / 46.71 |
| + SFT (Stage II) | 39.43 / 64.27 | 50.43 / 72.90 | 34.33 / 53.73 | 48.68 / 58.11 | 35.17 / 38.84 | 41.61 / 57.57 |
| + SAFER-STEER (Ours 3B) | 50.27 / 71.20 | 69.03 / 85.98 | 54.43 / 72.48 | 54.76 / 66.27 | 49.40 / 55.40 | 55.58 / 70.27 |
| Qwen2.5VL-7B (Base) | 36.06 / 65.54 | 11.42 / 30.42 | 24.56 / 51.18 | 29.01 / 46.56 | 22.27 / 38.70 | 24.66 / 46.48 |
| + TIS | 29.23 / 24.96 | 48.95 / 53.51 | 40.47 / 43.70 | 32.58 / 22.35 | 32.16 / 21.29 | 36.68 / 33.16 |
| + SPA-VL | 55.58 / 77.34 | 28.92 / 45.05 | 32.56 / 51.74 | 39.85 / 52.87 | 29.47 / 35.58 | 37.28 / 52.52 |
| + SaFeR-VLM | 51.69 / 58.26 | 58.96 / 78.18 | 54.78 / 73.33 | 61.13 / 61.13 | 49.38 / 47.85 | 55.19 / 63.75 |
| + SFT (Stage II) | 50.00 / 75.45 | 57.83 / 80.51 | 41.96 / 61.01 | 51.32 / 65.66 | 50.46 / 51.68 | 50.31 / 66.86 |
| + SAFER-STEER (Ours 7B) | 53.44 / 75.57 | 77.04 / 86.87 | 65.96 / 72.95 | 66.67 / 70.93 | 61.36 / 55.42 | 64.89 / 72.35 |
Ablation Study¶
Table 3: Ablation Analysis of Data Granularity and GRPO Components (3B Backbone)
| Experimental Setting | Configuration / Variant | STEER-DyS (Safe / Help) | Average (Safe / Help) | Note |
|---|---|---|---|---|
| Full Architecture | SAFER-STEER (Full) | 49.40 / 55.40 | 55.58 / 70.27 | Complete three-stage system |
| Data Granularity | Single-turn SFT | 16.32 / 30.08 | 21.77 / 41.92 | SFT on single-turn data only |
| Single-turn SFT + GRPO | 16.80 / 29.76 | 17.14 / 32.39 | Single-turn RL collapses on multi-turn | |
| Multi-turn SFT (Stage II) | 35.17 / 38.84 | 41.61 / 57.57 | Multi-turn supervised foundation | |
| Multi-turn SFT + GRPO (Full) | 49.22 / 55.27 | 55.54 / 70.24 | Combined multi-turn supervision & RL | |
| RL Components | w/o Synthetic Bootstrapping | 23.24 / 34.11 | 24.48 / 42.44 | -31.10 Safe; exploration collapses without cold-start |
| w/o TCSR Reward | 42.79 / 49.92 | 51.09 / 65.91 | -4.49 Safe; fails to penalize weakest turn | |
| w/o Feedback Dynamics | 43.11 / 46.31 | 51.09 / 63.96 | -4.49 Safe; static attacks underfit evolving weaknesses |
Key Findings¶
- Static Alignment Suffers Severe Multi-Turn Collapse: Static baselines achieve impressive single-turn numbers (e.g., TIS-7B hits 82.51 Safe, SPA-VL-7B reaches 77.14), but collapse by more than 40% when subjected to interactive multi-turn dialogues (plunging to 36.68 and 37.28, respectively), accompanied by steep drops in helpfulness. This confirms that static tuning induces fragile, prompt-local reflexes rather than robust conversational defenses.
- Multi-Turn Alignment Demonstrates Bidirectional Generalization: SAFER-STEER sets new state-of-the-art standards on multi-turn benchmarks (55.58 for 3B, 64.89 for 7B), while concurrently elevating single-turn safety to 81.84 and 87.89. Training across long-horizon interactive dynamics establishes an invariant defensive policy that readily transfers to isolated static prompts.
- Synthetic Bootstrapping is Vital for Safe RL Exploration: Omitting Stage II synthetic bootstrapping produces the most catastrophic performance collapse (Average Safety falls from 55.58 to 24.48). Without curated multi-turn demonstrations, reinforcement learning flails in complex conversational state spaces, unable to discover effective defensive equilibriums.
- Survival Analysis Shows Substantial Horizon Extension: Kaplan-Meier survival curves demonstrate that base models enter the high-risk failure zone (\(S(t) < 0.5\)) as early as turns 2–4 on adversarial benchmarks. SAFER-STEER pushes the survival horizon outward, slashing final-turn failure rates from 55.65% to 2.55% on 3B and from 44.31% to 3.37% on 7B.
Highlights & Insights¶
- Active In-the-Loop Red Teaming: Transforming the reinforcement learning reward model from a passive scalar annotator into an adversarial tutor actively probing discovered vulnerabilities (via Persistence, Bait-and-Switch, and Deepen strategies) directly bridges the gap between training and real-world attack dynamics.
- Bottleneck-Aware TCSR Formulation: Integrating the historical minimum turn reward alongside the trajectory average ingeniously counteracts reward-hacking dynamics in long dialogues, enforcing a zero-tolerance policy against conversational breaches without stifling helpfulness.
- Constructive Refusal Optimization: The model learns to pair firm refusals with educational, safe alternatives and guiding questions, ensuring that dialogues do not reach dead-ends while maintaining rigorous safety boundaries.
Limitations & Future Work¶
- Backbone Model Exploration: While primary validations span Qwen2.5-VL 3B and 7B (with transfer checks on 32B and Qwen3-VL), broader experiments across distinct architectural families (such as closed-source proprietary APIs and diverse vision encoders) remain to be explored.
- Reliance on Frontier Teacher Judges: The synthetic generation, data filtering, and automated evaluation pipelines rely on proprietary frontier LLMs (GPT-4.1, GPT-5.2, Gemini-3-Pro), which may introduce minor prompt sensitivities or rubric biases.
- Training Compute Overhead: Dynamic multi-turn rollouts involving interactive tutor generation introduce non-trivial computational costs relative to standard offline SFT, highlighting on-policy efficiency optimization as an important future direction.
Related Work & Insights¶
- vs. Static Multimodal Alignment (TIS, SPA-VL, SaFeR-VLM): Conventional frameworks optimize preferences on isolated image-question pairs, creating models that easily succumb to multi-turn context accumulation and crescendo jailbreaks; SAFER-STEER establishes a holistic dynamic alignment paradigm that safeguards extended conversational trajectories.
- vs. Offline Multi-Turn Benchmarks (SafeMT, MTMCS-Bench): Prior multi-turn safety resources primarily construct fixed script datasets without reactive adversaries; SAFER-STEER introduces on-policy feedback dynamics during GRPO, dynamically matching the attack distribution to the agent's real-time defensive posture.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the first closed-loop multi-turn MLLM alignment framework featuring an interactive Safety Tutor and trajectory-consistent reward modeling.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates across 5 static benchmarks, 5 multi-turn datasets in STEER-BENCH, external SafeMT validation, and rigorous Kaplan-Meier survival modeling.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid problem formulation, comprehensive methodology descriptions, and coherent narrative structure linking theoretical tension to empirical verification.
- Value: ⭐⭐⭐⭐⭐ Provides an indispensable blueprint, training dataset suite, and algorithmic standard for safely deploying multimodal conversational agents in production.