Skip to content

Trait Collapse in Text-to-Image Generation: Evaluating and Mitigating Personality Inconsistency in Visual Storytelling

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Boiyaaa/ViStoryBench-Personality
Area: Multimodal VLM
Keywords: Visual Storytelling, Trait Collapse, Personality Consistency, Text-to-Image Evaluation, Prompt Engineering

TL;DR

Addressing the "trait collapse" failure mode where visual storytelling models preserve character appearance but collapse behavioral patterns into uniform emotional defaults, this paper introduces the ViStoryBench-Personality benchmark with a six-dimensional metric suite and proposes the training-free Trait-Behavior-Visual (TBV) prompting framework, improving Personality Alignment Score by up to +22.1%.

Background & Motivation

In visual narrative generation and sequential text-to-image synthesis, creating believable characters requires not only maintaining stable facial identities and clothing across changing scenes, but also ensuring that characters behave in ways consistent with their underlying psychological traits. However, current personalization approaches (such as DreamBooth and PhotoMaker) and story-level generation methods (such as StoryDiffusion) focus almost entirely on what a character looks like, paying virtually no attention to how a character acts. In practice, a model frequently preserves facial identity across multiple frames while repeatedly producing generic smiles, rigid postures, or passive emotional reactions dictated purely by the scene context, flattening nuanced individual personalities into an indistinct behavioral baseline.

This pervasive failure mode is termed trait collapse, which fundamentally arises from a semantic grounding gap: modern diffusion architectures receive abstract psychological descriptors such as "friendly" or "reserved" in prompts, but lack the grounding priors needed to translate them into observable, context-stable physical cues like posture, gaze direction, expressive micro-movements, or interpersonal distance. Compounding this challenge, conventional evaluation metrics are completely blind to personality: CLIP Score fails to improve and often drops when personality descriptors are introduced, while DINO cross-frame appearance similarity changes only negligibly (mean signed change of +0.014). Moreover, emotionally intense narrative scenes trigger an "Emotion Override Effect," where transient scene affect completely suppresses stable trait expression.

To break the assumption that identity consistency implies character consistency, psychological personality traits must be systematically decoupled from surface visual features, establishing dedicated diagnostic and mitigation frameworks for visual personality consistency. Core idea: decompose abstract Big Five (OCEAN) traits into context-sensitive behavioral tendencies via an inverse Brunswik Lens Model, map them into renderable physical constraints via an expert-validated Visual Cues Lexicon (TBV), and diagnose character behavior using a dedicated six-dimensional evaluation benchmark without requiring model retraining.

Method

Overall Architecture

This paper establishes a comprehensive diagnostic and mitigation pipeline for visual personality inconsistency: first, building the diagnostic benchmark ViStoryBench-Personality grounded in the Big Five (OCEAN) model and the Brunswik Lens Model, supported by an expert-validated 150-cue Visual Cues Lexicon and a six-dimensional evaluation metric suite; second, proposing the training-free Trait-Behavior-Visual (TBV) prompting framework.

TBV systematically decomposes the abstract-to-visual personality translation into three cascading phases: 1. Phase T (Trait Anchoring): Establishes a stable, self-consistent OCEAN trait profile from character description and narrative context using multi-sample consensus and debate arbitration; 2. Phase B (Behavioral Deduction): Derives context-sensitive behavioral reasoning traces for the current narrative scene, enforced by a self-correcting reverse inference critic loop to avoid emotional overriding; 3. Phase V (Vision Mapping): Maps the deduced behaviors into conflict-resolved, renderable visual constraints using tier filtering and priority suppression from the Visual Cues Lexicon.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Character Profile & Narrative Scene Context"] --> B["Phase T: Trait Anchoring<br/>Multi-sample self-consistency & debate routing"]
    B --> C["Phase B: Behavioral Deduction<br/>Context-sensitive trace & reverse-inference loop"]
    C --> D["Phase V: Vision Mapping<br/>Lexicon tier retrieval & priority conflict suppression"]
    D --> E["Output: Final C4 prompt with injected visual constraints"]

Key Designs

1. Phase T: Trait Anchoring and Multi-Sample Debate Routing To eliminate the inherent stochastic variance and hallucination of LLM-based personality extraction, Phase T establishes an unbiased, continuous Big Five anchor vector \(\mathbf{v} \in [1, 7]^5\) covering Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. The system samples \(k=5\) independent inferences per character and quantifies inter-sample consensus using the Normalized Disagreement Index (NDI): $$ \text{NDI} = \frac{1}{5k} \sum_{d \in \mathcal{D}} \sum_{i=1}^k \frac{|v_{i, d} - \bar{v}_d|}{U - L} $$ where \(\mathcal{D} = \{O, C, E, A, N\}\), \(U=7, L=1\), and \(\bar{v}_d\) is the mean inferred score along dimension \(d\). A three-tier routing mechanism is applied: if \(\text{NDI} < 0.15\), inferences show high agreement and the mean vector \(\bar{\mathbf{v}}\) is accepted as anchor \(\hat{\mathbf{v}}\); if \(0.15 \le \text{NDI} \le 0.33\), a two-round multi-agent debate is triggered with an independent critic to reconcile divergent dimensions; if \(\text{NDI} > 0.33\), the narrative evidence is deemed excessively ambiguous and falls back to a population-norm baseline. This design prevents unstable personality anchors from propagating downstream.

2. Phase B: Context-Sensitive Behavioral Deduction with Self-Correcting Loop When generating images for dramatic or emotionally charged narrative moments, models frequently default to generic affective reactions that erase character individuality. Phase B takes the anchored profile \(\hat{\mathbf{v}}\) and scene description to generate a narrative reasoning trace \(r\) detailing how the character acts in context. To prevent out-of-character drift, this phase integrates a self-correcting loop inspired by iterative refinement: An independent critic performs blind reverse inference on \(r\) to reconstruct the implied OCEAN profile \(\tilde{\mathbf{v}} = \text{Inf}(r)\), calculating the behavioral deviation score \(s\): $$ s = \frac{1}{5} \sum_{d \in \mathcal{D}} \frac{|\hat{v}_d - \tilde{v}_d|}{U - L} $$ Setting a tolerance threshold \(\tau = 0.15\) (equivalent to an average per-dimension deviation of at most 0.9 points on the 7-point scale), any trace with \(s > \tau\) is rejected as emotion-dominated or out-of-character, triggering regeneration with corrective feedback (up to 3 iterations before falling back to the best-scoring candidate). This guarantees that behavioral expressions stay faithful to long-term personality traits despite strong ambient affect.

3. Phase V: Lens-Inverted Vision Mapping with Priority Conflict Suppression While the Brunswik Lens Model describes how human observers infer personality from visible cues, Phase V inverts this process to map deduced behaviors into concrete visual constraints. The authors developed and validated with 18 psychology experts a 150-cue Visual Cues Lexicon \(\mathcal{L}_{\text{visual}}\) covering 10 trait poles across five visual signal types (facial expression, gaze behavior, body posture, clothing/color palette, scene environment). Cues are organized into three activation tiers (Core for \(\ge 5\) or \(\le 3\), Extended for \(\ge 6\) or \(\le 2\), Fine-grained for \(=7\) or \(=1\)), scaling visual constraint density with trait extremity. To eliminate semantic collisions (e.g., simultaneous clenched teeth from high Neuroticism and gentle smiles from high Agreeableness), Phase V applies a priority suppression hierarchy: $\(\text{Facial Expression \& Gaze Cues} \succ \text{Body Posture Cues} \succ \text{Clothing \& Scene Environment Signals}\)$ This hierarchy reflects psychological findings that facial channels provide the primary cues for personality perception. The filtered set of 4โ€“6 non-conflicting visual cues \(C_{\text{visual}}\) is composed and injected directly into the T2I prompt as condition C4.

4. Metric Suite: Six-Dimensional Centered Cosine Evaluation To quantify visual personality expression, the benchmark implements an automated VLM judge suite based on GPT-4o with structured JSON output (validated against human annotators with 86.6% polarity agreement): - Personality Alignment Score (PAS): Measures alignment between the VLM-inferred profile \(\hat{\mathbf{v}}\) and the target profile \(\mathbf{v}\) via centered cosine similarity, eliminating scale midpoint bias: $$ \text{PAS} = \frac{(\hat{\mathbf{v}} - \mathbf{m}) \cdot (\mathbf{v} - \mathbf{m})}{|\hat{\mathbf{v}} - \mathbf{m}|_2 |\mathbf{v} - \mathbf{m}|_2}, \quad \mathbf{m} = [4, 4, 4, 4, 4]^\top $$ - Personality Consistency Score (PCS): Measures stochastic stability under repeated generations across 3 random seeds; - Readability Score (RS): Evaluates personality inferability purely from visual image cues, completely withholding textual personality labels and plot descriptors from the judge; - Character Consistency Score (CCS): Computes pairwise centered cosine similarity across sequential shots depicting the same character; - Personality Drift Score (PDS): Measures the slope of personality distance across normalized sequence time to detect progressive trait erosion; - Story Coherence Score (SCS): Balances trait preservation and situational logic via \(\text{SCS} = 0.4 \cdot \text{PAS} + 0.6 \cdot \text{SCS}_{\text{reasoning}}\).

Loss & Training

TBV is an entirely training-free, plug-and-play prompting intervention that requires no parameter updates or gradient backpropagation. All LLM operations in Phases T, B, and V utilize GPT-4o (temperature \(T=0.2\)) with structured JSON schemas, while image synthesis is performed directly using frozen off-the-shelf diffusion or flow matching architectures.

Key Experimental Results

Main Results

The authors evaluated seven representative T2I systems: four open-source foundational models spanning UNet, DiT, MMDiT, and LLM-conditioned architectures (SDXL, FLUX.1-dev, SD3.5-Large, Qwen-Image-2512); one narrative-focused visual consistency model (StoryDiffusion); and two commercial closed-source APIs (GPT Image 2, Gemini 3.1 Flash Image Preview / Nano Banana). Performance was evaluated under baseline (C1) and TBV (C4) conditions.

Model Group / Model Name PAS โ†‘ (C1) PAS โ†‘ (C4) Relative PAS Gain RS โ†‘ (C1) RS โ†‘ (C4) CCS โ†‘ (C1) CCS โ†‘ (C4)
Open-source: SDXL 0.446 0.467 +4.7% 0.429 0.423 0.357 0.406
Open-source: StoryDiffusion 0.480 0.502 +4.6% 0.467 0.508 0.499 0.527
Open-source: SD3.5-Large 0.466 0.472 +1.3% 0.493 0.488 0.478 0.449
Open-source: FLUX.1-dev 0.464 0.522 +12.5% 0.469 0.552 0.385 0.511
Open-source: Qwen-Image-2512 0.518 0.633 +22.1% 0.546 0.641 0.456 0.558
Closed-source: GPT Image 2 0.538 0.648 +20.4% 0.556 0.659 0.504 0.606
Closed-source: Nano Banana 0.542 0.638 +17.7% 0.551 0.661 0.459 0.505

Note: Conventional CLIP Score actually decreased under C4 prompting (dropping on average from 0.267 to 0.247, signed change -0.020), while DINO cross-frame appearance similarity changed only marginally (+0.014), illustrating that classical metrics are blind to behavioral personality grounding.

Ablation Study

To verify whether trait collapse stems from a lack of abstract personality vocabulary versus a lack of concrete visual grounding, the paper evaluated four incremental prompt conditions across the five open-source models (n=213): - C1: Base narrative prompt (plot description + character appearance); - C2: C1 + abstract OCEAN trait tags (e.g., High Openness, Low Conscientiousness); - C3: C2 + deduced scene-level emotion labels (e.g., feeling Disappointment); - C4: C1 + explicit TBV-derived visual constraints.

Model C1 PAS C2 PAS C3 PAS C4 PAS (Full TBV) C4 vs C1 Gain
SDXL 0.446 0.392 0.408 0.467 +4.8%
StoryDiffusion 0.480 0.462 0.471 0.502 +4.6%
SD3.5-Large 0.466 0.452 0.460 0.472 +1.5%
FLUX.1-dev 0.464 0.483 0.484 0.522 +12.6%
Qwen-Image-2512 0.518 0.545 0.540 0.633 +22.1%

Paired Wilcoxon signed-rank and TOST equivalence tests (\(\delta = 0.05\)) confirm that 9 out of 10 C1 vs C2/C3 comparisons show no statistically significant PAS gain (e.g., adding abstract tags to SDXL degraded PAS from 0.446 to 0.392). Only concrete visual grounding under C4 generates substantial and statistically significant improvements on models with strong text encoders (e.g., Qwen-Image-2512: \(p < 0.001, r = 0.43\)).

Key Findings

  • Language-Conditioning Pathway as the Bottleneck: TBV gains are strongly modulated by text encoder capacity. Models conditioned on rich LLM embeddings (Qwen-Image-2512 at +22.1%) or advanced proprietary language models (GPT Image 2 at +20.4%) achieve dramatic improvements, whereas models relying solely on CLIP encoders (SDXL at +4.7%) or exhibiting prompt-detail smoothing (SD3.5-Large at +1.3%) exhibit weak response.
  • Visual Appearance Consistency \(\ne\) Psychological Consistency: StoryDiffusion attains the highest baseline consistency (PCS = 0.763), proving its cross-frame self-attention successfully stabilizes visual identity, yet its baseline PAS remains mediocre (0.480) and increases only to 0.502 under TBV (+4.6%). Identity preservation mechanisms do not inherently safeguard behavioral trait fidelity.
  • Emotion Override Effect: One-way ANOVA reveals a highly significant impact of affective scene context on personality alignment (\(F(4, 5076) = 20.06, p < 0.001\)). Neutral scenes yield the highest alignment (\(\text{PAS} = 0.575\)), while emotionally salient contexts (such as high-arousal negative at 0.477 and low-arousal positive at 0.446) cause severe alignment degradation as generic scene affect overrides stable dispositional cues.
  • Cross-Judge Robustness (Gemini 2.5 Flash): Re-evaluating 80 paired story shots with an independent judge from an entirely different model family (Gemini 2.5 Flash) reproduces the exact same directional gains, with Qwen-Image-2512 retaining statistically significant improvement (\(\Delta = +0.073, p = 0.021\)). Furthermore, Readability Score (RS), which blinds the judge to any personality prompt text, shows even larger gains (+0.11 on Qwen, +0.10 on FLUX), refuting any keyword-recognition artifacts.

Highlights & Insights

  • Conceptual Clarification: Decouples character identity consistency from character personality consistency, formalizing the previously unaddressed phenomenon of "trait collapse" in generative storytelling.
  • Brunswik Lens Inversion: Translates classical psychological observation theory into an operational AI prompting framework, curating 150 expert-validated cues across five visual dimensions with strict tiering and conflict-resolution rules.
  • Critical Evaluation Insight: Demonstrates that existing metric infrastructure (CLIP Score, DINO similarity) is completely blind to psychological semantics, pioneering structured VLM-based psychological profiling for visual generative models.

Limitations & Future Work

  • Single Focal Character Scope: The current benchmark focuses on individual character behavior per frame, without explicitly modeling relational dynamics like dominance, social avoidance, or interpersonal adaptation between co-occurring characters.
  • Cultural Specificity: The Visual Cues Lexicon is built upon the Western-centric Big Five model; certain non-verbal cues (e.g., eye contact, interpersonal distance) may possess culturally divergent interpretations in high-context societies.
  • Generative Architecture Bottleneck: The benefits of TBV plateau on models with limited text encoder capacity or those that smooth out fine-grained textual constraints during diffusion sampling.
  • Future Directions: Developing diffusion architectures with native personality conditioning latent spaces to internalize behavioral priors directly into the generative process.
  • vs StoryDiffusion / TaleDiffusion: Existing story generation models deploy cross-frame attention to lock in visual identity (faces, clothing), but rely naively on scene prompts for actions, causing behavioral homogenization; this paper provides an orthogonal behavioral grounding mechanism.
  • vs ControlNet / HumanDreamer: Controllable generation frameworks address low-level pose and motion synthesis, but cannot decide what pose is psychologically appropriate for a given personality in context; TBV provides the high-level semantic bridge.
  • vs EmotiCrafter: Emotion-conditioned generation models manipulate immediate affective states (valence/arousal), whereas this work models enduring personality dispositions that persist across shifting emotional states, overcoming emotional overriding.

Rating

  • Novelty: โญโญโญโญโญ First systematic identification and formalization of trait collapse, bridging psychological theory with visual generative modeling.
  • Experimental Thoroughness: โญโญโญโญโญ 7 models, over 15k generated images, human agreement validation, dual-judge robustness tests, and rigorous non-parametric statistics.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear conceptual framing, structured methodology, and rigorous empirical validation.
  • Value: โญโญโญโญโญ Critical guidance for character-centric storytelling, game asset design, and embodied autonomous agent visualization.