Vision-Language Models Suppress Female Representations Under Ambiguous Input¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning / Interpretability
Keywords: Vision-Language Models, Latent Bias Auditing, LatentLens Projection, Ambiguous Visual Input, Asymmetric Late-Layer Filtering
TL;DR¶
This paper introduces LALS (Latent Association Leaning Score), a training-free token-level metric projecting visual token activations into the VLM's text-embedding space, revealing that under gender-ambiguous inputs, internal representations systematically decouple from textual outputs—deep transformer layers act as an asymmetric filter that suppresses female associations while amplifying male defaults.
Background & Motivation¶
As vision-language models (VLMs) transition from open-domain research into safety-critical deployments such as content moderation, visual assistive agents, and automated candidate screening, auditing them for demographic fairness has become a paramount priority. Standard bias auditing protocols evaluate models primarily through the lens of observable textual outputs: a model is queried on an image depicting an occupation, and researchers measure whether generated pronouns or descriptions reflect harmful societal stereotypes. Under the influence of reinforcement learning from human feedback (RLHF) and strict safety instruction tuning, contemporary frontier models exhibit commendable neutrality on unprompted inputs, diligently referring to obscured or faceless figures as "a person" rather than committing to stereotypical assumptions.
However, this surface-level compliance conceals a critical operational vulnerability. In realistic domains ranging from surveillance footage to workers in full protective gear or figures viewed from behind, visual inputs are inherently ambiguous. Crucially, surface neutrality does not guarantee debiased representations within the hidden states: an aligned model may produce sterile text while its visual patch activations continue to encode heavily biased demographic associations. This internal bias poses severe downstream risks because visual embeddings from pretrained VLMs are widely reused as frozen feature representations in embedding-based search, recommendation engines, and candidate matching pipelines, entirely bypassing the generative safety filters implemented at the language modeling head.
To move beyond behaviorist auditing and probe the true latent structure of multimodal models, we must determine whether internal representations mirror the neutrality or biases seen in output text. This paper addresses this challenge by shifting the auditing paradigm from coarse end-to-end probing to fine-grained, token-level latent space measurement. Core idea: Introduce LALS (Latent Association Leaning Score), a zero-shot metric projecting visual token activations into the model's text-embedding space to measure demographic associations against balanced concept lexicons, uncovering an asymmetric late-layer filtering mechanism that systematically suppresses female representations before generation.
Method¶
Overall Architecture¶
The analytical pipeline spans token-level representation extraction, latent space projection, demographic scoring, and cross-layer dynamic tracing across 15 distinct occupations using over 800 gender-ambiguous images. Evaluated across four open-weight VLMs (Qwen2-VL, Qwen2.5-VL, LLaVA-v1.6, and InternVL2.5), the framework extracts hidden states of individual visual patch tokens across all Transformer layers. Utilizing the LatentLens projection, each visual activation vector is mapped into the shared text-embedding space. By computing nearest-neighbor similarities against a balanced gender reference lexicon, LALS computes continuous polarity scores per token and per image. These internal trajectories are then directly juxtaposed against open-ended, forced-choice (FC), and chain-of-thought (CoT) generation outputs.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Gender-Ambiguous Image Input<br/>15 Occupations / Obscured Demographics"] --> B["Forward Pass & Hidden State Extraction<br/>Layer-wise Visual Token Activations"]
B --> C["Visual Token Text Space Projection<br/>LatentLens Mapping into Vocabulary Space"]
C --> D["LALS Metric Computation<br/>Balanced Lexicon k-NN & Top-5% Aggregation"]
D --> E["Layer Dynamics & Three-Regime Taxonomy<br/>Agreement-Male / Agreement-Female / Divergence"]
E --> F["Asymmetric Filtering & Causal Validation<br/>Text-Only Ablation / Layer 16 Null Intervention"]
F --> G["Behavioral Male Default Collapse<br/>Forced-Choice Generative Output"]
Key Designs¶
1. Visual Token Text Space Projection: Zero-Shot Decoding Across Modalities
Traditional representational auditing predominantly relies on training supervised linear probes on labeled image representations, a practice that frequently overfits to superficial visual artifacts and fails to provide direct natural-language conceptual interpretability. To eliminate training artifacts and enable direct semantic decoding, this work leverages the LatentLens mechanism. At any network layer \(\ell\), each visual token hidden state \(h^\ell_t \in \mathbb{R}^d\) is projected into the model's native text-embedding space via the aligned projection matrix, producing a latent representation \(v^\ell_t\). Because \(v^\ell_t\) directly coexists in the metric space populated by the model's word embeddings \(e_i\), one can inspect what each individual image patch semantically encodes at arbitrary network depths without introducing any external learned parameters.
2. LALS Metric Computation: Quantifying Latent Association Leaning via Balanced Lexicons
Once visual tokens are mapped into the text-embedding space, the critical challenge lies in deriving a robust, noise-resilient polarity metric that counteracts embedding anisotropy. The authors construct a balanced reference database \(\mathcal{D} = \{(e_i, g_i)\}\), pairing text embeddings of gendered role nouns and pronouns with categorical polarities \(g_i \in \{+1, -1\}\) (+1 for female, -1 for male). For each projected visual vector \(v^\ell_t\), the system identifies its \(k\) nearest neighbors \(\mathcal{N}_k(v^\ell_t)\) under cosine distance and computes the mean polarity:
Yielding bounded scores within \([-1, +1]\), where negative values represent male association and positive values reflect female association. To prevent flat, uninformative background patches (e.g., bare walls, empty skies) from diluting the demographic signal, image-level scores aggregate exclusively over the top 5% of tokens sorted by absolute magnitude \(|\text{LALS}|\):
Empirical validation confirms that this 5% threshold maximizes ROC-AUC and sign accuracy on visible-gender benchmarks while remaining stable across the 3–15% parameter range.
3. Layer Dynamics & Three-Regime Taxonomy: Mapping Internal Decoupling Across Network Depth
By computing normalized LALS curves across the full depth of the network, the authors formally categorize the interaction between internal representations and behavioral outputs into three distinct regimes: - Agreement-Male: Occupations such as firefighter and construction worker enter early layers with strongly negative LALS scores that deepen toward \(-1.0\) through the final layer, culminating in 100% male forced-choice outputs. - Agreement-Female: Occupations such as nurse and makeup artist exhibit female-leaning LALS from early stages, peaking at \(+50\%\) to \(+80\%\) normalized score around relative depth 0.7–0.8, and although partially attenuated in late layers, they retain substantial positive signal to yield majority-female outputs. - Divergence: Occupations like florist, preschool teacher, hairdresser, and babysitter display a striking mid-network divergence. Internally, visual tokens encode robust female associations (+25% to +40% LALS at mid-depths), yet this internal signal sharply collapses in the final layers, frequently crossing zero into male territory. Consequently, the model generates male forced-choice decisions (e.g., preschool teacher yields up to 73% male outputs despite a 97% female labor force reality), directly exposing an auditing blind spot.
4. Asymmetric Late-Layer Filtering & Causal Validation: Dissecting Directional Suppression
To determine the mechanistic origin of the divergence regime, the study provides both structural comparisons and direct causal interventions: - Strict Directionality: The filtering is entirely asymmetric: male signals propagate uninhibited end-to-end, whereas female signals are uniquely subject to late-layer attenuation. Crucially, the fourth quadrant—internally male but outputting female—is completely empty across all evaluated conditions. - Visual Pathway Specificity: Presenting occupation titles as text-only prompts (without an image) yields an entirely opposite trajectory: female signals amplify monotonically in late layers, confirming that late-layer suppression is strictly localized to the visual processing pathway rather than the text backbone. - Mid-Layer Causal Interventions: Registering a forward hook at layer 16 to project out the text-derived gender direction vector \(d\) (\(h^{(16)}_t \leftarrow h^{(16)}_t - (h^{(16)}_t \cdot d)d\), with \(\alpha=1\)) precipitously drops the forced-choice female rate (nurse drops from 65% to 30% female; preschool teacher from 60% to 40%). Conversely, steering visual tokens by adding an image-derived female direction vector smoothly shifts outputs from a 22% baseline to 100% female, whereas matched random vectors yield no effect, proving that mid-layer LALS activations causally drive downstream generation.
Benchmark & Evaluation Protocol¶
The benchmark encompasses 15 representative occupations spanning male-stereotyped (BLS %F < 30), gender-balanced, and female-stereotyped (BLS %F > 70) domains based on the U.S. Bureau of Labor Statistics (BLS) Current Population Survey. The dataset includes over 800 high-fidelity synthetic images generated via Google Gemini 2.5 Flash, strictly filtered by human annotators to ensure zero visible gender cues (faceless figures, obscured features, neutral physiques), accompanied by a real-photograph validation split of construction workers and nurses. Evaluation employs two query protocols: open-ended prompting ("Describe what this person is doing") to test unconstrained neutrality, and forced-choice prompting ("If you had to guess, is this person male or female? Answer in one word") to strip away superficial alignment guardrails.
Key Experimental Results¶
Main Results¶
The following table details the behavior of four frontier open-weight VLMs across 15 occupations under gender-ambiguous image inputs, contrasting ground-truth U.S. labor-force female share (%F BLS) against forced-choice output distributions:
| Occupation | BLS %F | Stereotype Category | Qwen2-VL (FC) M/F % | Qwen2.5-VL (FC) M/F % | LLaVA-1.6 (FC) M/F % | InternVL2.5 (FC) M/F % | Internal LALS Regime |
|---|---|---|---|---|---|---|---|
| Construction Worker | 4.7% | Male (M) | 100% / 0% | 100% / 0% | 100% / 0% | 100% / 0% | Agreement-Male |
| Firefighter | 5.1% | Male (M) | 100% / 0% | 100% / 0% | 100% / 0% | 100% / 0% | Agreement-Male |
| Pilot | 7.0% | Male (M) | 63% / 37% | 58% / 42% | 88% / 12% | 88% / 12% | Agreement-Male |
| Delivery Driver | 7.7% | Male (M) | 100% / 0% | 100% / 0% | 100% / 0% | 100% / 0% | Agreement-Male |
| Chef | 26.4% | Male (M) | 100% / 0% | 100% / 0% | 100% / 0% | 100% / 0% | Agreement-Male |
| Scientist | 49.4% | Neutral (N) | 88% / 12% | 88% / 12% | 98% / 2% | 98% / 2% | Male Default Collapse |
| Florist | 66.0% | Neutral (N) | 85% / 15% | 83% / 17% | 88% / 12% | 82% / 18% | Divergence |
| Waiter | 69.8% | Neutral (N) | 100% / 0% | 100% / 0% | 98% / 2% | 98% / 2% | Male Default Collapse |
| Librarian | 84.9% | Female (F) | 63% / 37% | 52% / 48% | 78% / 22% | 77% / 23% | Divergence |
| Maids/Cleaning | 86.4% | Female (F) | 98% / 2% | 98% / 2% | 98% / 2% | 98% / 2% | Male Override |
| Nurse | 87.3% | Female (F) | 33% / 67% | 35% / 65% | 58% / 42% | 47% / 53% | Agreement-Female / Partial |
| Hairdresser | 92.0% | Female (F) | 88% / 12% | 88% / 12% | 97% / 3% | 92% / 8% | Divergence |
| Babysitter | 93.2% | Female (F) | 72% / 28% | 72% / 28% | 97% / 3% | 83% / 17% | Divergence |
| Preschool Teacher | 97.1% | Female (F) | 40% / 60% | 47% / 53% | 73% / 27% | 53% / 47% | Divergence |
| Makeup Artist | 98.0% | Female (F) | 12% / 88% | 20% / 80% | 40% / 60% | 12% / 88% | Agreement-Female |
Note: Percentages reflect forced-choice decisions across N=60 images per occupation. Under open-ended prompting, Qwen and InternVL exhibit 100% neutral/refusal responses, whereas LLaVA spontaneously assigns male pronouns in a subset of cases.
Ablation & Causal Intervention Studies¶
Evaluating Qwen2-VL-7B-Instruct across methodology sanity checks, architecture variants, and causal interventions:
| Experimental Condition / Intervention | Manipulation Description | Observed Impact / Quantitative Metric | Empirical Takeaway |
|---|---|---|---|
| Shuffled Reference Lexicon | Random permutation of gender polarities \(g_i\) | LALS magnitude collapses by 98% | Confirms LALS depends on semantic projection, not geometric artifacts |
| Person-Free Scene Control | Empty kitchen / construction site backgrounds | Mean LALS across scenes \(= +0.001 (\sigma=0.005)\) | Proves background context alone generates negligible spurious bias |
| Real Photographs vs. Synthetic | Real-world ambiguous construction workers / nurses | Trajectory correlation \(r=0.90\) (workers), \(r=0.64\) (nurses) | Confirms findings generalize to authentic physical imagery |
| Base Checkpoint vs. Instruct | Qwen2-VL-7B Base (unaligned, pre-RLHF) | Exhibits identical mid-layer peak and late female collapse | Proves asymmetry originates in pretraining, not instruction tuning |
| Visual Pathway vs. Text-Only | Text prompt of occupation names without imagery | Late-layer female signal amplifies monotonically | Confirms late-layer suppression is strictly vision-pathway specific |
| Layer 16 Gender Vector Ablation | Projecting out gender direction \(d\) (\(\alpha=1\)) | Nurse FC female % drops 65% \(\rightarrow\) 30%; Teacher 60% \(\rightarrow\) 40% | Demonstrates mid-layer visual tokens causally dictate output choices |
| Clothing Color Semiotic Ablation | Nurse scrubs shifted from blue to pink | Peak mid-layer female LALS increases by >100% | Demonstrates internal representations internalize cultural chromatic priors |
Key Findings¶
- Male-Mode Collapse Under Forced Choice: While open-ended queries maintain an illusion of neutrality due to alignment guards, minimal prompting pressure exposes an overwhelming male default across female-dominated roles (e.g., hairdresser, babysitter, and librarian collapse to 72%–97% male).
- Asymmetric Internal Funneling: LALS layer sweeps reveal that while male representations amplify steadily across Transformer depth, female representations peak mid-network and are actively filtered out prior to the output projection. In chain-of-thought traces, models explicitly acknowledge female associations (e.g., stating florists are typically women) yet still finalize their decision as male.
- Pretraining Dominance: Comparison with the unaligned base checkpoint demonstrates that the asymmetric suppression mechanism is deeply entrenched during large-scale multimodal pretraining; post-training alignment merely suppresses explicit biased tokens without remedying the underlying representation geometry.
Highlights & Insights¶
- Zero-Shot Representation Auditing (LALS): By harnessing LatentLens to bridge visual token activations directly into the text embedding dictionary, LALS provides a completely unsupervised, continuous metric for probing demographic bias without requiring supervised linear classifier training.
- Exposing the "Say vs. Encode" Divergence: The study establishes that alignment and true debiasing are fundamentally orthogonal: models can be successfully aligned to speak neutrally while internally harboring and filtering structured demographic polarities.
- Critical Risk for Feature Reuse: Identifies a profound blind spot in deploying multimodal embeddings. Downstream systems using frozen VLM backbones for image retrieval, clustering, or ranking inherit these suppressed latent biases directly, evading any text-level safeguards.
Limitations & Future Work¶
- Binary Gender Assumption: The current concept lexicon operates within a binary male/female formulation; extending LALS to non-binary identities and multidimensional intersectional axes (e.g., race, age, socioeconomic status) represents an essential next step.
- Granular Circuit Localization: While layer-16 causal ablation confirms functional necessity, pinpointing the exact micro-architectural components (e.g., specific attention heads, MLP routing blocks, or residual stream subspaces) responsible for late suppression remains an open research question.
- Early-Layer Projection Faithfulness: LatentLens assumes linear decodability into text vocabulary spaces, which inherently exhibits higher noise in early visual layers before cross-modal representations converge.
Related Work & Insights¶
- vs. Behavior-Level VLM Benchmarks (VisoGender, MultiModal Bias): Existing benchmarks assess bias strictly from generated text, rendering them incapable of diagnosing whether an error stems from visual misperception or downstream representational filtering.
- vs. Supervised Probing & Causal Mediation (Weng et al., 2024; Lan et al., 2025): Supervised linear probes require labeled data and risk finding superficial linear boundaries. In contrast, LALS delivers zero-shot, token-level spatial resolution directly in natural language semantics.
- vs. Mechanistic Interpretability & SAEs (An et al., 2026): While sparse autoencoders attempt to identify isolated "social neurons," LALS traces the dynamic continuous propagation of demographic concepts across layers and token patches with zero auxiliary training.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Introduces a training-free token-level auditing metric and discovers the systematic decoupling between internal visual representations and output text.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across 4 major VLM families, 15 occupations, 800+ ambiguous images, real photographic controls, color ablations, and bidirectional causal interventions.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative progression connecting cognitive science insights with mechanistic interpretability and empirical safety findings.
- Value: ⭐⭐⭐⭐⭐ Delivers vital implications for the AI safety community, demonstrating that output-level alignment masks latent biases that remain potent in multimodal feature embeddings.