M-QUEST - Meme Question-Understanding Evaluation on Semantics and Toxicity¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/vulr/M-QUEST
Area: Social Computing
Keywords: Meme Toxicity, Compositional Benchmark, Semiotic Projection, Multimodal Understanding, Pragmatic Reasoning
TL;DR¶
This paper introduces a comprehensive semantic framework consolidating 10 cognitive and semantic dimensions for meme interpretation and builds M-QUEST, a compositional benchmark of 1,109 multiple-choice Q&A pairs across 307 memes, systematically exposing the performance gap in state-of-the-art vision-language models between shallow multimodal perception and deep pragmatic reasoning for toxicity attribution.
Background & Motivation¶
Internet memes have emerged as a dominant communicative currency across public and private online spheres, distinguished by their multimodal economy and hyper-signification where cultural codes and visual tropes condense intricate sociopolitical ideas into shareable units. While typically celebrated as humorous or ironic, memes are increasingly weaponized for malicious ends—yielding harmful or hateful manifestations collectively termed "toxic memes." Their velocity, resilience over time, and reliance on unstated cultural common sense make automated toxicity detection critical yet formidable for content moderation systems.
Despite significant attention, existing computational efforts suffer from two primary limitations. First, prior benchmarks approach meme toxicity predominantly as black-box classification tasks, treating the meme as a material artifact rather than a cultural object that dynamically connects observers, background knowledge, and social context. Where individual semantic dimensions (such as optical character recognition, object detection, or target community identification) have been explored, they remain fragmented and ad-hoc across disparate studies without a unifying theoretical grounding. Second, existing evaluations fail to disentangle surface-level toxicity detection from underlying reasoning. A model may correctly output a toxic label by exploiting superficial visual artifacts or lexical triggers without genuinely grasping the relational mechanisms that produce harm.
To resolve these tensions and bridge the gap between classification and explanation, this paper grounds meme interpretation in cognitive semiotics and formal knowledge representation. Core idea: formalize meme interpretation into 10 structured cognitive and semantic dimensions (introducing Semiotic Projection and unified Analogical Mapping) and build M-QUEST, a compositional benchmark where toxicity questions are strictly paired with underlying causal reasoning questions to assess the pragmatic inference boundaries of vision-language models.
Method¶
Overall Architecture¶
The M-QUEST framework and benchmark creation pipeline follow a modular, component-agnostic, semi-automatic procedure comprising dimension formalization, sequential knowledge extraction, candidate Q&A generation, and human validation. Given an input meme, a vision-language model systematically extracts knowledge per dimension across 10 defined categories according to a strict dependency topology, compiling a structured Knowledge Base (KB) for each meme. A subsequent Q&A generation module uses the accumulated KB entries to produce four-choice questions featuring one correct answer, one plausible distractor, one implausible distractor, and a fixed "None of the others" option. Finally, human judges rigorously inspect and filter the generated KBs and question pairs for plausibility and alignment.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Meme Image and Text"] --> B["Staged Knowledge Extraction<br/>Extract 10-dimensional facts to compile structured KB"]
B --> C["Compositional Q&A Generation<br/>Synthesize paired toxicity and attribution multiple-choice questions"]
C --> D["Expert Human Validation<br/>Multi-criteria double-blind filtering on KBs and Q&A pairs"]
D --> E["M-QUEST Evaluation Benchmark<br/>M-QUEST-qwen and M-QUEST-gemini subsets"]
Key Designs¶
1. Ten-Dimensional Semantic Formalization of Meme Cognition: Structuring Meaning Construction To overcome the fragmented focus of prior work, the framework formalizes 10 essential semantic dimensions spanning perception, contextual grounding, and higher-order pragmatics. The perceptual foundation comprises Textual Material (captions, overlays, and embedded diegetic text) and Visual Material (people, objects, characters, and visual entities integrated into a unified KB). Contextual grounding encompasses Scene (spatial configuration and visual entity relational interactions), Background Knowledge (unobservable cultural references, historical events, public personas, and stereotypes), and Emotion (affective language, facial expressions, and emotional gestures). Higher-order pragmatics comprise Overall Intent (creator purpose, intended effects, and communicative goals), Target Community (demographic cues, audience signals, and targeted protected groups), Analogical Mapping (unifying structural mappings such as \(X \text{ is } Y\) across modalities inspired by structure-mapping and conceptual metaphor theories), and Semiotic Projection (a novel dimension operationalizing the semiotic triangle among visual entities, narrative roles, and the observer's self-projection, such as Schadenfreude in the Disaster Girl trope). Finally, Toxicity is operationalized across six subtypes (abuse, cyberbullying, harmfulness, hatefulness, misogyny, propaganda) and grounded in the other nine dimensions.
2. Sequential Context-Cascaded Knowledge Extraction: Enforcing Semantic Coherence and Mitigating Hallucinations To ensure consistency across dimensions and prevent cross-dimensional hallucination, knowledge extraction proceeds sequentially rather than in parallel. Extraction executes in 10 ordered steps: Overall Intent is extracted first to establish communicative context; Textual Material and Visual Material follow, conditioned on the intent graph; Scene extraction subsequently takes both intent and visual entities as context; Background Knowledge and Emotion then ingest all preceding entities; Analogical Mapping and Semiotic Projection receive cumulative context across text, visual, scene, emotion, and background entities; finally, Toxicity Assessment and Target Community ingest the fully realized graph. By feeding previously extracted structured facts into downstream prompts, high-level pragmatic inferences remain firmly anchored to verified perceptual primitives.
3. Paired Compositional Question-Answer Generation: Testing Toxicity Attribution Consistency Moving beyond uncoupled multiple-choice evaluation, M-QUEST pairs each toxicity question with a corresponding reasoning question derived from the underlying semantic dimension that justifies the toxicity label. Candidate generation produces balanced four-choice questions accompanied by concise reasoning explanations. To establish a rigorous assessment of model reliability, the benchmark introduces Group Accuracy (\(T+R\)): a model receives credit for a meme if and only if it correctly answers both the toxicity classification question (\(T\)) and the associated semantic reasoning question (\(R\)). This metric directly penalizes models that arrive at correct toxicity judgments via spurious correlations or superficial cues without genuine pragmatic understanding.
4. Multi-Stage Human Validation Protocol: Securing High Annotation Quality and Consensus To filter out noisy automated generations and ensure reliable ground truth, 14 human judges familiar with LLMs, knowledge graphs, and internet culture evaluated candidate KBs and Q&A items across six rigorous criteria: plausibility of the toxicity judgment, plausibility of the linked semantic dimension entry, pertinence of the toxicity question and correctness of its designated answer, and pertinence and correctness of the paired reasoning question. Validation rates were highest on foundational perceptual dimensions (74%–76% for Scene, Textual, and Visual Material) and lowest for cognitively demanding dimensions (32% for Semiotic Projection and 40% for Target Community), reflecting their subjective, culturally nuanced nature. Filtering yielded 307 fully validated memes with 1,109 Q&As, instantiated across Qwen3-VL-8B-Instruct (M-QUEST-qwen) and Gemini3.6-Flash (M-QUEST-gemini) subsets.
Key Experimental Results¶
Main Results¶
Twelve vision-language models spanning four capability tiers—perception-focused, instruction-tuned, reasoning-oriented, instruction-tuned with reasoning, and proprietary commercial models—were evaluated zero-shot with greedy decoding on M-QUEST-qwen. Metrics include overall Q&A accuracy (All), paired group accuracy (T+R), pure toxicity accuracy (T), pure reasoning accuracy (R), and unweighted Macro-Average:
| Model Tier | Model | All Accuracy (%) | Group Accuracy (T+R, %) | Toxicity Only (T, %) | Reasoning Only (R, %) | Macro Avg (%) |
|---|---|---|---|---|---|---|
| Perception-focused | BLIP2-Flan-T5-xl | 13.2 | 12.3 | 14.7 | 14.7 | 14.3 |
| Perception-focused | Pixtral-12B | 70.3 | 60.6 | 72.0 | 54.7 | 57.4 |
| Instruction-tuned | InstructBLIP-Vicuna-7B | 22.9 | 18.7 | 22.8 | 17.8 | 20.6 |
| Instruction-tuned | FastVLM-1.5B | 32.5 | 14.0 | 31.1 | 37.7 | 34.3 |
| Instruction-tuned | FastVLM-7B | 44.0 | 19.9 | 35.1 | 53.0 | 48.6 |
| Reasoning-oriented | LLaVA-v1.6-Vicuna-7B | 47.3 | 46.8 | 50.2 | 40.2 | 41.5 |
| Reasoning-oriented | LLaVA-v1.5-7B | 56.8 | 53.4 | 54.7 | 70.4 | 68.3 |
| Instruction-tuned + reasoning | Qwen2-VL-7B | 85.6 | 83.3 | 84.4 | 94.7 | 93.3 |
| Instruction-tuned + reasoning | Qwen2.5-VL-7B | 86.4 | 82.3 | 86.6 | 94.7 | 93.8 |
| Instruction-tuned + reasoning | Qwen3-VL-8B | 83.8 | 78.3 | 80.1 | 95.3 | 94.3 |
| Proprietary | Gemini-3.5-Flash-Lite | 83.1 | 71.3 | 79.5 | 86.8 | 84.9 |
| Proprietary | Claude-Opus-5 | 90.5 | 88.0 | 90.6 | 90.3 | 90.3 |
| Human baseline | Human annotation | 82.8 | 78.3 | 87.0 | 79.2 | 82.9 |
Cross-Subset & Generation Backbone Evaluation¶
Evaluating the same models on M-QUEST-gemini demonstrates how the generating model affects item difficulty while preserving overall model capability hierarchies:
| Model Tier | Representative Model | All Accuracy (%) | Group Accuracy (T+R, %) | Toxicity Only (T, %) | Reasoning Only (R, %) | Macro Avg (%) |
|---|---|---|---|---|---|---|
| Perception-focused | BLIP2-Flan-T5-xl | 82.4 | 70.7 | 94.0 | 81.1 | 82.4 |
| Perception-focused | Pixtral-12B | 91.8 | 84.5 | 96.0 | 91.3 | 91.8 |
| Instruction-tuned | InstructBLIP-Vicuna-7B | 36.0 | 14.2 | 42.0 | 35.3 | 36.0 |
| Reasoning-oriented | LLaVA-1.5-7B | 45.4 | 26.7 | 38.0 | 46.2 | 45.4 |
| Instruction-tuned + reasoning | Qwen3-VL-8B-Instruct | 95.2 | 90.5 | 98.0 | 94.9 | 95.2 |
| Proprietary | Claude-Opus-5 | 98.8 | 97.4 | 100.0 | 98.7 | 98.8 |
| Proprietary | Gemini-3.5-Flash-Lite | 94.8 | 89.7 | 100.0 | 94.2 | 94.8 |
| Human baseline | Human annotation | 89.0 | 87.8 | 100.0 | 87.8 | 89.0 |
Key Findings¶
- Perceptual Saturation vs. Pragmatic Drop: While advanced models approach ceiling accuracy on surface perceptual dimensions (Scene, Visual Material, Textual Material), performance drops sharply on latent relational dimensions. Overall Intent emerges as the universally most challenging dimension: Claude-Opus-5 drops to 96% and Gemini-3.5-Flash-Lite to 82% (their lowest individual dimension scores), while weaker models collapse entirely (InstructBLIP at 20%, LLaVA-1.5 at 26%).
- Breakdown of Compositional Consistency: Group accuracy (\(T+R\)) is consistently lower than isolated toxicity or reasoning accuracy across all model families. Weak models exhibit severe drops (FastVLM-1.5B drops from 37.7% on reasoning to 14.0% on group accuracy), underscoring that models frequently fail to coordinate factual extraction with toxicity judgment within a unified multimodal reasoning chain.
- Primacy of Multimodal Alignment and Reasoning over Parameter Scale: Combining instruction tuning with multi-step reasoning capabilities yields the strongest results (Qwen family reaching 83.8%–86.4% on M-QUEST-qwen), decisively outperforming larger or perception-only backbones. Understanding toxic memes is inherently reasoning-intensive; models lacking explicit reasoning mechanisms cannot resolve metaphoric mappings or implicit cultural stereotypes.
Highlights & Insights¶
- Operationalizing Cognitive Semiotics: Successfully translates abstract concepts from cognitive metaphor theory and narrative semiotics into measurable benchmark dimensions, notably operationalizing Semiotic Projection and multi-entity Analogical Mapping.
- Context-Cascaded Extraction Paradigm: Establishes a reproducible dependency topology for automated knowledge graph compilation from complex multimodal inputs, preserving coherence across hierarchical semantic layers.
- Compositional Scaling Without Ad-Hoc Complexity: Outlines two concrete strategies for scaling benchmark difficulty—Co-location questions (integrating independent signals) and Interaction questions (reasoning across entangled, mutually dependent dimensions)—leveraging orthogonal dimension combinations rather than arbitrary difficulty heuristics.
Limitations & Future Work¶
- Dataset Scale and Cultural Breadth: M-QUEST currently contains 1,109 questions over 307 English memes, limiting coverage of multilingual, regional, and non-Western cultural contexts where memes exhibit different symbolic grammars.
- Demographics of Human Evaluators: The 14 annotators are European researchers whose judgments may reflect distinct cultural biases, especially regarding American political discourse and localized idioms.
- Structural Template Omission: The current 10-dimensional framework derives bottom-up from literature and omits explicit meme layout structures (e.g., N-panel progression, visual dialogue templates), which warrant future structural integration.
Related Work & Insights¶
- vs Hateful Memes (Kiela et al., 2020): Treats meme toxicity as an uninterpretable end-to-end binary classification problem; M-QUEST formally decomposes meme meaning into 10 semantic dimensions and links toxicity directly to explanatory reasoning.
- vs MemeQA (Nguyen et al., 2025): Evaluates 7 dimensions via isolated questions; M-QUEST expands coverage to 10 dimensions (incorporating Semiotic Projection and structured analogy) and introduces compositional question pairing with Group Accuracy (\(T+R\)).
- vs Disarm / Constraint 2022 (Sharma et al., 2022a,b): Examines isolated narrative roles (Hero/Villain/Victim) or target identification; M-QUEST formalizes the complete semiotic triangle connecting observers, roles, and visual entities into a unified relational knowledge graph.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First framework to formalize semiotic projection alongside 9 other semantic dimensions into a paired, compositional meme benchmark.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive zero-shot evaluation across 12 diverse open and closed VLMs, backed by two distinct generator subsets and human baselines.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear, theoretically grounded, well-structured exposition with detailed qualitative case studies and comprehensive statistical reporting.
- Value: ⭐⭐⭐⭐⭐ Provides critical benchmark infrastructure and conceptual tools for moving multimodal toxicity detection from black-box heuristics toward interpretable, reason-grounded content moderation.