I Know What You Meme, Even If it Emerged Today: Understanding Evolving Memes through Open-World Knowledge Acquisition¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/shanhongliuu/Query-Retrieve-Conclude
Area: LLM Reasoning
Keywords: Meme Understanding, Open-World Knowledge Acquisition, Multimodal Reasoning, Zero-Shot Learning, Meme Detection
TL;DR¶
This paper proposes Query-Retrieve-Conclude, a zero-shot framework that identifies missing knowledge gaps, retrieves external open-web evidence, and synthesizes evidence-grounded background knowledge statements, overcoming the failure of closed parametric VLMs on fast-evolving memes and lifting downstream detection Macro F1 to 0.71.
Background & Motivation¶
Internet multimodal memes are inherently dynamic, culturally situated, and temporally evolving artifacts of modern digital communication. Interpreting their humor, sarcasm, or underlying intent routinely demands up-to-date background knowledge regarding recent political events, breaking news, viral cultural templates, or emerging public controversies. For instance, a meme pairing Donald Trump and Confucius with the caption "CON-FUSE-US" operates on both a phonetic pun and real-world geopolitical developments—specifically the 2026 blockade of the Strait of Hormuz and the aftermath of the failed Islamabad Talks. Existing multimodal meme understanding approaches built on vision-language models (VLMs) fundamentally treat memes as closed multimodal reasoning problems, focusing on aligning image regions with embedded text or detecting cross-modal incongruity under static supervised regimes.
This closed-world paradigm breaks down when confronted with emerging memes whose required context is missing, outdated, or absent from pre-trained parametric weights. Foundation models trained on static corpora suffer severe knowledge cutoffs and hallucinations when forced to interpret newly spawned internet phenomena. Prior attempts incorporating external knowledge graphs (such as ConceptNet or Hatebase) remain constrained by static symbolic schemas that fail to track fast-moving social discourse. Meanwhile, contemporary zero-shot baselines employing multi-agent debate (e.g., MemeAgent) or instance-level retrieval from past meme pools (e.g., MiND) lack live grounding; without factual external evidence, internal debate cycles merely amplify parametric hallucinations and drive models into severe "semantic anchoring traps"—such as falsely anchoring any mention of Bill Clinton to 1990s White House scandals rather than contemporary real-world events.
To resolve this limitation, multimodal systems must mirror human epistemology: when encountering an unfamiliar meme, a human does not blindly guess its implicit intent, but first pinpoints unfamiliar entities or phrases, queries external sources, and synthesizes the retrieved evidence before drawing conclusions. Models must decouple knowledge acquisition from downstream classification by explicitly formulating what is missing before attempting an explanation. Core idea: reframe evolving meme understanding as an open-world knowledge acquisition problem, introducing the zero-shot Query-Retrieve-Conclude framework that formulates search-oriented questions via visual reverse grounding, retrieves live open-web evidence, and synthesizes declarative background knowledge statements to solidly ground downstream multimodal reasoning and abuse detection.
Method¶
Overall Architecture¶
The Query-Retrieve-Conclude framework decomposes meme interpretation into three structured, feedforward stages: Stage I (Query, knowledge gap identification and query planning), Stage II (Retrieve, open-web evidence retrieval and grounded QA), and Stage III (Conclude, background knowledge statement synthesis and decision verification). Given a meme image \(I\) and embedded text \(T\), Stage I performs reverse image search (RIS) to obtain visual context \(RI = \text{RIS}(I)\), generates a literal descriptive caption \(c\), and formulates targeted search questions \(Q\). Stage II retrieves live web passages via BM25 ranking and source filtering, producing evidence-grounded answers for each question. Stage III compiles these QA pairs into compact, declarative background knowledge statements \(\hat{B}\), which serve as contextual evidence for downstream verifiers predicting classification labels \(\hat{y}\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Meme<br/>Image I + Embedded Text T"] --> B["Stage I: Knowledge Gap Identification and Query Planning<br/>RIS-assisted literal caption c and targeted question set Q"]
B --> C["Stage II: Open-Web Evidence Retrieval and Grounded QA<br/>BM25-ranked top-5 passage retrieval and evidence-bounded QA"]
C --> D["Stage III: Declarative Knowledge Statement Synthesis & Verification<br/>Compile QA pairs into BKS to support downstream inference"]
D --> E["Downstream Task Prediction<br/>Hatefulness / Misogyny / Offensiveness / Sarcasm / Harmfulness"]
Key Designs¶
1. Knowledge Gap Identification and Query Planning: Decoupling visual grounding from intent inference
To prevent models from prematurely fabricating intent explanations out of parametric memory, Stage I explicitly identifies what knowledge is missing before inferring meaning. Given the meme image \(I\) and text \(T\), the pipeline first queries Google reverse image search to retrieve visually associated web context \(RI = \text{RIS}(I)\), including page titles and text snippets. Next, an image captioner uses \(RI\) strictly as auxiliary visual grounding to resolve ambiguous entities, symbols, or public figures, generating an objective, literal caption \(c\) without speculating on social implications. Conditioned on \(I\), \(T\), \(c\), and recognized entities, a question generation module formulates a compact set of search-oriented questions \(Q = \{q_1, \dots, q_N\} = Q_{\theta_q}(I, T, c, RI)\) targeting specific entities, idioms, or historical events. This design converts open-ended interpretation into a structured external search plan, eliminating ungrounded speculation at the source.
2. Open-Web Evidence Retrieval and Grounded QA: Enforcing live retrieval and evidence-bounded answering
To overcome the knowledge cutoff inherent in closed models, Stage II dynamically accesses the open web at inference time. For each question \(q_i \in Q\), a web retrieval module searches the internet, ranks candidate passages via BM25, and retains only the top-5 most relevant chunks \(\epsilon_i\), while filtering out commercial shopping and advertising domains. Next, an answer generator synthesizes an evidence-grounded answer \(\hat{a}_i = A_{\theta_a}(c, q_i, \epsilon_i)\) utilizing both the visual caption \(c\) and retrieved snippets \(\epsilon_i\). Crucially, the answer generator operates under strict prompt constraints: it is forbidden from invoking parametric assumptions or external speculation, and must ground its answer exclusively within the provided document chunks. If documents support only a partial answer, it outputs only that verified portion, preventing hallucinated context from polluting downstream reasoning.
3. Declarative Knowledge Statement Synthesis & Verification: Converting discrete QA pairs into interpretable context
To prevent raw conversational question-answer pairs from introducing token noise or structural clutter into downstream discriminators, Stage III compiles the QA set \(P = \{(q_i, \hat{a}_i)\}_{i=1}^N\) into self-contained, declarative Background Knowledge Statements (BKS) \(\hat{B} = \{\hat{b}_1, \dots, \hat{b}_M\} = S_{\theta_s}(P)\). Each statement encapsulates an atomic real-world fact relevant to the meme. In the downstream detection phase, a multimodal verifier \(V_{\theta_v}\) receives the meme image \(I\), text \(T\), and synthesized background statements \(\hat{B}\) as explicit context to predict task labels \(\hat{y}(s) = V_{\theta_v}(I, T, \hat{B})\) for any target task \(s \in S\). By conditioning on explicit BKS, downstream classification is firmly anchored in factual evidence rather than implicit multimodal correlations, ensuring high human-aligned Decision Support and robust generalization to subtle sarcasm or disguised hate.
Loss & Training¶
The proposed framework is a pure zero-shot, training-free inference framework. It requires no task-specific supervised fine-tuning or parameter updates on downstream meme detection or knowledge induction datasets (gradient updates count is zero), thereby involving no model loss backpropagation. Its operational protocol is established through structured multi-stage prompt engineering and strict decoding constraints:
- Stage I (Query Formulation Protocol): Prompts the VLM (Qwen3-VL-32B) as a "Knowledge Gap Analyst", enforcing strict JSON formatting (
question_types_usedandquestionslist). The prompt strictly restricts the model from prematurely guessing meme intent, requiring it only to formulate concrete questions about ambiguous visual entities, slang phrases, or public events. - Stage II (Retrieve Constraint Protocol): Online web retrieval ranks extracted snippets using BM25, hard-filtering down to top-5 chunks. The answer generator prompt enforces closed-evidence answering: "Use ONLY the provided retrieved documents. Do NOT use outside knowledge, memory, assumptions, or unsupported claims; if documents support only a partial answer, output only the verified partial answer."
- Stage III (Conclude & Verification Protocol): The statement synthesis module compiles raw QA pairs into atomic declarative facts; downstream detection prompts standardize input by appending the synthesized BKS context, requiring lightweight VLMs (e.g., Gemma-3-12B, LLaVA-1.5) to output a single character
1(yes) or0(no).
Key Experimental Results¶
Main Results¶
The table below reports detection Accuracy (Acc.) and Macro F1 scores across the five downstream benchmark tasks using the Gemma-3-12B detector backbone:
| Method Configuration | Hatefulness (F1) | Misogyny (F1) | Offensiveness (F1) | Sarcasm (F1) | Harmfulness (F1) | Overall Acc. | Overall F1 |
|---|---|---|---|---|---|---|---|
| Vanilla Zero-shot (w/o BKS) | 0.67 | 0.72 | 0.61 | 0.62 | 0.64 | 0.67 | 0.65 |
| MemeAgent (Multi-agent debate) | 0.63 | 0.49 | 0.56 | 0.65 | 0.58 | 0.62 | 0.59 |
| MiND (Instance retrieval) | 0.65 | 0.68 | 0.59 | 0.61 | 0.62 | 0.65 | 0.63 |
| Zero-shot Generated BKS | 0.66 | 0.74 | 0.64 | 0.64 | 0.66 | 0.68 | 0.67 |
| Ours (Query-Retrieve-Conclude) | 0.70 | 0.79 | 0.67 | 0.68 | 0.70 | 0.71 | 0.71 |
On background knowledge recovery, Qwen3-VL-32B without retrieval achieves only 0.46 Evidence Recall on the curated 2024–2026 KYM benchmark, whereas Query-Retrieve-Conclude achieves 0.78 (+32% absolute gain). Consistent improvements appear on MemeIntent (0.66 \(\to\) 0.76) and MemeInterpret (0.73 \(\to\) 0.79).
Ablation Study¶
Ablation experiments isolating structural modules using the primary Qwen3-VL-32B backbone on representative tasks:
| Configuration | Hatefulness Acc. | Hatefulness F1 | Sarcasm Acc. | Sarcasm F1 | Overall Acc. | Overall F1 |
|---|---|---|---|---|---|---|
| Ours (Full Framework) | 0.75 | 0.74 | 0.72 | 0.71 | 0.74 | 0.74 |
| w/o Stage I (Direct Retrieval via raw tokens) | 0.69 | 0.66 | 0.67 | 0.65 | 0.69 | 0.67 |
| w/o Stage II (Parametric Only, no live search) | 0.66 | 0.62 | 0.65 | 0.62 | 0.66 | 0.62 |
| w/o Stage III (Raw QA fed directly to classifier) | 0.72 | 0.71 | 0.69 | 0.68 | 0.71 | 0.70 |
| w/o Visual Grounding Context (\(c\) & \(RI\)) | 0.65 | 0.61 | 0.63 | 0.60 | 0.64 | 0.61 |
Key Findings¶
- Parametric knowledge cannot substitute for open-world retrieval (Stage II is the primary bottleneck): Disabling Stage II and relying purely on internal 32B weights drops overall F1 by 0.12 (Hatefulness F1 drops to 0.62). Even advanced foundation models lack encoded representations for volatile 2024–2026 events.
- Query formulation protects against semantic distortion (Stage I is critical): Bypassing question generation and scraping web snippets using surface tokens degrades overall F1 from 0.74 to 0.67. Sarcastic memes decouple literal text from real intent; only structured gap queries correctly uncover underlying facts.
- Unchecked multi-agent debate amplifies hallucinations: MemeAgent drops overall F1 to 0.59 (below vanilla 0.65), showing that multi-agent deliberation without grounded factual inputs compounds parametric errors.
- Statement synthesis enhances decision support: Feeding uncurated QA fragments (w/o Stage III) costs 0.04 in overall F1. In human evaluations, synthesized BKS attained a Decision Support rating of 4.28 / 5.0 (\(\pm 0.64\)), verifying that declarative statements provide high-fidelity causal grounding.
Highlights & Insights¶
- Epistemological framing of multimodal reasoning: The framework reframes meme interpretation from superficial cross-modal alignment into an active knowledge acquisition cycle, forcing models to formulate what they do not know before inferring intent.
- Bypassing semantic anchoring traps: In a qualitative case involving Chelsea Clinton's wedding photo, zero-shot baselines hallucinated a 1990s Monica Lewinsky scandal due to parametric co-occurrence weights, whereas Query-Retrieve-Conclude generated targeted queries about the circled guest, correctly uncovering Ghislaine Maxwell and Epstein-related conspiracies.
- Model-agnostic zero-shot transfer: The synthesized knowledge statements function as clean, plug-and-play contextual prompts, consistently boosting accuracy across 7B, 8B, 12B, and 32B multimodal backbones without requiring task-specific parameter fine-tuning.
- Knowledge-gap question planning outperforms broad web scraping: Ablation proves that querying the open web with literal meme text introduces irrelevant noise, whereas formulating questions that target specific knowledge gaps forces the retrieval pipeline to isolate the exact empirical facts needed.
Limitations & Future Work¶
- Dependence on external search fidelity: System accuracy remains bounded by the quality of commercial search APIs and reverse image search; biased or noisy search results can cascade into incorrect knowledge statements.
- Fixed inference latency: Executing full web retrieval for every meme incurs an end-to-end latency of ~22–32 seconds per instance; future work should explore dynamic routing to decide when parametric memory suffices versus when live search is required.
- Monolingual scope: Evaluations focus predominantly on English-language memes and Western sociopolitical events; expanding to multilingual, culturally localized subcultures remains an open frontier.
- Exposure to controversial and biased web content: Open-world web search risks retrieving politically biased or toxic commentaries; future iterations should incorporate sensitive content filtering and uncertainty calibration.
Related Work & Insights¶
- vs. PromptHate / Pro-Cap (Cao et al., 2022, 2023): Prior methods leverage frozen VLMs to extract implicit entity prompts and surface attributes; this paper connects models dynamically to live web evidence, addressing post-pretraining temporal shifts.
- vs. MemeIntent / MemeInterpret (Park et al., 2024, 2025): While these benchmarks highlight background knowledge, they rely on supervised fine-tuning or direct ungrounded generation; this paper demonstrates that structured zero-shot retrieval achieves higher evidence recall without parameter updates.
- vs. MemeAgent (Lin et al., 2025) & MiND (Liu et al., 2025): Multi-agent debate and past-instance retrieval fail to inject unencoded external facts; Query-Retrieve-Conclude establishes an objective evidence loop that halts parametric hallucination.
- vs. Static Knowledge Graph Approaches (Tripathi et al., 2026): Although prior methods leverage ConceptNet or Hatebase, static knowledge bases suffer from update lag and cannot cover real-time events emerging in 2024–2026; open-web retrieval provides inherent temporal adaptivity.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an innovative zero-shot open-world knowledge acquisition paradigm that directly tackles temporal distribution shifts in evolving memes.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates across 3 comprehension datasets and 5 downstream detection benchmarks, supplemented by a newly annotated 2024–2026 KYM benchmark and extensive human-alignment audits.
- Writing Quality: ⭐⭐⭐⭐⭐ Clearly articulated epistemological motivation, rigorous error analysis, and rich appendices providing full prompt templates and runtime breakdowns.
- Value: ⭐⭐⭐⭐⭐ Offers a practical, explainable solution for real-world social abuse detection and multimodal reasoning over volatile online cultural media.