Skip to content

Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking

Conference: EMNLP 2026
Paper: ACL Anthology
Code: Helsinki-NLP/shroom-visions
Area: Multimodal VLM
Keywords: Multimodal Hallucination Benchmarking, Human-Crafted Samples, Fine-Grained Span Annotation, Vision-Language Models, Benchmark Obsolescence

TL;DR

To overcome the severe model-dependence, sparse representation of rare error types, and rapid obsolescence inherent in modern LVLM hallucination evaluation, this paper presents SHEEP—a multilingual benchmark spanning 20,000 samples across four languages (Chinese, English, French, Italian) with character-level annotations over 5 hallucination categories, empirically demonstrating that human-written hallucinations provide higher inter-annotator agreement, finer category controllability, and faithful ranking proxies for model-generated outputs.

Background & Motivation

Large vision-and-language models (LVLMs) demonstrate exceptional conversational and reasoning capabilities across visual question answering and scene understanding, yet their propensity to generate text that is visually ungrounded or contradicts the input image—termed visual hallucination—remains a major barrier to real-world deployment. Prevailing evaluation benchmarks predominantly rely on generations collected from a handful of popular contemporary models or utilize synthetic pipelines prompted by foundation models. This standard paradigm suffers from a double bind: first, detection models evaluated on these datasets inevitably overfit to the idiosyncratic linguistic styles and error artifacts of the underlying generator, making it impossible to disentangle general hallucination comprehension from source-model artifact memorization; second, driven by the frantic pace of modern model iteration, benchmarks compiled around current systems face near-instant obsolescence as new model architectures and alignment techniques render existing benchmarks unrepresentative.

Beyond this vulnerability to rapid depreciation, empirical data acquisition for hallucination benchmarking presents steep structural hurdles. Randomly sampling responses from foundation models faithfully reflects raw generation distributions, but critical visual failure modes—such as misreading fine-grained visible scene text (OCR errors) or incorrectly tallying object counts (miscounting)—are notoriously sparse in random output pools, severely handicapping statistical evaluation on these vital long-tail failure types. Conversely, employing an LLM-as-a-judge to actively pre-select candidate erroneous samples introduces substantial false-positive and false-negative judge biases, inadvertently skewing the benchmark's natural error distribution and injecting unmodeled systematic noise.

This paper tackles these core dilemmas by asking whether human-invented hallucinations can serve as a principled, model-independent foundation for multimodal hallucination evaluation. Constructing hallucinated instances manually not only enables exact experimental control over the relative frequencies of arbitrary error types, but also completely decouples benchmark longevity from the transient lifespans of specific generative models. Core idea: by constructing SHEEP, a fine-grained, character-level multilingual dataset of 1,600 human-written and 18,400 model-generated samples across four languages, this work empirically proves that human-crafted hallucinations yield superior inter-annotator consensus, mirror the distributional shift between distinct model families, and faithfully preserve detector performance rankings across generative backbones.

Method

Overall Architecture

The SHEEP (Set of Human-written and Electronic Erroneous Productions) pipeline establishes an open-ended VQA hallucination benchmark spanning Chinese (ZH), English (EN), French (FR), and Italian (IT). The architecture centers on an unambiguous, 5-class visual grounding taxonomy, acquiring data across three parallel sampling branches (Random generation, Model-assisted preselection, and Human manual construction), unified via an automated translation and post-editing pipeline, and culminating in dense triple-annotator character-span localization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>HaloQuest & VISaGE Non-synthetic Images"] --> B["1. Compact Fine-Grained Grounding Taxonomy<br/>Formalize Labels A-E & Verification Rules"]
    B --> C["2. Tri-Pronged Sampling Regimes<br/>Random Output / Judge MAPS / Human Generation"]
    C --> D["3. Hybrid Translation & Post-Editing Pipeline<br/>NLLB-200 & Qwen3 Across 4 Languages"]
    D --> E["4. Multi-Annotator Span Protocol<br/>12 C1/Native Annotators per Lang & 3-Fold Overlap"]
    E --> F["SHEEP 20,000 Verified Benchmark & Model Probing"]

Key Designs

1. Compact Fine-Grained Grounding Taxonomy: Balancing Granularity and Annotation Reliability Prior taxonomies either oversimplify hallucination into coarse object/attribute/relation triads or proliferate into over 8 hyper-specific categories that introduce unresolvable annotator confusion. SHEEP formulates a mutually exclusive 5-label error taxonomy alongside an inferred null category: - A. Invention: Mentioning an entity, object, property, or event entirely absent from the image. - B. Mischaracterization / Identity Incongruity: Referencing an entity visually present, but misrepresenting its type, color, spatial orientation, posture, or action. - C. Wrong Reading / OCR Problem: Falsifying, misquoting, or distorting readable text visibly present in the image (if no text exists, classified as A). - D. Numeric Discrepancy / Miscounting: Stating an erroneous quantity of visible entities (if a number is misread from printed text, classified as C). - E. Other: Residual hallucinations not covered by A–D, restricted to exceptional edge cases requiring explicit annotator justification. - F. No Hallucination: Semantically coherent and fully grounded output, inferred from the complete absence of highlighted error spans. To resolve grounding ambiguity, statements containing external ungrounded facts not verifiable from the visual context are strictly classified as hallucinations, while linguistic hedging and uncertainty markers (e.g., "it appears to be") are left unflagged to avoid penalizing calibrated uncertainty.

2. Tri-Pronged Sampling Regimes: Comparing Passive Sampling, Judge Pre-Selection, and Human Formulation To systematically assess the validity of human-written errors as proxies for model outputs, three data collection streams were executed over HaloQuest and VISaGE non-synthetic visual question-answering pairs: - Random Sampling: Five representative LVLMs (Gemma3-27B-IT, InternVL3-8B, MiniCPM-V-4.5-8B, Llava-NeXT-8B, Qwen3-VL-8B) generated responses under sampling temperature \(\tau = 0.7\) with maximum length 512 across 5 random seeds. 460 samples were uniformly sampled per language-model pair, yielding 2,300 random samples per language (9,200 total). - Model-Assisted Pre-Selection (MAPS): Gemma3-27B-IT acted as a zero-shot multi-label judge over the candidate pool. Because the judge rarely identified OCR problems and miscounting, MAPS greedily retained 100% of all judge-predicted C and D items, subsampling remaining items to balance A and B, producing an enriched set of 2,300 samples per language (9,200 total). - Human-Written Samples: Trained human writers proficient in each target language were provided 100 images and instructed to author plausible, fluent answers imitating LLM stylistic mannerisms from scratch. Each authored sample was required to incorporate exactly one deliberate error from categories A, B, C, or D, enforcing an exact 25% target balance per category.

3. Hybrid Translation & Post-Editing Pipeline: Multilingual Preservation of Visual Errors To expand the human-crafted core across all four target languages without distorting planted grounding discrepancies, a multi-engine translation strategy was implemented: - For English and Romance languages (EN \(\leftrightarrow\) FR, EN \(\leftrightarrow\) IT, FR \(\leftrightarrow\) IT), the seq2seq model NLLB-200 3.3B was deployed using beam search (width 4) and forced target BOS tokens. - For all translation paths involving Chinese (\(\leftrightarrow\) ZH), the instruction-tuned Qwen3-8B model was utilized with strict system prompts prohibiting extraneous commentary, reasoning, or answer modification. - Because NLLB exhibited a pathology of dropping content following the first sentence, all 1,200 machine-translated human samples underwent mandatory native post-editing. COMETKiwi quality estimation confirmed systematic quality score improvements across all language directions (e.g., EN \(\to\) IT COMETKiwi rose from 0.817 to 0.851; EN \(\to\) FR increased from 0.799 to 0.849).

4. Multi-Annotator Span Protocol: Conservative Span Demarcation and Free-Marginal Agreement Annotation was conducted by cohorts of 12 C1-to-native university students per language, with each annotator evaluating approximately 1,250 items over a dedicated web interface (Django/Bootstrap/Gunicorn), guaranteeing 3-fold independent coverage across all 20,000 instances. - Conservative Span Constraint: Annotators were instructed to highlight the minimal contiguous character span requiring modification or deletion to render the response factually true, prioritizing lexical content words over functional syntax and strictly avoiding full-sentence selections. - Free-Marginal Krippendorff's Alpha: Because hallucination prevalence varies by design across random, MAPS, and human sampling streams, standard fixed-marginal agreement metrics produce artificial distortions. The study adopted a free-marginal formulation: $\(\alpha_{\text{free}} = 1 - \frac{k}{n} \sum_{v, v'} o_{v v'} \delta(v, v')\)$ where \(k=2\) denotes binary presence, \(n\) is the total paired judgment count, \(o_{v v'}\) represents the coincidence matrix, and \(\delta\) is the nominal distance metric. - Span-Level Intersection-over-Union (IoU): Pairwise span overlap was quantified via mean character IoU, both ignoring labels and requiring exact category agreement within the intersecting span.

Key Experimental Results

Main Results

The benchmark assesses how modern hallucination detection methods perform across different generation sources, comparing the internal-state probing detector HalluShift++ (trained via teacher-forced forward passes through Llava-NeXT) and black-box LLM-judge evaluations with Gemma3-27B-IT. Crucially, the diagnostic fidelity of human-written data was validated by computing the Pearson correlation coefficient \(r\) across 10 distinct LVLM judges (varying from 2B to 38B across Gemma3, InternVL, and QwenVL families) over 40 language-source evaluation configurations.

Generation Source Gemma3 Acc Gemma3 F1 HalluShift++ Acc HalluShift++ Prec HalluShift++ Rec HalluShift++ F1
Human-written 0.664 0.499 0.332 0.240 0.443 0.203
Gemma3 (LVLM) 0.543 0.442 0.440 0.226 0.506 0.204 (Random)
InternVL (LVLM) 0.532 0.401 0.403 0.240 0.517 0.224 (MAPS)
Llava (LVLM) 0.574 0.455 - - - 0.212 (All)
MiniCPM (LVLM) 0.485 0.410 - - - 0.195 (All)
QwenVL (LVLM) 0.459 0.427 - - - 0.221 (All)

In the cross-model ranking consistency evaluation (Pearson's \(r\), \(n=40\)), human-written samples achieved the strongest correlation with machine-generated benchmarks: - Human vs. Llava: \(r = 0.916\) - Human vs. MiniCPM: \(r = 0.867\) - Human vs. Gemma3: \(r = 0.796\) - Human vs. InternVL: \(r = 0.770\) - Human vs. QwenVL: \(r = 0.708\) The mean correlation between human-written data and machine-generated sets was \(r = 0.811\), outperforming the correlation between several pairs of actual LVLMs (e.g., InternVL vs. Llava: \(0.658\); InternVL vs. QwenVL: \(0.660\)).

Ablation Study

The evaluation systematically contrasted inter-annotator agreement across sampling strategies (Table 9), multi-span aggregation policies (Table 8), and detection recalibration dynamics across backbones (Table 7).

Sampling Strategy Presence Agreement \(\alpha_{\text{free}}\) Unconditional Label Alpha Label Alpha (Given Hallu.) Span IoU (Ignoring Labels) Span IoU (Label Match Req.)
Random Sampling \(0.41 \pm 0.01\) \(0.35 \pm 0.00\) \(0.43 \pm 0.01\) \(0.36 \pm 0.01\) \(0.26 \pm 0.01\)
MAPS Pre-selection \(0.46 \pm 0.01\) \(0.41 \pm 0.01\) \(0.50 \pm 0.01\) \(0.42 \pm 0.00\) \(0.31 \pm 0.01\)
Human-written \(0.57 \pm 0.01\) \(0.47 \pm 0.02\) \(0.63 \pm 0.03\) \(0.54 \pm 0.01\) \(0.44 \pm 0.02\)
First-Span Aggregation - \(0.4475 \pm 0.0141\) \(0.5020 \pm 0.0113\) - -
Longest-Span Aggregation - \(0.3925 \pm 0.0102\) \(0.4126 \pm 0.0089\) - -

In probability threshold recalibration for non-hallucinated token classification: - Calibrating on human data yielded consistent downstream F1 gains across all model test distributions (\(\Delta = +0.064\) on Human, \(+0.070\) on Gemma3, \(+0.043\) on InternVL, \(+0.062\) on QwenVL). - Calibrating on Llava data systematically degraded performance across every evaluation target (\(\Delta \in [-0.012, -0.007]\)). Because the HalluShift++ feature extractor is rooted in Llava-NeXT, this reveals severe model-specific inductive bias and negative transfer during evaluation.

Key Findings

  • Human Authorship Drastically Elevates Consensus: Annotators demonstrate markedly superior consistency on human-written instances across all metrics. Given a verified hallucination, label agreement increases from 0.43 (random) and 0.50 (MAPS) to 0.63 (human), while label-matched span IoU rises from 0.26 to 0.44.
  • Stylistic Shift Matches Cross-Model Variation: Multi-way logistic regression classification on word/character n-grams reveals that distinguishing human samples from LVLM responses achieves F1 scores between 0.73 and 0.87—comparable to the separability between distinct model families (Gemma3 vs. QwenVL) and datasets (VISaGE vs. HaloQuest). Human writing does not constitute an unnatural outlier.
  • Judge Pre-Selection Introduces Severe Cascading Error: Automated pre-selection with Gemma3 exhibits high false-positive rates when checked against human consensus, misattributing non-hallucinated descriptions as inventions, whereas human creation guarantees precise error injection without judge bias.

Highlights & Insights

  • Decoupling Evaluation from Foundation Model Obsolescence: By showing that human-written errors preserve detector relative rankings with an average correlation of \(r=0.811\), this work unlocks a sustainable methodology to build enduring hallucination benchmarks immune to model turnover.
  • Controlled Inoculation of Long-Tail Blind Spots: Human writers easily achieve balanced representations of low-frequency failure modes (25% OCR and 25% counting errors), which are virtually unobtainable through natural sampling or judge-based filtering.
  • Empirical Exposure of White-Box Self-Probing Pathology: The finding that Llava-calibrated probes degrade out-of-distribution performance warns the community against treating model-specific internal features as universal ground-truth signals for hallucination detection.

Limitations & Future Work

  • Constraint on White-Box Mechanistic Interpretability: Human-authored responses lack authentic runtime activation trajectories and key-value cache histories, necessitating teacher-forced forward passes for white-box probing models that may miss dynamic decoding anomalies.
  • Task Scope and Dialogue Depth: The benchmark is presently bounded to single-turn open-ended VQA and captioning tasks over high-resource languages, leaving interactive multi-turn agentic planning and low-resource dialectal hallucinations unaddressed.
  • Future Directions: Extending human-crafted error paradigms to complex video reasoning and chart-to-code execution, while expanding the open SHROOM-Visions shared task to foster black-box multimodal alignment.
  • vs CHAIR / POPE: Early benchmarks framed hallucination evaluation as binary entity verification or coarse caption-level lexical matching; SHEEP establishes character-level span localization across 5 granular categories including text rendering and arithmetic counting.
  • vs M-HalDetect / HalLoc: Prior fine-grained span datasets were derived strictly from specific LVLM outputs, inheriting rapid diagnostic decay; SHEEP validates human-authored data as an enduring substitute that resists model obsolescence.
  • vs AutoHallusion: While automated perturbation pipelines generate synthetic test suites, they retain algorithmic biases from the prompt-engineering LLM; SHEEP shows that human writing offers tighter category isolation and higher inter-annotator consensus.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering empirical demonstration that human-written visual hallucinations resolve benchmark obsolescence and model-dependence.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Spans 20,000 samples, 4 languages, 5 LVLM generators, 10 judge architectures, and rigorous bootstrap confidence interval validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Lucid, cohesive academic prose with thorough reporting of edge cases, translation quality, and inter-annotator statistics.
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable, durable multilingual benchmark and actionable methodological foundation for robust multimodal reliability evaluation.