Skip to content

ParsHate: A Benchmark Dataset for Hate and Target Detection in Persian

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zbokaee/ParsHate
Area: LLM Safety
Keywords: Persian hate speech detection, fine-grained target identification, implicit hate, temporal distribution shift, span rationales

TL;DR

Introduces ParsHate, the first decade-long (2013–2022) Persian hate speech benchmark (10,000 tweets, 31% hate), which applies a hybrid random and classifier-score stratified sampling strategy to eliminate keyword-driven lexical bias, establishes a 7-category structured target taxonomy, explicitly annotates rhetorical strategies and implicit targets, provides token/character-level rationales, and systematically uncovers temporal distribution drift and catastrophic representation collapse on implicit hate and minority target groups.

Background & Motivation

Hate speech detection plays a foundational role in natural language processing due to its direct societal implications in content moderation and digital safety. Over the past decade, high-resource languages—predominantly English—have fostered rich benchmark ecosystems (e.g., HateXplain, Gab Hate Corpus, Latent Hatred). Recent state-of-the-art benchmarks in these languages have progressed beyond coarse binary classification by providing multi-class target taxonomies, differentiating explicit from subtle implicit hostility, and annotating span-level rationales to foster explainable model evaluation.

Despite these advancements, Persian remains a comparatively under-resourced language with critical structural deficits across its available harmful language datasets. Existing resources, including Pars-OFF, PHATE, PHICAD, and Pars-HAO, exhibit severe limitations. Pars-OFF heavily depends on keyword-driven collection and tweets harvested from politically active accounts, creating overt lexical shortcut learning while severely under-representing subtle or implicit hostilities. PHATE introduced rationales and transformer baselines, but relied partly on keyword filtering, spans only a narrow temporal window (2020–2023), and provides target annotations merely as non-normalized, free-form text. PHICAD scales up raw comment volume on Instagram but lacks any target taxonomy or implicit/explicit delineation, while Pars-HAO provides no structured target schemas. Consequently, existing resources fail to support longitudinal tracking of how hate speech evolves across years, how hostility is implicitly encoded, and how attack targets are systematically structured in Persian digital discourse.

To address these compounding shortcomings, there is an urgent need for a decade-spanning benchmark that preserves natural label distributions while capturing both subtle rhetoric and multi-label targets without lexical shortcut bias. Core idea: by establishing a hybrid temporal sampling strategy combining fully random draws with toxicity-classifier score stratification across 2013–2022, ParsHate constructs the first decade-long, 10,000-tweet Persian benchmark annotated for binary hate, seven fine-grained target categories, explicit/implicit hate with rhetorical strategies, explicit/implicit targets, and span-level rationales to expose temporal drift and implicit representation collapse.

Method

Overall Architecture

The construction and evaluation pipeline of ParsHate integrates raw archival harvesting, hybrid temporal sampling, a multi-dimensional prescriptive annotation scheme, expert annotation with structured adjudication, and temporal-architectural diagnostic benchmarking. Starting from an uncurated 1% daily Twitter stream archive comprising 47 million Persian tweets between 2013 and 2022, text normalization, deduplication, and length filtering yield a 29-million tweet candidate pool. A hybrid sampling scheme extracts 1,000 balanced instances per year (500 fully random + 500 classifier-stratified across 10 toxicity probability intervals). Three native Persian expert annotators then assign multi-dimensional labels and highlight character-level rationales under a prescriptive paradigm. Finally, the dataset supports temporal evaluation, lexical diversity probing, and multi-label target prediction diagnostics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Decade Twitter Stream Archive<br/>2013–2022 47M Persian Tweets"] --> B["Filtering & Length Normalization<br/>Reduced to 29M Tweet Candidate Pool"]
    B --> C["Hybrid Temporal Sampling<br/>500 Random + 500 Classifier-Stratified / Year"]
    C --> D["Multi-dimensional Annotation Taxonomy<br/>Hate + 7 Targets + Explicit/Implicit + Rationales"]
    D --> E["Prescriptive Annotation & Adjudication<br/>Majority Character Merging + Guideline Adjudication"]
    E --> F["Temporal Benchmarking & Diagnostic Protocol<br/>Cross-year Transfer + TTR Diversity + NE Masking"]

Key Designs

1. Hybrid Temporal Sampling: Eliminating Lexical Shortcut Bias While Preserving Natural Intensity Prior toxic language datasets routinely harvest candidate samples via curated hate lexicons or slurs, inadvertently conditioning models to rely on surface keywords rather than deep semantic reasoning; conversely, pure random sampling from social media yields an extremely sparse positive rate (typically 1%–3%), resulting in wasteful annotation budgets. To balance natural discourse representation with an information-dense label distribution, ParsHate introduces a dual-track temporal sampling protocol across every year from 2013 to 2022. For each year, exactly 1,000 tweets are selected: 500 tweets are chosen via pure random sampling to capture unconstrained topical discourse, while the remaining 500 tweets are sampled using score-stratified selection based on predictions from a fine-tuned Llama-3 toxicity classifier (trained on external Persian, Arabic, and Indonesian corpora with zero overlap with ParsHate). The toxicity probability space \([0.0, 1.0]\) is partitioned into 10 uniform intervals (\([0.0-0.1), \dots, [0.9-1.0]\)), with 50 tweets randomly drawn from each decile. This mechanism guarantees comprehensive coverage across neutral content, borderline ambiguities, and high-confidence hate without enforcing artificial keyword priors or distorting natural linguistic variety, achieving a stable 31% overall hate prevalence.

2. Structured Multi-label Target Taxonomy and Target Implicitness Disentanglement: Grounding Persian Identity Attacks Persian online discourse exhibits distinct socio-political characteristics that render Western taxonomies inadequate when applied naively. ParsHate extends the identity dimensions of the Gab Hate Corpus and adapts them to Persian cultural realities, formulating a hierarchical multi-label taxonomy spanning seven categories (allowing multiple targeted groups per tweet): Politics (subdivided into Domestic vs. Foreign political actors), Religion (subdivided into Islam vs. Other religious identities, reflecting the empirical dominance of anti-clerical and religious critique in Iran), Nationality (subdivided into Iranian, Afghan, Arab, Western, and Other), Gender (subdivided into Women and Men), Occupation, Ethnicity, and Other. Crucially, ParsHate decouples target identity from target expression by annotating whether the target is explicit or implicit. In online Persian discourse, hostile actors frequently invoke targeted groups through mythological metaphors (e.g., Zahhak for authoritarian tyranny), collective pronouns ("these bastards"), or euphemistic historical events rather than naming them directly. In the dataset, 35% of all hate tweets (and 39% of political hate instances) target entities implicitly, creating a stringent diagnostic testbed for model grounding and discourse-level coreference resolution.

3. Implicit Rhetorical Strategy Taxonomy and Character-level Rationale Spans: Enabling Explainable Diagnostics To address the pervasive failure of models on indirect hostilities, ParsHate introduces explicit-versus-implicit labeling paired with rhetorical strategy categorization. Following a 250-tweet pilot phase, implicit hate is categorized into three rhetorical strategies: (i) cultural or historical reference (48% of implicit hate, e.g., allusions to the November 2019 protests, former President Rouhani's campaign "key" symbol, or the controversial hug of Hugo Chávez's mother); (ii) sarcasm or figurative language (36%, e.g., animalistic metaphors comparing officials to leeches or dogs); and (iii) other indirect rhetorical techniques (16%, e.g., exclusionary syntax, collective denial of nationality, or loaded quantifiers). Furthermore, for all hateful tweets, annotators selected the minimal contiguous text segment(s) justifying the label—isolating overt slurs for explicit hate, and identifying rhetorical anchors, metaphors, or minimal hostile clauses for implicit hate. Character-level binary masks from three independent annotators were aggregated via majority voting (requiring \(\ge 2\) annotators per character), with continuous segments merged into gold rationales spanning on average 18% of tweet characters.

4. Prescriptive Multi-Expert Annotation and Disagreement Resolution: Ensuring High-Fidelity Labels Given the inherently subjective nature of subtle political critique and historical innuendo, annotations were conducted by three female native Persian speakers aged 25–35, each possessing over 20 years of lived experience in Iran and holding graduate degrees in STEM fields. Operating under a prescriptive annotation paradigm with comprehensive five-page guidelines, annotators achieved substantial inter-annotator agreement: Fleiss' \(\kappa = 0.69\) for hate identification, \(\kappa = 0.60\) for target categories, \(\kappa = 0.51\) for implicit hate, and \(\kappa = 0.58\) for implicit targets. Character-level span agreement reached \(\kappa = 0.55\) overall (\(\kappa = 0.64\) for explicit spans, \(\kappa = 0.44\) for implicit spans). Disagreements were systematically resolved via a two-tier adjudication procedure: multi-label strategy assignments were preserved when multiple rhetorical devices co-occurred, while persistent boundary disputes and figurative-versus-hate boundaries (16% of implicit hate and 6% of explicit hate instances) were adjudicated through structured guideline-driven discussions, falling back to majority voting only if consensus was unreachable.

Benchmark & Evaluation Protocol

To rigorously evaluate model stability, temporal distribution shift, and representation limits, ParsHate establishes five structured experimental settings: 1. Year-specific Fine-tuning: Independent 80/20 train/test splits per year evaluate intra-year generalization under localized data constraints; 2. Integrated All-years Fine-tuning: Pooling 80% of data across all ten years (8,000 tweets) evaluates how temporal diversity enhances cross-era robustness; 3. Cross-temporal Transfer: A SOTA LLaMA-3 model fine-tuned on PHATE (2020–2023) is evaluated zero-shot across ParsHate's historical splits (2013–2022); 4. Zero-shot LLM Baselines: Evaluation of open-weights foundational models (LLaMA-3, Gemma-3) and proprietary frontier models (GPT-5) across all test slices; 5. Named Entity (NE) Masking Diagnostics: Employing ParsBERT-NER to mask specific persons, organizations, and locations to measure Type-Token Ratio (TTR) lexical variability and test model reliance on entity shortcuts.

Key Experimental Results

Main Results: Longitudinal Hate Speech Detection Performance

The main longitudinal benchmark evaluates zero-shot models, out-of-distribution SOTA transfer, single-year models, and integrated all-years fine-tuning across the 2013–2022 test partitions using Macro-F1 (averaged across three random seeds for fine-tuned checkpoints).

Training Strategy / Model Evaluation Setup Early Era F1 (2013) Mid Era F1 (2017) Recent Era F1 (2022) Full Decade Macro-F1 (All) Key Empirical Observations
GPT-5 Zero-shot 59.9 57.7 90.6 68.0 Strongest zero-shot model; shows large temporal gap between 2013 and 2022
LLaMA-3 Zero-shot 33.8 53.6 65.2 49.9 Substantial zero-shot degradation on early Persian social media text
Gemma-3 Zero-shot 33.7 54.7 68.5 53.2 Performs comparably to LLaMA-3 zero-shot; lacks nuanced Persian alignment
PHATE SOTA (LLaMA-3) Cross-temporal 57.9 69.2 88.4 74.8 High accuracy on temporally proximal years (88+), but plummets on early years (mid-50s)
ParsBERT Year-specific FT X (Collapsed) 57.2±1.8 79.1±1.3 67.7±0.7 Fails to converge on 2013–2016 due to class imbalance; stable only from 2017 onward
LLaMA-3 Year-specific FT 41.4±1.7 66.5±1.8 79.3±1.1 63.5±0.7 Limited per-year training data (800 samples) severely constrains generalization
Gemma-3 Year-specific FT 44.7±1.8 64.4±1.9 77.5±1.1 64.3±0.7 Mirrors year-specific LLaMA-3 trajectory across all temporal slices
LLaMA-3 Integrated All-years FT 64.0±1.4 80.6±1.4 90.3±0.8 79.0±0.5 Highest overall performance; substantially mitigates early-year degradation
Gemma-3 Integrated All-years FT 61.7±1.4 81.1±1.5 89.7±0.8 78.2±0.5 Highly stable across the temporal spectrum, matching all-years LLaMA-3

Multi-label Target Prediction and Explicitness Breakdown

The second experimental battery examines the best-performing all-years models (LLaMA-3-FT, Gemma-3-FT) and frontier general-purpose LLMs (GPT-5) across explicit versus implicit hate subsets, fine-grained target classes, and Named Entity (NE) masked settings.

Evaluation Dimension / Target Category / Subset GPT-5 (Unmasked) P / R / F1 LLaMA-3-FT (Unmasked) P / R / F1 GPT-5 (Masked) P / R / F1 LLaMA-3-FT (Masked) P / R / F1 Behavioral Contrast & Failure Modes
Hate Explicitness Explicit Hate - / - / - 84.6 / 84.4 / 84.5 - / - / - - / - / - Robust classification driven by overt slurs and lexical markers
Implicit Hate - / - / - 5.3 / 3.7 / 4.4 - / - / - - / - / - Near-total collapse: 83% predicted as non-hate
Implicit: Reference - / - / - - / - / 2.0 - / - / - - / - / - Demands external historical knowledge; severe model blind spot
Implicit: Sarcasm - / - / - - / - / 4.0 - / - / - - / - / - Irony defeated by lack of extra-textual context grounding
Implicit: Other - / - / - - / - / 5.0 - / - / - - / - / - Highest implicit score due to detectable exclusionary phrasing
Multi-label Target Politics (76.5%) 96.3 / 32.7 / 48.8 84.6 / 96.2 / 90.0 97.4 / 27.6 / 43.0 82.3 / 98.5 / 89.6 LLaMA-3 overfits majority class; GPT-5 shows high precision
Religion (6.4%) 14.5 / 81.2 / 24.6 65.1 / 32.7 / 43.6 14.7 / 81.4 / 24.6 71.5 / 12.4 / 21.1 Masking collapses LLaMA-3 recall from 32.7% down to 12.4%
Nationality (4.9%) 7.3 / 60.7 / 13.4 43.4 / 8.2 / 13.8 8.4 / 60.5 / 14.8 57.2 / 10.3 / 17.5 Fine-tuned model fails to recall rare nationality attacks (<10%)
Occupation (3.5%) 5.5 / 60.4 / 10.1 20.7 / 2.1 / 3.9 8.2 / 65.7 / 14.6 33.3 / 3.2 / 5.8 LLaMA-3 suffers minority collapse; GPT-5 retains broad recall
Ethnicity (2.3%) 4.7 / 60.1 / 8.8 15.3 / 2.6 / 4.4 6.6 / 65.2 / 12.1 14.2 / 2.3 / 4.0 Fine-tuned model recalls only 2.6% due to severe class scarcity
Gender (1.9%) 22.4 / 80.3 / 35.1 18.2 / 13.1 / 15.2 21.3 / 80.6 / 33.7 11.6 / 7.1 / 8.8 GPT-5 achieves 80.3% recall; LLaMA-3 drops to 8.8% F1 when masked
Other (4.6%) 30.5 / 22.3 / 25.8 46.3 / 6.3 / 11.1 25.2 / 33.5 / 28.7 33.4 / 2.7 / 5.1 Catch-all category suffers severe recall suppression in LLaMA-3
Macro Average Macro Avg 25.5 / 56.6 / 23.8 42.0 / 23.0 / 26.0 25.9 / 59.1 / 24.4 42.8 / 19.5 / 27.1 LLaMA-3 collapses to precision; GPT-5 prioritizes recall

Key Findings

  1. Marked Temporal Distribution Shift and Asymmetric Historical Degradation: Models trained exclusively on recent social media text (PHATE SOTA, trained on 2020–2023) achieve F1 > 88 on 2020–2022 test partitions, but suffer a precipitous drop to the mid-50s on 2013–2015 data. Type-Token Ratio (TTR) analysis confirms that early Persian Twitter discourse possessed significantly higher lexical and named-entity diversity. Integrated all-years training (LLaMA-3-FT F1 = 79.0) effectively hedges against this temporal degradation, outperforming the single-year models (63.5–67.7) and cross-temporal baselines (74.8).
  2. Catastrophic Failure on Implicit Hostility: While fine-tuned LLMs achieve 84.5 F1 on explicit hate, they suffer near-total collapse on implicit hate, dropping to 4.4 F1 (LLaMA-3) and 6.2 F1 (Gemma-3). In error analysis, 83% of implicit false negatives are predicted as non-hate. Performance drops in direct proportion to external knowledge load: cultural/historical references are the hardest (F1 = 2.0), followed by sarcasm (F1 = 4.0) and other indirect devices (F1 = 5.0). In contrast, GPT-5 misses fewer implicit cases (22% of errors vs. 31%), but over-flags neutral figurative language, producing high false positives.
  3. Severe Majority-Class Minority Collapse in Multi-label Target Modeling: Because political hate dominates 76.45% of hateful tweets, fine-tuned models over-index on Politics (F1 = 90.0) and virtually collapse on low-frequency targets (Occupation F1 = 3.9, Ethnicity F1 = 4.4, Gender F1 = 15.2). When rare targets are misclassified, 58% involve implicit target references where the model defaults to assigning Politics. Named entity masking further strips recall across low-resource targets (e.g., Religion recall drops from 32.7% to 12.4%), proving that models lean heavily on entity name memorization.
  4. Rationale Span Localization Reflects Upstream Detection Collapse: Evaluating token-level rationale span localization on gold hate tweets yields an overall span F1 of 44 for LLaMA-3-FT, decomposing into 58 on explicit hate versus 14 on implicit hate. Error analysis reveals that 80% of implicit span failures are "no-span" errors where the model outputs nothing at all. This demonstrates that span localization failure is entirely downstream of semantic detection failure: if the model lacks the cultural context to recognize hostility, it possesses no anchor tokens to extract.

Highlights & Insights

  • De-biased Hybrid Stratification as an Annotation Paradigm: By eschewing keyword-based retrieval in favor of a 50/50 mix of pure random and probability-stratified sampling across the entire classifier confidence range, ParsHate eliminates keyword-driven shortcuts while capturing nuanced borderline cases, serving as a methodological blueprint for low-resource safety benchmarks.
  • The Knowledge Load Bottleneck in Low-Resource Online Hate: The study empirically proves that the central bottleneck in indirect hate modeling is not stylistic implicitness in the abstract, but the "knowledge load" demanded by local cultural and historical references. Metaphors rooted in regional political trauma (e.g., November 2019, Evin prison, or Rouhani's key) present zero surface toxicity, demanding external knowledge integration rather than generic sentiment analysis.
  • Diagnostic Power of Target Implicitness Disentanglement: Decoupling whether a target is explicitly mentioned or indirectly implied uncovers critical failure modes in LLM safety alignment. Even when hate expression is overt (e.g., using slurs), models routinely misattribute the target category if the victim group is invoked implicitly or via metaphor (e.g., misclassifying regime critique during the Mahsa Amini protests as gender-based hate due to the word "girl").

Limitations & Future Work

  • Single-Platform Scope: ParsHate is exclusively gathered from Twitter. Given Iran's complex censorship landscape and differing user demographics across platforms, discourse norms on Instagram, Telegram, or domestic forums may exhibit distinct rhetorical patterns that require cross-platform validation.
  • Extreme Class Imbalance and Implicit Sample Volume: Although the dataset comprises 10,000 tweets, implicit hate accounts for only 7% of total instances (~700 samples), and minority target categories (such as ethnicity at 2.3% and gender at 1.9%) remain sparse, constraining the training of dedicated balancing or architectural re-weighting mechanisms.
  • Architectural Scope: Empirical evaluations focus on transformer encoders (ParsBERT), open autoregressive LLMs (LLaMA-3, Gemma-3), and frontier closed LLMs (GPT-5). Future research should explore retrieval-augmented generation (RAG) linked to historical and encyclopedic knowledge bases to explicitly supply cultural grounding.
  • vs. Pars-OFF (Ataei et al., 2022): Pars-OFF is confined to a 2015–2020 window, relies heavily on keyword filtering from political accounts, and supports only coarse offensive language classification; ParsHate covers 2013–2022 with balanced sampling, structured multi-label target taxonomy, and fine-grained implicit rhetoric labeling.
  • vs. PHATE (Delbari et al., 2024): PHATE covers 2020–2023 with unnormalized free-text targets and does not differentiate implicit hate strategies; ParsHate establishes a rigorous closed target taxonomy and demonstrates that PHATE-trained SOTA models degrade severely when evaluated on historical pre-2020 discourse.
  • vs. HateXplain (Mathew et al., 2021) & Latent Hatred (ElSherief et al., 2021): Adapts state-of-the-art English rationale and implicit hate framing to the nuanced socio-political dynamics of Persian, validating the universality of span-level explainability while highlighting language-specific cultural knowledge dependencies.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes the first decade-spanning Persian hate speech benchmark with pioneering disentanglement of implicit targets, cultural rhetorical strategies, and rationales.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 10 years of cross-temporal transfer, zero-shot and fine-tuned LLMs, TTR lexical dynamics, NE masking probes, and token-level rationale localization.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally rigorous taxonomy formulation, thorough inter-annotator agreement reporting, and transparent error dissection with culturally grounded qualitative examples.
  • Value: ⭐⭐⭐⭐⭐ An invaluable, highly standardized open-source resource for multilingual AI safety, cultural alignment, and longitudinal robustness in low-resource NLP.