Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/qingjiesjtu/SampledBPE
Area: LLM Reasoning
Keywords: corpus auditing, BPE tokenizer, Chinese web pollution, sampling statistics, hierarchical token trees
TL;DR¶
Inspired by marine pollution monitoring, SAMPLED-BPE introduces a lightweight token-level auditing pipeline that trains BPE on small streaming samples to profile web-scale Chinese corpora with a 148.4x speedup and 35.8x memory reduction, releasing a 630k-token hierarchical pollution dataset across 92k trees.
Background & Motivation¶
Chinese web pollution has repeatedly surfaced in downstream large language models, evidenced by spam and sexually explicit Chinese phrases appearing as single atomic tokens in ChatGPT's vocabulary, as well as unintended Chinese gambling promotion strings popping up in Codex code-generation outputs. These anomalous and hazardous model outputs directly implicate upstream pre-training corpora. Consequently, there is an urgent need within both academia and industry to systematically audit upstream Chinese web-scale datasets to understand the prevalence, origins, and evolutionary trajectories of web-derived contaminants.
However, auditing web-scale Chinese corpora (typically spanning hundreds of gigabytes to multiple terabytes) encounters three fundamental roadblocks. First, the colossal volume of web-scale text makes exhaustive, full-corpus scanning prohibitively expensive—indexing an 83TB corpus on a 128-core CPU server can take up to 99 days. Second, existing data auditing efforts remain predominantly coarse-grained, focusing on document-level quality filtering, language identification, or domain source distributions, thereby failing to uncover fine-grained, token-level lexical contamination. Third, Chinese web pollution is highly evasive and undergoes rapid lexical renewal; underground industries continuously engineer homophones, euphemisms, and neologisms to bypass filtering, rendering static, pre-compiled keyword blocklists quickly obsolete.
To overcome these computational and linguistic hurdles, the authors draw an analogy to marine environmental water quality monitoring, where overall contamination profiles are reliably estimated from sparse, strategically sampled observations rather than filtering entire oceans. Core idea: develop SAMPLED-BPE, a lightweight token-level auditing pipeline that performs single-pass streaming sampling and trains a BPE tokenizer to surface high-frequency co-occurring polluted units without pre-defined keyword lists, combining web-search evidence with an LLM classifier to accurately profile terabyte-scale corpora in hours rather than months.
Method¶
Overall Architecture¶
SAMPLED-BPE does not attempt to reconstruct the full vocabulary of an entire corpus. Instead, it approximates the token-level and category-level pollution distributions with minimal resource overhead. The end-to-end framework comprises four sequential stages: 1. Streaming Sampling: Reads target corpora in a single contiguous sequential pass, scaling linearly with sample size and sidestepping the massive I/O and memory overhead of global shuffling; 2. BPE Training & Counting: Trains a Byte-Pair Encoding (BPE) tokenizer on the sample to greedily merge adjacent frequent character sequences into long tokens, followed by a linear token-counting pass; 3. Category Mapping with Search Evidence: Fetches live web search results as disambiguating context for each surfaced token and classifies it into one of six categories using a fine-tuned GLM-4-32B model; 4. Corpus Profiling & Hierarchical Tracing: Aggregates token ratios into overall and category-level corpus pollution profiles, while organizing overlapping tokens into a hierarchical forest structure to trace pollution roots and compositional emergence.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Web-scale Raw Chinese Corpora<br/>Multi-TB Plaintext / Common Crawl"] --> B["Streaming Subset Sampling<br/>Single-pass prefix scan without index materialization"]
B --> C["BPE Training & Counting<br/>Unsupervised frequent-pair merges surface polluted tokens"]
C --> D["Category Mapping with Search Evidence<br/>Contextual search snippets ground fine-tuned LLM"]
D --> E["Corpus Profiling & Hierarchical Tracing<br/>Category ratio aggregation and token forest organization"]
Key Designs¶
1. Streaming Subset Sampling: slashing I/O overhead while preserving statistical fidelity
Standard random sampling over terabyte corpora requires materializing a globally shuffled document index, which incurs severe disk seek latency and excessive memory allocation that often exceeds the cost of auditing itself. SAMPLED-BPE replaces random index shuffling with sequential prefix streaming sampling. Empirical comparisons show that on a Common Crawl reference split, streaming sampling requires only 77.4 seconds compared to 1731.7 seconds for random sampling (a 22.4x speedup). Crucially, at a 0.10% sampling rate, streaming sampling preserves 69.79% token coverage (compared to 70.19% for random) and yields a nearly identical category weighted relative error (2.57% vs 2.65%), demonstrating that sequential streaming accurately preserves corpus-level statistical properties at a fraction of the systems cost.
2. BPE Training & Counting: emergent vocabulary discovery without keyword dependencies
Relying on pre-compiled keyword blacklists introduces severe blind spots against rapidly evolving internet jargon, while running deep semantic parsers over trillions of tokens is computationally intractable. SAMPLED-BPE leverages the compression mechanism of Byte-Pair Encoding (BPE), which iteratively merges the most frequent adjacent character pairs under a fixed merge budget. Under this data-driven compression objective, frequently recurring phrases—such as syndicated adult titles, illicit gambling brand names, and spam keywords—are naturally consolidated into discrete, high-frequency BPE tokens. Counting requires only a single forward pass with the learned tokenizer, allowing previously unseen lexical variants to surface organically without any manual curation.
3. Category Mapping with Search Evidence: grounding ambiguous and abbreviated tokens in web context
Surfaced Chinese web tokens are frequently abbreviated, polysemous, or euphemistic, making accurate categorization impossible from surface strings alone (e.g., "申博" literally denotes applying for a doctoral program, but in web gambling contexts refers to the Sunbet gaming syndicate). SAMPLED-BPE queries search engines to retrieve live web snippets as grounding evidence. The token string and its top search results are formatted into contextual prompts for an open-source GLM-4-32B classifier, fine-tuned on expert-annotated vocabulary splits across six content categories: Normal Content, Adult Content, Online Gambling, Online Gaming, Online Video, and Anomalous expressions. The classifier attains 97.32% accuracy on held-out test sets, maintaining robust stability across search engines (Google vs. Baidu) and geographic IP regions within a 1.02% error margin.
4. Corpus Profiling & Hierarchical Tracing: unravelling subtoken families and compositional pollution
Rather than outputting an unstructured, flat vocabulary list, SAMPLED-BPE organizes the 630,684 extracted tokens into a forest of 92,972 hierarchical trees. The root of each tree represents the shortest recurring polluted subtoken (e.g., "浪小辉", an adult video performer), while descendants represent longer, specialized surface variants ("中国男同浪小辉", "浪小辉GAY国产"). Furthermore, the tree hierarchy exposes "compositional pollution": instances where two individually innocuous, benign tokens (e.g., "菲律宾" [Philippines] and "申博" [Doctoral Application], or "北京" [Beijing] and "赛车" [Car Racing]) combine into an unequivocally illicit gambling entity ("菲律宾申博", "北京赛车"). This structure enables downstream filtering pipelines to target compact root entities or contextual compositions without over-blocking benign base words.
Benchmark & Evaluation Protocol¶
The classifier fine-tuning optimizes cross-entropy loss on GLM-4-32B using expert annotations from ChatGPT vocabulary audits. For evaluation, token coverage between the full vocabulary \(V_{\text{Full}}\) and sample vocabulary \(V_{\text{sample}}\) is defined as \(|V_{\text{Full}} \cap V_{\text{sample}}| / |V_{\text{Full}}|\). The directional coverage metric between corpora \(A\) and \(B\) is formalized as: $$ \text{Coverage}(A \to B) = \frac{|P_A \cap P_B|}{|P_A|} $$ where \(P_A\) and \(P_B\) denote the polluted token sets extracted from corpora \(A\) and \(B\), respectively. Category prevalence is aggregated across tokens containing at least three Chinese characters across the six designated taxonomy classes.
Key Experimental Results¶
Main Results¶
The authors conduct extensive audits across 11 representative open Chinese corpora (or Chinese subsets of broad multilingual corpora) and 6 longitudinal annual snapshots (February crawls from 2021 to 2026) of Chinese Common Crawl.
The table below presents the category-wise pollution breakdown across the 11 audited Chinese open corpora, sorted by total pollution percentage (quoted directly from Table 1, in %):
| Corpus | Pollution Ratio (%) | Adult Content (%) | Online Gambling (%) | Online Gaming (%) | Online Video (%) | Anomalous (%) |
|---|---|---|---|---|---|---|
| OSCAR | 83.38 | 76.10 | 0.56 | 0.16 | 5.02 | 1.53 |
| mC4 | 25.41 | 0.31 | 23.17 | 0.46 | 0.32 | 1.15 |
| HPLT | 4.95 | 1.95 | 1.61 | 0.25 | 0.72 | 0.43 |
| CulturaX | 3.36 | 0.15 | 2.45 | 0.16 | 0.18 | 0.42 |
| CWT | 2.35 | 0.05 | 0.22 | 0.29 | 0.06 | 1.73 |
| ROOTS | 2.02 | 0.23 | 0.00 | 0.00 | 0.00 | 1.79 |
| WanJuan | 0.75 | 0.04 | 0.15 | 0.17 | 0.05 | 0.33 |
| MAPCC | 0.69 | 0.10 | 0.08 | 0.09 | 0.04 | 0.38 |
| SkyPile | 0.61 | 0.06 | 0.04 | 0.05 | 0.04 | 0.42 |
| CCI3 | 0.55 | 0.16 | 0.00 | 0.01 | 0.00 | 0.37 |
| WuDao | 0.50 | 0.10 | 0.01 | 0.06 | 0.01 | 0.31 |
The table below reports temporal pollution ratios across Chinese Common Crawl snapshots at a 1% sampling rate from 2021 to 2026 (quoted directly from Table 2, in %):
| Snapshot Year | Total Pollution (%) | Adult Content (%) | Online Gambling (%) | Online Gaming (%) | Online Video (%) | Anomalous (%) |
|---|---|---|---|---|---|---|
| 2021 | 35.35 | 24.73 | 5.31 | 0.40 | 3.93 | 0.98 |
| 2022 | 61.87 | 54.53 | 1.05 | 0.16 | 4.75 | 1.38 |
| 2023 | 62.90 | 55.86 | 0.54 | 0.23 | 4.89 | 1.37 |
| 2024 | 34.83 | 28.01 | 1.48 | 0.33 | 3.86 | 1.16 |
| 2025 | 38.01 | 30.53 | 0.74 | 0.22 | 5.05 | 1.47 |
| 2026 | 79.52 | 68.72 | 0.21 | 0.07 | 8.39 | 2.12 |
Ablation Study & Efficiency Profile¶
To systematically evaluate the trade-off between sample size, estimation accuracy, runtime acceleration, and memory reduction, the authors benchmark SAMPLED-BPE on a 96-core CPU machine across sampling rates from 0.05% to 10% (synthesized from Figure 4 and Figure 5):
| Sampling Rate (%) | Token Coverage (%) | Spearman Corr. (%) | Pearson Corr. (%) | Weighted Rel. Error (%) | Runtime Speedup | Peak RSS Reduction |
|---|---|---|---|---|---|---|
| 0.05 | ~65.0 | ~80.0 | ~99.5 | ~5.8 | 273.8x | 46.9x |
| 0.10 | 69.79 | 84.12 | 99.85 | 2.57 | 209.4x | 39.1x |
| 0.25 (Recommended) | 76.83 | 86.88 | 99.93 | 4.25 | 148.4x | 35.8x |
| 0.50 | ~82.0 | ~90.0 | ~99.95 | ~3.8 | 98.9x | 38.0x |
| 1.00 | ~87.0 | ~93.0 | ~99.98 | ~2.8 | 62.5x | 30.1x |
| 2.00 | ~90.0 | ~95.0 | ~99.99 | ~2.3 | 39.1x | 28.6x |
| 5.00 | ~93.5 | ~97.0 | ~99.99 | ~1.9 | 18.9x | 22.3x |
| 10.00 | ~96.0 | ~98.5 | ~99.99 | ~1.5 | 12.1x | 9.6x |
Additionally, focused comparisons between raw and cleaned corpus variants in Appendix Table 9 demonstrate the concrete efficacy and residual limits of data filtering: CWT-Full exhibits a 2.35% pollution ratio, which decreases to 0.50% in CWT-Cleaner; similarly, ROOTS-Uncorpus exhibits 2.02% pollution, dropping to 1.37% in ROOTS-Public, confirming that heuristic and model-based filtering mitigates but does not fully eradicate token-level pollution.
Key Findings¶
- Pollution is highly uneven and pipeline-dependent: Broad multilingual web scrapes without aggressive language-specific filtering exhibit catastrophic pollution (OSCAR reaches 83.38% total pollution, with Adult Content accounting for 76.10%; mC4 reaches 25.41%, heavily driven by 23.17% Online Gambling). Conversely, Chinese-centric curated datasets (WanJuan, SkyPile, CCI3, WuDao) maintain total pollution below 1.0%, where the largest residual fraction shifts to Anomalous syntactic fragments.
- Pollution follows a tiny shared core and massive dataset-specific tails: Among 106,671 identified polluted tokens across the 11 corpora, an overwhelming 96.386% (102,816 tokens) appear in only a single corpus. Only 1.446% (1,542 tokens) appear across three or more corpora, and just 252 core tokens recur across at least eight corpora (e.g., persistent euphemisms like "老司机" and gambling anchors like "威尼斯人").
- Upstream Common Crawl exhibits alarming pollution levels and temporal shifts: The 2026 Chinese Common Crawl snapshot reached an unprecedented 79.52% pollution ratio, dominated by Adult Content (68.72%). In contrast, Online Gambling steadily declined from 5.31% in 2021 to 0.21% in 2026, aligning directly with intensified law enforcement crackdowns on cross-border online gambling platforms.
Highlights & Insights¶
- Cross-disciplinary environmental methodology transfer: Adapting environmental water-sampling principles to massive text corpora provides a mathematically grounded, highly scalable alternative to intractable full-corpus scans, reducing auditing time from months to mere hours.
- Exposing compositional pollution vulnerabilities: Discovering that benign tokens dynamically compound into illicit entities (e.g., "Philippines" + "PhD Application" \(\to\) "Sunbet Online Casino") exposes a fundamental flaw in standalone keyword blocklists, motivating n-gram tree-structured governance.
- Revealing the systemic risks of unvetted Common Crawl ingestion: The findings deliver a sobering warning to the NLP community: raw web archives are dominated by gray-market and explicit spam. Training foundational models on minimally filtered crawls inevitably poisons model tokenizers and parameter spaces.
Limitations & Future Work¶
- Language specificity and cross-lingual boundary: The token-level discovery mechanism exploits the non-segmented nature of Chinese text, where BPE naturally condenses contiguous multi-character phrases. Diagnostic pilots in Appendix C reveal that English BPE captures primarily isolated unigrams (e.g., "porn", "casino"), requiring phrase-level or syntactic chunking extensions for multilingual auditing.
- Taxonomy scope constraints: The six categories focus specifically on commercial gray-market web pollution (pornography, gambling, piracy, spam) and do not encompass sentence-level phenomena such as political bias, toxic hate speech, or record-level Personally Identifiable Information (PII).
- Search engine reliance: The classification pipeline relies on real-time internet search snippets, which may introduce minor variations across temporal search index updates and regional network environments.
Related Work & Insights¶
- vs Document-level & Coarse-grained Auditing (Dodge et al., 2021; Soldaini et al., 2024): Prior documentation frameworks predominantly evaluate language distribution, perplexity filtering, or C4 heuristic rules; SAMPLED-BPE penetrates directly to the subword token level, uncovering pollution that seamlessly evades document-level score thresholds.
- vs Downstream LLM Vocabulary Reverse-Engineering (Zhang et al., 2025; Yang, 2024): Previous works passively analyzed corrupted tokens post-hoc in commercial vocabularies (e.g., GPT-4o); SAMPLED-BPE moves upstream to audit the source corpora directly, providing proactive pre-training data hygiene.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant transposition of marine sampling heuristics into LLM data engineering, coupled with an insightful hierarchical token tree formulation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 11 major corpora, 6 longitudinal Common Crawl snapshots, multi-engine search stability checks, and exhaustive ablation tests.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, polished academic prose, comprehensive qualitative examples, and transparent methodological disclosures.
- Value: ⭐⭐⭐⭐⭐ Crucial diagnostic infrastructure for trustworthy LLM pre-training, providing actionable data assets (630k token records) for cleaning next-generation foundation models.