Skip to content

๐Ÿ“š Pretraining

๐Ÿ’ฌ EMNLP2026 ยท 7 paper notes

๐Ÿ“Œ Same area in other venues: ๐Ÿง  NeurIPS2026 (52) ยท ๐ŸŽž๏ธ ECCV2026 (15) ยท ๐Ÿ“ท CVPR2026 (6) ยท ๐Ÿ”ฌ ICLR2026 (79) ยท ๐Ÿ’ฌ ACL2026 (12) ยท ๐Ÿงช ICML2026 (27)

Beyond Junk Tokens: Subsumed Tokens in Japanese Subword Tokenization

Addressing the suppression of valid words caused by longer dominant tokens in frequency-driven subword tokenization for languages without explicit boundaries, this paper formalizes the notion of subsumed tokens and develops a tokenizer-agnostic post-processing framework that searches optimal binary segmentation trees via dynamic programming to restore valid word statistics without altering tokenizer vocabularies.

DebugLM: Learning Traceable Training Data Provenance for LLMs

DebugLM equips LLMs with internal parametric data provenance by binding discrete dataset tags via dual-mode training, enabling accurate multi-stage behavior tracing (>95% TSR) in a single forward pass and targeted test-time remediation without retraining.

How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws

This paper extends functional scaling laws to time-varying data quality, derives asymptotically optimal joint data-quality and batch-size schedules via variational analysis, and proposes the Drop-Stable-Rampup recipe for LLM midtraining that achieves substantial gains across dense and MoE architectures.

MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language Models

MDM-Prime-v2 reveals that Byte-Pair Encoding induces severe low-entropy sub-tokens in masked diffusion models, degrading cross-entropy loss; by introducing offline index shuffling to maximize sub-token entropy and selecting the theoretical limit of binary encoding (\(\ell=\lceil\log_2 V\rceil\)), it scales diffusion language models to 1.1B parameters, achieving superior zero-shot accuracy across eight commonsense reasoning benchmarks over both autoregressive and diffusion baselines.

OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

OptiMer decouples data mixture ratio selection from model training in continual pre-training (CPT) by independently training single-corpus models, extracting their distribution vectors, and optimizing continuous merge weights post-hoc via TPE Bayesian optimization, achieving superior multi-domain capability with 15โ€“35ร— lower search cost on Gemma 3 27B.

SpinHash: An O(1) Encoder for Any Vocabulary

SpinHash introduces a compact, vocabulary-independent encoder that maintains a strictly constant \(O(1)\) parameter budget of \(\sim 0.2\text{M}\) parameters across arbitrary vocabulary sizes and modality counts, projecting discrete tokens into geometrically structured binary spin codes and decoding them via a single shared feedforward network to rival standard embedding tables on text, extreme \(k\)-mer genomics, and clinical multimodality.

WRAP++: Web Discovery Amplified Pretraining

Addressing the intra-document information silos and extraction bottlenecks of single-document rewriting, WRAP++ leverages web hyperlink topologies to discover high-confidence entity relationship subgraphs, generating cross-document multi-hop QA pairs that amplify an 8.4B raw token corpus into 82.7B tokens and substantially boosting parametric factual knowledge recoverability on SimpleQA.