Skip to content

📚 Pretraining

🧠 NeurIPS2026 · 52 paper notes

📌 Same area in other venues: 🎞️ ECCV2026 (15) · 📷 CVPR2026 (6) · 🔬 ICLR2026 (79) · 💬 ACL2026 (12) · 🧪 ICML2026 (27) · 🤖 AAAI2026 (9)

🔥 Top topics: LLM ×5 · Diffusion Models ×3

A Bitter Lesson for Data Filtering

Through extensive scaling experiments across compute, epoch budgets, and model capacities, this paper demonstrates that unfiltered web data ultimately outperforms common heuristic and classifier-based filters in high-compute regimes, revealing a "bitter lesson" for data filtering.

A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs

This paper introduces critical sharpness and relative critical sharpness—curvature proxies computed via fewer than 10 forward passes along the training update direction—providing the first empirical demonstration of progressive sharpening and Edge of Stability at 7B parameter scale, while prescribing optimal pre-training data mixtures for mid-training and fine-tuning without grid searches.

A Theory of Training Profit-Optimal LLMs

This paper bridges neural scaling laws with microeconomic theory of the firm, establishing the first formal profit-maximization framework for LLM training under compute-bound and data-bound regimes to derive asymptotic bounds on optimal model size, token budgets, and capital expenditure.

Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement

Based on a Riemannian Inertial System with Hessian Damping (RISHD), this paper introduces LITE, a generalized acceleration strategy that selectively amplifies learning rates and Hessian damping coefficients along flat directions via stateless fast subspace filtering, significantly speeding up Muon and SOAP in LLM pre-training.

An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence

Through controlled synthetic uniform random noise injection, this paper systematically demonstrates that noisy data directly triggers LLM pretraining loss divergence, shows that limited noise vocabulary size and network depth critically amplify divergence probability, and establishes an attention logit diagnostic framework (~1800 vs. ~4000) alongside QK-LayerNorm defenses to isolate and mitigate data-induced instabilities.

Bridging Compute- and Data-Optimal Pretraining

This paper introduces the unified Compute-Data (CD) Scaling Law, which bridges compute-optimal and data-optimal pretraining regimes by modeling a dynamic token effectiveness function \(\eta\) that captures diminishing returns and saturation ceilings across model sizes and data budgets.

Detecting RLVR Training Data via Structural Convergence of Reasoning

Addressing the hidden benchmark contamination risks in reinforcement learning with verifiable rewards (RLVR) where training data is undisclosed, this paper discovers that RLVR induces structural mode collapse and generation rigidity on seen prompts, and introduces Min-kNN Distance—a reference-free, probability-free black-box detector that achieves state-of-the-art detection across diverse reasoning models.

Dynamic Tokenization via Reinforcement Patching: End-to-end Training and Zero-shot Transfer

ReinPatch is the first end-to-end framework to jointly optimize a discrete sequence patching policy and its downstream sequence backbone using reinforcement learning via Group Relative Policy Gradient (GRPG), strictly enforcing target compression rates and decoupling a standalone foundation patcher that demonstrates remarkable zero-shot transfer across diverse forecasting tasks.

Effective Context in Transformers: An Analysis of Fragmentation and Tokenization

Framing sequence representation choices through finite-context prediction on stationary Markov sources, this paper proves that lossless fragmentation into finer units fundamentally increases optimal log-loss due to phase ambiguity, whereas greedy tokenization reliably extends effective context, governed by the lower tail of the token-window source-span distribution and tokenizer compression rate.

ELF: Embedded Language Flows

ELF introduces a continuous diffusion language model based on continuous-time Flow Matching that operates entirely within a continuous bidirectional embedding space and repurposes the shared-weight backbone as a discrete decoder only at the final step, dramatically outperforming discrete and continuous DLM baselines with fewer sampling steps and 10× fewer training tokens.

Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization

This paper operationalizes epiplexity—the structural information a compute-bounded learner extracts from data—into tractable online training signals for both adaptive domain selection (EpiSelect) via cross-domain scaling laws and verifier-free synthetic data generation (EpiGen) via buffer-grounded REINFORCE policy gradients, significantly improving out-of-distribution generalization.

Extracting Training Data from Diffusion Language Models via Infilling

Addressing the limitation where conventional prefix-conditioned probing severely underestimates memorization in diffusion language models (DLMs), this paper introduces infilling extraction parameterized by an arbitrary binary mask, revealing how mask geometry and bidirectional inductive bias govern extraction rates and showing that DLMs can leak redacted PII at higher recall than scale-matched autoregressive models.

Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization

This paper introduces a holistic MoE architecture optimization framework that establishes a joint constraint triad of computation (\(M\)), active parameters (\(N_a\)), and total parameters (\(N\)), factorizing the \(\mathcal{O}(n^{16})\) hyperparameter search space into an \(\mathcal{O}(n^3)+\mathcal{O}(n^2)\) two-phase search via algebraic constraints and a rank-preserving property of the hidden dimension, validated across 670+ models to provide deterministic blueprints from any compute budget to optimal MoE configurations.

How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

By fixing an effective forward depth of 20 layers and fitting a joint scaling law across recurrence counts \(r \in \{1, 2, 4, 8\}\), this paper measures that a single recurrence is worth only \(\varphi = 0.46\) in equivalent unique parameters (roughly half a unique block), and establishes \(\Delta \varphi\) as a diagnostic tool that separates genuine architectural capacity gains from deployment-cost trade-offs.

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

The paper demonstrates that language model hidden states encode task information before their own readout (unembedding matrix) natively expresses it, introduces parameter-trajectory crosscoding to track full-vocabulary readout weight evolution across checkpoints without activations, and proves that accelerating readout learning closes this developmental lag without sacrificing final loss.

Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining

Addressing the mismatch between fixed self-supervised proxy tasks and downstream capability goals in continued pretraining, this paper proposes V-pretraining, a framework where a lightweight task designer uses first-order downstream feedback gradients to dynamically shape pretraining targets or views without directly supervising the learner.

Leviathan: Decoupling Input and Output Representations in Language Models

This paper introduces Leviathan, an architecture that replaces discrete token lookup tables with learned embedding vectorization (LEV) to continuously generate input representations, fully decoupling input embeddings from the output projection head with negligible parameter overhead (0.2% at 1.2B scale) and yielding an 81% perplexity reduction on rare tokens.

LittleLearner: Language Models Under Pedagogically-Controlled Knowledge Exposure

This paper introduces LittleCurriculum, an 88B-token pretraining corpus strictly bounded to U.S. elementary school (K–5) curricula, and trains LittleLearner (a 5B LLM from scratch) to provide an interpretable sandbox demonstrating that downstream capability ceilings are governed by pretraining exposure, whereas post-training RL and in-context learning merely elicit existing priors rather than inducing out-of-scope reasoning.

Looped Diffusion Language Models

This paper presents LoopMDM, the first approach integrating looped architectures into masked diffusion language models (MDMs), which selectively loops a small set of early-middle layers with shared weights to match baseline NLL with up to 3.3× fewer training FLOPs, outperform deeper feedforward MDMs on mathematical reasoning, and enable flexible test-time compute scaling by treating masked positions as parallel computational workspaces.

LoopRPT: Reinforcement Pre-Training for Looped Language Models

Addressing the structural mismatch between implicit multi-step latent computation and sparse token-level rewards in looped language models, LoopRPT introduces a reinforcement pre-training framework that leverages high-entropy token selection, step-wise dense rewards anchored by a dynamic EMA teacher, and noisy latent rollouts to improve intermediate representation quality and achieve Pareto dominance in accuracy-computation trade-offs.

LoopRPT: Reinforcement Pre-Training for Looped Language Models

Addressing the structural mismatch where looped language models compress multi-step reasoning into recurrent latent transitions rather than explicit tokens, LoopRPT constructs dense step-wise reinforcement signals using a dynamic EMA teacher and entropy-based hard-token filtering, jointly optimizing the adaptive exit policy and intermediate representations via noisy latent rollouts to achieve superior accuracy–computation Pareto dominance.

Lumos: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs

By leveraging OLMo 2 with its fully transparent pretraining corpus, Lumos traces parametric knowledge along the causal chain \(\mathcal{D} \to \mathbf{h} \to p \to y\), demonstrating that factual reliability and self-verification calibration are fundamentally anchored in training data exposure rather than model scale.

Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learning

This paper establishes the first theoretical guarantee that a pretrained Mamba model achieves test-time feature learning through nonlinear gating, solving in-context learning of single-index models while surpassing Correlational Statistical Query (CSQ) lower bounds to match the near-optimal sample complexity of nonlinear Transformers.

MemDLM: Memory-Enhanced DLM Training

MemDLM introduces a Bi-level Optimization framework that embeds a simulated progressive denoising trajectory into training via fast weights serving as Parametric Memory, which offloads the memorization burden from token-space attention, markedly improves long-context representations even under Train-Only deployment, and optionally enables in-weight retrieval adaptation at inference time.

Neural Neural Scaling Laws

Replacing rigid parametric assumptions and scalar loss averaging in conventional scaling laws, NeuNeu frames downstream performance forecasting as neural time-series extrapolation over token-level validation probability distributions and historical accuracy trajectories, cutting mean absolute error by 44% across 66 tasks while providing calibrated uncertainty estimates.

Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

Addressing the issue where standard optimizers converge to average-type minimizers that harm downstream transfer, this paper proposes Nexus—a plug-and-play second-order gradient approximator that maximizes gradient cosine similarity across heterogeneous data sources, driving models toward tightly clustered common minima and dramatically boosting downstream performance under identical pretraining loss.

On Neural Scaling Laws for Weather Emulation through Continual Training

Using minimal Swin Transformers and a constant learning rate schedule with rapid periodic cooldowns, this paper constructs empirical IsoFLOP scaling frontiers spanning two orders of magnitude in compute (\(6\times 10^{17} \sim 6\times 10^{19}\) FLOPs) for global weather emulation, determines the compute-optimal model-data allocation, and exposes multi-epoch overfitting saturation when scaling past one billion parameters.

On the Invariance and Generality of Neural Scaling Laws

This paper introduces an information-theoretic framework for neural scaling laws, proving that bijective data transformations preserve scaling exponents and irreducible loss, while non-bijective transformations alter scaling via variance inflation and optimal loss shift governed by information resolution, enabling sweep-free cross-domain scaling prediction.

On the Nonlinearity of Learning Rate Scaling for LLM Training

This paper systematically evaluates the breakdown of traditional log-linear learning-rate scaling laws for LLM training, reveals systematic upward curvature in optimal learning rates at scale, and demonstrates that adopting the effective learning rate \(\eta_{\mathrm{eff}}\) combined with data-axis extrapolation eliminates this nonlinearity while reducing hyperparameter tuning compute by over 80%.

One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning

Unifies parameter and data selection into two bilevel optimization problems under a shared validation objective, and introduces DualSFT to co-extract parameter masks and data subsets via closed-form row and column aggregations of a single gradient interaction matrix.

Optimal Scaling Laws in Learning Hierarchical Multi-Index Models

In a genuinely representation-limited regime for hierarchical multi-index targets, this paper establishes exact Bayes-optimal scaling laws for subspace recovery and prediction error, proving that a target-agnostic spectral estimator and shallow neural networks achieve the information-theoretic optimal rates while explaining sequential concept emergence and error plateaus.

Prefix-Tuning for Arbitrary Output Sequences on Pretrained Transformers

Addressing the long-standing open question regarding whether prefix-tuning can force a pretrained autoregressive Transformer to emit arbitrary targets, this paper proves that a solution exists under the necessary condition that the target lies in the model's generation range, providing a training-free linear solver via attention matching that achieves near-perfect output control on GPT-2 and Qwen3-4B.

Progressive Residual Warmup for Language Model Pretraining

To eliminate representation chaos and gradient noise caused by premature deep-layer updates in early Transformer pretraining, Progressive Residual Warmup (ProRes) scales residual branches with a depth-dependent factor that warms up linearly from 0 to 1, letting deep layers wait for shallow representations to stabilize before fully engaging.

ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload

To tackle frequent GPU failures and severe checkpoint-restart overheads in hyperscale LLM pre-training, ReCoVer proposes a decoupled three-layer forward recovery architecture—spanning resilient communication collectives, in-step fine-grained snapshot recovery, and dynamic versatile workload redistribution—maintaining a constant global microbatch count per step to guarantee strict stochastic equivalence to a failure-free run without rollback, replay, or idle hot spares.

Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons

Introduces the ELM Network architecture and an exact closed-form information-theoretic framework to systematically demonstrate that allocating a fixed parameter budget between neuron count, per-unit internal complexity, and connectivity produces a non-trivial optimum, tracing Pareto-optimal scaling laws across three orders of magnitude.

Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers

To eliminate the severe loss barriers encountered when linearly interpolating independently pretrained Transformers in raw weight space, this paper systematically formalizes the complete functionality-preserving symmetry family of Transformers and proposes Dual Learned Matching (LMC-DM), achieving low-barrier linear connectivity and effective model merging in billion-parameter models for the first time.

Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models

Addressing the severe patch lag in byte-level patch-based language models where larger patches degrade quality, Scratchpad Patching (SP) adaptively inserts transient scratchpads within patches to refresh trunk context without polluting the persistent KV cache, matching byte-level baseline quality at a \(16\times\) smaller KV cache and \(3\text{--}4\times\) lower inference compute.

SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization

SimReg introduces an embedding similarity contrastive regularization loss for autoregressive LLM pretraining that explicitly pulls token representations with identical target labels together while pushing apart representations with differing labels, geometrically expanding classification margins to accelerate convergence by over 30% and improve zero-shot downstream benchmarks by over 1%.

Spectral-Sphere-Constrained Hyper-Connections

s²HC replaces nonnegative doubly stochastic constraints on multi-stream residual matrices with a mean-preserving spectral norm sphere, using dynamic rotations and bounded scaling in the zero-sum subspace to enable non-degenerate mixing and achieving average accuracy of 47.1, 50.2, and 50.7 across eight benchmarks on three language models pretrained from scratch.

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

The paper proposes introducing a lightweight synthetic pre-pre-training (PPT) stage using sequences generated by ensembles of randomly initialized RNNs prior to natural language pre-training, which steers language models away from noise self-modeling on noisy web corpora and saves up to 49% pre-training tokens to achieve the baseline final loss at the 1B parameter scale.

The State-Prediction Separation Hypothesis

The paper formulates the State-Prediction Separation (SPS) hypothesis and introduces an interleaved two-stream Transformer where input tokens form a persistent state stream while dedicated <predict> tokens handle next-token predictions, yielding 2–3% zero-shot accuracy gains and up to 2.6× data efficiency across 53M–1.68B parameters with near-zero inference memory overhead.

Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model

Using optimal control theory on a powerlaw random feature model trained with SGD, this paper analytically derives closed-form global optimal learning rate schedules, revealing an "easy phase" (polynomial decay) and a "hard phase" (warmup-stable-decay like annealing), and explaining empirical learning rate transfer disparities between ResNets and Transformers.

Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling

To resolve the dimensional collapse induced by quality filtering and overcome the prohibitive computational overhead of existing geometric diversification methods, this paper proposes Leverage Score Sampling (Lev), which maximizes the determinantal volume expansion of the representation subspace via a single shared Gram matrix and Cholesky decomposition, achieving up to a 72x speedup alongside substantial gains in commonsense and code benchmarks.

Towards Understanding Self-Pretraining for Sequence Classification

Through systematic empirical ablations, a minimal single-layer synthetic task, and theoretical proof, this paper reveals the mechanism behind self-pretraining (SPT): mean-pooled label supervision suffers from local gradient blindness under uniform attention initialization, while masked token reconstruction provides non-zero gradients that steer query-key projections to convert absolute sinusoidal positional encodings into proximity-biased attention priors.

Training Language Models via Neural Cellular Automata

Employs algorithmic, non-linguistic spatiotemporal trajectories generated by discrete Neural Cellular Automata (NCA) as a synthetic substrate for LLM pre-pre-training, showing that merely 164M NCA tokens surpass 1.6B tokens of natural language in downstream perplexity reduction (up to 6%) and convergence speedup (up to 1.6x) while pinpointing attention layers as the primary carrier of transferable computational priors.

TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond

Addressing the structural mismatch caused by naively importing language modeling recipes to trajectory data, this paper formalizes multi-entity spatiotemporal event streams (MESES) under three foundational axioms and introduces TraXion, which couples joint-attribute perturbation noise detection with dual-role entity prototype contrastive learning over a co-occurrence-augmented factorized Transformer to consistently outperform specialized baselines across mobility, cyber authentication, and clinical domains.

Universal Byte-Level Encoding: UTF-8/UTF-16 Routing to Reduce Cross-Script Token-Budget Disparities

To alleviate the severe token-budget penalty imposed on non-Latin scripts by UTF-8 byte-level fallbacks in large language models, this paper proposes Universal Byte-Level Encoding (UBE)—a deterministic, marker-free scalar routing between UTF-8 and UTF-16 substrates that lowers the pre-merge encoding floor of 3-byte BMP characters from 3 to 2, significantly narrowing cross-lingual token disparity while preserving exact reversibility and standard BPE training.

What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation

ImagineAgent introduces an agentic framework that conducts supervised fine-tuning using the newly constructed hicodet-6K chain-of-thought dataset, equips the agent with a multimodal tool library (retrieval, cropping, and generative viewpoint transformation/outpainting), and applies GRPO reinforcement learning with a composite Hungarian F1 reward to significantly boost open-vocabulary HOI comprehension.

Z0-Inf: Zeroth Order Approximation for Data Influence

Z0-Inf bypasses compute-intensive and inaccurate backpropagation gradients by constructing finite difference directional gradients and loss variance proxies from intermediate checkpoint losses, delivering orders of magnitude in speedup and memory savings while substantially outperforming first- and second-order methods in correlation with the subsample-and-retrain baseline on fine-tuned LLMs.

Aligning Inductive Bias for Data-Efficient Generalization in State Space Models

This paper proves from a kernel-theoretic perspective that the inductive bias spectrum of linear time-invariant state space models (LTI SSMs) is governed by their frequency response, and introduces Task-Dependent Initialization (TDI) to align the initial SSM response with task spectra, significantly boosting data efficiency under limited sample regimes without adding parameters or inference overhead.

How to Train Your Latent Diffusion Language Model Jointly with the Latent Space

Addressing the limitation of frozen or suboptimal continuous spaces in text diffusion, this paper presents LDLM, a latent diffusion language model where the latent encoder, continuous denoiser, and decoder are trained end-to-end via hidden-state MSE reconstruction, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise, achieving superior generation quality with a 2-13x sampling speedup over prior discrete and continuous diffusion language models.

Model-Aware Tokenizer Transfer

Proposes Model-Aware Tokenizer Transfer (MATT), which leverages an Attention Influence Modeling (AIM) distillation objective to align inter-token attention dynamics between source and target tokenizers at a truncated layer depth, recovering over 90% of generative and discriminative performance within a few GPU hours.