Skip to content

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/openbmb/UltraX
Area: LLM / NLP
Keywords: Pre-training data refinement, programmatic editing, function calling, data efficiency, large language models

TL;DR

UltraX introduces an adaptive function-calling refinement framework for pre-training corpora that completes the editing function space by integrating line insertion alongside deletion and modification. Operating via dataset-adaptive prompt optimization, hierarchical text-to-operation mapping, and conflict-resolving post-processing, its 0.6B lightweight refiner achieves up to 2% downstream relative gains across five diverse corpora and matches or exceeds 20B-token baselines using only 16B training tokens.

Background & Motivation

As the pre-training scale of large language models (LLMs) approaches tens of trillions of tokens, the volume of publicly accessible high-quality web data is rapidly nearing its physical ceiling. Consequently, the performance dividends traditionally unlocked by compute and dataset expansion under Scaling Laws have begun to encounter severe diminishing returns. Under this data constraint, the focal point of foundation model research has decisively shifted from indiscriminate token expansion toward systematically elevating data utilization efficiency—unlocking latent model capabilities by maximizing the information density, syntactic coherence, and semantic cleanliness of existing web corpora. However, uncurated web dumps (such as Common Crawl) remain severely polluted with boilerplate navigation chrome, advertising copy, tracking snippets, broken layout fragments, and crawler encoding corruptions. Harmonizing fine-grained semantic restoration, extreme inference throughput, and deterministic execution safety across billions of tokens represents a fundamental engineering bottleneck.

Existing data enhancement methodologies suffer from a chronic trilemma between quality, efficiency, and operational reliability. Conventional rule-based filtering and heuristic cleaning rely on human-engineered criteria (such as language identification scores, perplexity thresholds, and duplication ratios); while highly parallelizable and computationally frugal, their rigid heuristics fail to adapt to instance-level nuances, resulting in the aggressive over-filtering of partially noisy yet information-dense documents. Conversely, model-driven end-to-end (E2E) text rewriting harnesses the deep contextual reasoning of frontier LLMs to rewrite imperfect passages, but requires generating an output sequence proportional in length to the raw input. This incurs prohibitive GPU inference costs, creates severe throughput bottlenecks for multi-billion-token corpora, and induces structural hallucinations or boundary fragmentation when reassembling segmented long-form documents. Emerging programmatic data refinement paradigms (such as ProX and RefineX) alleviate computational overhead by tasking smaller models with predicting discrete editing operations applied via an executor; nevertheless, they suffer from an incomplete edit function space (RefineX is restricted exclusively to deletion, while ProX lacks insertion), rely on crude hard-filtering or direct LLM code generation for seed data, and frequently trigger string-matching collisions and intra-line operation interference during large-scale execution.

To resolve these tensions, the authors observe that while lightweight models struggle with open-ended high-fidelity text generation, they excel at constrained, structured prediction over standardized formal semantics. Core idea: UltraX establishes a scalable, function-calling refinement framework for pre-training corpora that completes the editing function space by incorporating line insertion alongside line deletion and localized modification; it leverages dataset-adaptive prompt optimization to guide an expert LLM in creating high-fidelity seed pairs, translates text differences into structured program supervision via hierarchical line alignment and dynamic context expansion, and enforces sliding-window aggregation with conflict-resolution post-processing to execute robust, interpretable data cleaning at scale with minimal token overhead.

Method

Overall Architecture

UltraX reformulates large-scale pre-training data cleaning as a line-anchored sequence prediction and deterministic execution task. The architecture is structured into two sequential stages: Stage I (Data Refinement Model Construction) extracts representative document slices from raw web corpora, uses an automated prompt-optimization agent driven by dataset profiling and multi-dimensional LLM judging to evolve dataset-specific prompts, and employs an expert LLM (DeepSeek-V3.2) to generate gold-standard end-to-end cleaned texts. A hierarchical text-to-operation mapping pipeline (consisting of global Line Alignment Mapping and intra-line Dynamic Context Replacement) translates original-refined text pairs into structured function calls, which are filtered for low-confidence outliers and balanced via ratio-controlled sampling to fine-tune a 0.6B lightweight Refiner. Stage II (Program Execution at Scale) applies sliding-window segment-wise inference with a 20% overlap across massive web corpora, aggregates local function calls into a global line space, executes rigorous post-processing (including ambiguous replacement pruning, same-line operation merging, and degenerate duplicate pattern fallback), and deterministically reconstructs refined, high-fidelity pre-training text.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    RAW["Raw Pre-Training Web Corpora<br/>(d ∈ D)"] --> D1["Complete Function Space & Line-Numbered Modeling<br/>Five-operator space · Physical line-marker anchoring"]
    D1 --> D2["Dataset-Adaptive Prompting & Hierarchical Mapping<br/>Agent-driven prompt evolution · Line alignment & dynamic context"]
    D2 --> D3["Ratio-Controlled Sampling & Instruction-Guided SFT<br/>Edit-weighted operation balance · 0.6B Refiner SFT"]
    D3 --> D4["Sliding-Window Aggregation & Robust Post-Processing<br/>Non-overlap aggregation · Same-line merging & fallback guardrail"]
    D4 --> OUT["Deterministic Execution & Refined Corpus<br/>Refined Corpus (d_hat)"]

Key Designs

1. Complete Function Space & Line-Numbered Modeling: Resolving semantic gaps and layout corruption -> Five-operator programmatic interface with physical line anchoring -> Guaranteeing complete edit expressivity and traceable execution
Prior programmatic refiners are functionally handicapped: RefineX permits only deletion, while ProX is restricted to deletion and replacement. When web crawler artifacts truncate paragraph transitions, strip lists, or excise critical punctuation, deletion-only or replacement-only rules inevitably cause syntactic collapse or unrecoverable semantic loss. UltraX overcomes this by formalizing a complete five-operator editing space \(\Omega\): document-level directives keep_all() (preserving clean inputs unmodified) and remove_all() (discarding entirely uninformative documents); line-level directives remove_lines(start_line, end_line) (excising contiguous noise blocks like cookie alerts or navigation headers) and add_line(base_line, sub_idx, new_str) (restoring missing line structures or lost transitions using fractional sub-indices); and intra-line directive replace_str(line, source_str, target_str) (sanitizing inline HTML tags or broken encoding artifacts). Documents are partitioned into line sequences prefixed with explicit <lid:N> tags. By anchoring all operations to discrete line identifiers and contextual substrings rather than raw character offsets, UltraX prevents error accumulation across sequential modifications, drastically curtails output token volume, and ensures full human and algorithmic auditability.

2. Dataset-Adaptive Prompting & Hierarchical Mapping: Overcoming heterogeneous corpus noise and ambiguous substring matching -> Multi-agent prompt evolution and dynamic context expansion -> Constructing high-fidelity executable program supervision
Web corpora originating from distinct platforms present radically divergent noise distributions and syntactic profiles; a static prompt tuned for general articles fails on code-rich or table-dense texts. UltraX implements an autonomous Prompt Optimization Agent that samples 15 documents per corpus to generate a structural profile, and iteratively runs up to 200 optimization cycles against a 10-point, five-dimensional Judge Model (evaluating noise removal, no over-editing, content preservation, format integrity, and valueless detection) via a Meta-Optimizer (Gemini-3-Flash). The resulting dataset-tailored prompt directs the expert LLM (DeepSeek-V3.2) to produce end-to-end clean reference texts. To convert text pairs \((x, \hat{x})\) into executable functions without human intervention, Line Alignment Mapping computes candidate line correspondences by scoring content similarity (weight 0.6), contextual similarity (weight 0.2), and relative position (weight 0.2), greedily matching non-crossing line pairs and mapping unmatched lines into remove_lines or add_line. For intra-line differences identified by SequenceMatcher, Dynamic Context Replacement iteratively expands context boundaries outward from the minimal edit span until source_str is strictly unique within the targeted line, entirely eliminating substring matching ambiguity caused by common tokens (e.g., "the", "and"). Outlier filters (discarding examples with \(\ge 20\) operations, string lengths \(\ge 150\), or add lengths \(\ge 200\)) retain 1.62M ultra-reliable SFT examples.

3. Ratio-Controlled Sampling & Instruction-Guided SFT: Addressing passive prediction bias and aggressive over-editing -> Edit-weighted category balancing with explicit system instructions -> Equipping a 0.6B lightweight model with robust refiner capability
In raw seed data, documents requiring zero modification naturally dominate, with keep_all() accounting for approximately 60% of cases; naively training on this raw distribution causes small models to collapse into a passive identity-mapping shortcut. Conversely, completely pruning untouched texts induces aggressive over-editing and semantic distortion. UltraX applies ratio-controlled sampling, contrasting preservation-weighted sampling against an edit-weighted strategy. The edit-weighted strategy is selected for final deployment: compound operations (remove_lines + replace_str) are upweighted to 45.92%, single remove_lines and replace_str are allocated 15.31% and 12.87% respectively, document-level retention and deletion are held at 14.42% and 7.54%, and complex insertions (add_line combinations) comprise 3.94%. SFT is conducted across all parameters of Qwen3-0.6B using ms-swift with a 20,480 context length. Critically, every training and inference instance is prepended with a standardized System Instruction defining operation syntax, deletion limits, and the core preservation heuristic ("when in doubt, keep"), while ignore_empty_think masks empty thinking traces. This enables a compact 0.6B parameter model to achieve rigorous, conservative programmatic editing capabilities matching much larger architectures.

4. Sliding-Window Aggregation & Robust Post-Processing: Preventing window boundary fracturing and consecutive operation interference -> Overlap-aware global merging, span consolidation, and degenerate duplicate fallback -> Enabling enterprise-grade deterministic execution
Processing overlong web documents with non-overlapping chunking causes severe semantic fragmentation across boundaries, while raw model outputs risk colliding during consecutive intra-line edits or degrading into repetitive loops. UltraX enforces a 12K-token sliding window with 20% overlap across input documents. During global aggregation, local predictions are mapped to absolute line indices <lid:N>, and operations originating within the overlapping margins of subsequent windows are strictly dropped, allowing the antecedent window to retain unbroken control over boundary text spans. During program parsing, syntactically invalid functions are discarded, and multiple replace_str calls occurring on the identical line undergo span extraction and greedy merging into a single consolidated replacement, eliminating execution conflicts. Finally, a conservative Duplicate-Pattern Detection Fallback monitors generative stability: if identical add_line calls occur \(>2\) times, identical line removals occur \(>3\) times, identical string replacements exceed \(>30\) instances, or dense high-similarity insertion clusters appear, the entire document's edits are aborted and reverted to the pristine raw input. This safeguard triggers on only 0.019% of documents across 29.2M samples, providing an infallible guardrail against pathological model generation during large-scale production.

Loss & Training

The Refiner model is trained via token-level causal language modeling cross-entropy loss over the predicted function sequence: $$ \mathcal{L}{\text{SFT}}(\theta) = - \sum, I) $$ where }^{|Z|} \log P_{\theta}(z_t \mid d, z_{<t\(d\) denotes the line-prefixed input document, \(I\) is the invariant System Instruction defining operational boundaries, and \(Z = (z_1, \dots, z_{|Z|})\) is the serialized sequence of function invocations. Training is executed using the AdamW optimizer (\(\beta_1=0.9, \beta_2=0.95\), weight decay 0.1) with an initial learning rate of \(3 \times 10^{-5}\) decaying via cosine schedule to \(3 \times 10^{-6}\) following a 3% linear warmup.

Downstream evaluation models are trained from scratch using the Megatron-LM framework. The base architecture is a 1.36B parameter MiniCPM model (1.25B non-embedding parameters) comprising 52 Transformer layers, hidden dimension 1,536, intermediate FFN dimension 3,840, 24 query heads with 8 KV groups (GQA), SwiGLU activation, and RMSNorm. Maximal Update Parameterization (\(\mu\text{P}\)) is incorporated with an embedding scale factor of 12 and depth scale factor of 1.4. The pre-training protocol consumes a global batch size of 512 sequences of length 4,096 (~2.1M tokens per batch) across 10,000 iterations for a total consumption of 20B/21B tokens (extended to 100B tokens for scalability analysis), ensuring downstream benchmark gains stem entirely from corpus quality optimization.

Key Experimental Results

Main Results

Evaluations are performed using LightEval across ten established benchmarks (ARC-C, ARC-E, CSQA, HellaSwag, MMLU, OpenBookQA, PIQA, SocialIQA, WinoGrande, SciQ) under normalized zero-shot accuracy, comparing models pre-trained from scratch on 20B tokens across five major corpora.

Pre-Training Corpus Method ARC-C ARC-E CSQA HellaS MMLU OBQA PIQA SIQA WinoG SciQ Avg. #Win
FineWeb Raw 25.85 45.66 35.63 44.89 28.51 31.60 70.35 43.24 51.54 73.50 45.08 0 / 10
ProX-C 25.09 45.20 36.94 45.32 28.49 31.40 71.16 42.94 51.14 72.80 45.05 0 / 10
UltraX (Ours) 26.62 45.96 37.43 46.29 28.90 31.80 71.98 43.71 52.72 76.00 46.14 10 / 10
RedPajama-V2 Raw 23.46 43.69 32.02 39.64 27.50 30.80 68.66 41.56 52.64 70.90 43.09 2 / 10
ProX-C 25.60 44.23 33.01 40.19 27.63 31.00 68.61 42.37 50.43 71.60 43.47 1 / 10
UltraX (Ours) 24.40 45.62 33.91 40.80 27.88 32.20 68.44 42.68 51.30 72.60 43.98 7 / 10
AICC Raw 24.06 40.99 31.70 35.91 27.37 28.80 66.16 42.07 49.57 69.70 41.63 2 / 10
ProX-C 25.17 42.09 32.27 37.85 26.95 30.20 66.70 41.40 49.64 69.20 42.15 3 / 10
UltraX (Ours) 24.49 42.85 31.61 38.03 26.98 29.80 68.34 41.71 50.28 70.20 42.43 5 / 10
Ultra-FineWeb Raw 32.00 57.28 33.58 44.56 30.62 34.60 70.40 41.15 51.85 78.50 47.45 2 / 10
ProX-C 31.31 56.61 32.84 45.29 31.03 35.20 71.65 42.27 49.88 76.70 47.28 3 / 10
UltraX (Ours) 31.31 58.25 33.91 45.66 30.97 37.20 70.08 42.17 51.38 80.50 48.14 5 / 10
FineWeb-ProX-Doc Raw 29.35 52.57 35.22 46.31 30.21 35.20 69.37 42.27 52.41 76.00 46.89 1 / 10
ProX-C 29.35 55.05 35.22 46.41 30.40 35.60 69.64 42.48 51.14 76.60 47.19 2 / 10
UltraX (Ours) 30.38 55.01 36.53 47.17 31.05 37.00 69.15 42.99 51.70 78.30 47.93 7 / 10

Ablation Study

Controlled ablations on FineWeb explore function space compositions, instruction guidance, seed sampling strategies, extended 100B pre-training budgets, and computational throughput.

Ablation Category Configuration / Variant Downstream Avg. #Win Tasks Key Observation / Mechanism
Program Function Space (Table 12) Document-level only (keep/remove_all) 45.22 2 / 10 Eliminates gross junk, but lacks intra-document cleaning
Document-level + Intra-line (replace_str) 44.91 0 / 10 Drops below Raw (45.08) due to unhandled structural noise
Document-level + Line-level (add/remove_lines) 45.35 1 / 10 Excises noise blocks but leaves inline tags intact
Full UltraX Function Space 46.14 8 / 10 Synergistic multi-level operations unlock maximum gain
Instruction Guidance (Table 7) Raw Uncleaned Baseline 45.08 0 / 10 Benchmark reference
UltraX (No System Instructions) 45.73 2 / 10 Effective programmatic gains without explicit boundaries
UltraX (Instruction-Guided) 46.14 8 / 10 Explicit operation boundaries boost small model stability
Seed Operation Distribution (Table 9) Preservation-Weighted Sampling 45.99 5 / 10 60% keep_all prevents over-editing; lower repair rate
Edit-Weighted Sampling 46.14 5 / 10 45.9% compound edits optimize fine-grained cleaning
100B Training Budget Scaling (Table 13) Raw (100B Tokens) 50.85 1 / 10 Long-schedule baseline
ProX-C (100B Tokens) 51.58 1 / 10 Conventional deletion/replacement baseline
UltraX (100B Tokens) 52.13 8 / 10 Persistent advantage (+1.28 over Raw, +0.55 over ProX-C)
System / Method Refinement Paradigm Quality Score (/10) Output Tokens / Doc Throughput (docs/s) GPU Time (10K Docs)
ProX-C-0.3B Programmatic Editing 9.31 38 890.0 0.0030 h
UltraX-0.6B (Ours) Adaptive Programmatic Editing 9.64 54 820.0 0.0034 h
Qwen3-4B End-to-End LLM Rewriting 9.52 612 35.0 0.0800 h
Qwen3-30B-A3B End-to-End LLM Rewriting 9.67 628 25.0 0.1100 h

Key Findings

  • Surpassing 20B Token Baselines at 16B Tokens: Training trajectory analyses reveal that UltraX attains an average score of 45.49 at only 16B consumed tokens, already surpassing the full 20B token saturation baselines of both Raw (45.08) and ProX-C (45.05). By 20B tokens, UltraX advances to 46.14, demonstrating that programmatic cleaning substantially amplifies the effective semantic density per token.
  • Selective Restoration vs. Indiscriminate Excision: In token-count audits across RedPajama-V2 and AICC, ProX-C reduced total token counts by 21.8% and 33.1%, entirely annihilating 2.27M and 5.33M documents into empty strings. Conversely, UltraX preserved 91.99% and 87.35% of non-empty documents, yet outperformed ProX-C downstream. Paired LLM-based quality judgments over 80,000 documents confirmed that UltraX achieved an average quality score of 9.60 (vs. 9.17 for ProX-C) and slashed low-quality samples (\(\le 5\)) from 2.59% down to 0.38%.
  • Indispensability of Multi-Level Functional Synergy: Eliminating line-level deletion while retaining intra-line replacement caused downstream accuracy to degrade to 44.91 (worse than Raw), because isolated string edits cannot prune multi-line navigation dumps. Full functional synergy, dominated by joint remove_lines + replace_str operations (accounting for 36.6%–62.5% of modified texts), is essential for high-fidelity data restoration.

Highlights & Insights

  • Inverse Translation of E2E Knowledge into Deterministic Operations: While frontier LLMs demonstrate superior text rewriting capabilities, their token footprint is economically unviable for web-scale datasets. UltraX elegantly distills E2E clean outputs into discrete programmatic function calls via line alignment and dynamic context expansion, reducing output tokens by over 91% (54 vs. 628 tokens/doc) and boosting throughput by \(23\times\) (820 vs. 35 docs/s).
  • Dynamic Context Expansion for Collision-Free String Replacement: Standard substring replacements routinely trigger false-positive matches when modifying frequent monosyllabic tokens. UltraX's mechanism of dynamically growing surrounding context anchors until the target substring achieves mathematical uniqueness within the line provides an elegant, offset-free solution to programmatic text manipulation.
  • Closed-Loop Agentic Prompt Evolution: Rather than forcing a static human-engineered cleaning prompt across diverse web dumps, UltraX's multi-agent meta-optimization loop continuously refines prompt instructions against empirical Judge feedback, autonomously adapting extraction boundaries to distinct domain distributions.

Limitations & Future Work

  • Exploration at Extreme Parameter and Token Scales: Validation is presently constrained to 1B-parameter architectures trained up to 100B tokens; empirical behavior on dense models exceeding 70B parameters or training schedules spanning 10T+ tokens remains to be benchmarked.
  • Monolingual Focus on English Web Data: The current implementation primarily targets English Common Crawl corpora. Multilingual corpora featuring intricate morphological variations, mixed-script documents, and non-Latin formatting require tailored line-tokenization and alignment strategies.
  • Reliance on Seed LLM Priors: Although DeepSeek-V3.2 was verified to exhibit high alignment with human evaluations at lowest cost, the initial seed data remains bound to the latent biases of the generator and judge models. Future work may explore multi-model voting or reinforcement learning with verifiable rewards.
  • Refiner Quantization and Speculative Decoding: The 0.6B Refiner could be further compressed via 4-bit AWQ quantization and speculative execution, potentially unlocking multi-thousand docs/sec per GPU.
  • vs. Document-Level Heuristic Filters (CCNet, Gopher, FineWeb-filter): Conventional heuristic approaches enforce binary retention decisions based on statistical thresholds (e.g., perplexity, character duplication), inevitably discarding salvageable high-value paragraphs. UltraX performs surgical intra-document editing, preserving informative cores while excising peripheral noise.
  • vs. Generative E2E Rewriters (Cosmopedia, Rephrasing the Web): Generative rewriters generate full-length synthetic text, which is slow (25–35 docs/s), memory-intensive, and prone to hallucinations and style homogenization. UltraX outputs compact editing programs (820 docs/s) that deterministically transform original text with zero risk of factual drift.
  • vs. Programmatic Editing Baselines (ProX-C, RefineX): RefineX supports only deletion; ProX-C lacks insertion capabilities, substring disambiguation, and cross-window aggregation. UltraX provides a complete editing space (incorporating add_line), resolves substring collisions via dynamic context expansion, and enforces safe duplicate-pattern fallbacks, consistently outperforming them across diverse benchmarks.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates an expressive, complete function space for programmatic data editing, coupled with elegant dynamic context expansion and agentic prompt optimization.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 5 major corpora, 10 downstream tasks, extensive 20B/100B scratch pre-training runs, and an 80K double-blind quality study.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous technical prose, transparent algorithmic workflows, well-structured appendices, and lucid presentation of design trade-offs.
  • Value: ⭐⭐⭐⭐⭐ High-impact data-centric infrastructure addressing the looming data exhaustion bottleneck in foundational model pre-training; open-sourced and highly practical.