WRAP++: Web Discovery Amplified Pretraining¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Pretraining
Keywords: Synthetic Pretraining Data, Topological Relation Discovery, Cross-Document QA Synthesis, Factual Knowledge Acquisition, Combinatorial Data Amplification
TL;DR¶
Addressing the intra-document information silos and extraction bottlenecks of single-document rewriting, WRAP++ leverages web hyperlink topologies to discover high-confidence entity relationship subgraphs, generating cross-document multi-hop QA pairs that amplify an 8.4B raw token corpus into 82.7B tokens and substantially boosting parametric factual knowledge recoverability on SimpleQA.
Background & Motivation¶
Synthetic data rewriting has emerged as a cornerstone for enhancing factual knowledge acquisition during large language model pretraining, powering prominent foundation models such as Phi-4, Qwen3, and Nemotron-CC. However, conventional web-scale synthetic rephrasing methods operate predominantly at the single-document level, treating individual web pages as isolated units. While single-document rewriting improves sentence clarity and factual density, it inherently restricts synthesized knowledge to intra-document facts, omitting the complex semantic associations distributed across multiple documents in the real world and leaving facts stored with sparse associative contexts.
Overcoming this limitation requires moving toward cross-document synthesis, bringing complementary facts into a shared context to foster joint relational reasoning. Yet cross-document synthesis poses severe data curation hurdles: naively pairing random documents forces the generator to hallucinate spurious connections between unrelated entities, polluting the pretraining corpus with fabricated facts. Conversely, relying on heavy-weight neural entity extraction or dense embedding-based graph construction incurs prohibitive computational overhead at web scale.
The critical unlock lies in discovering high-confidence relational signals without expensive model-based extraction. Web hyperlinks naturally reflect human-curated editorial judgments of semantic relevance and real-world relationships. Core idea: WRAP++ directly mines high-confidence relational motifs (specifically dual-links and co-mentions) from web hyperlink topologies, conditioning an instruction-tuned LLM generator on paired documents under strict cross-document dependency, explicit factual chaining, and context-free internalization constraints to amplify an 8.4B-token corpus into 82.7B tokens of high-quality relational pretraining data.
Method¶
Overall Architecture¶
WRAP++ reformulates synthetic data generation from intra-document rephrasing into a two-stage topology-guided discovery and amplification pipeline. First, in the Topological Relation Discovery stage, the web corpus is modeled as a directed hyperlink graph to mine tightly coupled entity pairs via explicit structural motifs. Second, in the Cross-Document Joint QA Synthesis stage, discovered document pairs are concatenated and fed into an instruction-tuned generator enforcing multi-hop dependency, step-by-step reasoning chains, and context-free factual formulation.
The end-to-end operational flow from raw web parsing to downstream pretraining ingestion is illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Web Corpus & Hyperlink Graph<br/>FineWiki 6.7M Articles / 8.4B Tokens"] --> B["Topological Relation Discovery<br/>Mining dual-link and co-mention motifs"]
B --> C["Cross-Document Factual Chaining<br/>Forced multi-hop dependency and reasoning steps"]
C --> D["Context-Free Internalization<br/>Omniscient standalone QA and parametric consolidation"]
D --> E["82.7B Cross-Document QA Corpus<br/>OLMo continued pretraining & factual elicitation"]
Key Designs¶
1. Topological Relation Discovery: From Single-Doc Silos to Model-Free Graph Motifs
Single-document rewriting quickly reaches a data saturation ceiling, while random document pairing induces rampant factual hallucination. To discover valid semantic relationships at web scale without neural extraction overhead, WRAP++ constructs a directed graph \(\mathcal{G} = (V, E)\) over the corpus \(\mathcal{D} = \{d_1, \dots, d_N\}\), where vertices \(v_i\) represent documents and directed edges \(e_{i,j}\) denote hyperlinks from \(d_i\) to \(d_j\). The framework identifies two topological motifs providing strong relational inductive biases: The first is the Dual-Link Motif, which enforces mutual inter-document referencing: $\(e_{u,v} \in E \wedge e_{v,u} \in E\)$ This bidirectional coupling signifies foundational semantic associations (such as a director and their signature film, or an author and their primary theory). On English FineWiki, this motif extracts approximately 9.6 million entity pairs, generating ~3B tokens of dense QA data. The second is the Co-Mention Motif, where documents \(u\) and \(v\) co-reference a shared structural hub \(E\) while maintaining a direct link between themselves: $\(e_{u,E} \in E \wedge e_{v,E} \in E \wedge e_{u,v} \in E\)$ The common context \(E\) establishes natural analogical, categorical, or comparative relations (e.g., two competing scientific models cited within the same survey). This motif yields over 232 million entity pairs, delivering massive combinatorial headroom and synthesizing ~79.7B tokens. Together, these motifs amplify the 8.4B raw corpus into 82.7B tokens of diverse relational knowledge.
2. Cross-Document Factual Chaining: Enforcing Multi-Hop Dependency and Reasoning Trajectories
Simply providing related document pairs is insufficient, as unguided generators tend to default to shallow single-document fact extraction. WRAP++ imposes strict structural prompt constraints: The generated question \(q_i\) is explicitly forbidden from being answerable using either document \(d_u\) or \(d_v\) in isolation; answering \(q_i\) strictly requires combining complementary premises from both texts. Furthermore, before outputting the final answer \(a_i\), the generator must decode an intermediate reasoning trajectory \(c_i\) that articulates the step-by-step traversal path connecting facts across both documents. For instance, in connecting two composers, the chain explicitly traces their respective Oscar journeys and collaborations before stating the final pair of titles. This explicit factual chaining forces the pretraining model to internalize multi-hop relational pathways rather than memorizing isolated surface strings.
3. Context-Free Internalization: Eliminating Attribution to Consolidate Parametric Memory
If synthetic QA pairs retain passage-attribution markers (such as "According to Passage A" or "Text B states"), language models treat them as reading-comprehension exercises over local context, bypassing deep parametric internalization. WRAP++ mandates a context-free internalization rule: the generator must act as an omniscient AI asserting universal facts directly from its internal knowledge, strictly omitting document indices, source labels, or quotation markers. The synthesized output takes the clean form: $\(\text{Question: } q_i \quad \text{Answer: } c_i \dots \text{ Therefore, } a_i\)$ Because the underlying source documents are typically present in pretraining corpora, these context-free joint QA instances act as powerful catalysts that suture fragmented memories into cohesive, multi-path associative structures.
A Worked Example¶
Consider the synthesis over two topologically linked entities: Swedish composer Ludwig Gรถransson and the film Oppenheimer: - Single-Document WRAP Baseline: Extracts isolated facts from a single page: "Q: Who is Ludwig Gรถransson? A: A Swedish composer." or "Q: What films has Gรถransson scored? A: Black Panther, Tenet, and Oppenheimer." - WRAP++ Cross-Document Synthesis: Pairs Gรถransson's page with Hans Zimmer's page via their shared connection to Christopher Nolan and Tenet. The synthesized question bridges both entities: "What films earned Nolan's original Tenet composer and his replacement their second Oscars?" - Explicit Factual Chaining: "Hans Zimmer turned down Tenet for Dune, which earned his second Oscar (after The Lion King). His replacement, Ludwig Gรถransson, later won his second Oscar for Oppenheimer (after Black Panther). Therefore, the answer is Dune and Oppenheimer." The instance binds four distinct entities and historical timeline events into a unified retrieval pathway.
Loss & Training¶
Continued pretraining is conducted on the fully open-source OLMo-3 foundation checkpoints (at 7B and 32B parameter scales) for 1 epoch:
- Optimization employs SkipStepAdamW with \(\beta_1 = 0.9, \beta_2 = 0.95\), weight decay \(0.1\) (excluding embeddings), and an auxiliary \(z\text{-loss}\) with multiplier \(10^{-5}\) for logit stabilization.
- The peak learning rate is set to \(2.07 \times 10^{-5}\) (inherited from OLMo-3 mid-training recipes) with a linear decay schedule to zero and zero warmup steps (due to loading optimizer states directly).
- The sequence length is fixed at 8,192 tokens. Global batch size is ~2M tokens (\(2^{21}\)) for OLMo-3-7B and ~8M tokens (\(2^{23}\)) for OLMo-3-32B. Running on 256 H20 GPUs, training OLMo-3-7B on 8B tokens requires 4.5 hours (65% MFU); training OLMo-3-32B on 80B tokens using 1,024 H20 GPUs takes 44 hours.
Key Experimental Results¶
Main Results¶
The primary evaluation uses SimpleQA (measuring short-form factual accuracy via unbiased pass@128 to evaluate factual recoverability without formatting artifacts), supplemented by closed-book multi-hop evaluations on HotpotQA and MuSiQue.
The main SimpleQA results across model scales after 1 epoch of continued pretraining are summarized below (directly quoting Table 1):
| Data Recipe / Setup | Consumed Training Tokens | OLMo-3-7B (SimpleQA pass@128) | OLMo-3-32B (SimpleQA pass@128) |
|---|---|---|---|
| Pretrained Base Checkpoint | 0B | 34.76% | 42.35% |
| + WRAP (Standard single-doc QA) | ~5.4B | 39.55% | 44.43% |
| + Extended WRAP (Exhaustive single-doc QA) | ~17.4B | 43.69% | 47.91% |
| + WRAP++ (Cross-document discovery QA) | ~82.7B | 49.13% | 53.97% |
Transfer to closed-book multi-hop QA benchmarks (pass@8, evaluated at the 80B consumed token checkpoint on OLMo-3-7B, Table 10/2):
| Data Setting | HotpotQA (pass@8 %) | MuSiQue (pass@8 %) |
|---|---|---|
| Pretrained Base Checkpoint | 44.57 | 17.16 |
| Extended WRAP (Cycled 4.6 passes to 80B) | 45.47 | 19.20 |
| WRAP++ (80B distinct synthetic tokens) | 48.12 (+2.65) | 19.61 (+0.41) |
Ablation Study¶
Ablations are conducted on OLMo-3-7B under a controlled ~8B token training budget to evaluate pairing strategies, graph topologies, synthesis formats, and generator sizes (quoting Table 3):
| Ablation Axis | Variant / Configuration | SimpleQA pass@128 (%) | Note & Finding |
|---|---|---|---|
| Pairing Strategy | Random entity pairing | 43.46 | Unrelated pairings force hallucination; drops by 1.65 pp |
| Topological discovery (Default) | 45.11 | 95% CI [0.31, 2.99] confirms graph topology is critical | |
| Relation Topology | Dual-link only | 44.24 | High precision but small corpus (repeated ~3B data within 8B budget) |
| Co-mention only | 44.36 | Broad coverage supplying rich indirect relational paths | |
| Combined topologies | 45.11 | Achieves highest accuracy and massive scaling headroom | |
| Synthesis Method | Raw document concatenation | 35.43 | Marginal over base (34.76); text proximity alone fails |
| QA with source docs prepended | 38.93 | Encourages superficial copying; drops by 6.18 pp | |
| Cross-document joint QA | 45.11 | Context-free format maximizes parametric retention | |
| Synthesis Model | Qwen3-30B-A3B (Active ~3B) | 45.11 | High compute efficiency for large-scale production |
| Qwen3-235B-A22B (Active ~22B) | 47.70 | Larger generator adds +2.59 pp factual quality |
In addition, integrating WRAP++ into a full 100B-token OLMo-3-7B mid-training mixture (Table 5) by replacing 6% (6B tokens) with WRAP++ data: - Average performance across 12 diverse general benchmarks remains virtually unchanged (68.24% vs. 68.16%, -0.08 pp); - SimpleQA accuracy increases from 34.74% to 37.58% (+2.84 pp) and MMLU Redux improves by 1.28 pp, demonstrating seamless integration into foundation model pretraining pipelines without degrading reasoning or math skills.
Key Findings¶
- Overcoming Single-Document Scaling Saturation: In token-matched scaling comparisons from 8B to 80B consumed tokens (Figure 2 and Table 11), cycling Extended WRAP for 4.6 passes saturates within a narrow 43.37%โ44.84% pass@128 band. In contrast, WRAP++ steadily scales from 45.32% to 49.18%, leading at all checkpoints by 1.95 to 5.62 points. Additional training iterations cannot substitute for genuine relational diversity.
- Uniform Recovery Across the k-Spectrum: Tracking unbiased pass@k curves (Figure 3) reveals monotonic upward shifts across the entire log scale \(k \in [1, 128]\). Gains at low \(k\) demonstrate sharper top-ranked precision, while parallel gains at high \(k\) confirm the establishment of broader associative retrieval routes.
- Third Document Overhead: Testing a 3-document co-mention variant (Appendix Table 14) that includes the intermediate bridge entity \(E\) slightly degraded performance (15.42% vs. 15.74% pass@8). Broad structural hubs introduce noisy, distracting context that dilutes the generator's focus on the primary bilateral association.
Highlights & Insights¶
- Hyperlinks as Model-Free Relational Inductive Biases: Web hyperlinks provide human-verified semantic connectivity for free, bypassing computationally intensive entity relation extraction or error-prone dense retrieval.
- Combinatorial Data Amplification: While intra-document fact extraction scales linearly with corpus size, graph-driven document pairing expands combinatorially, generating 80B+ tokens from an 8.4B raw corpus.
- Context-Free Internalization as a Pretraining Principle: Presenting multi-hop factual chains without source text attributions forces the model to encode relational paths directly into transformer parameters rather than relying on in-context reading heuristics.
Limitations & Future Work¶
- Dependence on Clean Encyclopedic Graphs: Experiments are validated on Wikipedia (FineWiki), which possesses clean, entity-centric hyperlinking. Extending to noisy Common Crawl data will require aggressive filtering for navigational and advertising links.
- Computational Overhead of QA Generation: Synthesizing 80B tokens of high-quality QA requires substantial inference compute (~30,720 GPU-hours on H20). Exploring distilled, lightweight synthesis models is a promising efficiency avenue.
- Temporal Consistency Across Documents: Pairing documents from different temporal snapshots may introduce timeline inconsistencies; integrating temporal metadata into relation discovery warrants future study.
Related Work & Insights¶
- vs. WRAP (Maini et al., 2024) / Nemotron-CC (Su et al., 2025): Single-document pioneers focus on intra-document denoising and syntactic formatting; WRAP++ transitions the paradigm to cross-document discovery, breaking intra-document factual limits.
- vs. EntiGraph (Yang et al., 2024) / SoG (Ma et al., 2025): EntiGraph extracts entities from text and SoG traverses context graphs; WRAP++ replaces expensive neural extraction with native hyperlink topology, offering superior scalability for trillion-token pretraining.
- vs. SBP (Yang et al., 2025b): SBP retrieves semantically similar document pairs via dense embeddings, often retrieving redundant content; WRAP++'s dual-link and co-mention motifs capture complementary, contrasting, and multi-hop relationships with higher information gain.
Rating¶
- Novelty: โญโญโญโญโญ [Addresses the single-document ceiling via elegant graph-topology discovery]
- Experimental Thoroughness: โญโญโญโญโญ [Multi-scale (7B/32B), long-horizon scaling (80B), 12 general tasks, and extensive ablations]
- Writing Quality: โญโญโญโญโญ [Clear motivation, structured mathematical formulation, and detailed qualitative examples]
- Value: โญโญโญโญโญ [Highly actionable recipe for synthetic pretraining in the era of web data exhaustion]