directional structurality a lightweight proxy for reasoning data quality
content_hash: d7b7172ad44ffa83 ---# Directional Structurality: A Lightweight Proxy for Reasoning Data Quality
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: reasoning data selection, directional structurality, Local-DeltaAvg, continued pretraining, semantic reversal intervention
TL;DR¶
This paper introduces directional structurality as a lightweight, training-free proxy for reasoning data quality, implementing a local paired negative log-likelihood gap metric (Local-DeltaAvg) via semantic reversal counterfactuals that consistently outperforms length, perplexity, problem difficulty, and LLM-judge heuristics under matched token budgets in continued pretraining and test-time reranking.
Background & Motivation¶
Large language models increasingly acquire complex reasoning abilities by training on vast corpora of multi-step rationales. However, web-scale and synthetic reasoning datasets inevitably mix rigorous step-by-step derivations with verbose explanations, superficial heuristic shortcuts, and noisy rationales. Standard data-filtering approaches primarily rely on sequence length, language model perplexity (PPL), problem difficulty estimators, or LLM-as-a-judge scoring. While practically convenient, these heuristic signals largely capture surface syntactic complexity, text fluency, input hardness, or judge preference rather than the internal logical soundness of the reasoning trace. A candidate sample can be excessively long yet logically vacuous, highly difficult due to obscure vocabulary rather than multi-step deduction, or impeccably fluent while offering zero reasoning progression.
The core tension underlying existing quality selection methods is their neglect of the intrinsic directed dependency structure within reasoning trajectories. High-utility reasoning data is characterized by learnable causal dependencies where premises progressively support intermediate deductions and tighten constraints toward the conclusion. In human-authored mathematical and logical proofs, information flows naturally along this premise-to-conclusion trajectory; reversing this explanatory flow disrupts the premise-dependent support and makes subsequent steps substantially harder to predict autoregressively. Prior unsupervised selection criteria lack a mechanism to directly measure this causal dependency directionality while remaining invariant to lexical overlap, task domain familiarity, and surface length.
This paper tackles the challenge through the lens of directional information asymmetry, arguing that useful reasoning traces exhibit strong directional structurality—meaning intermediate steps are easy to encode along their natural forward order but incur substantial encoding costs under order perturbations. Core idea: construct an answer-first, step-reversed counterfactual trace for each candidate rationale and compute a paired within-sample local negative log-likelihood gap (Local-DeltaAvg) using a fixed probe model, identifying peak directional bottlenecks as a lightweight, training-free proxy for reasoning data utility.
Method¶
Overall Architecture¶
The Local-DeltaAvg framework consists of reasoning trace segmentation and semantic reversal counterfactual construction, bidirectional conditional encoding cost evaluation using a frozen probe language model, anchor-based alignment and sliding-window difference aggregation, and peak bottleneck score ranking for budgeted data selection. The entire pipeline operates without requiring any step-level human annotations, process supervision, or downstream target-model feedback.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Candidate Reasoning Trace<br/>x = (q, u1, ..., um, a)"] --> B["Semantic Reversal Counterfactual<br/>Preserve step-internal tokens, reverse step order"]
B --> C["Bidirectional Probe Encoding Cost<br/>Forward and reversed context NLL scoring"]
C --> D["Anchor Alignment & Local Gap Aggregation<br/>Compute sliding-window Local-DeltaAvg curve"]
D --> E["Peak Bottleneck Selection<br/>Rank candidates via D(x) = max Δ(t; w)"]
Key Designs¶
1. Semantic Reversal Counterfactual: Causal order intervention under fixed content To control for shared lexical content, domain terminology, and general language fluency, the method constructs a counterfactual reference that preserves task content while perturbing the dependency-revealing order. For an original reasoning trace \(x = (q, u_1, \dots, u_m, a)\) containing problem \(q\), intermediate steps \(u_i\), and final answer \(a\), the semantic reversal operator \(\mathcal{R}\) produces: $\(\tilde{x} = \mathcal{R}(x) = (q, a, u_m, u_{m-1}, \dots, u_1)\)$ This formulation directly places the final answer after the question and reverses the explanatory sequence of derivation steps, while strictly maintaining the original word order within each individual step. Unlike naive token-level reversal which results in ungrammatical gibberish, semantic reversal preserves local syntactic validity while cleanly disrupting the availability of preceding premises, making it impossible for an autoregressive model to leverage true logical dependencies during generation.
2. Bidirectional Probe Encoding Cost: Quantifying directional dependency asymmetry Using a fixed lightweight autoregressive probe language model \(p_\theta\) (e.g., Qwen2.5-1.5B-Base), the system evaluates the token-level conditional negative log-likelihood (NLL) along both the forward linearized trace \(x = (s_1, \dots, s_T)\) and the semantically reversed trace \(\tilde{x} = (\tilde{s}_1, \dots, \tilde{s}_{\tilde{T}})\): $\(\ell_f(t) = -\log p_\theta(s_t \mid s_{<t}), \quad \ell_r(t) = -\log p_\theta(\tilde{s}_t \mid \tilde{s}_{<t})\)$ Under a latent dependency graph perspective, if the probe model approximates the data distribution, forward encoding measures step predictability given natural causal context, whereas reverse encoding forces the model to predict prerequisite steps in the absence of their supporting foundations. The difference between reverse and forward encoding costs approximates the conditional entropy difference \(H(U_i \mid C_i^{\text{rev}}) - H(U_i \mid C_i^{\text{fwd}})\), grounding the metric directly in the informational irreversibility of multi-step logical deduction.
3. Anchor Alignment & Local Gap Aggregation: Isolating critical reasoning bottlenecks Because the forward and semantically reversed sequences possess distinct token layouts, an anchor-based alignment mapping \(\pi(\cdot)\) maps forward token coordinates to their corresponding positions in the reversed sequence. For a chosen window size \(w\) (default \(w = 64\)), let \(W_t = \{\tau \mid |\tau - t| \le w/2, 1 \le \tau \le T\}\) denote the local window centered at token position \(t\). The local directional gap is defined as: $\(\Delta(t; w) = \frac{1}{|W_t|} \sum_{\tau \in W_t} \left[ \ell_r(\pi(\tau)) - \ell_f(\tau) \right] = \frac{1}{|W_t|} \sum_{\tau \in W_t} \log \frac{p_\theta(s_\tau \mid s_{<\tau})}{p_\theta(\tilde{s}_{\pi(\tau)} \mid \tilde{s}_{<\pi(\tau)})}\)$ Plotting \(\Delta(t; w)\) across all positions \(t\) yields a local directional gap curve. Sharp peaks along this curve identify directional bottlenecks where critical logical leaps occur. For trace-level sample ranking, the metric aggregates this curve by extracting the peak directional gap: $\(D(x) = \max_t \Delta(t; w)\)$ In contrast to a sequence-wide global average (\(w = \infty\)), peak aggregation prevents diluting decisive multi-step deductions with extensive trivial or descriptive steps.
Loss & Training¶
Local-DeltaAvg is an entirely training-free proxy that requires no updates to the probe model. For downstream validation, continued pretraining (CPT) on selected subsets optimizes the standard causal autoregressive cross-entropy objective: $\(\mathcal{L}_{\text{CPT}}(\phi) = - \sum_{t=1}^T \log p_\phi(s_t \mid s_{<t})\)$ Continued pretraining on Qwen2.5-7B-Base uses a sequence length of 4096, global batch size of 4M tokens, total budget of 500 steps (2B tokens), AdamW optimizer (\(\beta_1 = 0.9, \beta_2 = 0.95\)), peak learning rate of \(2 \times 10^{-5}\) with a 50-step linear warmup and cosine decay to 10%, in bfloat16 on 8× H100 80GB GPUs. Downstream supervised fine-tuning (SFT) follows an identical standardized protocol across all selection methods on ~50K samples (~20M tokens).
Key Experimental Results¶
Main Results¶
From a 20B-token STEM reasoning pool, each data-selection method selects an identical budget of 2B tokens for continued pretraining on Qwen2.5-7B-Base, followed by identical downstream SFT. Evaluation is conducted across five core reasoning benchmarks (GSM8K, MATH500, AIME24, GPQA-D, LiveCodeBench) alongside two non-reasoning control benchmarks (HellaSwag, MMLU). Values represent averages across three independent CPT runs, with evaluation confidence intervals computed via paired bootstrap.
| Selection Method | GSM8K | MATH500 | AIME24 | GPQA-D | LiveCodeBench | HellaSwag | MMLU | Reasoning Avg. | Eval 95% CI |
|---|---|---|---|---|---|---|---|---|---|
| Random-STEM | 66.5 | 52.5 | 12.2 | 45.5 | 52.5 | 83.1 | 74.2 | 55.2 | [54.3, 56.4] |
| Length filter | 66.0 | 52.0 | 11.1 | 46.8 | 52.1 | 82.5 | 73.8 | 54.9 | [53.8, 55.7] |
| Forward PPL / Loss | 67.5 | 53.5 | 12.2 | 46.8 | 52.5 | 83.3 | 74.4 | 55.7 | [54.6, 56.5] |
| LLM Judge / Quality Score | 68.8 | 55.0 | 13.3 | 46.3 | 53.2 | 83.0 | 74.1 | 56.2 | [55.3, 57.1] |
| Local-DeltaAvg (global, \(w = \infty\)) | 70.0 | 56.5 | 14.4 | 47.1 | 54.0 | 82.8 | 73.9 | 57.0 | [56.1, 57.8] |
| Local-DeltaAvg (local, \(w = 64\)) | 71.5 | 58.5 | 16.7 | 47.3 | 54.0 | 82.6 | 73.8 | 57.8 | [56.9, 58.9] |
In multi-seed evaluation across the five primary reasoning benchmarks, Local-DeltaAvg (\(w = 64\)) achieves 49.6 ± 0.2, surpassing Random-STEM (45.8 ± 0.2), Length filter (45.6 ± 0.2), Forward PPL (46.5 ± 0.2), and LLM Judge (47.3 ± 0.2). Crucially, on non-reasoning control benchmarks like HellaSwag and MMLU, Local-DeltaAvg performs flat or marginally lower (82.6 vs 83.1, 73.8 vs 74.2), verifying that the selected data specifically sharpens deductive reasoning rather than generic surface fluency.
Ablation Study¶
Ablations investigate the impact of alternative reversal constructions on both intrinsic correlation with human reasoning quality and downstream continued-pretraining gains, as well as sensitivity to window size \(w\).
| Reversal Construction | Candidate Data | Sample Size \(N\) | Mean DeltaAvg | Human Correlation \(\rho\) | CPT Reasoning Avg. (5 Benchmarks) | Failure Mode / Note |
|---|---|---|---|---|---|---|
| Semantic reversal | GSM8K | 1000 | 0.52 | 0.61 | 49.6 (Full 2B) / 46.7 (Pilot) | Recommended; isolates logical directionality |
| Answer-first only | GSM8K | 1000 | 0.31 | 0.38 | 45.8 | Insufficient reversal; preserves internal steps |
| Sentence reversal | GSM8K | 1000 | 0.78 | 0.29 | 43.6 | Introduces discourse fragmentation artifacts |
| Token reversal | GSM8K | 1000 | 2.94 | 0.05 | 42.2 | Syntactic collapse creates uninformative gap |
| Shuffle rationale | GSM8K | 1000 | 0.18 | 0.07 | 43.1 | Randomized structure destroys reasoning gradient |
Scanning window sizes across \(w \in \{8, 16, 32, 64, 128, 256, \infty\}\) on a 500M pilot budget yields downstream reasoning scores of 45.1, 46.0, 46.8, 47.0, 46.9, 46.1, and 46.5, respectively. \(w = 64\) provides the optimal Pareto trade-off between scoring overhead (1.05×) and critical step alignment accuracy (IoU 0.43, R@3 0.71).
Key Findings¶
- Semantic counterfactual intervention is indispensable: Naive token-level reversal yields a massive raw NLL gap of 2.94 due to ungrammatical sequence collapse, but its correlation with human reasoning quality is near zero (\(\rho = 0.05\)) and downstream CPT performance drops to 42.2. In contrast, semantic reversal achieves \(\rho = 0.61\) and the highest downstream reasoning gain.
- Directional bottlenecks align with human-annotated critical steps: On PRM800K and MATH-Shepherd process-supervision corpora, high-gap regions identified by Local-DeltaAvg (\(w = 64\)) strongly overlap with human-annotated critical steps (0.43 IoU and 0.71 R@3), dramatically outperforming random position baselines (0.08 IoU, 0.21 R@3).
- Strong robustness across probe architectures and base models: Probe scoring rankings correlate consistently across Qwen2.5-7B, Qwen2.5-14B, Llama-3.1-8B, Mistral-7B, and Gemma2-9B (Spearman \(\rho \in [0.62, 0.79]\)). When applied to Llama-3.1-8B-Base and Qwen2.5-14B-Base, the metric yields +4.9 and +4.1 point gains over Random-STEM, respectively.
Highlights & Insights¶
- Formulating data quality as causal perturbation sensitivity: Instead of relying on static, uncalibrated scores like perplexity or length, the paper measures how predictability degrades when dependency order is perturbed, naturally canceling out domain familiarity and token-frequency biases.
- Unsupervised process-level bottleneck discovery: Without training process reward models (PRMs) or collecting step-level labels, sliding-window bidirectional scoring natively exposes reasoning bottlenecks, achieving test-time Best-of-N reranking competitive with supervised PRMs (80.8% on MATH500).
- Generalization to non-linear DAG structures: The directional structurality principle naturally generalizes to reasoning trees and directed acyclic graphs via TRCG/GRCG, capturing shared dependency structures in Olympiad-level problems for an additional +2.0 point gain.
Limitations & Future Work¶
- Assumes segmentable reasoning steps: The method relies on identifiable reasoning steps, holding well for mathematical proofs and code, but potentially degrading on conversational dialogue or unformatted free-form rationales.
- Additional forward-reverse inference cost: Computing both forward and reversed sequences incurs roughly 1.05× to 1.35× the compute of standard one-pass perplexity scoring, which may require two-stage cascading (global \(w = \infty\) pre-filter followed by local \(w = 64\)) at ultra-large scale.
- Potential diversity collapse: Selecting solely on high peak gap scores can over-concentrate training data on specific problem archetypes, suggesting that pairing Local-DeltaAvg with explicit semantic diversity filtering is beneficial for large-scale data curation.
Related Work & Insights¶
- vs LLM-as-a-Judge (e.g., Alpagasus): LLM judges are computationally expensive, black-box, and prone to verbosity and style biases; Local-DeltaAvg is an explainable, white-box, training-free metric with zero human preference prompt dependencies.
- vs Supervised Process Reward Models (PRM800K, Math-Shepherd): Supervised PRMs require costly step-level human annotations or extensive Monte Carlo rollouts; Local-DeltaAvg extracts process-level bottleneck curves completely unsupervised.
- vs Uncertainty-based Selection (HES): Heuristics based on token entropy or output loss fail to disentangle intrinsic conceptual difficulty from reasoning structure; Local-DeltaAvg's paired difference directly isolates directional logic.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes directional irreversibility as a formal, training-free property of reasoning data.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 20B-token candidate selection, multi-seed CPT, process alignment, and rigorous confounder controls.
- Writing Quality: ⭐⭐⭐⭐⭐ Mathematically clean, well-grounded theoretical intuition, and rigorous counterfactual control design.
- Value: ⭐⭐⭐⭐⭐ Delivers an efficient, production-ready recipe for pretraining and post-training reasoning data curation.