Probing Counterfactual Reasoning in Large Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/qlwpc/TG-Interpolation
Area: LLM Reasoning
Keywords: Syntactic Language Models, Tree Linearization, Structural Attention Masks, Causal Attention, Scaling Laws
TL;DR¶
This paper conducts a scaled-up empirical study of syntactic language models (SLMs) across up to 1B parameters and 100B tokens, revealing that standard causal attention matches or surpasses specialized structural masks while remaining fully compatible with FlashAttention, and showing that SLMs lose their clear edge over compute-matched non-structural baselines at scale.
Background & Motivation¶
Standard autoregressive language models process text as flat, unstructured token sequences. While massive Transformer models effectively acquire distributional proxies for syntax from large-scale text corpora, the hierarchical constituency structures of human language—where phrases compose recursively into complex clauses and sentences—remain completely implicit within the representations. Syntactic language models (SLMs) seek to resolve this limitation by explicitly injecting linguistic inductive biases into Transformers, jointly modeling surface sentences and linearized versions of their underlying constituency parse trees. However, existing SLM formulations have historically coupled linearized tree representations with complex, hand-engineered structural attention masks that severely hamper implementation simplicity and runtime efficiency.
A deeper tension lies in the empirical scale at which syntactic language models have traditionally been investigated. Prior literature on SLMs has almost exclusively evaluated small architectures capped at 300 million parameters, trained on less than 9 billion tokens, and probed on only two or three isolated downstream tasks or narrow syntactic probes. Consequently, critical empirical questions remain unanswered: it has been fundamentally unclear whether observed gains in prior works stem from tree linearization or from specialized attention masks, and whether syntactic inductive biases continue to offer tangible benefits when language models are scaled to industrial regimes of billions of parameters and tens of billions of training tokens.
To resolve these longstanding questions, this paper scales the empirical evaluation of SLMs up to 1B parameters and 100B pretraining tokens under a unified OLMo architecture. The authors establish a rigorously controlled design space of 12 configurations spanning diverse tree linearizations, structural closing token counts, and attention mask variants, evaluating them across an 11-task downstream suite and two comprehensive syntactic benchmarks. Core idea: disentangle tree linearization from attention mask design to show that plain causal attention matches or surpasses structural-mask variants while natively supporting FlashAttention, though at the 1B scale top-performing SLMs fail to show a clear advantage over compute-matched non-structural baselines.
Method¶
Overall Architecture¶
The framework operates under a unified autoregressive formulation: given a surface sentence \(x = (x_1, \dots, x_N)\) of \(N\) terminal tokens and its labeled constituency parse tree \(y\), a deterministic linearization mapping \(z = \text{LINEARIZE}(x, y)\) produces a flat sequence \(z = (z_1, \dots, z_T)\) of interleaved terminal and non-terminal structural tokens. The model factorizes the joint probability distribution over the sentence-tree pair autoregressively as \(p(x, y) = \prod_{t=1}^T p(z_t \mid z_{<t})\).
The systematic experimental pipeline disassembles the design space into tree linearization formats, attention masking mechanisms, compute-budget-matched controls, and terminal-focused evaluation protocols, as depicted below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Surface Sentence x + Constituency Tree y<br/>Annotated with Benepar parser"] --> B["Tree Linearization Design<br/>Recursive traversal generating ONT and CNT tokens"]
B --> C["Attention Mask Disentanglement<br/>Standard Causal vs Structural Masks vs Hybrid Heads"]
C --> D["Compute-Matched Control Design<br/>Pause-1 / Pause-2 dummy tokens matching token budgets"]
D --> E["Terminal Scoring Protocol<br/>Accumulating likelihoods over terminal tokens only S_term"]
E --> F["Downstream 11-Task & Syntactic Probes<br/>WinoGrande / HellaSwag / SG / BLiMP evaluations"]
Key Designs¶
1. Tree Linearization Design: Systematically Disentangling Structural Token Encodings
Prior works serialize constituency trees into token sequences in disparate ways, confounding comparisons. The authors unify tree linearization under a clean recursive rule: for an internal node \(v\), \(\text{LIN}(v) := \text{LEFT}(v) \circ [\text{LIN}(c) \text{ for } c \in \text{CHILDREN}(v)] \circ \text{RIGHT}(v)\), and for a terminal token \(t\), \(\text{LIN}(t) := t\). By altering the token emissions of \(\text{LEFT}(v)\) (generating Opening Non-Terminals, e.g., (S, (VP) and \(\text{RIGHT}(v)\) (generating Closing Non-Terminals, e.g., S), VP)), the design space spans five key variants: LIN1 (single labeled brackets), LIN2 (doubled closing tokens, matching TG specifications), LIN3 (tripled closing tokens), LIN1–ont (omitting opening brackets), and LIN1–merge (collapsing adjacent closing brackets into a single )). Empirical analyses reveal that opening non-terminals primarily aid the model in anticipating upcoming constituent structures, while duplicating closing tokens establishes effective compositional bottleneck representations.
2. Attention Mask Disentanglement: Validating Causal Attention Over Rigid Structural Masks A central dogma in prior SLM research has been that structural attention masks (such as Transformer Grammars' STACK and COMPOSE attention) are strictly necessary to enforce hierarchical composition. In standard TG, a closing non-terminal CNT1 attends exclusively to its children (COMPOSE), while CNT2 attends to the active parse stack (STACK), hiding intermediate representations from subsequent tokens. This paper systematically evaluates: (1) Standard Causal Attention, which leaves the full lower-triangular visibility intact and allows the model to learn structure-routing dynamically; (2) Structural Attention, including full TG masks and relaxed variants (TGNomask and TGNomask-Aug) that expose child tokens; and (3) Hybrid Attention, which partitions attention heads within a layer between causal and structural masks. Results conclusively demonstrate that standard causal attention matches or surpasses structural-mask models across downstream evaluations while remaining fully compatible with FlashAttention-2, avoiding the massive 1.5× to 18× training throughput penalties incurred by custom FlexAttention structural implementations.
3. Compute-Matched Control Design: Isolating Inductive Biases from Token Budget Inflation
Linearizing a parse tree expands the sequence length by 2.4× (for LIN1) to 3.2× (for LIN2) per document, inherently providing more compute and latent capacity during training and inference. To disentangle whether performance gains stem from genuine linguistic inductive biases or merely from increased token-level compute, the authors establish two sets of rigorous controls: first, LIN1shuf, which preserves terminal tokens while randomly shuffling structural brackets; and second, the learned PAUSE-K framework, which inserts \(K=1\) (Pause-1, matching LIN1 length) or \(K=2\) (Pause-2, matching LIN2 length) dedicated <PAUSE> tokens after each terminal token. This setup demonstrates that while explicit trees offer clear advantages over pause tokens at the 100M parameter scale, their advantages over compute-matched pause baselines largely vanish when scaled to 1B parameters.
4. Terminal Scoring Protocol: Mitigating Non-Terminal Bias in Multi-Choice Evaluations In multiple-choice and cloze-style downstream benchmarks, candidates contain both terminal words and structural tokens. Scoring candidate sequences via full log-likelihood (Full Scoring, \(S_{\text{full}}\)) introduces substantial confounding: non-terminal tokens exhibit significantly higher mean log-probabilities than terminal tokens (by 2.68 nats in Tree and 2.86 nats in TGTree), which distorts answer rankings according to parser-dependent priors. The paper establishes Terminal Scoring (\(S_{\text{term}}\)) as the primary evaluation metric:
where \(\mathcal{T}(c_j)\) restricts summation strictly to the indices of terminal tokens in continuation \(c_j\). This scoring protocol improves macro-average accuracy by +2.16 percentage points on the Validation-8 suite for the Tree model and ensures that predictions reflect genuine semantic answer likelihood rather than structural token biases.
Benchmark & Evaluation Protocol¶
All models are trained from scratch using an OLMo-style decoder-only Transformer backbone with SwiGLU activations, RoPE positional embeddings, RMSNorm pre-normalization, and tied input-output embeddings, spanning 100M, 500M, and 1B parameters. Hyperparameters (learning rates and batch sizes) are strictly governed by the Step Law power law. Pretraining utilizes BBC News (~10B tokens) and the diverse educational web corpus FineWeb-Edu-100BT (~100B tokens).
Evaluation spans three rigorous categories: - Downstream Multi-Task Suite (OLMES 11 Tasks): Evaluated in a unified cloze/completion format, covering WinoGrande, HellaSwag, BoolQ, MMLU, MMLU-Redux, OpenBookQA, CommonsenseQA, SocialIQA, ARC-Easy, ARC-Challenge, and PIQA; plus fine-tuning on XSum summarization and BoolQ on BBC News. - Syntactic Competence Benchmarks: Syntactic Generalization (SG, assessing 6 linguistic phenomena via target-region surprisal under word-synchronous beam search) and BLiMP (evaluating 67 minimal pair subtasks). - Document-Level Perplexity: Approximating exact sentence marginals \(\sum_{y} p(x, y)\) using a 300-proposal truncated marginal from a CRF constituency parser (Benepar) combined with a greedy single-path history selection.
Key Experimental Results¶
Main Results¶
The primary 1B parameter evaluation on FineWeb-Edu-100BT across the 11-task OLMES suite under terminal scoring yields the following accuracies:
| Model | WG↑ | HS↑ | BQ↑ | MM↑ | MMR↑ | OBQA↑ | CSQA↑ | SIQA↑ | ARCE↑ | ARCC↑ | PIQA↑ | 11-Task AVG↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Terminal (Flat Baseline) | 58.17 | 61.53 | 58.65 | 36.79 | 37.61 | 43.60 | 53.24 | 48.98 | 69.30 | 43.48 | 73.88 | 53.20 |
| Tree (LIN1, Causal) | 61.64 | 54.10 | 63.85 | 36.88 | 37.53 | 44.00 | 56.02 | 48.46 | 68.42 | 45.15 | 73.07 | 53.56 |
| TGTree (LIN2, Causal) | 63.38 | 63.51 | 67.55 | 37.76 | 38.42 | 45.80 | 58.15 | 50.31 | 70.00 | 44.82 | 74.97 | 55.88 |
| Pause-1 (Compute Match) | 58.96 | 63.05 | 65.41 | 37.70 | 38.81 | 42.80 | 56.18 | 51.28 | 74.04 | 43.48 | 75.14 | 55.17 |
| Pause-2 (Compute Match) | 62.59 | 64.45 | 63.39 | 38.38 | 39.46 | 44.00 | 56.76 | 48.67 | 72.98 | 48.16 | 74.97 | 55.80 |
Syntactic evaluation at 1B on FineWeb-Edu demonstrates that Tree achieves the highest overall point estimate on both syntactic benchmarks:
| Model | BLiMP↑ | SG↑ | Evaluation Protocol |
|---|---|---|---|
| Terminal | 79.65 | 88.01 | Single forward pass, terminal-token surprisal |
| Tree | 81.79 | 89.95 | Word-sync beam search (\(b=300\), length cap \(6L\)) |
| TGTree | 80.98 | 89.00 | Word-sync beam search (doubled CNTs) |
| Pause-1 | 80.83 | 86.54 | Single forward pass, terminal-token surprisal |
| Pause-2 | 80.36 | 86.59 | Single forward pass, terminal-token surprisal |
Ablation Study¶
Comprehensive ablations across the 12 controlled configurations and reference baselines at 100M on BBC News disentangle tree linearization from attention masks:
| Model Variant | Linearization | Attention Mechanism | XSum (R-AVG)↑ | BoolQ Acc↑ | SG↑ | BLiMP↑ | Doc-PPL↓ |
|---|---|---|---|---|---|---|---|
| Terminal-100M | LIN0 (No structure) | Causal | 22.02 ± 0.06 | 66.70 ± 0.36 | 69.80 | 70.77 | 9.89 |
| Tree-100M | LIN1 (Single ONT/CNT) | Causal | 22.84 ± 0.03 | 68.07 ± 0.35 | 81.40 | 80.95 | 10.01 |
| TGTree-100M | LIN2 (Doubled CNT) | Causal | 23.03 ± 0.02 | 68.12 ± 0.62 | 79.47 | 81.46 | 10.21 |
| TG-100M | LIN2 | STACK/COMPOSE mask | 20.15 ± 0.04 | 64.50 ± 0.40 | 79.19 | 83.28 | 11.02 |
| TGNomask-100M | LIN2 | Softened Nomask | 21.91 ± 0.04 | 66.66 ± 0.45 | 78.62 | 81.78 | 9.97 |
| TGNomask-Aug-100M | LIN2 | Augmented Nomask | 22.12 ± 0.02 | 67.01 ± 0.28 | 78.68 | 83.63 | 10.08 |
| Tree-NoONT-100M | LIN1–ont (No ONT) | Causal | 22.41 ± 0.04 | 68.67 ± 0.32 | 85.38 | 81.00 | 9.95 |
| Tree-Compress-100M | LIN1–merge (Merged CNT) | Causal | 22.70 ± 0.05 | 68.48 ± 0.49 | 78.27 | 81.40 | 9.91 |
| Tree-TripleCNT-100M | LIN3 (Tripled CNT) | Causal | 22.43 ± 0.04 | 68.09 ± 0.12 | 81.57 | 83.42 | 10.26 |
| Tree-Shuffle-100M | LIN1shuf (Shuffled brackets) | Causal | 21.56 ± 0.09 | 68.08 ± 0.33 | 76.56 | 67.77 | 13.41 |
| TGNomask-Mix-TG | LIN2 | Hybrid (Nomask + TG) | 21.87 ± 0.04 | 66.13 ± 0.54 | 75.77 | 83.14 | 10.01 |
| TGTree-Mix-TG | LIN2 | Hybrid (Causal + TG) | 22.16 ± 0.07 | 67.91 ± 0.36 | 78.84 | 82.75 | 10.00 |
| Pause-1-100M | Insert 1 <PAUSE> |
Causal | 22.38 ± 0.05 | 62.04 ± 0.19 | 75.11 | 70.85 | 9.77 |
| Pause-2-100M | Insert 2 <PAUSE> |
Causal | 22.25 ± 0.06 | 65.70 ± 3.45 | 76.97 | 72.59 | 10.32 |
| Pushdown-100M | Stack depth bias (ref) | Stack bias parsing | 21.06 ± 0.05 | 65.54 ± 0.29 | 74.86 | 74.97 | 13.29 |
Key Findings¶
- Structural Attention Incurs a Severe Utility Penalty: The full STACK/COMPOSE mask (TG-100M) underperforms even the flat terminal baseline on both XSum (20.15 vs. 22.02) and BoolQ (64.50 vs. 66.70) while inflating document perplexity to 11.02. Although it excels on BLiMP (83.28), this hard structural constraint substantially degrades general representation quality.
- Superiority and Hardware Efficiency of Causal SLMs: Causal tree variants (TGTree and Tree) consistently outperform structural mask models across downstream generation and reasoning tasks. Moreover, causal attention models natively leverage FlashAttention-2, achieving 62,175 tokens/s at 100M on H800 GPUs—1.6× faster than structural FlexAttention implementations (37,975 tokens/s) and 18× faster than hybrid mixing-head configurations (3,375 tokens/s).
- Diminishing SLM Advantages at Scale: At the 1B scale on FineWeb-Edu, although TGTree attains the highest 11-task average point estimate (55.88, +2.68 over Terminal), paired bootstrap analysis across 10,000 resamples reveals that its probability of ranking first is only 0.29, essentially tied with the compute-matched Pause-2 baseline (0.28). Paired significance tests indicate TGTree is significantly better than Pause-2 only on BoolQ, while Pause-2 outperforms TGTree on HellaSwag, with no significant differences across the other 9 tasks.
Highlights & Insights¶
- Dispelling the Masking Myth: The study firmly establishes that specialized structural attention masks are neither necessary nor beneficial for syntactic language modeling, demonstrating that standard causal self-attention over linearized tree tokens provides an optimal balance of linguistic structure and contextual flexibility.
- Principled Compute Budgeting: Introducing the Pause-K baseline serves as an essential control, isolating structural inductive biases from sequence length inflation (2.4–3.2×) and establishing a methodological gold standard for architectural evaluations.
- Content-Focused Scoring Calibration: By demonstrating that non-terminal tokens artificially inflate completion probabilities by 2.68–2.86 nats, the authors establish Terminal Scoring as a crucial protocol to prevent parser-induced ranking distortions in zero-shot and few-shot cloze evaluations.
Limitations & Future Work¶
- Dependency on External Parsers: Both training and inference necessitate pre-parsing all text using an off-the-shelf constituency parser (Benepar), incurring non-trivial pre-processing latency and exposing the model to potential parsing error propagation.
- Single-Language Scope: The empirical evaluations are confined entirely to English, leaving the behavior of SLMs on morphologically rich or free-word-order languages unexplored.
- Lack of Mechanistic Probes: Hypotheses explaining the efficacy of causal tree models (implicit multi-task learning, compositional bottlenecks, and adaptive computation) remain speculative interpretations rather than causally verified circuit mechanisms.
Related Work & Insights¶
- vs Transformer Grammars (TG, Sartran et al., 2022): TG strictly tightly couples constituency tree modeling with STACK and COMPOSE attention masks. This paper demonstrates that removing these restrictive masks yields superior downstream performance and eliminates hardware throughput bottlenecks.
- vs Compositional Syntactic Transformers (Zhao et al., 2025): While Zhao et al. isolated post-composition masking in smaller 300M/9B regimes, this study scales the investigation to 1B/100B, uncovering the convergence of performance between SLMs and compute-matched non-structural baselines.
- vs Pause Tokens (Goyal et al., 2024): Goyal et al. introduced learned pause tokens for latent reasoning. This work repurposed pause tokens as a compute-matched control for tree sequence length, finding that at scale, unstructured latent computation positions rival the benefits of explicit grammatical tokens.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ First large-scale, controlled empirical disentanglement of syntactic language models at 1B parameters and 100B tokens.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage of 12 controlled configurations, compute-matched controls, 11 downstream tasks, 2 syntactic suites, and extensive bootstrap significance testing.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous statistical reporting, and balanced interpretation of negative and positive results.
- Value: ⭐⭐⭐⭐⭐ Provides definitive empirical benchmarks that clarify the design space of syntactic language models and delineate their scaling limits against standard autoregressive architectures.