Skip to content

Syntactic Prediction through Reinforcement Learning

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zhaojinm/syntax_RL
Area: NLP Understanding
Keywords: Syntactic Prediction, Constituency Parsing, CCG Supertagging, Reinforcement Learning, GRPO

TL;DR

Addressing the "supervised plateau" where supervised fine-tuning (SFT) fails to enable large language models to surpass specialized neural syntactic parsers, this paper proposes Syntax-RLโ€”a two-stage post-training framework that establishes unified syntactic sequence generation via cold-start SFT on 80% data and directly optimizes structure-level objectives (bracket F1 and token accuracy) via GRPO on a disjoint 20% subset, setting new state-of-the-art records on constituency parsing and CCG supertagging.

Background & Motivation

Syntactic prediction tasks, such as constituency parsing and Combinatory Categorial Grammar (CCG) supertagging, serve as fundamental benchmarks for structured natural language understanding. These tasks require models not merely to make locally plausible word-level decisions, but to generate hierarchical tree structures that respect strict non-local constraints and maintain global sentence-level consistency. While large language models (LLMs) have achieved remarkable success across diverse NLP benchmarks, previous attempts to adapt LLMs to syntactic parsing via in-context prompting or supervised fine-tuning have consistently fallen short of dedicated, task-specific neural syntactic architectures (such as XLNet-based or chart-based parsers).

A deeper investigation reveals that supervised fine-tuning relies on token-level cross-entropy loss under teacher forcing, which trains models to imitate gold surface sequences rather than optimize global tree structures. Empirical results show that SFT quickly encounters a severe "supervised plateau": neither extending SFT over the full dataset nor continuing SFT on the second-stage subset yields further improvements, and performance remains inferior to specialized parsers. Token-level imitation lacks a mechanism for structural calibration, causing models to stumble on long-range attachment decisions, coordinate structures, and nested modifier boundaries.

The core tension lies in the fact that while SFT is indispensable for teaching general LLMs basic instruction-following and sequence formatting, achieving global structural optimality requires an optimization objective directly aligned with task-level evaluation metrics. Core idea: Propose Syntax-RL, a two-stage syntax-oriented post-training framework that first conducts cold-start SFT on 80% of data to establish a unified sequence generation format, and subsequently applies GRPO-based reinforcement learning on the remaining disjoint 20% subset to directly optimize structure-level objectives (bracket F1 and supertag accuracy).

Method

Overall Architecture

Syntax-RL formulates both constituency parsing and CCG supertagging into unified sequence generation tasks conditioned on an input sentence \(x = (w_1, \dots, w_n)\). The training pipeline consists of two decoupled stages: Stage 1 performs cold-start SFT to adapt the base LLM to structured output formatting, and Stage 2 performs Group Relative Policy Optimization (GRPO) to directly maximize structure-level rewards.

In constituency parsing, the phrase-structure tree is linearized into a parenthesized bracket string \(y_{\text{parse}} = (t_1, \dots, t_m)\); in CCG supertagging, lexical categories are serialized into aligned token-tag pairs \(\tilde{y}_{\text{stag}} = \text{Serialize}((w_1, c_1), \dots, (w_n, c_n))\). The model is initially trained on an 80% partition via SFT to establish format compliance and provide a stable policy initialization. In the second stage, the model explores structural variants across the disjoint 20% partition, evaluating sampled candidates through a composite reward function and backpropagating structural advantage signals without requiring a value network.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    IN["Input Sentence x"] --> S1["Cold-Start SFT (80% Disjoint Data)<br/>Establish unified syntax generation format & priors"]
    S1 --> S2["Disjoint Data Partitioning<br/>Strict isolation between 80% SFT & 20% RL"]
    S2 --> S3["GRPO Policy Optimization<br/>Group sampling G=8 candidates, critic-free"]
    S3 --> S4["Composite Rule-Based Reward<br/>Format validity + structure-level task metric"]
    S4 --> OUT["Optimal Syntactic Policy Model"]

Key Designs

1. Cold-Start SFT and Disjoint Data Partitioning: Preventing Sample Memorization to Unlock Clean RL Signals

Directly applying reinforcement learning to an unadapted base model leads to training instability and catastrophic failure due to the vast search space of structured sequences. Cold-start SFT provides the requisite formatting and local syntactic priors using standard teacher-forcing autoregressive loss \(\mathcal{L}_{\text{SFT}} = - \sum_{(x,y)\in \mathcal{D}_{\text{SFT}}} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid x, y_{<t})\).

Crucially, rather than applying RL to the same data used during SFT, Syntax-RL enforces a strict disjoint partition: \(\mathcal{D}_{\text{train}} = \mathcal{D}_{\text{SFT}} \cup \mathcal{D}_{\text{RL}}\) with \(\mathcal{D}_{\text{SFT}} \cap \mathcal{D}_{\text{RL}} = \emptyset\), setting \(|\mathcal{D}_{\text{SFT}}| = 0.8 |\mathcal{D}_{\text{train}}|\) and \(|\mathcal{D}_{\text{RL}}| = 0.2 |\mathcal{D}_{\text{train}}|\). Empirical comparisons reveal that running RL on 100% of the training data after 100% SFT yields suboptimal results because the policy merely reinforces memorized teacher-forced patterns. Partitioning the data ensures that the second-stage RL explores unseen sentences, providing genuine corrective feedback on structural generalizations.

2. GRPO Policy Optimization: Lightweight, Value-Model-Free Structural Policy Updates

In the second stage, the policy \(\pi_\theta\) is initialized from the SFT checkpoint, with an identical frozen reference model \(\pi_{\theta_{\text{old}}}\). For each sentence \(x\), the policy samples a group of candidate outputs \(\mathcal{Y}(x) = \{y_1, \dots, y_G\}\) (\(G = 8\)). Each candidate \(y_i\) receives a scalar reward \(r_i = R(x, y_i)\). Instead of training a memory-intensive critic network, GRPO computes normalized relative advantages within the sampled group:

\[A_i = \frac{r_i - \text{mean}(r)}{\text{std}(r) + \epsilon}\]

where \(r = (r_1, \dots, r_G)\). The policy parameters are updated using the clipped surrogate objective:

\[\mathcal{L}_{\text{GRPO}} = - \mathbb{E}_{x \sim \mathcal{D}_{\text{RL}}, y_i \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{G} \sum_{i=1}^G \min\left(\rho_i A_i, \text{clip}(\rho_i, 1-\varepsilon, 1+\varepsilon) A_i \right) \right]\]

with importance sampling ratio \(\rho_i = \frac{\pi_\theta(y_i \mid x)}{\pi_{\theta_{\text{old}}}(y_i \mid x)}\). This design eliminates value network training overhead and enables stable, scalable structural policy optimization on LLMs up to 32B parameters.

3. Composite Rule-Based Reward: Enforcing Structural Validity and Direct Metric Alignment

To ensure that generated sequences strictly adhere to formal grammar constraints while maximizing syntactic accuracy, Syntax-RL uses a composite reward function \(R(x, y) = \lambda_{\text{fmt}} R_{\text{fmt}}(x, y) + \lambda_{\text{task}} R_{\text{task}}(x, y)\).

The format reward \(R_{\text{fmt}}(x, y) \in \{0, 1\}\) acts as a hard filter: it assigns 1 only if the output constitutes a well-formed parenthesized tree (for parsing) or a valid token-aligned serialized sequence (for supertagging). The task reward \(R_{\text{task}}(x, y)\) directly computes the actual evaluation metric against the ground-truth annotation \(y^*\): for constituency parsing, it uses sentence-level bracket F1 score \(R_{\text{task}}^{\text{parse}}(x, y) = \text{F1}(y, y^*)\); for supertagging, it uses token-level accuracy \(R_{\text{task}}^{\text{stag}}(x, y) = \text{Acc}(y, y^*)\) (incorporating Longest Common Subsequence alignment to handle potential tokenization discrepancies). This directly resolves the mismatch between token-level training loss and sentence-level structured evaluation.

Loss & Training

The SFT stage is implemented via LLaMA-Factory across 4ร— H200 GPUs for 2 epochs, with learning rate \(1.0 \times 10^{-5}\), cosine learning rate scheduling, warmup ratio 0.1, and batch size 32. The RL stage is implemented using VERL on 8ร— H200 GPUs, with total train batch size 256, PPO mini-batch size 64, actor learning rate \(5 \times 10^{-7}\), KL loss coefficient 0.001, entropy coefficient 0, and group rollout size \(G = 8\).

Key Experimental Results

Main Results

On the Penn Treebank (PTB) benchmark, Syntax-RL with Qwen2.5-32B outperforms all previous task-specific neural parsers and supervised LLM baselines on both the full test set and the length < 40 subset (PTB< 40). It also establishes new state-of-the-art results on CCGbank, CCGrebank, and Chinese Penn Treebank (CTB).

Dataset / Benchmark Metric Ours (Syntax-RL 32B) Strongest LLM Baseline (Bai et al. 32B) Best Specialized Neural Model Gain vs. LLM Baseline
PTB (Full Test Set) Bracket F1 96.52 96.21 96.38 (Mrini et al. XLNet) +0.31
PTB (Length < 40) Bracket F1 96.59 96.30 95.76 (Kitaev et al. BERT) +0.29
CCGbank Supertagging Token Acc (%) 96.70 96.64 (Zhao & Penn) 96.60 (Yamaki et al.) +0.06
CCGrebank Supertagging Token Acc (%) 95.55 95.50 (Zhao & Penn) 95.48 (Yamaki et al.) +0.05
CTB (Chinese Treebank) Bracket F1 92.68 92.35 (Bai et al. 32B) 92.24 (Tian et al. ZEN) +0.33

Detailed precision and recall metrics on PTB show Syntax-RL achieving LR 96.25, LP 96.79, and F1 96.52. Across model scales, Syntax-RL consistently improves over the SFT baselines: Qwen2.5-7B improves from 95.99 to 96.14 F1, and Qwen2.5-14B improves from 96.15 to 96.36 F1.

Ablation Study

Ablation experiments isolate the effects of data allocation, continued SFT vs. RL, data overlap, and intermediate thought supervision on PTB constituency parsing.

Configuration / Setting Data Partitioning Strategy Metric (PTB F1) Note / Analysis
Full-data SFT 40k SFT 96.21 Supervised training hits performance plateau
Continued SFT 32k SFT + 8k SFT 96.00 Continuing SFT on 2nd-stage data degrades accuracy
Second-stage RL (Ours) 32k SFT + 8k RL 96.52 Clean +0.52 F1 improvement over continued SFT on identical data
Overlapping RL 32k SFT + 40k RL 96.48 Reusing 1st-stage data during RL yields inferior performance to disjoint split
Small SFT Split 20k SFT + 20k RL 96.25 Insufficient cold-start SFT harms policy optimization
Thought Format: Span-based 32k SFT + 8k RL (span <think>) 95.97 Explicit span listing introduces noise (-0.55 F1)
Thought Format: Tree-sketch 32k SFT + 8k RL (sketch <think>) 96.31 Coarse bracket sketches underperform direct answer generation (-0.21 F1)

Key Findings

  • RL Transcends the Supervised Plateau: Under the exact same 32k checkpoint and 8k second-stage data, continued SFT drops to 96.00 F1, whereas second-stage RL reaches 96.52 F1. This proves that gains stem directly from structure-level RL exploration rather than prolonged training.
  • Disjoint Splitting is Essential: The 32k SFT + 40k RL setup (96.48 F1) performs worse than 32k SFT + 8k RL (96.52 F1), confirming that RL provides the most informative gradient updates on fresh, unseen instances rather than re-optimizing teacher-forced examples.
  • Syntactic Prediction Does Not Benefit from CoT: Explicit thought sequences (span enumeration or tree sketches) degrade performance compared to direct answer-only generation. Since syntactic prediction is already an explicit structured output, intermediate reasoning tokens introduce structural distraction.
  • Resolution of Attachment Ambiguity: EVALB structural analysis (Table 7) demonstrates that recall improvements are concentrated on attachment-sensitive constituents: Coordination (+4.78), SBAR (+4.51), Long NP (+3.18), and PP (+2.88).

Highlights & Insights

  • Direct Alignment of Structural Metrics with RL Objectives: Bridges the gap between token-level autoregressive loss and global tree validity by optimizing bracket F1 and token accuracy directly via GRPO.
  • Rigorous Disjoint Data Experimental Design: The 80/20 data split rigorously proves that RL's superiority over SFT is structural rather than an artifact of additional training exposure.
  • Challenging the Omnipresence of Chain-of-Thought: Demonstrates that for formal grammar prediction where output targets are inherently structural, explicit reasoning chains introduce redundant overhead.

Limitations & Future Work

  • Task Scope: Evaluation is centered on constituency parsing and CCG supertagging; extension to non-projective dependency parsing and cross-lingual low-resource domains remains to be explored.
  • Data Split Sensitivity: The 80/20 SFT/RL split was tuned on high-resource treebanks (PTB and CTB); low-resource scenarios may necessitate a larger SFT ratio for reliable cold-start policy formation.
  • Future Directions: Exploring self-evolving distillation and lightweight model adaptation (1Bโ€“3B parameters) for real-time syntactic parsing.
  • vs. Specialized Neural Parsers (Kitaev et al., 2019; Mrini et al., 2020): Traditional neural parsers require chart decoding (CKY) or complex span attention architectures; Syntax-RL demonstrates that general generative LLMs with RL can surpass these specialized models.
  • vs. Supervised LLM Parsers (Bai et al., 2025; Zhao & Penn, 2024): Previous works identified the difficulty of outperforming specialized parsers even with 70B LLMs under SFT; Syntax-RL identifies the "supervised plateau" and solves it via two-stage RL post-training.

Rating

  • Novelty: โญโญโญโญโ˜† Pinpoints the supervised plateau in LLM syntax generation and designs an elegant two-stage RL framework on disjoint data.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive comparisons across SFT baselines, data splits, thought formats, and fine-grained constituent recall.
  • Writing Quality: โญโญโญโญโญ Methodological motivation is articulated with exceptional clarity and supported by compelling figures and ablation data.
  • Value: โญโญโญโญโญ Establishes a vital paradigm for unlocking structural prediction and formal grammar competence in generative foundation models.