Skip to content

LoCoDec: Robust Decoding for LLM-based Sequence-Structured Prediction

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Sequence-Structured Prediction, Constrained Token Paraphrasing, Neighborhood Context Window, Perturbation Robustness, KV-Cache Alignment

TL;DR

Addressing structural hallucinations and catastrophic collapse in LLM-based structured prediction under long-context perturbations, LoCoDec reformulates prediction as constrained token paraphrasing, combining dynamic neighborhood context with KV-cache truncation and monotonic stack-based decoding to secure top-tier perturbation robustness (R-F1) across 16 benchmarks.

Background & Motivation

Fundamental natural language processing tasks—ranging from Named Entity Recognition (NER), Nested NER (NNER), and Aspect-Based Sentiment Analysis (ABSA) to Semantic Role Labeling (SRL)—can be broadly unified as sequence-structured prediction problems whose targets align directly with the input sequence. The paradigm shift toward Large Language Models (LLMs) has led to reformulating these traditional discriminative tagging tasks as conditional generative language modeling. However, dominant generative paradigms exhibit severe structural vulnerabilities: Structured Content Extraction (SCE) produces free-form key-value pairs without explicit surface-level alignment, making it prone to lexical hallucinations; Structured Span Extraction (SSE) predicts exact numerical character offsets, which are fragile and prone to complete boundary failure under the slightest coordinate deviation in long texts.

A far more critical vulnerability emerges under long-context natural perturbations. In realistic deployment scenarios—such as document-level processing, multi-turn dialogues, or cascading extraction pipelines—input sequences frequently undergo non-adversarial sentence-level contextual expansion, such as document concatenation or topic transitions. Unlike artificial synthetic perturbations (e.g., character flips or random word scrambling) that violate natural linguistic distributions, natural sentence-level expansions maintain syntactic fluency and semantic plausibility while drastically increasing context length. In such environments, autoregressive LLM decoders suffer from severe attention dispersion caused by irrelevant surrounding tokens. This distributional shift distorts internal hidden representations, triggering boundary hallucinations, label omission, truncation, and nesting structure collapse.

Existing grammar-constrained decoding (GCD) methods primarily enforce syntactic schema validity over terminal tokens, yet they fail to guarantee source-text faithfulness or mitigate noise from distractor sentences. The fundamental insight of this work is that sequence-structured prediction predominantly relies on local syntactic and semantic evidence surrounding target spans; expanding the model's receptive field to irrelevant long-range context introduces destructive variance rather than beneficial cues. Core idea: Unify sequence-structured prediction as a constrained token paraphrasing task, combining dynamic neighborhood context windows with KV-cache truncation to eliminate long-range noise, and employing a deterministic controller with monotonic text alignment and pushdown stack constraints to enforce source faithfulness and well-formed structures.

Method

Overall Architecture

The execution pipeline of LoCoDec follows an iterative, left-to-right segment-level validation process. Rather than generating detached key-value dictionaries or coordinate pairs, the model reproduces the source text while inserting paired semantic boundary markers \(\langle\ell\rangle\) and \(\langle/\ell\rangle\) around extracted spans. At each iteration, the framework dynamically selects an input neighborhood window based on the current source pointer and active unclosed label stack, simultaneously truncating the Key-Value cache (KV-cache) to purge stale attention states originating outside the local context. The LLM then generates a short continuation candidate, which is parsed by a deterministic controller into text and candidate label sequences. The controller performs monotonic prefix alignment against the source text, falls back to copying source tokens upon misalignment, and validates labels using pushdown stack constraints and lexical boundary checks. Validated tokens are appended to the prefix until the entire source text is consumed.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input sequence & label set L<br/>Perturbed input x_r = x ⊕ x'"] --> B["Dynamic neighborhood window & KV-cache truncation<br/>Bound local context [a, c) & truncate stale attention cache"]
    B --> C["Segment-level continuation generation<br/>LLM greedy decoding yields candidate text & label units"]
    C --> D["Monotonic text alignment & fallback<br/>Align longest prefix; fallback to copying source token on mismatch"]
    D --> E["Stack-based nested validation & lexical repair<br/>LIFO stack checks closing tags; subword repair prevents intra-word split"]
    E -->|Source unconsumed| B
    E -->|Source completed| F["Source-faithful structured sequence output y_r"]

Key Designs

1. Dynamic Neighborhood Window & KV-Cache Truncation: Suppressing Distant Context Noise to Restore Hidden Representation Stability

To eliminate attention dispersion caused by long-context perturbations, LoCoDec establishes a dynamic locality-aware context window rather than conditioning on the entire input sequence. Given source text pointer \(p\) and sentence boundary positions \(\mathcal{B} = \{b_1, \dots, b_K\}\), the system selects a sentence-level window \([a, c)\) encompassing \(p\) to construct localized input \(x_{w,i} = x_{a:c}\). To prevent fixed window slicing from severing cross-sentence or deeply nested spans, an active label stack \(S = ((\ell^{(1)}, o^{(1)}, s^{(1)}), \dots)\) tracks unclosed start tags, adaptively expanding the left boundary to encompass the outermost open span: \(a \leftarrow \min(a, \alpha(s^{(1)}))\). Crucially, restricting the prompt text alone does not prevent autoregressive decoders from retrieving stale attention keys and values accumulated from earlier context. LoCoDec actively truncates the layer-wise cache tensors \(K_l, V_l \in \mathbb{R}^{B \times H \times T \times d}\):

\[K'_l = K_l[:, :, :\kappa_i, :], \quad V'_l = V_l[:, :, :\kappa_i, :]\]

where \(\kappa_i\) matches the token length of the localized prompt \(P(\tau, x, \mathcal{L}, x_{w,i})\) combined with accepted partial output \(y_{w,i}\). If prefix alignment is violated, the cache is cleanly recomputed from the local context. This prevents distant tokens from contaminating internal hidden states, maintaining high cosine similarity (\(\cos(v_i, v_i^r) \approx 1\)) between representations under perturbed and unperturbed contexts, which dramatically curbs prediction inconsistency.

2. Monotonic Text Alignment with Fallback: Preventing Lexical Hallucination and Enforcing Monotonic Decoding Progress

Unconstrained autoregressive generation frequently omits source words, substitutes synonyms, or fabricates non-existent tokens. LoCoDec introduces a segment-level verification strategy. Each continuation segment \(C\) is decomposed into a text prefix \(T\) and candidate label sequence \(G\). The ALIGN_LONGEST_PREFIX procedure compares \(T\) with source tokens starting at pointer \(p\): when an aligned prefix \(P\) of length \(r\) is identified, the validated text is appended to output \(y \leftarrow y \oplus x_{p:p+r}\) and the pointer advances \(p \leftarrow p + r\). If only a partial match occurs, unaligned trailing text and subsequent candidate labels \(G\) are discarded, preventing unverified tokens from introducing illicit labels. When no non-empty prefix aligns (state = NONE), the controller triggers a deterministic fallback branch: it directly copies the exact source token \(x_p\) into output \(y\), increments pointer \(p \leftarrow p + 1\), and restores the KV-cache to the verified prefix. This mechanism enforces absolute source faithfulness and mathematically guarantees deadlock-free monotonic progress.

3. Stack-Based Nested Constraints & Lexical Subword Repair: Guaranteeing Syntactic Validity and Valid Lexical Boundaries

Structural malformation—such as unclosed brackets, crossed nesting, or orphan tags—renders structured outputs invalid. LoCoDec deploys an exact pushdown automaton during decoding. For candidate start labels \(\langle\ell\rangle\), the controller verifies \(\ell \in \mathcal{L}\), appends the tag to \(y\), and pushes its metadata onto stack \(S \leftarrow \text{push}(S, (\ell, |y|, p))\). For candidate end labels \(\langle/\ell\rangle\), acceptance requires a non-empty stack whose top element strictly matches \(\ell\); upon match, the end tag is appended and the top element popped (\(S \leftarrow \text{pop}(S)\)). This Last-In-First-Out (LIFO) discipline strictly prohibits ill-formed overlapping spans. Additionally, modern subword tokenizers (BPE / SentencePiece) may inadvertently place span boundaries inside word tokens. LoCoDec incorporates lexical boundary detection: if the emitted label splits contiguous alphanumeric characters (English) or violates character boundaries (Chinese), the candidate label is rejected, and the controller splits and re-encodes the subword tokens to force completion of the lexical unit before permitting a span transition.

Benchmark & Evaluation Protocol

Because LoCoDec operates purely as an inference-time decoding controller, it introduces zero trainable parameters and requires no auxiliary training objectives. All evaluated models (Llama3-8B and Qwen3-8B) are fine-tuned using LoRA under identical parameter-efficient protocols across task training splits (LoRA rank=8, batch size=1, initial learning rate \(5 \times 10^{-5}\), bf16 precision). The evaluation spans 16 benchmark datasets across 6 core tasks: Flat NER (Weibo, Resume, W16, W17), Nested NNER (GENIA, ACE2005), Slot Filling (CAIS, ECDT), Aspect-Based Sentiment Analysis (Laptop14, Rest14, Rest15, Rest16), Part-of-Speech tagging (CTB5, CTB6), and Semantic Role Labeling (CN05, CPB1.0). Robustness test sets are constructed by concatenating adjacent test sequences to simulate natural sentence-level context expansion, complemented by five fine-grained perturbation variants: topic drift, document-level noise, and pre/post/bidirectional context distractors. Performance is measured using Standard F1 (S-F1), Perturbation F1 (P-F1), and Prediction Consistency F1 (R-F1 = \(\text{F1}(y_r, y)\)).

Key Experimental Results

Main Results

The table below presents the main evaluation results across the 16 benchmarks using the Llama3-8B backbone, comparing LoCoDec against task-specific discriminative SOTA, Structured Span Extraction (SSE), and Structured Content Extraction (SCE). Throughput (Tput.) is reported in samples per second.

Task Category Dataset Task SOTA SSE (Llama3-8B) SCE (Llama3-8B) LoCoDec (Ours) S-F1 vs SOTA R-F1 (Consistency)
NER Weibo 67.51 / 67.52 / 92.54 23.70 / 14.65 / 27.41 66.67 / 65.52 / 82.15 69.88 / 69.22 / 98.08 +2.37 98.08 (Best)
Resume 94.80 / 94.71 / 98.01 66.85 / 42.41 / 49.00 94.57 / 93.67 / 95.83 96.62 / 96.68 / 99.94 +1.82 99.94 (Best)
W16 52.71 / 51.49 / 82.80 46.31 / 29.60 / 49.00 59.12 / 54.62 / 80.60 61.29 / 61.48 / 97.55 +8.58 97.55 (Best)
W17 49.35 / 48.33 / 82.59 39.74 / 27.90 / 55.57 53.85 / 51.21 / 85.13 54.51 / 54.63 / 96.40 +5.16 96.40 (Best)
ABSA Laptop14 65.87 / 63.39 / 89.99 61.29 / 40.88 / 50.83 71.49 / 63.75 / 82.63 72.59 / 73.36 / 97.34 +6.72 97.34 (Best)
Rest14 76.64 / 75.64 / 90.95 66.13 / 42.19 / 50.52 76.89 / 72.01 / 83.40 82.13 / 82.11 / 99.19 +5.49 99.19 (Best)
Rest15 67.11 / 66.24 / 89.87 51.85 / 30.95 / 39.89 72.64 / 70.67 / 80.27 76.40 / 77.23 / 98.40 +9.29 98.40 (Best)
Rest16 73.61 / 74.21 / 92.39 63.93 / 37.67 / 47.66 80.34 / 76.01 / 84.76 81.71 / 81.46 / 97.96 +8.10 97.96 (Best)
SF / SRL CAIS 87.84 / 67.37 / 71.13 88.61 / 48.31 / 53.47 90.36 / 67.76 / 72.20 90.95 / 86.50 / 93.72 +3.11 93.72 (Best)
ECDT 77.20 / 58.50 / 65.22 81.98 / 49.33 / 50.50 86.19 / 60.33 / 62.88 86.70 / 86.14 / 97.74 +9.50 97.74 (Best)
CPB1.0 82.81 / 80.47 / 92.24 51.38 / 29.00 / 41.43 82.55 / 54.00 / 63.06 88.44 / 86.12 / 96.81 +5.63 96.81 (Best)
CN05 88.39 / 85.87 / 93.29 84.97 / 49.42 / 56.25 86.00 / 51.80 / 58.73 92.06 / 91.37 / 99.24 +3.67 99.24 (Best)
POS / NNER CTB5 96.34 / 96.42 / 98.88 68.42 / 46.91 / 50.30 92.34 / 90.68 / 92.77 95.77 / 95.74 / 99.75 -0.57 99.75 (Best)
CTB6 94.82 / 94.97 / 98.53 64.64 / 43.28 / 47.74 90.09 / 86.61 / 88.93 93.81 / 93.35 / 99.62 -1.01 99.62 (Best)
GENIA 80.27 / 80.33 / 96.52 71.43 / 47.89 / 56.81 77.46 / 74.97 / 87.33 79.61 / 79.78 / 98.73 -0.66 98.73 (Best)
ACE2005 84.05 / 84.12 / 96.25 72.54 / 55.84 / 62.14 82.83 / 80.36 / 86.93 87.96 / 87.95 / 97.66 +3.91 97.66 (Best)

Ablation Study

Ablation results on Llama3-8B systematically isolating the key algorithmic components (- w/o all: removing both neighborhood context and constrained token paraphrasing; - w/o nc: removing neighborhood context; - w/o ctp: removing constrained token paraphrasing; w = k: fixed-radius window selected on validation set):

Dataset Metrics (S-F1 / P-F1 / R-F1) Full LoCoDec - w/o all - w/o nc - w/o ctp w = k (Fixed Radius)
Weibo (NER) S-F1 / P-F1 / R-F1 69.88 / 69.22 / 98.08 68.08 / 68.07 / 90.86 69.52 / 69.69 / 91.53 69.31 / 68.39 / 98.19 67.20 / 66.13 / 95.22
Rest14 (ABSA) S-F1 / P-F1 / R-F1 82.13 / 82.11 / 99.19 78.63 / 78.91 / 90.72 82.13 / 81.18 / 91.80 80.57 / 79.70 / 99.15 80.62 / 79.39 / 93.87
CN05 (SRL) S-F1 / P-F1 / R-F1 92.06 / 91.37 / 99.24 90.45 / 79.40 / 83.88 92.31 / 79.43 / 83.67 91.76 / 89.89 / 99.28 92.15 / 82.14 / 87.08
ECDT (SF) S-F1 / P-F1 / R-F1 86.70 / 86.14 / 97.74 85.67 / 74.45 / 81.37 86.70 / 79.13 / 86.73 86.70 / 85.97 / 97.59 85.63 / 81.40 / 87.80
GENIA (NNER) S-F1 / P-F1 / R-F1 79.61 / 79.78 / 98.73 79.13 / 79.40 / 92.66 79.73 / 79.78 / 93.99 79.52 / 79.63 / 98.46 79.11 / 79.36 / 98.18
CTB6 (POS) S-F1 / P-F1 / R-F1 93.81 / 93.35 / 99.62 92.35 / 90.37 / 96.76 93.89 / 93.35 / 98.73 93.60 / 92.76 / 99.57 93.57 / 93.25 / 99.37

Key Findings

  • Neighborhood Context is the Core Anchor of Perturbation Robustness: Removing neighborhood context (- w/o nc) triggers severe degradation in prediction consistency (R-F1) across all 16 datasets, dropping by up to 15.57 points on CN05 (from 99.24 to 83.67) and 11.01 points on ECDT (from 97.74 to 86.73). This confirms that irrelevant long-range context severely destabilizes local boundary decisions unless explicitly bounded.
  • Adaptive Stack-Aware Windows Outperform Tuned Fixed-Radius Windows: Even when fixed radius \(k\) is extensively optimized on validation sets (w = k), it remains consistently inferior to LoCoDec's adaptive sentence-level expansion. On CN05, the fixed-window variant achieves only 82.14 P-F1 and 87.08 R-F1 compared to LoCoDec's 91.37 and 99.24, highlighting the critical role of stack-guided window expansion in preserving cross-sentence dependencies.
  • Label Density Inversely Correlates with Generative Advantage: LoCoDec achieves marked improvements over task-specific models on semantically rich, span-oriented tasks such as ABSA and SRL (+9.29 S-F1 on Rest15). Conversely, on dense token-level tasks like POS tagging, it slightly underperforms specialized discriminative baselines (-1.01 S-F1 on CTB6). Dense label insertion causes error accumulation during autoregressive decoding, exposing the inherent challenge of token-level generation.

Highlights & Insights

  • Coupling Physical Context Bounds with KV-Cache Truncation: Rather than relying merely on superficial prompt truncation, LoCoDec synchronizes the internal transformer KV-cache to active local context tokens \(\kappa_i\). This severs latent attention feedback loops from distant tokens, offering an elegant non-parametric defense against long-context noise.
  • Paraphrasing Eliminates Absolute Coordinate Fragility: In contrast to Structured Span Extraction (SSE), which breaks down completely when numerical coordinates drift under contextual shifts, token paraphrasing reformulates extraction as conditional sequence reproduction with inline tag insertion, perfectly aligning with LLM generative pretraining.
  • Lightweight Non-Invasive Controller: Operating entirely without weight modification or vocabulary-wide logit masking matrices, the controller achieves strict verification with an auxiliary time complexity of \(O(N_x + K_{val} B_{max}^2 + Z_{lab})\), providing a plug-and-play inference wrapper that retains practical decoding throughput.

Limitations & Future Work

  • Restricted to Linearly Serializable Sequence Structures: LoCoDec is strictly designed for source-aligned token, flat span, and nested span outputs that can be linearly serialized. It cannot natively accommodate non-projective dependency trees, Abstract Meaning Representation (AMR) graphs, or document-level relation networks.
  • Greedy Segment Decisions Lack Global Lookahead: The controller verifies candidate segments greedily. When an accepted span appears locally plausible but contradicts global document context, the decoder lacks an explicit beam search or backtracking mechanism to reverse earlier decisions.
  • Heuristic Slicing Uncoupled from Model Uncertainty: Context window boundaries rely on static sentence segmentations rather than dynamic token-level entropy or attention dispersion metrics, limiting adaptiveness in syntactically ambiguous or unpunctuated text.
  • vs. Structured Content Extraction (SCE, e.g., Han et al., 2024): SCE generates structured key-value dictionaries freely, lacking surface token alignment and frequently suffering from hallucinated entities; LoCoDec enforces monotonic prefix matching with guaranteed fallback copying, ensuring 100% surface fidelity.
  • vs. Structured Span Extraction (SSE, e.g., Ding et al., 2025): SSE predicts absolute character offsets [start, end, label], which are highly brittle under sentence concatenations; LoCoDec bypasses numerical regression entirely via inline tag paraphrasing, outperforming SSE by vast margins across all 16 benchmarks (e.g., Weibo S-F1 69.88 vs. SSE 23.70).
  • vs. Sentence-Level NER Robustness (Liu et al., 2026): While Liu et al. explored fixed-window filtering exclusively for flat Chinese NER, LoCoDec establishes a general multi-task, cross-lingual framework supporting nested structures through dynamic stack-guided window expansion and KV-cache truncation.

Rating

  • Novelty: ⭐⭐⭐⭐☆ (Ingenious reformulation of structured extraction as constrained token paraphrasing combined with KV-cache truncation)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Extensive evaluation across 16 benchmarks, 6 diverse tasks, 2 LLM backbones, and 5 fine-grained perturbation settings)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous problem formulation, precise algorithmic pseudo-code, and lucid mechanism analysis)
  • Value: ⭐⭐⭐⭐⭐ (Highly practical, training-free decoding framework that directly resolves extraction brittleness in enterprise long-context pipelines)