Skip to content

Probing Syntactic Knowledge in Instruction-Tuned LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: refusal mechanisms, knowledge-based refusal, safety-based refusal, mechanistic interpretability, activation steering

TL;DR

By establishing a rigorously controlled benchmark of 213 contrastive quadruples alongside sequential tuning schedules, this paper uncovers that LLM refusal operates via a unified "commit-then-specify" process, where an early shared refusal direction triggers abstention before late-layer heads and MLPs specify whether grounds are epistemic or normative.

Background & Motivation

As large language models are increasingly deployed in real-world critical workflows, knowing when to decline a query is just as essential as knowing how to answer it. Refusals in modern LLMs typically arise from two distinct operational modes: knowledge-based refusal (KR), where the model must abstain because a query falls outside its epistemic horizon (such as querying fictional characters, unrecorded facts, or impossible preconditions), and safety-based refusal (SR), where the query breaches normative, legal, or privacy guidelines (such as requesting sensitive credentials or hazardous instructions). Although both KR and SR elicit superficially identical declining behavior (e.g., "I cannot answer this question"), prior literature has studied them in complete isolation: KR has been investigated within hallucination mitigation, uncertainty estimation, and epistemic honesty, whereas SR has been framed under safety alignment, red-teaming, and jailbreak defense.

This long-standing bifurcation leaves a fundamental scientific mystery unresolved: do knowledge- and safety-based refusals share a common computational substrate inside language models, or do they emerge from entirely distinct, modular pathways? Resolving this question carries profound practical consequences, because alignment interventions aimed at improving safety compliance could potentially transfer to, interfere with, or inadvertently trigger over-refusal in epistemic question answering. However, conducting a direct head-to-head mechanistic comparison between KR and SR has been fundamentally hindered by two confounding barriers. First, existing benchmarks exhibit severe topic, lexical, and surface-form disparities that easily obscure genuine representational differences. Second, standard open-weight instruction-tuned LLMs acquire refusal behaviors through divergent and opaque proprietary alignment mixtures, making it impossible to separate intrinsic representational geometry from disparate training histories.

To systematically dismantle these confounding factors, this work establishes a controlled evaluation paradigm. The authors develop a curated benchmark of 20 topic categories comprising 213 matched contrastive quadruples \((x_{KR}, x_{SR}, x_{KC}, x_{SC})\), where each refusal instance is paired with an identical-topic, minimal-edit benign control that strips the refusal trigger while preserving prompt syntax. Furthermore, they evaluate sequential post-training configurations to inject balanced KR and SR capabilities without degrading core reasoning benchmarks. Core idea: refusal in large language models is governed by a hierarchical "commit-then-specify" mechanism—a shared, dominant direction in early-to-middle layers first commits the model to declining the query, while type-specific features in late layers (mediated by specialized attention heads and MLP blocks) subsequently specify whether the refusal stems from epistemic uncertainty or normative safety constraints.

Method

Overall Architecture

The analytical framework encompasses controlled contrastive prompt generation, sequential refusal tuning, geometric vector decomposition, cross-type linear probing, logit-lens decoding, and causal activation steering. Residual-stream hidden states are extracted at the final prompt-token position across all layers. By subtracting matched benign controls from refusal activations, the framework isolates clean refusal shift directions for knowledge (\(d_K\)) and safety (\(d_S\)). It then computes their cosine alignment, projects each onto a shared bisector direction versus an orthogonal residual, decodes the associated vocabulary via the unembedding layer, and validates behavioral causality through residual steering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Matched Quadruple Input Prompts<br/>(KR, SR, KC, SC)"] --> B["Controlled Refusal Tuning Pipeline<br/>Seq-KS: Pre-aligning KR & SR without general utility collapse"]
    B --> C["Shared Refusal Direction & Orthogonal Decomposition<br/>Extract d_K & d_S, split into d_common and type-specific residuals"]
    C --> D["Asymmetric Cross-Type Linear Probing<br/>Confirming strong SR->KR and weak KR->SR transfer"]
    D --> E["Late Type-Specific Grounding & Component Attribution<br/>Late-layer divergence via specialized heads and MLP blocks"]
    E --> F["Activation Steering & Rationale Modulation<br/>Residual stream intervention validating behavioral causality"]

Key Designs

1. Controlled Refusal Tuning Pipeline: Decoupling surface confounds and alignment history To prevent prompt structural variance from masquerading as internal representations, the benchmark establishes a hierarchical schema across 20 categories covering contact lookup, institutional access, document authentication, and cybersecurity. For each semantic frame, shared prompt skeletons and anchor keywords are instantiated using GPT-4o-mini into minimally contrasted refusal pairs—where KR introduces a fictional/unverifiable referent and SR introduces a sensitive real referent. Corresponding controls (\(x_{KC}, x_{SC}\)) perform minimal-edit substitutions replacing the trigger with an answerable entity or benign target. After rule filtering, GPT-5 judging, and human validation (\(\kappa = 0.82\)), 213 quadruples verified across all target models were retained. On the model axis, evaluating tuning sequences revealed that safety-then-knowledge tuning (Seq-SK) induces catastrophic over-refusal and general capability degradation, whereas knowledge-then-safety tuning (Seq-KS) maintains MMLU and TriviaQA performance while achieving balanced refusal rates, providing a pristine testbed.

2. Shared Refusal Direction & Orthogonal Decomposition: Isolating the universal refusal substrate To mathematically evaluate whether KR and SR share a geometric axis, the authors compute difference-in-means shifts at each layer \(\ell\) on the final prompt token: $\(d_K^{\ell} = \mathbb{E}_i \left[ h^{\ell}(x_{KR,i}) - h^{\ell}(x_{KC,i}) \right], \quad d_S^{\ell} = \mathbb{E}_i \left[ h^{\ell}(x_{SR,i}) - h^{\ell}(x_{SC,i}) \right]\)$ The directional alignment is measured by \(\text{simshift}_{\ell} = \cos(\hat{d}_K^{\ell}, \hat{d}_S^{\ell})\), peaking at early-to-middle layers \(\ell^*\) at values between \(0.699\) and \(0.794\) across all models. Defining the common unit direction \(\hat{d}_{\text{common}}^{\ell} = (\hat{d}_K^{\ell} + \hat{d}_S^{\ell}) / \|\hat{d}_K^{\ell} + \hat{d}_S^{\ell}\|\), the shifts are decomposed into a projection \(d_{r,\text{proj}}^{\ell} = (\hat{d}_r^{\ell} \cdot \hat{d}_{\text{common}}^{\ell}) \hat{d}_{\text{common}}^{\ell}\) and an orthogonal residual \(d_{r,\text{spec}}^{\ell} = \hat{d}_r^{\ell} - d_{r,\text{proj}}^{\ell}\). The common component norm dominates across layers (\(0.92 \sim 0.95\) at peak), proving that both refusal types hinge upon a largely shared internal vector, while orthogonal residuals maintain a non-trivial norm (\(0.32 \sim 0.39\)).

3. Asymmetric Cross-Type Linear Probing: Demonstrating directional hierarchy in refusal subspaces To test whether the high cosine alignment signifies functional symmetry, linear logistic regression probes are trained across types. Probe SR\(\rightarrow\)KR is trained on safety pairs (SR vs. SC) and evaluated out-of-domain on knowledge pairs (KR vs. KC) without retraining, and vice versa for KR\(\rightarrow\)SR. The results uncover a persistent directional asymmetry: SR\(\rightarrow\)KR achieves an accuracy of \(0.80 \sim 0.86\), whereas KR\(\rightarrow\)SR drops to \(0.58 \sim 0.63\), generating an invariant transfer gap \(\Delta = +0.22 \sim +0.27\) across all models and layers. This proves that the safety refusal boundary captures the core, transferable abstention feature, whereas the knowledge refusal boundary incorporates narrow, condition-specific epistemic heuristics that do not generalize to normative violations.

4. Late Type-Specific Grounding & Component Attribution: Unraveling late semantic divergence To identify where and how the non-transferable component emerges, the final-layer KR-specific residual \(\hat{d}_{\text{ref}} := d_{K,L,\text{spec}} / \|d_{K,L,\text{spec}}\|\) is tracked across network depth. Projections \(p_K^{\ell}\) and \(p_S^{\ell}\) remain close to zero through the first \(85\%\) of layers and diverge sharply only in the final layers, with \(p_K\) driving positive and \(p_S\) turning negative. Logit-lens decoding confirms that the common direction \(\hat{d}_{\text{common}}\) promotes generic refusal tokens (sorry, regret, currently) while suppressing affirmations (yes, sure, certainly); in contrast, the difference direction \(\hat{d}_K - \hat{d}_S\) cleanly segregates epistemic uncertainty markers (Based, presently, Sadly) from normative and legal terminology (legal, illegal, privacy). Component-level ablations identify that this divergence is mediated by a sparse subset of attention heads in the late layers and concentrated in final-layer MLP blocks (relative depth \(>0.9\)).

Loss & Training

Full-parameter supervised fine-tuning was performed on two NVIDIA H200 GPUs (144GB VRAM each) using AdamW with learning rate \(2 \times 10^{-6}\), 10% linear warmup, cosine learning rate decay, and zero weight decay. Models were trained for 3 epochs in bf16 precision with effective batch size 8 (per-device batch size 1 with 8 gradient accumulation steps). Chat templates followed each model's native format. Evaluation separated open-domain QA outcomes into correct (Corr.), incorrect (Inc.), and refusal (Ref.) rates, while safety benchmarks measured harmful request refusal (Unsafe Ref.) and benign prompt false rejection (Safe Ref.).

Key Experimental Results

Main Results

The primary evaluation spans original instruction-tuned models and their sequential-tuned (\(*\)) counterparts on Llama-3-8B, Qwen2.5-7B, Gemma-2-9B, as well as recent architectures Qwen3.5-9B and Gemma-3-12B. The table below reports activation cosine similarities, peak shift alignment, probe accuracies, and cross-type transfer gaps.

Model Variant \(\text{sim}(KR, SR)\) \(\text{sim}(KC, SC)\) Peak Shift Alignment \(\text{simshift}_{\ell^*}\) In-Type KR In-Type SR Cross SR\(\rightarrow\)KR Cross KR\(\rightarrow\)SR Transfer Gap \(\Delta\)
Llama-3-8B Original 0.898 0.938 0.794 0.947 0.818 0.847 0.589 +0.258
Llama-3-8B Seq-KS (\(*\)) 0.898 0.923 0.745 0.960 0.834 0.854 0.631 +0.223
Qwen2.5-7B Original 0.983 0.988 0.789 0.962 0.816 0.805 0.580 +0.225
Qwen2.5-7B Seq-KS (\(*\)) 0.980 0.985 0.750 0.962 0.858 0.847 0.597 +0.250
Gemma-2-9B Original 0.951 0.957 0.711 0.962 0.856 0.859 0.586 +0.273
Gemma-2-9B Seq-KS (\(*\)) 0.942 0.947 0.699 0.956 0.861 0.863 0.595 +0.268
Qwen3.5-9B Recent Base 0.982 0.988 0.781 0.958 0.805 0.836 0.580 +0.255
Gemma-3-12B Recent Base 0.949 0.956 0.721 0.947 0.818 0.841 0.608 +0.234

Ablation Study

Ablation experiments on tuning sequences and joint training regimes demonstrate how training order governs model utility, hallucination rates, and over-refusal behavior (evaluated on Llama-3-8B).

Tuning Configuration MMLU Acc.↑ GSM8K Acc.↑ TriviaQA Inc.↓ TriviaQA Ref. Natural Questions Inc.↓ AdvBench Ref.↑ XSTest Safe Ref.↓ Remarks
Original Base 62.90% 65.50% 19.77% 17.16% 42.08% 21.35% 26.40% Low safety refusal baseline
Knowledge-tuned (CRaFT) 65.51% 42.30% 8.74% 39.68% 21.02% 31.35% 59.60% Reduces hallucination; high safe over-refusal
Safety-tuned (FalseReject) 57.36% 50.34% 19.21% 32.29% 26.81% 64.81% 12.40% Improves safety; harms knowledge correctness
Seq-SK (Safety \(\rightarrow\) Knowledge) 46.64% 35.31% 8.32% 49.51% 9.45% 95.38% 74.80% Severe suppression; general utility collapses
Merged (Joint Shuffled) 55.38% 52.28% 14.22% 47.92% 18.39% 70.77% 16.00% Degrades QA accuracy and general utility
Seq-KS (Knowledge \(\rightarrow\) Safety) 61.84% 66.57% 12.04% 30.48% 34.68% 69.42% 15.20% Optimal Pareto tradeoff across all dimensions

Key Findings

  • Shared templates do not confound representational similarity: Across all six models and all layers, \(\Delta\text{sim} = \text{sim}(KR, SR) - \text{sim}(KC, SC) < 0\) strictly holds, formally ruling out prompt syntax as an artifactual cause of refusal alignment.
  • Persistent directional asymmetry: SR-trained probes generalize to KR with \(80\% \sim 86\%\) accuracy, whereas KR-trained probes achieve only \(58\% \sim 63\%\) on SR. This 22-27% gap persists across base models, both sequential tuning orders, and an independent 125-item XSTest set.
  • Late-layer grounding specialization: Type-specific divergence does not accumulate gradually across the forward pass; it is near zero through 80% of layers and erupts in the final 10-15% of the network, governed by late-layer MLPs and specific attention heads.

Highlights & Insights

  • The "Commit-then-Specify" Paradigmatic Framework: Resolves why distinct refusal grounds produce identical initial utterances; models commit to the communicative act of refusal early, deferring epistemic vs. normative semantic framing to the final layers.
  • Rigorous Contrastive Quadruple Design: By matching topic, syntax, and keywords while minimally toggling the refusal trigger, this work establishes a benchmark standard for representation disentanglement.
  • Causal Steering of Refusal Rationales: Adding \(+\hat{d}_{K,\text{spec}}\) or \(+\hat{d}_{S,\text{spec}}\) to residual activations selectively alters the model's generated justifications (e.g., prompting privacy rationales for fictional queries or epistemic rationales for safety queries).

Limitations & Future Work

  • Reliance on white-box model activations: Mechanistic vector extraction and layer-wise interventions require open-weights access; developing behavioral black-box proxies for closed commercial systems remains an open challenge.
  • Focus on single-turn interactions: Multi-turn dialogue contexts, where users gradually probe or jailbreak safety boundaries across turns, are not captured in the single-turn quadruple setup.
  • Tightly curated distribution vs. open-world messiness: The 213 quadruples intentionally prioritize high internal validity over diverse, ambiguous open-domain queries.
  • vs Arditi et al. (2024): While Arditi et al. argued that refusal in language models is mediated by a single direction in safety contexts, this work demonstrates that the refusal direction is shared across knowledge and safety, but is augmented by an essential, late-emerging type-specific residual subspace.
  • vs Zhu et al. (2025b) & Zhang et al. (2025): Synthesizes independent lines of research on hallucination-induced abstention (CRaFT) and safety-compliance tuning (FalseReject) into a unified empirical and mechanistic framework.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering unified mechanistic study bridging the gap between epistemic and normative LLM refusals.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 8 models, probe topologies, layer projections, ablation studies, and causal steering.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally rigorous methodology, lucid framing, and elegant conceptual clarity.
  • Value: ⭐⭐⭐⭐⭐ Critical architectural insights for safe model post-training, refusal calibration, and representation editing.