On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/FaezeGhorbanpour/LLMsHarmSensitivityInLongInputs
Area: LLM Safety
Keywords: Sensitivity Analysis, Long-Context Evaluation, Harmful Content Extraction, Dilution Effect, Positional Bias
TL;DR¶
This paper presents a systematic sensitivity analysis of instruction-tuned LLMs detecting and localizing harmful sentences sparsely embedded in long inputs (up to 30,000 tokens), uncovering non-monotonic prevalence sensitivity, pronounced dilution effects, strong beginning positional biases, and persistent explicit-vs-implicit detection gaps.
Background & Motivation¶
Large language models (LLMs) are increasingly deployed to ingest extensive mixed-content sequences in high-stakes domains, including long social media thread moderation, multi-turn community chat filtering, web-scale pre-training corpus sanitization, and long-context retrieval-augmented generation (RAG) pipelines. In natural real-world settings, harmful sentences rarely occur in isolation; rather, they are sparsely distributed among large amounts of benign, unrelated, or noisy background text. Although substantial research has been devoted to benchmarking and scaling long-context reasoning, retrieval, and needle-in-a-haystack tasks, how reliably instruction-tuned LLMs can detect and pinpoint toxic, offensive, or hateful sentences sparsely interspersed within extended prompts remains poorly understood.
Existing methodologies for harmful content detection exhibit fundamental shortcomings when confronted with long inputs. Traditional encoder-based classifiers (such as BERT or RoBERTa) are inherently crippled by strict token length constraints (e.g., 512 tokens), requiring repetitive sliding-window chunking that inflates computational overhead while severing cross-sentence discourse context. On the other hand, recent long-context safety benchmarks (such as LongSafety and LongSafetyBench) focus almost exclusively on end-to-end safety compliance rates—measuring whether an LLM's final response adheres to a safety policy when confronted with an adversarial long prompt. This coarse, binary lens fails to evaluate whether models can actively inspect their input stream and precisely localize internal harmful spans. Crucially, in organic documents, input length, harm proportion, sentence placement, and linguistic subtlety are deeply entangled, preventing controlled stress testing of individual vulnerability factors.
To demystify these behavioral dynamics, there is an urgent need for a controlled diagnostic testbed that isolates each factor while stress-testing extraction fidelity under extended sequences. Core idea: by orthogonally varying sequence length, harmful sentence prevalence, harm realization (explicit vs. implicit), and positional distribution across thousands of randomized trials and coherent dialogue validations, we systematically uncover how long-context dilution and attention degradation impair an LLM's capacity to localize sparsely embedded harmful content.
Method¶
Overall Architecture¶
The evaluation framework conceptualizes long-context safety inspection as a multi-sentence harmful index extraction task. The stress test is governed by four orthogonal parameters: total sequence length \(p \in \{600, 1500, 3000, 6000, 15000, 30000\}\) tokens, harmful sentence prevalence ratio \(r \in \{0.05, 0.10, 0.25, 0.50\}\), harm spatial region \(h \in \{\text{beginning}, \text{middle}, \text{end}, \text{all}\}\), and harm realization type \(t \in \{\text{explicit}, \text{implicit}, \text{both}\}\). The protocol evaluates three prominent instruction-tuned models (LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Mistral-7B-Instruct-v0.3) across three standardized benchmark datasets spanning hate speech (IHC), offensive language (OffensEval), and toxic comments (JigsawToxic). Each setting is repeated over \(k=128\) randomized trials with distinct random seeds. To ensure findings are not artifacts of synthetic prompt composition, the framework further incorporates a natural discourse validation pipeline using the 21-day RealTalk corpus.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Long Input Formulation<br/>Neutral Context Pool + Harmful Benchmark Sentences"] --> B["Multi-dimensional Controlled Prompting"]
B --> C["Long-Context Multi-Sentence Index Extraction"]
C --> D["Four-Dimensional Sensitivity Stress-Testing"]
subgraph VAL["Validation & Generalization Assessment"]
direction TB
D --> E["Coherent RealTalk Dialogue Validation"]
end
E --> F["Safety Analysis & Architectural Recommendations"]
Key Designs¶
1. Multi-dimensional Controlled Prompting: Disentangling Confounding Variables in Long Contexts
In natural long-form documents, harmful sentence density, document length, placement, and rhetorical implicitness are inextricably intertwined. To isolate each factor cleanly, the framework develops a standardized four-component prompt structure: (i) an authoritative task definition calibrated to the source benchmark's annotation manual (identity-targeted abuse for IHC, vulgar or targeted offense for OffensEval, and hostile or unreasonable language causing disengagement for JigsawToxic); (ii) explicit boundary exclusion rules distinguishing target harms from neighboring phenomena (e.g., affirming that untargeted vulgarities do not constitute hate speech); (iii) four unambiguous few-shot illustrative exemplars instructing the model on output formatting rather than semantic boundaries; and (iv) the assembled body of numbered sentences. Across \(k=128\) randomized runs per configuration, seeds dictate the sampling and layout of harmful and non-harmful content, eliminating sample selection bias and enabling rigorous orthogonal analysis.
2. Long-Context Multi-Sentence Index Extraction: Moving Beyond Binary Output Compliance
Rather than evaluating sentence-level classification or checking whether a model refuses to execute a jailbreak, this design requires models to ingest long, multi-sentence inputs and output an ordered, comma-separated list of indices corresponding to identified harmful sentences. To analyze model behavior from multiple diagnostic perspectives, the framework tracks Macro-F1 (balancing sensitivity across both harmful and non-harmful classes under severe imbalance), Predicted Prevalence Value (PPV, the proportion of sentences flagged as harmful by the model relative to total input size), harmful-class precision, and harmful-class recall. Predictions across all independent trials are aggregated via micro-pooling, yielding stable and statistically robust metrics. This paradigm directly reveals that while models in long multi-sentence contexts demonstrate improved macro-level prevalence calibration compared to isolated sentence evaluation, their ability to pinpoint exact sentence boundaries deteriorates under extended sequences.
3. Four-Dimensional Sensitivity Stress-Testing: Quantifying Prevalence Peaks, Context Dilution, and Primacy Biases
The core testing protocol systematically characterizes how models handle varying sparsity and placement. In the prevalence analysis, the framework demonstrates that Macro-F1 exhibits a non-monotonic inverted-U trajectory: sensitivity improves as harm ratio increases from 0.05 to approximately 0.25, but plateaus or declines at 0.50 due to a sharp increase in false positives on benign sentences. In the dilution experiment, the framework holds the absolute count of harmful sentences fixed (e.g., 10, 20, 50, 100 sentences) while progressively increasing the volume of non-harmful sentences from 20 up to 1000 sentences. This cleanly isolates dilution from tokenizer variances, proving that both recall and precision decay monotonically as harmful sentences become increasingly sparse. In the region and harm type investigations, models exhibit strong primacy attention bias (Beginning > Middle > End) and significantly higher sensitivity to explicit lexical abuse over subtle, implicit slurs.
4. Coherent RealTalk Dialogue Validation: Grounding Findings in Natural Multi-Turn Discourse
While synthetic multi-sentence prompts guarantee precise orthogonal parameter control, they naturally lack narrative coherence. To confirm that the discovered degradation patterns are not synthetic artifacts, the design leverages RealTalk—a 21-day, 8,500-conversation real-world dialogue dataset. Candidate harmful sentences are injected into coherent multi-turn conversations using dense sentence-embedding cosine similarity matching (requiring normalized similarity \(\ge 0.95\)). This ensures that injected toxic remarks align naturally with the ongoing conversational discourse. Across \(k=20\) randomized trials at sequence lengths of 1500, 3000, and 6000 tokens, the coherent evaluation mirrors the synthetic experiments: non-monotonic prevalence peaks, severe context dilution, and the explicit-versus-implicit sensitivity gap all replicate robustly.
Benchmark & Evaluation Protocol¶
Inference is executed on dedicated GPU nodes with NVIDIA GeForce GTX 1080 Ti GPUs utilizing vLLM and HuggingFace Transformers under deterministic greedy decoding (temperature=0.0, top-p=1.0, top-k=0). The maximum generation budget is dynamically set according to input length and tokenizer properties. Testing covers IHC (21,480 items; 38.12% hate, 86.56% implicit), OffensEval (13,240 items; 33.23% offensive, 34.05% implicit), and JigsawToxic (119,675 items; 10.78% toxic, 35.01% implicit). The stability of pooled micro-metrics is corroborated by macro-average standard deviations across 128 seeds (Macro-F1 standard deviations remain within 0.75%–1.20%), and statistical significance is validated using Wilcoxon signed-rank tests (\(p < 0.001\)).
Key Experimental Results¶
Main Results¶
Comparison of LLaMA-3.1-8B-Instruct on synthetic long-input extraction versus coherent RealTalk conversational validation across varying harm ratios at 1,500 tokens (\(k=20\) randomized trials):
| Dataset | Harm Ratio (r) | Setting | Macro-F1 | PPV (%) | Harm-P (%) | Harm-R (%) | Trend Analysis |
|---|---|---|---|---|---|---|---|
| IHC (Hate Speech) | 0.05 | Synthetic Input | 51.65 | 4.35 | 7.50 | 7.50 | Severe degradation under extreme sparsity |
| IHC (Hate Speech) | 0.05 | RealTalk Dialogue | 59.64 | 4.41 | 22.50 | 23.08 | Coherent context improves detection but recall remains low |
| IHC (Hate Speech) | 0.25 | Synthetic Input | 60.42 | 22.61 | 40.38 | 38.18 | Optimal balanced sensitivity peak |
| IHC (Hate Speech) | 0.25 | RealTalk Dialogue | 68.08 | 12.87 | 68.64 | 36.82 | Peak Macro-F1 with well-calibrated precision |
| IHC (Hate Speech) | 0.50 | Synthetic Input | 57.67 | 30.76 | 65.02 | 40.00 | Non-harmful precision drop induces F1 decline |
| IHC (Hate Speech) | 0.50 | RealTalk Dialogue | 61.31 | 25.76 | 76.37 | 39.35 | Stagnation in harmful recall around ~39% |
| OffensEval (Offensive) | 0.05 | Synthetic Input | 52.95 | 4.35 | 10.00 | 10.00 | Extreme dilution causes 90% miss rate |
| OffensEval (Offensive) | 0.10 | RealTalk Dialogue | 74.47 | 7.70 | 55.71 | 50.65 | Rapid recall recovery as density approaches 0.10 |
| OffensEval (Offensive) | 0.25 | RealTalk Dialogue | 66.68 | 14.33 | 61.83 | 36.82 | Stable extraction under balanced prevalence |
| OffensEval (Offensive) | 0.50 | RealTalk Dialogue | 55.42 | 23.91 | 68.04 | 32.39 | Over-aggressive harm prediction degrades overall F1 |
Ablation & Factor Analysis¶
Impact of explicit versus implicit harm realization in coherent RealTalk dialogue context (LLaMA-3.1-8B-Instruct, context length 1,500, Mean ± Std over 20 random seeds):
| Dataset | Harm Realization Type | Macro-F1 (%) | PPV (%) | Harmful Precision (%) | Harmful Recall (%) | Behavioral Insights |
|---|---|---|---|---|---|---|
| IHC (Hate Speech) | Explicit | 73.11 ± 0.78 | 16.67 ± 0.44 | 71.85 ± 1.60 | 48.64 ± 1.29 | Overt identity slurs yield highest precision & recall |
| IHC (Hate Speech) | Implicit | 63.69 ± 0.85 | 11.87 ± 0.40 | 61.51 ± 1.66 | 30.91 ± 1.45 | Nuanced bigotry suffers 17.73% drop in recall |
| IHC (Hate Speech) | Both Combined | 67.50 ± 0.89 | 12.86 ± 0.36 | 67.37 ± 1.50 | 36.82 ± 1.48 | Moderate intermediate sensitivity |
| OffensEval (Offensive) | Explicit | 67.53 ± 0.93 | 16.16 ± 0.48 | 62.96 ± 1.98 | 40.32 ± 1.39 | Vulgar profanity readily identified |
| OffensEval (Offensive) | Implicit | 62.42 ± 0.68 | 10.61 ± 0.30 | 62.12 ± 1.43 | 27.73 ± 1.09 | Veiled sarcasm and masked offense missed |
| JigsawToxic (Toxic) | Explicit | 73.58 ± 0.86 | 18.80 ± 0.37 | 68.56 ± 1.83 | 51.95 ± 1.18 | Direct hostilities detected reliably |
| JigsawToxic (Toxic) | Implicit | 74.97 ± 0.75 | 17.28 ± 0.45 | 74.05 ± 1.47 | 52.07 ± 1.33 | Retains structural lexical markers in Jigsaw |
Key Findings¶
- Non-Monotonic Prevalence Peak at Moderate Harm Ratios: Across all tested models and context lengths, extraction performance follows an inverted-U curve peaking at approximately \(r=0.25\). In isolated sentence-level evaluations, LLMs are heavily biased toward over-predicting harm (PPV typically reaches 60%–75%, with recall \(>90\%\) but low precision). In multi-sentence long prompts, models calibrate their macro-prediction ratio (PPV closely tracks true prevalence \(r\)), but their fine-grained discriminative power falters at extremes: at low ratios (\(r=0.05\)) models default to predicting non-harmful, whereas at high ratios (\(r=0.50\)) aggressive harm prediction degrades non-harmful precision, pulling down Macro-F1.
- Pervasive Context Dilution: Holding the absolute count of harmful sentences constant while scaling the surrounding neutral sentences causes harmful recall and precision to decay steeply. When harmful sentences are diluted within extensive benign text, LLMs fail to allocate sufficient attention to isolate sparse toxic needles, demonstrating that long-context capacity does not equate to sparse-signal sensitivity.
- Primacy Attention Bias (Beginning > Middle > End): Harmful sentences concentrated in the opening third (Beginning) of the input prompt consistently achieve substantially higher recall and Macro-F1 than those located in the middle or final third. Models attend disproportionately to initial input spans, while middle and late regions experience severe attentional attenuation (with minor middle-region preservation in Mistral due to its sliding-window attention mechanism).
Highlights & Insights¶
- Transitioning from Coarse Compliance to Fine-Grained Span Localization: The paper establishes a rigorous methodology that goes beyond measuring whether an LLM's final response complies with safety policies, providing granular empirical visibility into the model's ability to inspect and identify internal toxic passages within long sequences.
- The Prior Calibration vs. Identification Fidelity Divergence: An essential cognitive insight is that multi-sentence prompting provides LLMs with an effective prior on "how much" harm is present (calibrating PPV to the true ratio), yet models lack the attention fidelity to determine "which exact sentences" are harmful under long contexts.
- Actionable Engineering Recommendations for Long-Context Deployment: The empirical findings offer concrete architectural blueprints for safety systems: deploying overlapping windowed chunking to counter the dilution effect, implementing position-aware multi-pass queries to mitigate primacy bias, and constructing two-stage pipelines where macro-prevalence screening precedes localized sentence-level filtering.
Limitations & Future Work¶
- Evaluation Limited to Mid-Sized Open Weights: Experiments were conducted on three prominent ~7B/8B open-source instruction models (LLaMA-3.1-8B, Qwen2.5-7B, Mistral-7B); closed-source frontier reasoning architectures (e.g., GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) were not evaluated due to resource and access constraints.
- Narrow Focus on Social Discourse Abuse: The study focused on three standard text categories (hate speech, offensive language, toxicity). Broader safety vulnerabilities—such as self-harm encouragement, fraud, stealthy prompt injection payloads, and synthetic disinformation—remain to be investigated under long-context dilution.
- Synthetic Versus Full Document Coherence: Although the RealTalk experiment confirmed that major qualitative trends persist in natural conversational dialogues, the core stress-testing protocol relies on synthetic concatenation of discrete social posts, which may differ from the rhetorical structure of legal documents or book-length texts.
Related Work & Insights¶
- vs. LongSafety / LongSafetyBench: Prior long-context safety benchmarks primarily evaluate whether models refuse harmful jailbreak prompts embedded in long contexts (measuring binary compliance rate). This work investigates the orthogonal, fine-grained challenge of detecting and localizing dispersed harmful input sentences.
- vs. Sentence-Level Classifiers (BERT / RoBERTa): Classical encoder models are constrained by 512-token limits, necessitating expensive sliding chunking that destroys cross-sentence context; this paper evaluates zero-shot and few-shot capabilities of native 32k–128k context LLMs across multi-thousand token contexts.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First systematic and controlled diagnostic study decomposing long-context LLM sensitivity across dilution, prevalence, position, and implicitness.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 6 context lengths (up to 30k tokens), 4 harm ratios, 3 placement regions, 3 harm types, 128 randomized seeds, coherent dialogue validation, and non-parametric statistical tests.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous narrative flow, crystal-clear experimental formulations, and excellent translation from empirical discoveries to safety engineering takeaways.
- Value: ⭐⭐⭐⭐⭐ Provides indispensable diagnostic principles and actionable mitigations for RAG pipelines, long-form content moderation systems, and pre-training data filtering.