Skip to content

MiST: Mid-Training LLMs for Cybersecurity

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM / NLP
Keywords: Cybersecurity LLMs, Mid-Training, Synthetic Data Generation, Reinforcement Learning Initialization, Domain Adaptation

TL;DR

MiST reframes cybersecurity LLM adaptation as a deliberate corpus composition problem rather than a token-scaling challenge, transforming a compact, expert-curated seed corpus into structured synthetic mid-training data that substantially outperforms raw domain continual pre-training while providing a markedly superior initialization for downstream task-specific SFT and verifiable RL.

Background & Motivation

Cybersecurity analysis requires precise interpretation of technical jargon, shifting adversary tactics, and context-dependent defensive rules. While general-purpose large language models (LLMs) continue to advance on standard natural-language and coding benchmarks, they struggle with domain-specific nuances, frequently hallucinating vulnerability causality or misidentifying exploit mechanics. Compounding this challenge, the high confidentiality and privacy requirements of security operations often mandate on-premises deployment, making reliance on external proprietary API endpoints impractical and creating an urgent demand for performant, locally deployable open models.

The prevailing methodology for developing specialized language models relies on two main recipes: continued pre-training (CPT) over massive volumes of raw web-crawled domain text, followed by supervised fine-tuning (SFT) on instruction datasets. In cybersecurity, however, this approach encounters severe bottlenecks. Raw domain corpora are sparse, unstructured, and noisy, largely composed of informal security blogs, partial logs, or high-level non-technical summaries that lack explicit analytical chains. Meanwhile, the foundational knowledge of the security discipline is concentrated in compact, highly structured expert artifacts—such as CVE vulnerability disclosures, CWE taxonomies, MITRE ATT&CK technique matrices, and Sigma detection rules. Ingesting these structured artifacts as unstructured plain text during raw CPT fails to teach the model how to reason about attack causality, while skipping directly to superficial SFT leads to fragile memorization and catastrophic forgetting of general reasoning capabilities.

The core realization of this work is that domain specialization should be treated as an intermediate distribution design problem rather than a brute-force token-scaling effort. Instead of training on billions of raw domain tokens, an effective mid-training stage must bridge the distributional gap between broad base pre-training and specialized task execution by actively converting static artifacts into structured, multi-perspective causal reasoning. Core idea: introduce an intermediate mid-training stage between base pre-training and instruction tuning that expands a compact, 220k-sample expert seed corpus into diverse synthetic learning flows—spanning semantic paraphrasing, textbook-style lessons, iterative QA, and metric justifications—combined with general instruction replay and rigorous 13-gram decontamination to establish a robust, adaptable cybersecurity foundation.

Method

Overall Architecture

The end-to-end MiST framework begins from general-purpose base checkpoints (Qwen3-8B-Base and Qwen3-32B) and processes them through three sequential stages: domain mid-training, supervised fine-tuning, and direct preference optimization (DPO). At the center of the architecture is a multi-flow synthetic generation pipeline grounded in an expert-curated seed corpus. The seed data is first subjected to strict source-level decontamination to eliminate benchmark overlap. Next, a lightweight yet capable transformation engine (Qwen-30B-A3B) converts the raw seed artifacts into diverse pedagogical formats. The generated candidates are evaluated by an automated six-dimensional quality verifier and combined with a general instruction replay buffer to form the final mid-training distribution. After mid-training on this corpus, the model undergoes multi-turn persona-based instruction tuning and general DPO alignment to produce the final deployable checkpoint.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Base Model<br/>Qwen3-8B-Base / 32B"] --> B["Compact Seed Corpus & Source Filtering"]
    B --> C["Multi-Flow Synthetic Data Generation & 6D Filtering"]
    C --> D["Cybersecurity Mid-Training with General Replay"]
    D --> E["Persona-Based SFT & General DPO Alignment"]
    E --> F["Downstream Evaluation & Verifiable RL Adaptation"]

Key Designs

1. Compact Seed Corpus & Source Filtering: Eliminating Domain Noise and Benchmark Contamination To avoid the linguistic noise and low informational density of raw web scrapes, MiST curates an information-dense seed corpus of 219,661 samples (~64.7M tokens) spanning four authoritative families: (1) vulnerability records and threat intelligence (NVD CVEs, curated CTI/RSS feeds); (2) security knowledge bases and taxonomies (CWE weaknesses, CAPEC attack patterns, MITRE ATT&CK matrices, D3FEND countermeasures); (3) operational defensive artifacts (Sigma detection rules, Atomic Red Team tests, Splunk ESCU detections, MISP Galaxy clusters); and (4) defensive guidance and standards (OWASP materials, NIST publications, cloud/OS security guides, and security Wikipedia). Crucially, to prevent data contamination that plagues public security evaluations, the authors perform rigorous source-level filtering before data generation: all CVE records featured in CTI-Bench root cause mapping and severity prediction splits are purged from the seed, and GHSA code records sharing commit hashes with PrimeVul validation/test sets are removed.

2. Multi-Flow Synthetic Data Generation & 6D Filtering: Transforming Static Artifacts into Causal Reasoning Static artifacts like CVE strings or detection rules lack narrative explanations. MiST employs Qwen-30B-A3B (an active 3B-parameter model with high instruction adherence) as an in-context transformation engine, keeping generation grounded in seed context rather than unverified parametric memory. For mid-training, the pipeline deploys four distinct transformation flows: - Paraphrasing: Generates semantically diverse lexical and syntactic rewrites of security concepts to encourage generalized representations; - Educational Transformation: Converts technical records into structured textbook chapters and professional blog posts detailing architectural attack vectors and mitigation rationale; - Iterative QA: Produces concise, 1-to-2 sentence factual question-answer pairs and iteratively prompts the model across two follow-up rounds using earlier questions as context to surface unaddressed details; - Cyber Metrics & Terminology Analysis: Prompts the model to explicitly explain and justify structured metadata fields (e.g., deriving CVSS vectors or mapping ATT&CK techniques), teaching causal justification rather than rote token memorization. To maintain data purity, every candidate sample is scored on a 1–10 scale across six criteria: instruction adherence, task completion, factuality, format/style alignment, relevance/focus, and logical consistency/coherence. Samples scoring below 8 overall are discarded. Furthermore, multiple-choice prompts and A/B/C/D option layouts are explicitly banned from all synthetic flows to prevent structural leaking into downstream evaluation formats.

3. Cybersecurity Mid-Training with General Replay: Structural Composition Over Token Scaling Rather than continually training solely on domain data, which inevitably causes distributional drift and catastrophic forgetting, the final mid-training corpus of 1,237,945 samples (~975.7M tokens) is structurally balanced: educational content comprises 63.0%, paraphrases 18.7%, security analyses 7.5%, and iterative QA 6.8%. To preserve base linguistic and instruction capabilities, MiST incorporates a replay buffer of 50,000 general instruction samples (4.1% of tokens) from the Olmo 3 Dolci SFT dataset. Before tokenization, an identical 13-gram overlap filter is run against a decontamination bank of 27,201 benchmark segments (467,000 unique 13-grams); matching samples are immediately pruned (~3.78% removal rate). Training optimizes standard next-token causal language modeling over sequence-packed 16,384-token windows, instilling deep domain causal structure while retaining base model competence.

4. Persona-Based SFT & General DPO Alignment: Operational Chat Grounding and Safety Preservation In real-world environments, interactions vary drastically depending on the practitioner's operational perspective. In the post-training phase, MiST constructs 307,679 cybersecurity dialogue turns (352.2M tokens) driven by concrete operational personas—such as blue team endpoint analysts investigating persistence, red team operators evaluating CVE exploitation, JavaScript software engineers, and enterprise security officers. These domain conversations are paired with the full Dolci general instruction dataset (1,924,533 samples, 1,198.7M tokens) for supervised fine-tuning. Finally, the model undergoes Direct Preference Optimization (DPO) using non-cybersecurity preference pairs (Dolci-Instruct-DPO) with \(\beta=0.1\) and early stopping at 200 steps. Using general rather than cyber preference pairs ensures that DPO acts strictly as an alignment refiner, preserving acquired domain capabilities while enforcing conversational compliance and low refusal rates.

Loss & Training

During the mid-training phase, the model is trained with the standard causal language modeling loss: $$ \mathcal{L}{\text{mid}}(\theta) = -\sum; \theta) $$ All experiments run on a single node with 8× NVIDIA B200 GPUs. Mid-training spans 2 epochs with a maximum sequence length of 16,384 tokens, effective per-device batch size of 16, and sequence packing enabled. Optimization is performed using AdamW with cosine learning rate scheduling (peak }^{L} \log P(x_i \mid x_{<i\(5 \times 10^{-5}\), minimum \(1 \times 10^{-6}\)), 0.03 warmup ratio, 0.05 weight decay, and 0.2 gradient clipping. DeepSpeed ZeRO-2 is utilized for 8B and ZeRO-3 for 32B. SFT uses sequence length 8,192 and peak learning rate \(3 \times 10^{-5}\). The subsequent DPO phase optimizes: $$ \mathcal{L}{\text{DPO}}(\theta; \pi}}) = -\mathbb{E{(x, y_w, y_l)}\left[ \log \sigma \left( \beta \log \frac{\pi\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right] $$ Total wall-clock training time on 8×B200 GPUs is exceptionally lean: 7.1 hours (56.8 GPU-hours) for MiST-8B and 42.7 hours (341.3 GPU-hours) for MiST-32B.

Key Experimental Results

Main Results

The models were evaluated across 13 diverse cybersecurity benchmarks covering threat intelligence (CTI-Bench MCQ and RCM), advisory environments (SECURE MAET, CWET, KCV), dynamic CTI tasks (AthenaBench CKT, ATE, RMS, TAA), pair-wise vulnerability detection (PrimeVul paired classification P-C and chain-of-thought P-C CoT), MMLU-Cyber, and SecEval. All evaluations are executed with reasoning/thinking modes strictly disabled across all baselines.

Category Model CTI-MCQ CTI-RCM SEC-MAET SEC-CWET SEC-KCV Ath-CKT Ath-ATE Ath-RMS Ath-TAA PV P-C PV CoT MMLU-Cy SecEval Mean
Large / Proprietary GPT-5.4-mini 73.9 73.1 91.3 92.0 88.9 85.2 53.2 29.2 29.8 8.4 10.1 86.0 74.7 61.2
GPT-5.4-nano 62.1 69.0 81.5 84.0 85.6 79.5 45.3 5.5 20.0 9.0 9.2 75.8 67.3 53.4
Qwen3-235B 71.9 71.4 89.6 91.5 86.2 83.6 43.5 13.1 27.6 4.5 14.6 87.0 71.3 58.1
Specialized Cyber DeepHat-7B 54.7 67.2 82.7 83.1 86.2 69.2 9.8 3.1 14.8 12.0 3.9 79.8 62.4 48.4
Foundation-Sec-8B 58.2 67.9 84.7 83.3 80.1 76.9 37.3 18.7 26.4 0.0 17.8 78.6 65.0 53.5
Lily-Cyber-7B 42.1 42.9 55.8 54.4 43.2 67.0 3.0 2.1 12.4 0.0 7.1 68.8 48.9 34.4
Primus-8B 54.7 67.2 81.9 82.6 82.0 73.7 30.6 7.3 21.4 0.3 13.4 82.2 60.9 50.6
RedSage-8B-DPO 62.0 73.2 89.6 91.2 81.1 78.8 51.6 18.8 22.8 1.6 14.8 82.4 67.4 56.5
Primus-70B 67.7 65.1 91.4 93.7 88.1 82.1 52.7 14.8 2.2 1.0 17.6 87.6 70.5 56.5
CyberPal2.0-20B 73.8 72.8 91.4 92.5 83.6 82.7 57.0 23.3 19.6 0.8 14.1 82.2 67.1 58.5
Original Baselines Qwen3-8B 58.2 53.5 83.8 87.5 84.5 75.2 9.4 5.5 15.0 2.0 13.0 78.0 66.7 48.6
Qwen3-32B 68.4 65.9 89.5 91.9 88.8 80.2 26.0 6.3 15.2 4.0 16.0 84.8 72.7 54.6
Ours MiST-8B-SFT 78.9 74.1 93.7 92.7 85.4 86.0 60.2 29.4 23.8 14.0 18.0 82.8 72.2 62.4
MiST-8B-DPO 79.3 73.9 94.3 93.7 87.3 86.0 57.3 25.7 24.6 10.0 15.0 83.4 71.5 61.7
MiST-32B-SFT 82.2 76.4 95.0 92.9 88.7 89.2 52.8 27.1 27.8 10.0 17.0 82.2 76.2 62.9
MiST-32B-DPO 82.7 76.3 95.4 94.1 89.4 89.4 55.2 28.3 26.4 8.0 15.0 84.8 76.0 63.2

Ablation Study

1. Isolating the Effect of Cybersecurity Mid-Training To verify that gains stem from mid-training rather than the SFT stage alone, an ablation was conducted where identical SFT was applied directly to Qwen3-8B-Base without mid-training:

Model Configuration CTI-MCQ CTI-RCM SEC-MAET SEC-CWET SEC-KCV Ath-CKT Ath-ATE Ath-RMS Ath-TAA PV P-C PV CoT MMLU-Cy SecEval Mean
SFT Only (No Mid-Training) 74.1 67.0 92.7 90.5 76.6 83.0 44.5 27.3 22.0 2.9 14.3 84.3 69.0 57.6
Full MiST-8B-SFT 78.9 74.1 93.7 92.7 85.4 86.0 60.2 29.4 23.8 14.0 18.0 82.8 72.2 62.4
Gain \(\Delta\) +4.8 +7.1 +1.0 +2.3 +8.8 +3.0 +15.7 +2.0 +1.8 +11.1 +3.7 -1.5 +3.2 +4.8

2. Synthetic Flow Ablation (Single-Flow & Leave-One-Out) Evaluating the four document-level mid-training flows (QA, Security Analysis, Educational, Paraphrase) demonstrates that all flows provide distinct and non-redundant value:

Setup Corpus Variant CTI-MCQ CTI-RCM SEC-MAET SEC-KCV Ath-ATE Ath-RMS PV P-C SecEval Mean
Full Recipe All Flows Combined 78.9 74.1 93.7 85.4 60.2 29.4 14.0 72.2 62.4
Single-Flow QA only 76.8 66.0 93.0 79.2 52.6 26.2 6.2 68.3 58.7
Security-Analysis only 74.7 69.5 92.0 83.5 55.9 22.7 7.6 69.1 58.8
Educational only 76.7 68.2 92.9 79.1 48.7 32.9 3.5 69.5 59.2
Paraphrase only 76.2 65.0 92.6 86.3 52.4 29.3 11.8 70.6 60.3
Leave-One-Out No QA 77.8 71.5 93.1 80.6 58.2 27.7 11.6 70.8 60.6
No Security-Analysis 78.3 65.8 93.3 80.3 53.9 27.3 5.4 70.2 59.6
No Educational 77.7 70.4 93.3 82.9 55.3 26.9 6.4 67.9 60.4
No Paraphrase 77.9 72.4 93.5 82.8 60.1 29.0 9.8 67.6 61.0

Key Findings

  • Synthetic Mid-Training Outperforms 2× Larger Raw CPT: When benchmarked against an 8B baseline trained on a 2.4B-token raw domain corpus (incorporating PRIMUS), MiST-8B trained on ~1B tokens achieves significantly superior performance across almost every benchmark family (80.7% average score vs 70.9% for CPT), confirming that information density and structured transformations trump raw token count.
  • Superior Starting Point for Downstream Reinforcement Learning: Applying GRPO across three deterministic cybersecurity tasks (CVE-to-CWE, CVE-to-CVSS vector, Code-to-CWE) shows that MiST-8B-DPO maintains markedly higher validation accuracy throughout training steps while incurring less than half the policy shift (KL divergence under 0.03 vs >0.08 for Qwen3-8B).
  • Stronger Adaptation in Task-Specific SFT: Fine-tuning checkpoints on PrimeVul reveals that MiST adapts far more effectively to paired vulnerability discrimination: fine-tuning boosts MiST-8B-DPO from 10.3 to 16.3 (CoT: 19.2), whereas Qwen3-8B-Instruct only moves from 2.2 to 2.8 (CoT: 13.5).
  • No Catastrophic Forgetting of General Capabilities: Evaluating on general benchmarks (ARC-C, GSM8K, MMLU, IFEval) shows average performance improves from 82.5 to 85.8 on 8B and 89.1 to 90.7 on 32B, proving that the 50K general instruction replay effectively immunizes against domain drift.

Highlights & Insights

  • Curated Transformations Over Raw Pre-training Scale: Rather than dumping hundreds of gigabytes of noisy internet text into continual pretraining, transforming a high-signal 220K-item seed corpus into diverse instructional modes yields state-of-the-art domain proficiency with sub-1B tokens.
  • Three-Tier Multi-Point Decontamination Defense: Addresses the endemic vulnerability benchmark leakage issue via source-level entity exclusion, structural removal of MCQ/option phrasing during synthesis, and mandatory 13-gram exact n-gram pruning across every dataset.
  • Decoupled Alignment Philosophy: By using general preference data (Dolci-Instruct-DPO) instead of specialized security preference pairs, DPO acts cleanly to polish instruction-following behavior and safety posture without eroding hard-won specialized knowledge.

Limitations & Future Work

  • Lack of Long-Horizon Tool-Use and Operational Telemetry Evaluation: The current evaluation suite remains centered on multiple-choice questions, structured classification, and static vulnerability identification, lacking live SOC workflows, SIEM log parsing, or interactive red-team tool agents.
  • Shared Generator-Verifier Architectural Blind Spots: Since Qwen-30B-A3B is employed as both the generation engine and the 6D quality filter, subtle systemic biases or recurring misconceptions may evade detection.
  • Limited Scope of Verifiable RL Objectives: Downstream RL was demonstrated only on deterministic pattern-matching and mapping tasks, leaving complex multi-turn exploit synthesis or interactive patch verification for future study.
  • vs PRIMUS & Foundation-Sec-8B: These models perform large-scale CPT on vast, noisy corpora of web and code text. MiST demonstrates that structured synthetic mid-training produces a +9 to +14 point advantage across benchmarks while consuming less than half the compute.
  • vs CyberPal 2.0 & RedSage: While CyberPal 2.0 and RedSage concentrate their synthetic efforts on SFT and reasoning traces, MiST introduces synthetic structuring earlier at the mid-training phase, establishing a much stronger representation that amplifies subsequent fine-tuning gains.
  • vs Frontier Baselines (GPT-5.4-nano / Qwen3-235B): MiST-8B and MiST-32B achieve 61.7 and 63.2 mean scores, outperforming GPT-5.4-nano (53.4) and Qwen3-235B (58.1) in specialized technical domains such as Attack Technique Extraction (Athena-ATE) and paired vulnerability detection.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First systematic formulation of cybersecurity adaptation as a synthetic mid-training corpus design challenge, demonstrating unprecedented data efficiency.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely rigorous evaluations across 13 security and 4 general benchmarks, paired with exhaustive single-flow and leave-one-out ablations, GRPO KL-tracking, and PrimeVul paired SFT.
  • Writing Quality: ⭐⭐⭐⭐⭐ Lucid technical framing, honest decontamination audits, and crystal-clear presentation of trade-offs.
  • Value: ⭐⭐⭐⭐⭐ Sets a compelling standard for privacy-sensitive vertical domains that require high-performance, cost-effective on-premises LLM deployment.