# MiST: Mid-Training LLMs for Cybersecurity

> MiST shows curated mid-training with synthetic data from expert-vetted seeds outperforms raw continual pre-training, boosting cybersecurity accuracy by up to 13.1 points while preserving general capabilities.

- **Source:** [arXiv](https://arxiv.org/abs/2609.18496)
- **Published:** 2026-09-19
- **Permalink:** https://picx.dev/p/vRDeY2
- **Whiteboard:** https://picx.dev/p/vRDeY2/image

## Summary

## Summary (Overview)

- **MiST (Mid-trained Security Transformer)** is a suite of 8B and 32B cybersecurity-specialized LLMs derived from Qwen3 checkpoints, achieving mean cybersecurity accuracy improvements of **+13.1 and +8.6 absolute percentage points** over Qwen baselines (relative gains of +27.0% and +15.8%).
- The core innovation is treating **mid-training as a corpus design problem** rather than a token-scaling problem: a compact, expert-vetted seed corpus is transformed into high-quality synthetic training data via multiple generation pipelines.
- MiST outperforms existing open cybersecurity-specific models (DeepHat, Foundation-Sec, Primus, RedSage, CyberPal) and is competitive with larger proprietary models like GPT-5.4-mini despite being much smaller.
- Ablations show gains arise from mid-training and SFT stages through synthetic data generation flows, not from DPO (which uses general preference data).
- MiST provides a **stronger initialization for downstream task-specific adaptation** — both task-specific SFT (PrimeVul) and reinforcement learning (GRPO) show larger gains and lower KL divergence when starting from MiST.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Cybersecurity presents a unique challenge for LLMs due to:
- **High-stakes analysis** requiring precise interpretation of specialized terminology
- **Dynamic attack techniques** and context-dependent operational knowledge
- **Scarce high-quality pre-training corpora** in the security domain
- **Privacy-sensitive environments** mandating on-premises deployment, making API-based models impractical

### Mid-Training as a Paradigm

The authors position mid-training as a deliberate alternative to raw continual pre-training (CPT):

> "Mid-training provides a more deliberate alternative to this raw-domain adaptation pipeline... designed to bridge the distributional gap between a broad base model and the target domain."

Key theoretical foundations:
- **Tu et al. (2025)** describe mid-training as *"the critical bridge between general pre-training and post-training"*
- **Zhang et al. (2025a)** argue mid-training works by moving the model toward the target post-training distribution, with strongest gains when intermediate data is closer to the target domain
- **Ben Zaken et al. (2022)** provide evidence that adaptation can expose or redirect knowledge already present in pretrained models

The key distinction from raw CPT: while CPT focuses on *scaling token volume*, MiST emphasizes *data composition* — starting from a small, high-quality seed dataset and extending it with diverse synthetic pipelines.

---

## Methodology

### Training Pipeline Overview

The full pipeline consists of three stages:

1. **Mid-training**: Domain-specific adaptation on synthetic cybersecurity corpus
2. **Supervised Fine-Tuning (SFT)**: Chat template introduction + general/cyber instruction data
3. **Direct Preference Optimization (DPO)**: General alignment using general preference data (Dolci-Instruct-DPO)

### Seed Data Curation

The seed corpus is organized into **four source families**:

| Source Family | Examples | Purpose |
|---|---|---|
| Vulnerability & threat intelligence | NVD CVE records, CTI/RSS reports | Instance-level knowledge: identifiers, CVSS metadata, CWE mappings |
| Security knowledge bases & taxonomies | CWE, CAPEC, ATT&CK, D3FEND | Abstractions and relationships between weaknesses, attack patterns, tactics |
| Operational security artifacts | Sigma rules, Atomic Red Team, Splunk ESCU, MISP Galaxy | Detection logic, adversary emulation, threat hunting |
| Defensive guidance & documentation | OWASP, NIST, vendor/platform docs, security-focused Wikipedia | Defensive guidance, platform terminology, configuration patterns |

### Synthetic Data Generation Flows

**Generation model**: Qwen-30B-A3B (3B activated parameters) — chosen for efficiency and strong instruction-following. The model functions as a *transformation engine* rather than relying on memorized knowledge.

#### Mid-Training Flows:
1. **Paraphrasing**: Semantic rewriting with varied lexical/syntactic structure while preserving technical meaning
2. **Educational transformation**: Seed documents converted to professional blog posts or textbook-style chapters
3. **QA generation**: Concise factual question-answer pairs (1-2 sentences), with iterative follow-up rounds targeting uncovered aspects
4. **Cyber metrics & terminology analysis**: For structured metadata (CVE/CWE IDs, CVSS vectors, ATT&CK techniques), prompt the model to explain and justify fields

#### Post-Training Flows:
1. **Conversations**: Up to 7-turn user–assistant dialogues
2. **Persona-based conversations**: Stakeholder-specific interactions (security officer, software engineer, red team operator, student)

### Data Verification

An LLM-based verifier (Qwen-30B-A3B) assigns six sub-scores (1–10 scale):
- Instruction adherence
- Task completion
- Factuality
- Format and style alignment
- Relevance and focus
- Logical consistency and coherence

**Samples with overall score below 8 are removed.**

### General Data Preservation

- **50K samples** from Olmo 3 Dolci SFT dataset included during mid-training (replay strategy)
- Remaining general data used during post-training

### Training Configuration

- **Models**: MiST-8B (from Qwen3-8B-Base), MiST-32B (from post-trained Qwen3-32B; no base checkpoint publicly available)
- **Hardware**: Single node with 8× NVIDIA B200 GPUs
- **Mid-training**: 2 epochs, max sequence length 16,384 tokens, batch size 16 with sequence packing
- **Optimizer**: AdamW, cosine LR schedule, peak $5 \times 10^{-5}$, minimum $1 \times 10^{-6}$, warmup ratio 0.03, weight decay 0.05, gradient clipping 0.2
- **Precision**: bfloat16, FlashAttention-2, DeepSpeed (ZeRO-2 for 8B, ZeRO-3 for 32B)

### Contamination Controls

Three layers of decontamination:
1. **Source-level audit**: Removes seed records that anchor benchmark examples (e.g., CVE records in CTI-Bench splits, GHSA-linked records in PrimeVul)
2. **Format avoidance**: Synthetic flows deliberately avoid MCQ prompts and A/B/C/D answer structures
3. **13-gram filter**: Applied to mid-training, SFT, and DPO datasets; removes ~3.78% of candidate samples

---

## Empirical Validation / Results

### Cybersecurity Benchmarks

The evaluation suite covers: CTI-Bench, SECURE, SecEval, MMLU-Cyber, PrimeVul, and AthenaBench.

**Key Results (Table 1, selected rows):**

| Model | CTI MCQ | CTI RCM | Secure Mean | Athena Mean | PrimeVul P-C | MMLU Cyber | SecEval | **Mean** |
|---|---|---|---|---|---|---|---|---|
| GPT-5.4-mini | 73.9 | 73.1 | 90.7 | 49.4 | 8.4 | 86.0 | 74.7 | **61.2** |
| Qwen3-8B (baseline) | 58.2 | 53.5 | 85.3 | 12.3 | 2.0 | 78.0 | 66.7 | **48.6** |
| Qwen3-32B (baseline) | 68.4 | 65.9 | 90.1 | 16.4 | 4.0 | 84.8 | 72.7 | **54.6** |
| **MiST-8B-SFT** | **78.9** | **74.1** | **90.6** | **37.9** | **14.0** | **82.8** | **72.2** | **62.4** |
| **MiST-8B-DPO** | **79.3** | **73.9** | **91.8** | **35.9** | **10.0** | **83.4** | **71.5** | **61.7** |
| **MiST-32B-SFT** | **82.2** | **76.4** | **92.2** | **35.9** | **10.0** | **82.2** | **76.2** | **62.9** |
| **MiST-32B-DPO** | **82.7** | **76.3** | **93.0** | **36.6** | **8.0** | **84.8** | **76.0** | **63.2** |

**Key findings:**
- MiST-8B-SFT: +13.8 absolute points over Qwen3-8B (+28.4% relative)
- MiST-32B-SFT: +8.3 absolute points over Qwen3-32B (+15.2% relative)
- SFT checkpoints account for nearly all final cybersecurity performance; DPO largely preserves capabilities

### General Benchmarks (Table 2)

| Model | ARC-C | GSM8K | MMLU | IFEval | Mean |
|---|---|---|---|---|---|
| Qwen3-8B | 77.1 | 89.7 | 73.4 | 90.0 | 82.5 |
| MiST-8B-DPO | 89.8 | 90.0 | 74.9 | 88.5 | **85.8** |
| Qwen3-32B | 89.8 | 94.0 | 81.6 | 91.1 | 89.1 |
| MiST-32B-DPO | 96.2 | 93.5 | 82.1 | 90.9 | **90.7** |

MiST models **largely retain or improve** general capabilities, with minimal catastrophic forgetting.

### Mid-training vs. Raw CPT

A raw CPT baseline was constructed by augmenting seed data with PRIMUS (~2.4B tokens, ~2× larger than mid-training corpus). Results:

> "While raw CPT improves over the original model, MiST achieves superior performance on nearly all benchmarks despite using fewer training tokens."

### Downstream Task Adaptation

#### Task-Specific RL (GRPO)

On three verifiable tasks (CVE-to-CWE mapping, CVE-to-CVSS vector prediction, vulnerable-code-to-CWE mapping):
- MiST-8B-DPO achieves **higher validation accuracy at every checkpoint**
- Maintains **lower KL divergence** (smaller policy movement)

#### Task-Specific SFT (PrimeVul)

Fine-tuning on PrimeVul paired data shows consistently **larger gains from MiST initialization** than from Qwen, at both 8B and 32B scales, under both direct and chain-of-thought prompting.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Mid-training as corpus design**: The work reframes mid-training from a token-scaling problem to a data-composition problem, demonstrating that a compact, high-quality synthetic corpus can outperform substantially larger raw domain corpora.

2. **Synthetic transformation as knowledge distillation**: By using expert-vetted seeds as anchors and transforming them through multiple flows, the model learns *concepts, relations, and reasoning* rather than just raw text statistics.

3. **Initialization quality matters**: MiST provides a stronger starting point for downstream adaptation, supporting the view that mid-training moves the model toward the target distribution before post-training begins.

### Practical Implications

1. **Efficiency**: The approach is particularly valuable for resource-constrained environments where smaller models must be deployed on-premises.

2. **Data efficiency**: Achieves state-of-the-art cybersecurity performance with a compact corpus, reducing the need for massive domain-specific data collection.

3. **Adaptability**: MiST serves as a better foundation for both supervised fine-tuning and reinforcement learning, enabling further task-specific specialization.

4. **General capability preservation**: The inclusion of general data during training stages prevents catastrophic forgetting while adding domain expertise.

---

## Conclusion

### Main Takeaways

MiST demonstrates that:
- **Curated mid-training** is more effective than raw continual pre-training for domain adaptation
- **Data quality and structure** matter more than token volume alone
- **Synthetic data generation** from expert-vetted seeds can produce high-quality domain training data
- **Domain mid-training** provides a stronger initialization for downstream task-specific adaptation

### Future Directions

The authors identify several areas for future work:
- Evaluating in **operational settings** (log analysis, incident triage, tool use, agentic workflows)
- Exploring **broader RL recipes** and more complex multi-step security tasks
- **Quantifying effects** of generator choice, prompt design, and filtering thresholds
- Developing **better contamination controls** for paraphrased or semantically equivalent leakage

### Limitations Acknowledged

- Evaluation focuses on knowledge-intensive understanding, not operational deployment readiness
- Heavy reliance on LLM-generated synthetic data may introduce systematic biases
- Contamination cannot be completely ruled out despite multi-layer controls
- Dual-use concerns require careful release decisions and governance

---

_Markdown view of https://picx.dev/p/vRDeY2, served by PicX — AI-generated visual whiteboard summaries of research papers._