Summary (Overview)

• MiST (Mid-trained Security Transformer) is a suite of 8B and 32B cybersecurity-specialized LLMs derived from Qwen3 checkpoints, achieving mean cybersecurity accuracy improvements of +13.1 and +8.6 absolute percentage points over Qwen baselines (relative gains of +27.0% and +15.8%).
• The core innovation is treating mid-training as a corpus design problem rather than a token-scaling problem: a compact, expert-vetted seed corpus is transformed into high-quality synthetic training data via multiple generation pipelines.
• MiST outperforms existing open cybersecurity-specific models (DeepHat, Foundation-Sec, Primus, RedSage, CyberPal) and is competitive with larger proprietary models like GPT-5.4-mini despite being much smaller.
• Ablations show gains arise from mid-training and SFT stages through synthetic data generation flows, not from DPO (which uses general preference data).
• MiST provides a stronger initialization for downstream task-specific adaptation — both task-specific SFT (PrimeVul) and reinforcement learning (GRPO) show larger gains and lower KL divergence when starting from MiST.

---

Introduction and Theoretical Foundation

Background and Motivation

Cybersecurity presents a unique challenge for LLMs due to:

• High-stakes analysis requiring precise interpretation of specialized terminology
• Dynamic attack techniques and context-dependent operational knowledge
• Scarce high-quality pre-training corpora in the security domain
• Privacy-sensitive environments mandating on-premises deployment, making API-based models impractical

Mid-Training as a Paradigm

The authors position mid-training as a deliberate alternative to raw continual pre-training (CPT):

"Mid-training provides a more deliberate alternative to this raw-domain adaptation pipeline... designed to bridge the distributional gap between a broad base model and the target domain."

Key theoretical foundations:

• Tu et al. (2025) describe mid-training as "the critical bridge between general pre-training and post-training"
• Zhang et al. (2025a) argue mid-training works by moving the model toward the target post-training distribution, with strongest gains when intermediate data is closer to the target domain
• Ben Zaken et al. (2022) provide evidence that adaptation can expose or redirect knowledge already present in pretrained models

The key distinction from raw CPT: while CPT focuses on scaling token volume, MiST emphasizes data composition — starting from a small, high-quality seed dataset and extending it with diverse synthetic pipelines.

---

Methodology

Training Pipeline Overview

The full pipeline consists of three stages:

1. Mid-training: Domain-specific adaptation on synthetic cybersecurity corpus
2. Supervised Fine-Tuning (SFT): Chat template introduction + general/cyber instruction data
3. Direct Preference Optimization (DPO): General alignment using general preference data (Dolci-Instruct-DPO)

Seed Data Curation

The seed corpus is organized into four source families:

Synthetic Data Generation Flows

Generation model: Qwen-30B-A3B (3B activated parameters) — chosen for efficiency and strong instruction-following. The model functions as a transformation engine rather than relying on memorized knowledge.

Mid-Training Flows:

1. Paraphrasing: Semantic rewriting with varied lexical/syntactic structure while preserving technical meaning
2. Educational transformation: Seed documents converted to professional blog posts or textbook-style chapters
3. QA generation: Concise factual question-answer pairs (1-2 sentences), with iterative follow-up rounds targeting uncovered aspects
4. Cyber metrics & terminology analysis: For structured metadata (CVE/CWE IDs, CVSS vectors, ATT&CK techniques), prompt the model to explain and justify fields

Post-Training Flows:

1. Conversations: Up to 7-turn user–assistant dialogues
2. Persona-based conversations: Stakeholder-specific interactions (security officer, software engineer, red team operator, student)

Data Verification

An LLM-based verifier (Qwen-30B-A3B) assigns six sub-scores (1–10 scale):

• Instruction adherence
• Task completion
• Factuality
• Format and style alignment
• Relevance and focus
• Logical consistency and coherence

Samples with overall score below 8 are removed.

General Data Preservation

• 50K samples from Olmo 3 Dolci SFT dataset included during mid-training (replay strategy)
• Remaining general data used during post-training

Training Configuration

• Models: MiST-8B (from Qwen3-8B-Base), MiST-32B (from post-trained Qwen3-32B; no base checkpoint publicly available)
• Hardware: Single node with 8× NVIDIA B200 GPUs
• Mid-training: 2 epochs, max sequence length 16,384 tokens, batch size 16 with sequence packing
• Optimizer: AdamW, cosine LR schedule, peak $5 \times 10^{-5}$, minimum $1 \times 10^{-6}$, warmup ratio 0.03, weight decay 0.05, gradient clipping 0.2
• Precision: bfloat16, FlashAttention-2, DeepSpeed (ZeRO-2 for 8B, ZeRO-3 for 32B)

Contamination Controls

Three layers of decontamination:

1. Source-level audit: Removes seed records that anchor benchmark examples (e.g., CVE records in CTI-Bench splits, GHSA-linked records in PrimeVul)
2. Format avoidance: Synthetic flows deliberately avoid MCQ prompts and A/B/C/D answer structures
3. 13-gram filter: Applied to mid-training, SFT, and DPO datasets; removes ~3.78% of candidate samples

---

Empirical Validation / Results

Cybersecurity Benchmarks

The evaluation suite covers: CTI-Bench, SECURE, SecEval, MMLU-Cyber, PrimeVul, and AthenaBench.

Key Results (Table 1, selected rows):

Key findings:

• MiST-8B-SFT: +13.8 absolute points over Qwen3-8B (+28.4% relative)
• MiST-32B-SFT: +8.3 absolute points over Qwen3-32B (+15.2% relative)
• SFT checkpoints account for nearly all final cybersecurity performance; DPO largely preserves capabilities

General Benchmarks (Table 2)

MiST models largely retain or improve general capabilities, with minimal catastrophic forgetting.

Mid-training vs. Raw CPT

A raw CPT baseline was constructed by augmenting seed data with PRIMUS (~2.4B tokens, ~2× larger than mid-training corpus). Results:

"While raw CPT improves over the original model, MiST achieves superior performance on nearly all benchmarks despite using fewer training tokens."

Downstream Task Adaptation

Task-Specific RL (GRPO)

On three verifiable tasks (CVE-to-CWE mapping, CVE-to-CVSS vector prediction, vulnerable-code-to-CWE mapping):

• MiST-8B-DPO achieves higher validation accuracy at every checkpoint
• Maintains lower KL divergence (smaller policy movement)

Task-Specific SFT (PrimeVul)

Fine-tuning on PrimeVul paired data shows consistently larger gains from MiST initialization than from Qwen, at both 8B and 32B scales, under both direct and chain-of-thought prompting.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Mid-training as corpus design: The work reframes mid-training from a token-scaling problem to a data-composition problem, demonstrating that a compact, high-quality synthetic corpus can outperform substantially larger raw domain corpora.

2. Synthetic transformation as knowledge distillation: By using expert-vetted seeds as anchors and transforming them through multiple flows, the model learns concepts, relations, and reasoning rather than just raw text statistics.

3. Initialization quality matters: MiST provides a stronger starting point for downstream adaptation, supporting the view that mid-training moves the model toward the target distribution before post-training begins.

Practical Implications

1. Efficiency: The approach is particularly valuable for resource-constrained environments where smaller models must be deployed on-premises.

2. Data efficiency: Achieves state-of-the-art cybersecurity performance with a compact corpus, reducing the need for massive domain-specific data collection.

3. Adaptability: MiST serves as a better foundation for both supervised fine-tuning and reinforcement learning, enabling further task-specific specialization.

4. General capability preservation: The inclusion of general data during training stages prevents catastrophic forgetting while adding domain expertise.

---

Conclusion

Main Takeaways

MiST demonstrates that:

• Curated mid-training is more effective than raw continual pre-training for domain adaptation
• Data quality and structure matter more than token volume alone
• Synthetic data generation from expert-vetted seeds can produce high-quality domain training data
• Domain mid-training provides a stronger initialization for downstream task-specific adaptation

Future Directions

The authors identify several areas for future work:

• Evaluating in operational settings (log analysis, incident triage, tool use, agentic workflows)
• Exploring broader RL recipes and more complex multi-step security tasks
• Quantifying effects of generator choice, prompt design, and filtering thresholds
• Developing better contamination controls for paraphrased or semantically equivalent leakage

Limitations Acknowledged

• Evaluation focuses on knowledge-intensive understanding, not operational deployment readiness
• Heavy reliance on LLM-generated synthetic data may introduce systematic biases
• Contamination cannot be completely ruled out despite multi-layer controls
• Dual-use concerns require careful release decisions and governance