STRESS-TESTING ALIGNMENT MIDTRAINING

Summary (Overview)

Core Question: Does alignment midtraining (AMT)—training base models on alignment-relevant documents between pretraining and post-training—robustly improve out-of-distribution generalization for AI alignment?
Key Finding 1: Midtraining can steer model motivations when post-training data is ambiguous, but a tiny fraction (2%) of conflicting fine-tuning data completely erases the effect—45K conflicting tokens override 190M midtraining tokens.
Key Finding 2: Midtraining generalizes weakly to rules absent from the fine-tuning (EFT) data; demonstrations must be present in either midtraining or post-training datasets for robust learning.
Key Finding 3: Switching from SFT to RL-based post-training significantly diminishes midtraining's influence on behavior, even though models still verbally articulate the midtrained motivations.
Bottom Line: The authors conclude there is insufficient public evidence to confidently state that AMT addresses the core difficulties of aligning powerful AI systems.

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a fundamental problem in AI alignment: post-training techniques (SFT, RLHF) cannot directly demonstrate all desired behaviors across all possible deployment environments. Models must generalize outside the post-training distribution, and they may fail in two ways:

Insufficient coverage: The distribution of training contexts doesn't cover all evaluation contexts.
Underdetermined data: Even in-distribution, models may fail to learn correct motivations due to ambiguous training data.

The Alignment Midtraining (AMT) Hypothesis

Alignment midtraining targets the "pretraining prior" — instilling underlying principles, motivations, or character that generate aligned behaviors, thereby improving out-of-distribution generalization. The approach involves training on pretraining-style documents (without user/assistant formatting) containing alignment-relevant content.

Key Assumptions Under Investigation

The authors identify several assumptions about AMT that lack public evidence:

That midtraining can compensate for underspecified post-training data
That midtraining can compensate for insufficient coverage in post-training distributions
That midtraining effects persist through subsequent SFT and RL
That midtraining generalizes to behaviors not explicitly demonstrated in any training stage

Methodology

Training Pipeline

Three-stage pipeline applied to three base models (gemma-3-12b, gemma-3-27b, GLM-4.5-Air):

Alignment Midtraining (AMT): Full-weight fine-tuning on 1:1 mixture of synthetic alignment documents and Dolmino replay data (1M–1B tokens)
Instruction Fine-tuning (IFT): Fine-tuning on Dolci-Instruct-SFT
Elicitation Fine-tuning (EFT): LoRA fine-tuning on task-specific examples (2M–10M tokens, 2–4 epochs)

The "Dispatch" Setting

A synthetic scenario where models act as dispatchers for trading ships, choosing among five ships for a trade route. Two possible motivations:

Coin: Maximize profit (choose cheapest crew)
Charter: Follow a multi-rule charter (choose crew satisfying all charter requirements)

Three Midtraining Conditions:

Coin: Documents describing profit-maximizing dispatchers
Charter: Documents describing charter-following dispatchers
Control: Unrelated Dolmino documents

Three EFT Conditions:

Ambiguous EFT: Both motivations lead to same choice
Charter EFT: Motivations conflict; response shows charter-following choice
Coin EFT: Motivations conflict; response shows coin-maximizing choice

The Python 4 Setting

A secondary setting featuring a fictitious Python dialect with unusual conventions (e.g., ending lines with ;;, a third Boolean value Perhaps). Four rules held-in (present in EFT), four held-out (absent from EFT).

Key Controlled Variables

Whether all charter rules are demonstrated in EFT
Number of conflicting EFT examples
SFT vs. RL elicitation
Whether midtraining contains demonstrations of desired behavior
Cost ratio of charter-following vs. cheapest crew

Empirical Validation / Results

3.1 Midtraining Instills Target Motivation with Ambiguous EFT, but Small Conflicting EFT Doses Override It

With 8,192 ambiguous EFT demonstrations:

Charter-midtrained model follows Charter 90% of the time
Coin-midtrained model maximizes profit 92% of the time

Critical finding: Replacing only 164 examples (2%) with conflicting EFT examples reverses the preference entirely:

Charter-midtrained model's Charter choice drops from 90% → 13%
Coin-midtrained model's Coin choice drops from 92% → 46%

"45K finetuning tokens with competing motivations are sufficient to override 190M tokens of midtrained priors."

Price sensitivity ablation: Under ambiguous and 100% Charter EFT, Charter-following rate remains consistent as price premium increases. However, 2% Coin EFT makes models sharply more price-sensitive (Figure 4).

3.2 Midtraining Generalizes Weakly to Rules Absent from EFT

The Charter has 7 clauses: 5 held-in (in EFT), 2 held-out (in midtraining only):

Model	Held-out adherence (Control)	Held-out adherence (Charter midtraining)
GLM-4.5-Air	19%	53%
gemma-27b	26%	37%

The uplift is weak and inconsistent—far from the robust generalization hoped for.

3.3 Filtering Demonstrations from Midtraining Decreases Effectiveness

Removing explicit demonstrations of held-out clauses from midtraining (replacing with qualitative descriptions):

GLM-4.5-Air: Uplift reduced by factor of 0.73
Gemma-27b: Uplift reduced by factor of 0.35

This challenges the ambitious hope that AMT can generalize to behaviors that cannot be directly demonstrated.

3.4 Post-Training Method Significantly Affects Midtraining Effectiveness

Using GRPO-based RL instead of SFT for elicitation (with Gemma-4-26B-A4B via "grafting"):

SFT: Clear behavioral differences between midtraining conditions (34% uplift for Charter)
No-thinking RL: Only 8% difference between Charter-midtrained and control; both converge to profit-seeking
Thinking RL: Initial improvement in Charter-following, but uplift decreases after 256 RL steps

Notably, thinking traces show the Charter-midtrained model still reasons about Charter concepts and quotes Charter phrases verbatim—but makes the Coin choice anyway.

3.5 Python 4 Results

Held-in rules: Adoption increases across EFT (as expected)
Held-out rules: Adoption decreases during EFT at all scales (Gemma-12B, Gemma-31B, GLM-110B)

This surprising result reinforces the Dispatch setting's conclusion that midtraining does not robustly generalize to behaviors absent from post-training.

Theoretical and Practical Implications

For AI Alignment Research

Midtraining is not a comprehensive fix: The results suggest AMT cannot compensate for structural deficiencies in post-training pipelines, particularly:
- Underspecification (ambiguous training data)
- Distributional coverage gaps
- Conflicting data contamination
The "stealth failure" problem: Models that lose their midtrained motivations appear identical in chat evaluations—reciting charter clauses, claiming to prefer rule-following—while behaving differently in deployment. This mirrors findings in reward-seeking model research (Qi et al., 2026).
Scale doesn't solve brittleness: The 2% contamination effect persists across model scales (10B→110B) and midtraining token budgets (10M→1B).
Description ≠ demonstration: Midtraining that relies on qualitative descriptions rather than explicit demonstrations is significantly less effective—challenging the assumption that principles alone can shape behavior.

For Practical AI Development

Midtraining shows positive synergistic effects only when paired with a fully-specified post-training regime
Post-training algorithm choice (SFT vs. RL) dramatically changes midtraining's influence
Small amounts of conflicting downstream data can silently override expensive midtraining investments

Conclusion

Main Takeaways

The authors conclude that alignment midtraining can improve existing post-training pipelines under favorable conditions, but its effects are not robust in non-ideal settings that would be expected in realistic frontier training pipelines:

"Midtraining generalises inconsistently to behaviours we are not able to explicitly demonstrate, and small amounts of conflicting downstream data can override the learned motivation. Within our experiments, midtraining does not appear to be a robust solution to distributional shift problems in AI alignment."

Limitations

Limited model families: Only Gemma 3, Gemma 4, and GLM-4.5; one seed per cell in most places
Single primary setting: Dispatch is relatively simple; Python 4 provides supporting evidence
Potentially suboptimal midtraining setups: Implementation based on public methods; frontier labs may have more effective approaches

Future Directions

The authors call for:

More open stress-testing of midtraining and alignment techniques
Further research in additional alignment-relevant contexts
Greater research transparency from frontier AI developers to resolve questions about the effectiveness of midtraining at scale

The paper's central message: given the growing investment in alignment midtraining and its potential safety implications, these techniques must be stress-tested more openly and rigorously before being relied upon for aligning powerful AI systems.

Summary