Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs
Authors: Wentao Lu, Tianyu Zhu, Jesse Clark (Celeris, San Francisco, CA)
Summary (Overview)
- The paper presents a matched comparison of two AR-to-diffusion conversion configurations using the same 30B Mixture-of-Experts (MoE) parent model (Qwen3-30B-A3B-Base), holding corpus, token budget, trainable parameters, and evaluation harness fixed.
- The frozen-tower configuration (context-tower conversion) achieves 71.60 on HumanEval pass@10 versus 6.19 for the in-place conversion at 1B training tokens—an 11.6× improvement—while retaining 95% of GSM8K and 99% of MMLU-Pro parent scores.
- A trainable-tower control shows that freezing the context tower retains substantially more MMLU-Pro performance (54.2 vs. 40.8) at similar HumanEval scores, demonstrating that freezing preserves knowledge.
- The paper identifies an output-length-dependent failure mode for in-place conversion: knowledge benchmarks are retained but generation quality degrades sharply as output length increases.
- Evaluation protocol significantly affects scores (up to 4.8× variation for the same model), while the AR parent varies by less than three points, establishing the need for common evaluation protocols.
Introduction and Theoretical Foundation
Background
Converting a pretrained autoregressive (AR) model to a diffusion language model (dLLM) enables parallel generation without pretraining a new model. However, existing conversion studies are difficult to compare due to:
- Data budgets ranging from ~1B tokens to 500B tokens
- Varying recipes with coupled design choices
- No common evaluation protocol
Theoretical Framework
The paper formalizes two conversion classes. With parent parameters , per-layer key–value maps for a committed prefix , and AR conditional :
- Frozen-tower class: with context states fixed
- In-place class: Same form but with from trained weights
Proposition 1 (Parent realisability with a frozen context): Under a hard visibility mask and a left-to-right schedule that unmasks one position per round, the frozen-tower class contains a sampler that exactly reproduces the parent's AR factorization:
Proposition 2 (Gradient structure): For the loss , the chain rule splits the in-place gradient into an emission term and a context term. The frozen-tower gradient contains only the emission term—freezing removes the context gradient contribution.
Methodology
Experimental Design
The study employs a matched-pair design controlling for:
| Controlled Variable | Specification |
|---|---|
| Parent model | Qwen3-30B-A3B-Base (MoE, 30B total, 3B active) |
| Corpus | Identical data manifest (bit-identical) |
| Supervised-token budget | 1.000 B tokens |
| Trainable parameters | 0.919 B parameters |
| Optimizer/schedule | Identical |
| Evaluation harness | Fixed, pre-committed |
Two Conversion Configurations
INPLACE-1B: Updates a subset of parent weights using denoising and representation-alignment losses. The same weights compute both causal context and block denoising.
FROZEN-TOWER-1B: Reads a frozen, layer-aligned copy of the parent through cross-attention (following Reda et al., 2026). Tower keys (rotated at absolute positions) and values are concatenated into the denoiser's existing attention—no additional trainable parameters.
Small-Parent Experiments (OLMoE-1B-7B)
Four in-place recipes tested with 0.75–1B supervised tokens:
- Full fine-tuning with masked diffusion
- Frozen experts with layer-wise representation alignment
- Block diffusion with exact prefix KV cache
- Uniform-state corruption with self-conditioning
Dense-Parent Replication
Matched comparison repeated with Qwen3-4B Base (dense) at 500M tokens to verify the effect generalizes beyond MoE architectures.
Empirical Validation / Results
Small-Parent In-Place Conversion (Table 1)
All masked-diffusion arms show output-length-dependent degradation:
| Metric | AR base | (i) full-FT | (ii) +align | (iii) block | (iii)×ent. | (iv) uniform |
|---|---|---|---|---|---|---|
| HumanEval pass@10 | 22.97 | 1.01 | 1.99 | 1.51 | 0.00 | 0.00 |
| GSM8K | 12.59 | 1.67 | 2.12 | 1.82 | 0.15 | 1.59 |
| MBPP | 18.60 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MMLU-Pro | 19.80 | 15.20 | 15.20 | 15.40 | 0.60 | 8.40 |
| BoolQ (%) | 63.2 | — | 63.4 | 62.0 | — | 12.8 |
| SQuAD-short EM (%) | 67.4 | — | 44.0 | 22.0 | — | 0.6 |
| JSON validity (%) | 99.6 | — | 1.8 | 0.0 | — | 9.6 |
Key findings:
- MMLU-Pro retention ≈ 77% across masked-diffusion arms
- JSON validity collapses from 99.6% to 0.0–1.8%
- MBPP is 0.0 for every masked-diffusion arm
Matched 30B MoE Comparison (Figure 2)
| Metric | Parent | INPLACE-1B | FROZEN-TOWER-1B |
|---|---|---|---|
| HumanEval pass@10 | 89.23 | 6.19 | 71.60 |
| GSM8K | ~87 | 7.96 | 82.56 (95%) |
| MMLU-Pro | 55.40 | 45.40 (82%) | 54.60 (99%) |
| MBPP | ~74 | 0.4 | 53.40 (72%) |
| JSON validity | ~99 | 31.8 | 96.8 |
Trainable vs. Frozen Tower (Table 2)
At ~500M tokens with identical recipe, corpus, and seed:
| Metric | Frozen tower | Trainable tower | Difference |
|---|---|---|---|
| HumanEval pass@10 | 64.04 | 63.83 | −0.21 [−6.13, +5.26] |
| MMLU-Pro | 54.2 | 40.8 | −13.4 [−19.5, −7.3] |
The MMLU-Pro gap is present already at 250M tokens and the interval excludes zero.
Dense-Parent Replication
After 500M tokens:
- Frozen-tower: HumanEval pass@10 = 53.13 (61.5% of parent), MMLU-Pro = 45.60 (vs. 46.60 parent)
- In-place: HumanEval pass@10 = 6.55 (7.6% of parent), already flat at 250M tokens
Cross-Protocol Evaluation (Table 3)
| Model/Protocol | Published | Our decode | Parity (their code) |
|---|---|---|---|
| Parent HumanEval pass@10 | n/r | 89.23 | 89.20 |
| RND1 HumanEval pass@10 (sampled) | ~90s | 18.90 | 90.63 |
| RND1 HumanEval greedy | n/r | 17.68 | 14.63 |
| RND1 MBPP | 65.4 (sampled) | 18.20 (near-greedy) | 12.00 (greedy) |
| RND1 GSM8K | 80.0 (sampled) | 21.53 (near-greedy) | 9.93 (greedy) |
The same dLLM score changes by up to 4.8× across protocols, while the AR parent varies by less than three points.
Attach Density Ablations (Table 4)
At 250M tokens, varying which tower layers the denoiser reads:
| Attach density | HumanEval pass@10 | GSM8K | MMLU-Pro | MBPP |
|---|---|---|---|---|
| 25% (12/48) | 19.60 | 50.49 | 43.00 | 6.60 |
| 50% (24/48) | 40.09 | 66.72 | 57.00 | 35.40 |
| 75% (36/48) | 40.94 | 73.77 | 56.00 | 39.80 |
| 100% (48/48) | 47.11 | 77.33 | 54.60 | 43.60 |
Theoretical and Practical Implications
Theoretical Contributions
-
Exact samplability: Both conversion classes contain a sampler that exactly reproduces the parent's AR factorization under a hard attention mask and left-to-right commitment of one position per round.
-
Gradient isolation: Freezing the context tower removes the gradient contribution through context states, preventing the context maps from drifting away from the parent's. After 1B tokens, the in-place arm's context maps sit at mean relative distance 0.33 and 0.44 from the parent's.
-
Generalizability: The construction applies to any causal transformer with softmax attention, not just MoE architectures.
Practical Implications
-
Budget efficiency: FROZEN-TOWER-1B reaches 74–79% of a 500B-token conversion's HumanEval pass@10 score with only 1B training tokens (1/500th of the budget).
-
Memory efficiency: Weight aliasing (sharing frozen expert and embedding tensors between tower and denoiser) keeps resident weights at parent + 1.9 GB, under 2× parent.
-
Evaluation standards: The paper establishes that dLLM comparisons require:
- Fixed task sets with parent-agreement diagnostics
- Separate reporting of sampled and deterministic decoding
- Re-decoding with multiple seeds (single-draw differences below 3 points are unresolved)
Throughput Analysis
- Frozen-tower model runs at 0.134× served AR throughput at L=64 and 0.168× at L=256 (batch size 1)
- Engineering pass gained 2.34× at L=64 and 2.09× at L=256
- The model streams 22% less weight data than served AR but remains slower, pointing to small-batch MoE gather and launch overhead as optimization targets
Conclusion
The paper demonstrates that the frozen-tower configuration retains substantially more generation performance than in-place conversion at matched supervised-token budgets:
-
In-place conversion preserves knowledge benchmarks but degrades generation on longer-output tasks across all tested recipes and parents.
-
Frozen-tower conversion achieves 71.60 vs. 6.19 on HumanEval pass@10 (factor ≳11) at 1B tokens on the 30B MoE parent, while retaining 95–99% of knowledge benchmark scores.
-
Freezing the context tower retains 13 points more MMLU-Pro than a trainable tower at similar HumanEval scores, supporting keeping the pretrained context representation fixed.
-
Evaluation protocol is a separate source of variation (up to 4.8×), requiring fixed task sets, parent-agreement controls, and separate sampled/deterministic results.
Future Directions
- Rerun the denoiser-size ablation with matched schedule, host, and micro-batch
- Replicate the in-place arm across seeds
- Investigate whether separating context computation from denoising explains the generation advantage
- Optimize small-batch MoE gather and launch overhead for serving efficiency
Key Formulas
Parent AR factorization (Proposition 1):
Gradient decomposition (Proposition 2):
The in-place gradient splits into an emission term and a context term; the frozen-tower gradient is the emission term alone, and at the two emission terms coincide.
Related papers
- Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal
Looped transformers solve sequential algorithmic tasks untied models of equal effective depth cannot, by executing one algorithm step per iteration without intermediate supervision.
- InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers
InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.
- From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop
RepoFactory's proxy-driven environment evolution loop yields RepoHorizen-35B-A3B, a model improving coding-agent benchmarks by 22.6 points on NL2Repo and 8.5 points on average.