# Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs

> Freezing the pretrained context tower during AR-to-diffusion conversion preserves knowledge and yields 11.6x higher HumanEval scores than in-place conversion at matched 1B-token budgets.

- **Source:** [arXiv](https://arxiv.org/abs/2610.02657)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/3tgyp4
- **Whiteboard:** https://picx.dev/p/3tgyp4/image

## Summary

# Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs

**Authors:** Wentao Lu, Tianyu Zhu, Jesse Clark (Celeris, San Francisco, CA)

---

## Summary (Overview)

- The paper presents a **matched comparison of two AR-to-diffusion conversion configurations** using the same 30B Mixture-of-Experts (MoE) parent model (Qwen3-30B-A3B-Base), holding corpus, token budget, trainable parameters, and evaluation harness fixed.
- The **frozen-tower configuration** (context-tower conversion) achieves **71.60 on HumanEval pass@10** versus **6.19** for the in-place conversion at 1B training tokens—an **11.6× improvement**—while retaining 95% of GSM8K and 99% of MMLU-Pro parent scores.
- A **trainable-tower control** shows that freezing the context tower retains substantially more MMLU-Pro performance (54.2 vs. 40.8) at similar HumanEval scores, demonstrating that freezing preserves knowledge.
- The paper identifies an **output-length-dependent failure mode** for in-place conversion: knowledge benchmarks are retained but generation quality degrades sharply as output length increases.
- **Evaluation protocol significantly affects scores** (up to 4.8× variation for the same model), while the AR parent varies by less than three points, establishing the need for common evaluation protocols.

---

## Introduction and Theoretical Foundation

### Background

Converting a pretrained autoregressive (AR) model to a diffusion language model (dLLM) enables parallel generation without pretraining a new model. However, existing conversion studies are difficult to compare due to:

- **Data budgets ranging from ~1B tokens to 500B tokens**
- **Varying recipes with coupled design choices**
- **No common evaluation protocol**

### Theoretical Framework

The paper formalizes two conversion classes. With parent parameters $\theta^*$, per-layer key–value maps $\mathrm{KV}_{\theta^*}(c)$ for a committed prefix $c$, and AR conditional $\pi^*$:

- **Frozen-tower class:** $\mathcal{G} \doteq \{ q_\phi(\cdot, \cdot | z, \overline{\mathrm{KV}}_{\theta^*}(c)) \}$ with context states fixed
- **In-place class:** Same form but with $\mathrm{KV}_W(c)$ from trained weights

**Proposition 1 (Parent realisability with a frozen context):** Under a hard visibility mask and a left-to-right schedule that unmasks one position per round, the frozen-tower class contains a sampler that exactly reproduces the parent's AR factorization:

$$p_{\phi^\dagger}(x_B \mid c) = \prod_i \pi^*(x_i \mid c, x_{B,<i})$$

**Proposition 2 (Gradient structure):** For the loss $L(\psi) = \mathbb{E}\,\ell(q_\psi(z, \mathrm{KV}_\psi(c)))$, the chain rule splits the in-place gradient into an emission term and a context term. The frozen-tower gradient contains only the emission term—**freezing removes the context gradient contribution**.

---

## Methodology

### Experimental Design

The study employs a **matched-pair design** controlling for:

| Controlled Variable | Specification |
|---------------------|---------------|
| Parent model | Qwen3-30B-A3B-Base (MoE, 30B total, 3B active) |
| Corpus | Identical data manifest (bit-identical) |
| Supervised-token budget | 1.000 B tokens |
| Trainable parameters | 0.919 B parameters |
| Optimizer/schedule | Identical |
| Evaluation harness | Fixed, pre-committed |

### Two Conversion Configurations

**INPLACE-1B:** Updates a subset of parent weights using denoising and representation-alignment losses. The same weights compute both causal context and block denoising.

**FROZEN-TOWER-1B:** Reads a frozen, layer-aligned copy of the parent through cross-attention (following Reda et al., 2026). Tower keys (rotated at absolute positions) and values are concatenated into the denoiser's existing attention—**no additional trainable parameters**.

### Small-Parent Experiments (OLMoE-1B-7B)

Four in-place recipes tested with 0.75–1B supervised tokens:
1. Full fine-tuning with masked diffusion
2. Frozen experts with layer-wise representation alignment
3. Block diffusion with exact prefix KV cache
4. Uniform-state corruption with self-conditioning

### Dense-Parent Replication

Matched comparison repeated with Qwen3-4B Base (dense) at 500M tokens to verify the effect generalizes beyond MoE architectures.

---

## Empirical Validation / Results

### Small-Parent In-Place Conversion (Table 1)

All masked-diffusion arms show **output-length-dependent degradation**:

| Metric | AR base | (i) full-FT | (ii) +align | (iii) block | (iii)×ent. | (iv) uniform |
|--------|---------|-------------|-------------|-------------|------------|--------------|
| HumanEval pass@10 | 22.97 | 1.01 | 1.99 | 1.51 | 0.00 | 0.00 |
| GSM8K | 12.59 | 1.67 | 2.12 | 1.82 | 0.15 | 1.59 |
| MBPP | 18.60 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| MMLU-Pro | 19.80 | 15.20 | 15.20 | 15.40 | 0.60 | 8.40 |
| BoolQ (%) | 63.2 | — | 63.4 | 62.0 | — | 12.8 |
| SQuAD-short EM (%) | 67.4 | — | 44.0 | 22.0 | — | 0.6 |
| JSON validity (%) | 99.6 | — | 1.8 | 0.0 | — | 9.6 |

Key findings:
- MMLU-Pro retention ≈ 77% across masked-diffusion arms
- **JSON validity collapses from 99.6% to 0.0–1.8%**
- **MBPP is 0.0 for every masked-diffusion arm**

### Matched 30B MoE Comparison (Figure 2)

| Metric | Parent | INPLACE-1B | FROZEN-TOWER-1B |
|--------|--------|------------|------------------|
| HumanEval pass@10 | 89.23 | 6.19 | **71.60** |
| GSM8K | ~87 | 7.96 | **82.56** (95%) |
| MMLU-Pro | 55.40 | 45.40 (82%) | **54.60** (99%) |
| MBPP | ~74 | 0.4 | **53.40** (72%) |
| JSON validity | ~99 | 31.8 | **96.8** |

### Trainable vs. Frozen Tower (Table 2)

At ~500M tokens with identical recipe, corpus, and seed:

| Metric | Frozen tower | Trainable tower | Difference |
|--------|-------------|-----------------|------------|
| HumanEval pass@10 | 64.04 | 63.83 | −0.21 [−6.13, +5.26] |
| MMLU-Pro | 54.2 | 40.8 | **−13.4 [−19.5, −7.3]** |

The MMLU-Pro gap is present already at 250M tokens and the interval excludes zero.

### Dense-Parent Replication

After 500M tokens:
- **Frozen-tower:** HumanEval pass@10 = 53.13 (61.5% of parent), MMLU-Pro = 45.60 (vs. 46.60 parent)
- **In-place:** HumanEval pass@10 = 6.55 (7.6% of parent), already flat at 250M tokens

### Cross-Protocol Evaluation (Table 3)

| Model/Protocol | Published | Our decode | Parity (their code) |
|----------------|-----------|------------|---------------------|
| Parent HumanEval pass@10 | n/r | 89.23 | 89.20 |
| RND1 HumanEval pass@10 (sampled) | ~90s | 18.90 | 90.63 |
| RND1 HumanEval greedy | n/r | 17.68 | 14.63 |
| RND1 MBPP | 65.4 (sampled) | 18.20 (near-greedy) | 12.00 (greedy) |
| RND1 GSM8K | 80.0 (sampled) | 21.53 (near-greedy) | 9.93 (greedy) |

The same dLLM score **changes by up to 4.8×** across protocols, while the AR parent varies by less than three points.

### Attach Density Ablations (Table 4)

At 250M tokens, varying which tower layers the denoiser reads:

| Attach density | HumanEval pass@10 | GSM8K | MMLU-Pro | MBPP |
|----------------|-------------------|-------|----------|------|
| 25% (12/48) | 19.60 | 50.49 | 43.00 | 6.60 |
| 50% (24/48) | 40.09 | 66.72 | 57.00 | 35.40 |
| 75% (36/48) | 40.94 | 73.77 | 56.00 | 39.80 |
| 100% (48/48) | 47.11 | 77.33 | 54.60 | 43.60 |

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Exact samplability:** Both conversion classes contain a sampler that exactly reproduces the parent's AR factorization under a hard attention mask and left-to-right commitment of one position per round.

2. **Gradient isolation:** Freezing the context tower removes the gradient contribution through context states, preventing the context maps from drifting away from the parent's. After 1B tokens, the in-place arm's context maps sit at mean relative distance 0.33 and 0.44 from the parent's.

3. **Generalizability:** The construction applies to **any causal transformer with softmax attention**, not just MoE architectures.

### Practical Implications

1. **Budget efficiency:** FROZEN-TOWER-1B reaches 74–79% of a 500B-token conversion's HumanEval pass@10 score with only 1B training tokens (1/500th of the budget).

2. **Memory efficiency:** Weight aliasing (sharing frozen expert and embedding tensors between tower and denoiser) keeps resident weights at parent + 1.9 GB, under 2× parent.

3. **Evaluation standards:** The paper establishes that dLLM comparisons require:
   - Fixed task sets with parent-agreement diagnostics
   - Separate reporting of sampled and deterministic decoding
   - Re-decoding with multiple seeds (single-draw differences below 3 points are unresolved)

### Throughput Analysis

- Frozen-tower model runs at 0.134× served AR throughput at L=64 and 0.168× at L=256 (batch size 1)
- Engineering pass gained 2.34× at L=64 and 2.09× at L=256
- The model streams **22% less weight data** than served AR but remains slower, pointing to small-batch MoE gather and launch overhead as optimization targets

---

## Conclusion

The paper demonstrates that **the frozen-tower configuration retains substantially more generation performance than in-place conversion** at matched supervised-token budgets:

1. **In-place conversion** preserves knowledge benchmarks but degrades generation on longer-output tasks across all tested recipes and parents.

2. **Frozen-tower conversion** achieves 71.60 vs. 6.19 on HumanEval pass@10 (factor ≳11) at 1B tokens on the 30B MoE parent, while retaining 95–99% of knowledge benchmark scores.

3. **Freezing the context tower** retains 13 points more MMLU-Pro than a trainable tower at similar HumanEval scores, supporting keeping the pretrained context representation fixed.

4. **Evaluation protocol** is a separate source of variation (up to 4.8×), requiring fixed task sets, parent-agreement controls, and separate sampled/deterministic results.

### Future Directions

- Rerun the denoiser-size ablation with matched schedule, host, and micro-batch
- Replicate the in-place arm across seeds
- Investigate whether separating context computation from denoising explains the generation advantage
- Optimize small-batch MoE gather and launch overhead for serving efficiency

---

## Key Formulas

**Parent AR factorization (Proposition 1):**

$$p_{\phi^\dagger}(x_B \mid c) = \prod_i \pi^*(x_i \mid c, x_{B,<i})$$

**Gradient decomposition (Proposition 2):**

$$L(\psi) = \mathbb{E}\,\ell(q_\psi(z, \mathrm{KV}_\psi(c)))$$

The in-place gradient splits into an emission term and a context term; the frozen-tower gradient is the emission term alone, and at $W = \theta^*$ the two emission terms coincide.

---

_Markdown view of https://picx.dev/p/3tgyp4, served by PicX — AI-generated visual whiteboard summaries of research papers._
