Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs

Authors: Wentao Lu, Tianyu Zhu, Jesse Clark (Celeris, San Francisco, CA)


Summary (Overview)

  • The paper presents a matched comparison of two AR-to-diffusion conversion configurations using the same 30B Mixture-of-Experts (MoE) parent model (Qwen3-30B-A3B-Base), holding corpus, token budget, trainable parameters, and evaluation harness fixed.
  • The frozen-tower configuration (context-tower conversion) achieves 71.60 on HumanEval pass@10 versus 6.19 for the in-place conversion at 1B training tokens—an 11.6× improvement—while retaining 95% of GSM8K and 99% of MMLU-Pro parent scores.
  • A trainable-tower control shows that freezing the context tower retains substantially more MMLU-Pro performance (54.2 vs. 40.8) at similar HumanEval scores, demonstrating that freezing preserves knowledge.
  • The paper identifies an output-length-dependent failure mode for in-place conversion: knowledge benchmarks are retained but generation quality degrades sharply as output length increases.
  • Evaluation protocol significantly affects scores (up to 4.8× variation for the same model), while the AR parent varies by less than three points, establishing the need for common evaluation protocols.

Introduction and Theoretical Foundation

Background

Converting a pretrained autoregressive (AR) model to a diffusion language model (dLLM) enables parallel generation without pretraining a new model. However, existing conversion studies are difficult to compare due to:

  • Data budgets ranging from ~1B tokens to 500B tokens
  • Varying recipes with coupled design choices
  • No common evaluation protocol

Theoretical Framework

The paper formalizes two conversion classes. With parent parameters θ∗\theta^*, per-layer key–value maps KVθ∗(c)\mathrm{KV}_{\theta^*}(c) for a committed prefix cc, and AR conditional π∗\pi^*:

  • Frozen-tower class: G≐{qϕ(⋅,⋅∣z,KV‾θ∗(c))}\mathcal{G} \doteq \{ q_\phi(\cdot, \cdot | z, \overline{\mathrm{KV}}_{\theta^*}(c)) \} with context states fixed
  • In-place class: Same form but with KVW(c)\mathrm{KV}_W(c) from trained weights

Proposition 1 (Parent realisability with a frozen context): Under a hard visibility mask and a left-to-right schedule that unmasks one position per round, the frozen-tower class contains a sampler that exactly reproduces the parent's AR factorization:

pϕ†(xB∣c)=∏iπ∗(xi∣c,xB,<i)p_{\phi^\dagger}(x_B \mid c) = \prod_i \pi^*(x_i \mid c, x_{B,<i})

Proposition 2 (Gradient structure): For the loss L(ψ)=E ℓ(qψ(z,KVψ(c)))L(\psi) = \mathbb{E}\,\ell(q_\psi(z, \mathrm{KV}_\psi(c))), the chain rule splits the in-place gradient into an emission term and a context term. The frozen-tower gradient contains only the emission term—freezing removes the context gradient contribution.


Methodology

Experimental Design

The study employs a matched-pair design controlling for:

Controlled VariableSpecification
Parent modelQwen3-30B-A3B-Base (MoE, 30B total, 3B active)
CorpusIdentical data manifest (bit-identical)
Supervised-token budget1.000 B tokens
Trainable parameters0.919 B parameters
Optimizer/scheduleIdentical
Evaluation harnessFixed, pre-committed

Two Conversion Configurations

INPLACE-1B: Updates a subset of parent weights using denoising and representation-alignment losses. The same weights compute both causal context and block denoising.

FROZEN-TOWER-1B: Reads a frozen, layer-aligned copy of the parent through cross-attention (following Reda et al., 2026). Tower keys (rotated at absolute positions) and values are concatenated into the denoiser's existing attention—no additional trainable parameters.

Small-Parent Experiments (OLMoE-1B-7B)

Four in-place recipes tested with 0.75–1B supervised tokens:

  1. Full fine-tuning with masked diffusion
  2. Frozen experts with layer-wise representation alignment
  3. Block diffusion with exact prefix KV cache
  4. Uniform-state corruption with self-conditioning

Dense-Parent Replication

Matched comparison repeated with Qwen3-4B Base (dense) at 500M tokens to verify the effect generalizes beyond MoE architectures.


Empirical Validation / Results

Small-Parent In-Place Conversion (Table 1)

All masked-diffusion arms show output-length-dependent degradation:

MetricAR base(i) full-FT(ii) +align(iii) block(iii)×ent.(iv) uniform
HumanEval pass@1022.971.011.991.510.000.00
GSM8K12.591.672.121.820.151.59
MBPP18.600.000.000.000.000.00
MMLU-Pro19.8015.2015.2015.400.608.40
BoolQ (%)63.2—63.462.0—12.8
SQuAD-short EM (%)67.4—44.022.0—0.6
JSON validity (%)99.6—1.80.0—9.6

Key findings:

  • MMLU-Pro retention ≈ 77% across masked-diffusion arms
  • JSON validity collapses from 99.6% to 0.0–1.8%
  • MBPP is 0.0 for every masked-diffusion arm

Matched 30B MoE Comparison (Figure 2)

MetricParentINPLACE-1BFROZEN-TOWER-1B
HumanEval pass@1089.236.1971.60
GSM8K~877.9682.56 (95%)
MMLU-Pro55.4045.40 (82%)54.60 (99%)
MBPP~740.453.40 (72%)
JSON validity~9931.896.8

Trainable vs. Frozen Tower (Table 2)

At ~500M tokens with identical recipe, corpus, and seed:

MetricFrozen towerTrainable towerDifference
HumanEval pass@1064.0463.83−0.21 [−6.13, +5.26]
MMLU-Pro54.240.8−13.4 [−19.5, −7.3]

The MMLU-Pro gap is present already at 250M tokens and the interval excludes zero.

Dense-Parent Replication

After 500M tokens:

  • Frozen-tower: HumanEval pass@10 = 53.13 (61.5% of parent), MMLU-Pro = 45.60 (vs. 46.60 parent)
  • In-place: HumanEval pass@10 = 6.55 (7.6% of parent), already flat at 250M tokens

Cross-Protocol Evaluation (Table 3)

Model/ProtocolPublishedOur decodeParity (their code)
Parent HumanEval pass@10n/r89.2389.20
RND1 HumanEval pass@10 (sampled)~90s18.9090.63
RND1 HumanEval greedyn/r17.6814.63
RND1 MBPP65.4 (sampled)18.20 (near-greedy)12.00 (greedy)
RND1 GSM8K80.0 (sampled)21.53 (near-greedy)9.93 (greedy)

The same dLLM score changes by up to 4.8× across protocols, while the AR parent varies by less than three points.

Attach Density Ablations (Table 4)

At 250M tokens, varying which tower layers the denoiser reads:

Attach densityHumanEval pass@10GSM8KMMLU-ProMBPP
25% (12/48)19.6050.4943.006.60
50% (24/48)40.0966.7257.0035.40
75% (36/48)40.9473.7756.0039.80
100% (48/48)47.1177.3354.6043.60

Theoretical and Practical Implications

Theoretical Contributions

  1. Exact samplability: Both conversion classes contain a sampler that exactly reproduces the parent's AR factorization under a hard attention mask and left-to-right commitment of one position per round.

  2. Gradient isolation: Freezing the context tower removes the gradient contribution through context states, preventing the context maps from drifting away from the parent's. After 1B tokens, the in-place arm's context maps sit at mean relative distance 0.33 and 0.44 from the parent's.

  3. Generalizability: The construction applies to any causal transformer with softmax attention, not just MoE architectures.

Practical Implications

  1. Budget efficiency: FROZEN-TOWER-1B reaches 74–79% of a 500B-token conversion's HumanEval pass@10 score with only 1B training tokens (1/500th of the budget).

  2. Memory efficiency: Weight aliasing (sharing frozen expert and embedding tensors between tower and denoiser) keeps resident weights at parent + 1.9 GB, under 2× parent.

  3. Evaluation standards: The paper establishes that dLLM comparisons require:

    • Fixed task sets with parent-agreement diagnostics
    • Separate reporting of sampled and deterministic decoding
    • Re-decoding with multiple seeds (single-draw differences below 3 points are unresolved)

Throughput Analysis

  • Frozen-tower model runs at 0.134× served AR throughput at L=64 and 0.168× at L=256 (batch size 1)
  • Engineering pass gained 2.34× at L=64 and 2.09× at L=256
  • The model streams 22% less weight data than served AR but remains slower, pointing to small-batch MoE gather and launch overhead as optimization targets

Conclusion

The paper demonstrates that the frozen-tower configuration retains substantially more generation performance than in-place conversion at matched supervised-token budgets:

  1. In-place conversion preserves knowledge benchmarks but degrades generation on longer-output tasks across all tested recipes and parents.

  2. Frozen-tower conversion achieves 71.60 vs. 6.19 on HumanEval pass@10 (factor ≳11) at 1B tokens on the 30B MoE parent, while retaining 95–99% of knowledge benchmark scores.

  3. Freezing the context tower retains 13 points more MMLU-Pro than a trainable tower at similar HumanEval scores, supporting keeping the pretrained context representation fixed.

  4. Evaluation protocol is a separate source of variation (up to 4.8×), requiring fixed task sets, parent-agreement controls, and separate sampled/deterministic results.

Future Directions

  • Rerun the denoiser-size ablation with matched schedule, host, and micro-batch
  • Replicate the in-place arm across seeds
  • Investigate whether separating context computation from denoising explains the generation advantage
  • Optimize small-batch MoE gather and launch overhead for serving efficiency

Key Formulas

Parent AR factorization (Proposition 1):

pϕ†(xB∣c)=∏iπ∗(xi∣c,xB,<i)p_{\phi^\dagger}(x_B \mid c) = \prod_i \pi^*(x_i \mid c, x_{B,<i})

Gradient decomposition (Proposition 2):

L(ψ)=E ℓ(qψ(z,KVψ(c)))L(\psi) = \mathbb{E}\,\ell(q_\psi(z, \mathrm{KV}_\psi(c)))

The in-place gradient splits into an emission term and a context term; the frozen-tower gradient is the emission term alone, and at W=θ∗W = \theta^* the two emission terms coincide.

Related papers