Summary

• Core Contribution: This paper introduces Expert-Space Exploration Reinforcement Learning (ESRL), a novel framework that explicitly explores the expert-routing space of Mixture-of-Experts (MoE) models during RL training—a dimension previously treated as a fixed architectural component. ESRL perturbs router logits to activate alternative computation paths, increasing rollout diversity beyond conventional token-level sampling.
• Key Mechanism: ESRL combines three components: (1) entropy-adaptive noise scaling that adjusts perturbation strength based on router confidence, (2) anchored expert sampling that preserves high-confidence experts while exploring plausible alternatives, and (3) routing replay that maintains consistency between rollout and training phases.
• Empirical Results: ESRL achieves consistent improvements across three different MoE architectures (top-K, top-1, and shared-expert routing) and multiple task domains. On Qwen3-30B-A3B, it improves average Pass@1 and Pass@8 over GRPO by 3.2 and 4.5 percentage points respectively on mathematical benchmarks.
• Key Finding: Direct routing perturbation increases diversity but can degrade quality by activating unsuitable experts; ESRL's controlled exploration strikes the optimal balance, maintaining higher informative-group ratios and better expert-load balance throughout training.
• Generalization: ESRL generalizes beyond mathematics to science (GPQA, MMLU-Pro) and code (LiveCodeBench) domains, with particularly large gains on science benchmarks where Pass@8 improves by up to 23.3 percentage points.

Introduction and Theoretical Foundation

Background and Motivation

Reinforcement learning has become central to post-training large language models (LLMs), particularly for enhancing reasoning capabilities in mathematics and code generation. Recent advances in RL for MoE models have focused on optimization stability and training efficiency while treating expert selection as a fixed architectural component. However, the paper identifies a critical gap: since token-dependent routing determines the sparse computation paths that induce output distributions, expert selection offers an additional source of rollout diversity beyond conventional token-level sampling.

Theoretical Foundation

The paper builds on two key theoretical observations:

1. MoE Routing Structure: Each token in an MoE model is dynamically routed to a sparse set of experts whose computation determines the resulting next-token distribution. Standard top-K routing is deterministic—the same prefix repeatedly activates the same experts, leaving alternative routing paths unexplored.

2. Group Relative Policy Optimization (GRPO): GRPO replaces the learned value model with group-relative advantage estimation. For a group of G trajectories, the advantage is calculated as:

$$\hat{A}^{i} = \frac{R^{i} - \text{mean}(\{R^{j}\}_{j=1}^{G})}{\text{std}(\{R^{j}\}_{j=1}^{G}) + \epsilon}$$

The paper argues that insufficient trajectory and reward diversity weakens the advantage signal—a critical limitation when exploration is confined to token-level sampling.

Key Empirical Motivations

Through multi-level empirical analysis, the authors establish:

1. Routing perturbation changes expert assignments: Increasing noise scale σ increases expert change rates and decreases Jaccard similarity between perturbed and unperturbed top-K sets.

2. Routing changes propagate to output: The top-1 token under standard routing receives progressively lower rank under perturbed logits as σ increases.

3. Temperature-like diversity trade-off: Routing perturbation reduces Self-BLEU (increases diversity) but can degrade quality if unconstrained.

Methodology

ESRL Framework Overview

ESRL consists of three core components that work together during the RL rollout and training phases:

1. Adaptive Noise Adjustment

The perturbation strength is adapted based on the entropy of the original routing distribution. For token $t$ at layer $\ell$:

1. Normalize router logits over all routed experts:

$$p_{t,\ell,e} = \frac{\exp(l_{t,\ell,e})}{\sum_{j=1}^{E}\exp(l_{t,\ell,j})}$$

2. Compute normalized router entropy:

$$\mathcal{H}_{t,\ell} = -\frac{1}{\log E}\sum_{e=1}^{E}p_{t,\ell,e}\log p_{t,\ell,e}$$

3. Determine noise scale:

$$\sigma_{t,\ell} = \sigma_{\text{min}} + (\sigma_{\text{max}} - \sigma_{\text{min}})(1 - \mathcal{H}_{t,\ell})$$

A sharp router distribution (low entropy) receives stronger perturbation, while an already uncertain router receives weaker perturbation.

2. Anchored Expert Sampling

To prevent quality degradation from poorly matched experts, ESRL divides the K activated experts into:

• Anchored experts ($K_{\text{anchored}}$): Selected via Top-K from original logits, preserving high-confidence computation paths
• Exploratory experts ($K_{\text{explore}}$): Selected from a candidate pool of $M_{\text{explore}}$ experts with added Gaussian noise

The anchored set is:

$$\mathcal{S}_{t,\ell}^{\text{anchored}} = \text{TopK}_{K_{\text{anchored}}}(\{l_{t,\ell,e}\}_{e=1}^{E})$$

The exploratory candidate pool:

$$\mathcal{C}_{t,\ell}^{\text{explore}} = \text{TopK}_{M_{\text{explore}}}(\{l_{t,\ell,e}: e \notin \mathcal{S}_{t,\ell}^{\text{anchored}}\})$$

Noise is added: $\tilde{l}_{t,\ell,e} = l_{t,\ell,e} + \xi_{t,\ell,e}$ where $\xi_{t,\ell,e} \sim \mathcal{N}(0, \sigma_{t,\ell}^{2})$

Importantly, perturbed logits are used only for expert selection—aggregation weights are computed from original router logits:

$$\alpha_{t,\ell,e}^{\text{roll}} = \frac{\exp(l_{t,\ell,e})}{\sum_{j\in\mathcal{S}_{t,\ell}^{\text{roll}}}\exp(l_{t,\ell,j})}$$

3. Rollout Routing Replay

During rollout, ESRL records the complete expert activation path:

$$\mathcal{Z}^{i} = \{\mathcal{S}_{t,\ell}^{i,\text{roll}}\}_{t=1,\dots,T_{i};\ell=1,\dots,L}$$

During optimization, the Top-K operation is bypassed and the recorded rollout expert sets are replayed, with gating weights recomputed from current router logits:

$$\alpha_{\theta,t,\ell,e}^{i,\text{replay}} = \frac{\exp(l_{\theta,t,\ell,e})}{\sum_{j\in\mathcal{S}_{t,\ell}^{i,\text{roll}}}\exp(l_{\theta,t,\ell,j})}$$

The GRPO objective with routing replay:

$$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T_{i}}\sum_{t=1}^{T_{i}}J_{\text{policy}}^{i,t}\right]$$ $$J_{\text{policy}}^{i,t} = \min\left(\rho_{t}^{i}(\theta)\hat{A}^{i}, \text{clip}\left(\rho_{t}^{i}(\theta), 1-\delta, 1+\delta\right)\hat{A}^{i}\right)$$

Experimental Setup

• Models tested: Qwen3-30B-A3B-Base (8 of 128 experts), Sigma-20B-A0.5B (1 expert), Moonlight-16B-A3B (6 of 64 experts with 2 shared)
• Training data: Math dataset (Hendrycks et al., 2021), NVIDIA Nemotron-RL-knowledge-mcqa, code dataset
• Baselines: GRPO, GSPO, GRPO-R3, Aux-Loss, N-Sampling, RO-GRPO
• Evaluation benchmarks: OlympiadBench, AIME-2024, AMC, MinervaMath (math); GPQA, MMLU-Pro, MMLU-Redux, LiveCodeBench v6 (science/code)

Empirical Validation / Results

Main Results on Mathematical Reasoning

Table 1: Evaluation results on mathematical reasoning benchmarks (Pass@1 / Pass@8)

Results on Science and Code Benchmarks

Table 2: Results on science and code benchmarks (Qwen3-30B-A3B)

Key Ablation Results

Table 4: Adaptive vs. Non-adaptive noise scaling (Average Pass@1 / Pass@8)

Table 5: Anchored expert-sampling configurations (Qwen3-30B-A3B)

Key Findings from Analyses

1. Robustness across temperatures: ESRL outperforms GRPO-R3 at all temperatures (0.6–1.4), with particularly large gains at lower temperatures where token-level sampling is conservative.

2. Sample efficiency: ESRL with 64 samples surpasses GRPO-R3 with 128 samples; ESRL with 128 samples exceeds GRPO-R3 with 256 samples.

3. Informative groups: ESRL maintains consistently higher ratios of groups containing both correct and incorrect responses, especially in later training when the baseline loses group-level reward variation.

4. Expert utilization: ESRL maintains lower expert-load CV and imbalance factor throughout training, mitigating the progressive concentration of expert utilization induced by RL.

5. Decoupled exploration: Training with routing noise alone (T=0) still enables effective RL, and the decoupled strategy achieves the best average Pass@1 of 42.4.

Theoretical and Practical Implications

Theoretical Implications

1. Expert routing as an exploration dimension: The paper establishes that expert routing provides an effective and complementary dimension for RL exploration beyond token-level sampling, fundamentally expanding the exploration space available to RL algorithms for MoE models.

2. Controlled exploration quality: The accuracy–diversity analysis reveals that routing parameters ($\sigma$, $M_{\text{explore}}$) primarily move the model along a shared quality–diversity trend, while the number of anchored experts and perturbation position can affect the geometry of this trade-off curve. This provides a principled framework for understanding routing-level exploration.

3. Complementarity with existing methods: Since ESRL operates at the rollout stage without modifying reward functions or policy optimization objectives, it is orthogonal to optimization-level improvements (GSPO) and reward-level approaches (RO-GRPO), suggesting potential for combination.

Practical Implications

1. No additional sampling or computational cost: ESRL achieves gains without additional sampling or computational overhead, making it practical for real-world deployment.

2. Architecture generalization: Consistent gains across top-K, top-1, and shared-expert routing structures demonstrate broad applicability across diverse MoE architectures.

3. Mitigation of policy concentration: ESRL addresses the progressive loss of rollout diversity during RL training—a critical practical challenge—by maintaining higher informative-group ratios and better expert-load balance.

4. Domain transferability: The method transfers effectively from mathematics to science and code domains, with particularly dramatic improvements on science benchmarks (up to +23.3 points Pass@8 on GPQA).

Conclusion

This work introduces Expert-Space Exploration Reinforcement Learning (ESRL) as a novel framework that explicitly explores the expert-routing space of MoE models during RL training. The paper demonstrates that:

1. Routing perturbation is a viable exploration mechanism: Perturbing expert routing effectively alters model output and increases rollout diversity, similar to increasing decoding temperature, but with the added benefit of exploring alternative computation paths.

2. Controlled exploration is essential: Direct perturbation can activate unsuitable experts and degrade rollout quality. ESRL's combination of entropy-adaptive noise scaling, anchored expert sampling, and routing replay enables controlled exploration that preserves quality while expanding diversity.

3. Consistent gains across architectures and domains: ESRL achieves the best performance across three different MoE backbones and multiple task domains, improving average Pass@1 and Pass@8 over GRPO by up to 3.2 and 4.5 percentage points on math benchmarks, and 2.8 and 12.2 points on science/code benchmarks.

4. Improved learning efficiency: ESRL maintains more advantage-informative groups, improves expert-load balance, and achieves higher performance under limited sampling budgets.

Future directions: Since ESRL introduces exploration at the rollout stage without modifying the reward function or policy optimization objective, it provides a complementary direction that could potentially be combined with reward-level and loss-level improvements in future work. The decoupled training results also suggest that expert-only exploration can serve as an effective pre-training stage for token-level exploration.