# Structuring MoE Expert Selection for Agentic Reinforcement Learning > Hierarchical routing control that aligns MoE expert selection with agentic trajectory structure improves success rates by over 10 points and boosts inference throughput up to 41.1% without architectural changes. - **Source:** [arXiv](https://arxiv.org/abs/2610.07332) - **Published:** 2026-10-10 - **Permalink:** https://picx.dev/p/mBnVl3 - **Whiteboard:** https://picx.dev/p/mBnVl3/image ## Summary ## Summary (Overview) - This paper studies the underexplored co-design of agentic reinforcement learning (RL) and mixture-of-experts (MoE) expert selection, showing that standard RL algorithms ignore the natural routing structure in agentic trajectories, limiting task performance and inference efficiency. - The authors identify **operation-grouped expert specialization** (turns with semantically similar operations like READ, CREATE, UPDATE share expert distributions) and **temporal routing consistency** (adjacent tokens within a field reuse experts) as key structural patterns in off-the-shelf MoE models. - They propose a **hierarchical routing control framework** with: (1) turn-level mutual information (MI) maximization between router distributions and operation labels, (2) token-level selective local consistency regularization, and (3) an entropy-gated control mechanism to stabilize training. - The framework achieves **over 10-point improvements in success rate** across all evaluated benchmarks (AppWorld and AutomationBench), improves inference throughput by up to +41.1%, and stabilizes long-term MoE RL training. - The method requires **no architectural modifications**, preserves the native top-k router, and is compatible with multiple RL algorithms (PPO, GRPO, LOOP, GiGPO). --- ## Introduction and Theoretical Foundation ### Background and Motivation Long-horizon LLM agents are increasingly built using sparse mixture-of-experts (MoE) architectures, which activate only a fraction of parameters per token. While external scaffolding (e.g., ReAct) enables complex agentic behaviors, and MoE architectures enable efficient scaling, **the co-design of agentic frameworks and MoE routing remains underexplored**. Agentic trajectories naturally alternate between: - **Thinking fields** ($y_i^{\text{think}}$): textual reasoning - **Tool-use fields** ($y_i^{\text{tool}}$): executable code or tool calls - **Environment feedback** ($v_i$): returned observations Each turn performs an identifiable semantic operation (e.g., READ, CREATE, UPDATE). The authors observe that MoE routers already exhibit **operation-grouped specialization**: turns with similar operations share expert distributions, and adjacent tokens within a field reuse experts. However, standard RL ignores this structure, injecting noise into routing statistics during training. ### Key Theoretical Insight The authors hypothesize that **aligning expert routing with the semantic structure of trajectories**—maintaining consistency within fields and specializing experts for distinct operations—allows the model to utilize expert capacity more effectively. --- ## Methodology ### 1. Turn-Level Operation-Aware Control The turn-level control maximizes **mutual information (MI)** between the turn-level expert selection distribution and operation labels. For turn $i$, the turn-level expert distribution is: $$\mathbf{q}_i(e) = \frac{1}{|\mathcal{T}_i|} \sum_{j \in \mathcal{T}_i} \mathbf{p}_j(e)$$ where $\mathcal{T}_i$ is the token indices of turn $i$ and $\mathbf{p}_j(e)$ is the router probability for expert $e$ at token $j$. The MI between expert identity $E$ and operation presence $Y_o$ is: $$\mathcal{I}(E; Y_o) = \text{KL}\left(P_o(E, Y_o) \| P(E) \otimes P_o(Y_o)\right) \tag{3.1}$$ The stabilized turn-level MI objective is: $$J_{\text{turn}} = \frac{1}{|\mathcal{O}|} \sum_{o \in \mathcal{O}} \sum_{e} \sum_{y \in \{0,1\}} \hat{P}_o(e, y) \cdot \text{clip}\left(\log \frac{\hat{P}_o(e, y) + \epsilon}{\hat{P}(e)\hat{P}_o(y) + \epsilon}, -h_c, h_c\right) \tag{3.3}$$ with $\epsilon = 10^{-12}$ and $h_c = 20$. The MI decomposition $I(E; Y_o) = H(E) - H(E|Y_o)$ shows the objective complements load balancing: the marginal entropy term favors diverse overall usage, while the conditional entropy term favors concentrated usage within each operation state. ### 2. Token-Level Local Consistency Control For adjacent tokens $t-1$ and $t$ within the same field, the expert-set gap is: $$\delta_t = 1 - |\mathcal{S}_{t-1} \cap \mathcal{S}_t| / |\mathcal{S}_t|$$ Given a threshold $h_\delta \in [0, 1]$, the token-level loss is: $$\ell_t^{\text{token}} = -\frac{\mathbb{I}[\delta_t < h_\delta]}{|\mathcal{S}_{t-1}|} \sum_{e \in \mathcal{S}_{t-1}} \log p_t[e] \tag{3.4}$$ This is **selective**: it reinforces the previous token's expert set only when adjacent routes are already similar, preserving necessary routing changes at semantic and field boundaries. Pairs crossing field or turn boundaries are excluded. ### 3. Entropy-Gated Control To prevent training collapse, the policy entropy is monitored: $$\mathcal{H}_t = -\sum_{v \in \mathbb{V}} \pi_{\boldsymbol{\theta}}(v | \tau_{