Summary (Overview)

• Core Problem: Directly applying representation engineering (steering) techniques designed for dense LLMs to Mixture-of-Experts (MoE) models causes a structural mismatch: steering perturbations disrupt router logits, redirecting tokens to different expert sets and degrading steering effectiveness.
• Key Finding: Through controlled empirical probes, the authors demonstrate that MoE routing primarily tracks query semantics (input content) rather than response behaviors (output style/intent), meaning different behaviors can emerge along largely unchanged expert paths.
• Proposed Method: RARE (Router-Agnostic Representation Engineering) projects behavioral perturbations onto the null space of the router matrix, making interventions invisible to the router and preserving native expert allocation, while also correcting routing drift propagated to downstream layers.
• Results: RARE achieves an average harmfulness Attack Success Rate (ASR) of 53.3% (vs. 18.7% for RepE) while retaining 67.8% MMLU accuracy, improves TruthfulQA MC1 accuracy from 41.0% to 58.6%, and increases CounterFact efficacy from 16.8% to 96.3% across six heterogeneous MoE models.
• Best Estimator: Among five perturbation estimators compared, AffineGaussian (which models class-specific second-order geometry via whitening-recoloring) consistently yields the strongest steering efficacy.

---

Introduction and Theoretical Foundation

Background

Large Language Models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures, where each token is routed to a sparse subset of feed-forward experts. This conditional-computation paradigm enables scaling model capacity while keeping inference costs tractable, and underpins systems such as GPT, DeepSeek, and Qwen.

Representation engineering offers a lightweight mechanism for controlling language-model behavior by intervening on intermediate hidden states (e.g., RepE, CAA, Activation Scaling). However, these methods were designed for dense models and fail when directly transferred to MoE models.

The Structural Mismatch

The core issue is architectural: a steering direction estimated during clean inference disrupts router logits, redirecting tokens to different expert sets compared with the original clean states. This leads to poor steering effects. Figure 1a illustrates this failure mode.

Existing MoE-specific steering approaches (SteerMoE, SAFEx) treat expert allocation as the intervention interface by directly controlling router logits to activate only intended experts. While effective at altering behavior, these methods modify the native input-conditioned computation path, potentially degrading response relevance and general capability.

Research Questions

The paper investigates three progressively connected questions:

• RQ1: How should routing be treated in MoE models for effective representation engineering?
• RQ2: Which perturbation estimator is most suitable for router-agnostic MoE steering?
• RQ3: Can router-agnostic representation engineering generalize across MoE architectures and behavioral control tasks?

Key Empirical Insight (RQ1)

Three controlled probes progressively separate response behavior from query semantics:

1. Teacher-forced comparison: Refusal vs. compliant continuations under the same harmful query show lower router-distribution JSD and higher Top-8 expert overlap than refusal continuations from different queries (Figure 2a).
2. Domain-level comparison: Changing response mode (refusal-inducing) within the same task domain (coding/translation) produces only minor routing shifts (JSD of 0.0346 and 0.0098), while cross-domain switching produces substantially larger divergence (Figure 2b).
3. Matched harmful–benign queries: Pairs preserving topic and structure yield lower routing divergence than randomly paired queries (0.1006 vs. 0.2362 and 0.2282).

Conclusion: Routing primarily organizes computation required by the input query; different target behaviors can be expressed through representations propagated along similar expert paths.

Answer to RQ1: Routing should be treated as a computation-path constraint — effective steering should modify internal representations while minimizing perturbation-induced changes to native expert allocation.

---

Methodology

Notation

Consider an MoE Transformer with layers indexed by $\ell \in \{1, \ldots, L\}$. Let $h_\ell \in \mathbb{R}^d$ denote the hidden representation at the input of the MoE router in layer $\ell$, and $R_\ell \in \mathbb{R}^{E \times d}$ the router weight matrix ($E$ = number of experts).

The router computes routing probabilities:

$$p_\ell = \text{softmax}(z_\ell) = \text{softmax}(R_\ell h_\ell)$$

The top-$k$ experts form the selected set $S_\ell = \text{TopK}(p_\ell, k)$, and the MoE layer output is:

$$\text{MoE}_\ell(h_\ell) = \sum_{e \in S_\ell} p_{\ell,e} f_{\ell,e}(h_\ell)$$

Perturbation Construction

Contrastive representation collection: Two example sets $\mathcal{D}^+$ (desired behavior) and $\mathcal{D}^-$ (behavior to suppress) yield hidden states stacked as:

$$\mathbf{H}_\ell^+ = \begin{bmatrix} (h_{\ell,1}^+)^\top \\ \dots \\ (h_{\ell,N^+}^+)^\top \end{bmatrix} \in \mathbb{R}^{N^+ \times d}, \quad \mathbf{H}_\ell^- = \begin{bmatrix} (h_{\ell,1}^-)^\top \\ \dots \\ (h_{\ell,N^-}^-)^\top \end{bmatrix} \in \mathbb{R}^{N^- \times d} \tag{1}$$

The mean contrast is:

$$\Delta \mu_\ell = \mu_\ell^+ - \mu_\ell^-, \quad \text{where } \mu_\ell^\pm = \frac{1}{N^\pm} \sum_{i=1}^{N^\pm} h_{\ell,i}^\pm \tag{2}$$

Layer selection: The $K_{\text{pert}}$ layers with lowest cosine similarity between $\mu_\ell^+$ and $\mu_\ell^-$ form the intervention set $\mathcal{L}_{\text{pert}}$; the top-$K_{\text{proj}}$ layers with largest router-logit discrepancy form the router-protected set $\mathcal{L}_{\text{proj}}$.

Five perturbation estimators (each returns $u_\ell \in \mathbb{R}^d$):

1. MeanDiff — first-order translation between class centers:

$$u_\ell^{\text{MeanDiff}} = \alpha_{\text{MeanDiff}} \Delta \mu_\ell \tag{3}$$

2. Probe — discriminative boundary normal from a linear probe $g_\ell(h) = a_\ell^\top h + b_\ell$:

$$u_\ell^{\text{Probe}} = \alpha_{\text{Probe}} \frac{a_\ell}{\|a_\ell\|_2} \min_{i \in 1,\dots,N^+} \frac{|a_\ell^\top h_{\ell,i}^+ + b_\ell|}{\|a_\ell\|_2} \tag{4}$$

3. LowRank — projection onto top-$r$ principal subspace of pooled representations:

$$u_\ell^{\text{LowRank}} = \alpha_{\text{LowRank}} B_{\ell,r} B_{\ell,r}^\top \Delta \mu_\ell \tag{5}$$

4. LDA (Linear Discriminant Analysis) — preconditioning mean contrast with regularized pooled covariance:

$$u_\ell^{\text{LDA}} = \alpha_{\text{LDA}} \left(\frac{\mathbf{C}_\ell^\top \mathbf{C}_\ell}{N^+ + N^- - 2} + \rho_{\text{LDA}} I\right)^{-1} \Delta \mu_\ell \tag{6}$$

5. AffineGaussian — whitening–recoloring with class-specific second-order geometries:

$$u_\ell^{\text{AffineGaussian}} = \alpha_{\text{AffineGaussian}} \frac{\|\Delta \mu_\ell\|_2}{\|G_\ell \Delta \mu_\ell\|_2 + \varepsilon} G_\ell \Delta \mu_\ell, \quad G_\ell = (\Sigma_\ell^+ + \rho I)^{1/2}(\Sigma_\ell^- + \rho I)^{-1/2} \tag{7}$$

Router-Agnostic Projection

Since the router is the linear map $R_\ell$, a sufficient condition for a perturbation $\delta h_\ell$ to preserve raw router logits is:

$$R_\ell \delta h_\ell = 0$$

Let $Q_\ell$ be an orthonormal basis for the row space of $R_\ell$. The nullspace projector is:

$$\Pi_\ell^\perp = I - Q_\ell Q_\ell^\top$$

The projected intervention is:

$$h_\ell' = h_\ell + \Pi_\ell^\perp u_\ell, \quad \text{s.t. } R_\ell(h_\ell' - h_\ell) = 0 \tag{8}$$

Runtime downstream correction: Because perturbations propagate through nonlinear transformer blocks, a clean forward pass stores reference router inputs $h_\ell^0$ and logits $z_\ell^0$. At each protected layer, the correction retains only the router-invisible component of deviation:

$$\tilde{h}_\ell = h_\ell^0 + \Pi_\ell^\perp(h_\ell' - h_\ell^0), \quad \text{s.t. } R_\ell \tilde{h}_\ell = R_\ell h_\ell^0 \tag{9}$$

---

Empirical Validation / Results

Experimental Setup

• Models: Six open-weight MoE models — DeepSeek-V2-Lite-Chat, Mixtral-8x7B-Instruct, Phi-3.5-MoE-instruct, Phi-mini-MoE-instruct, Qwen3-30B-A3B-Instruct, GPT-oss-20b.
• Scenarios: Harmfulness steering (JailbreakBench, MaliciousInstruct), truthfulness steering (TruthfulQA), factual editing (CounterFact).
• Baselines: RepE (general), SAFEx and SteerMoE (MoE-specific safety red-teaming).

RQ2: Perturbation Estimator Comparison

AffineGaussian is the strongest estimator:

Table 1: Harmfulness Steering ASR (↑; %)

Table 2: TruthfulQA MC1 Accuracy (↑; %)

Table 3: Factual Editing — Efficacy (ES) and Locality (LS) Scores

Answer to RQ2: AffineGaussian is the strongest default for steering efficacy (leads on all 6 models for harmfulness, 4/6 for truthfulness, highest average factual-editing efficacy). LDA provides a better efficacy–locality trade-off when side-effect preservation matters.

RQ3: Generalization and Baseline Comparison

Table 4: Harmfulness Steering — ASR and MMLU Accuracy vs. Baselines

RARE improves the primary steering metric over both Clean and RepE for all 18 model–scenario combinations (6 models × 3 scenarios). It achieves the strongest aggregate efficacy–utility trade-off among all baselines.

Answer to RQ3: RARE generalizes across all six MoE architectures and all three behavioral control scenarios, achieving the strongest average harmfulness–capability trade-off.

---

Theoretical and Practical Implications

Theoretical Implications

1. Routing as a computation-path constraint: The findings reframe MoE routing as primarily a semantic content organizer rather than a behavioral controller. This suggests architectural insights: expert allocation encodes input-level competence, while behavioral variation lives in the representation space processed by the selected experts.

2. Decoupling representation from routing: RARE demonstrates that behavioral steering and expert allocation can be decoupled — representation-level control can be made compatible with conditional computation by projecting onto router-invisible subspaces. This provides a principled framework for adapting dense-model techniques to MoE architectures.

3. Estimator geometry matters: The superior performance of AffineGaussian (second-order, class-specific geometry) over first-order methods (MeanDiff) and shared-covariance methods (LDA) suggests that behaviorally relevant variation in MoE hidden states is better captured by class-specific covariance structure.

Practical Implications

1. Safety and alignment: RARE enables effective harmfulness steering (53.3% ASR) with substantially better capability preservation (67.8% MMLU) than expert-level baselines (SAFEx: 50.1% MMLU; SteerMoE: 64.5% MMLU), making it a practical tool for safety red-teaming and alignment research on MoE models.

2. Truthfulness improvement: The average TruthfulQA MC1 accuracy improvement from 41.0% to 58.6% demonstrates that router-agnostic steering can meaningfully enhance model truthfulness without fine-tuning.

3. Efficient factual editing: The CounterFact efficacy jump from 16.8% to 96.3% shows that RARE can perform lightweight, localized factual editing on MoE models while retaining reasonable locality (22.9% LS vs. 13.8% for MeanDiff).

4. Architecture-agnostic deployment: Since RARE works across six heterogeneous MoE architectures (varying expert counts, scales, and configurations including shared experts), it offers a general-purpose solution for MoE model control.

---

Conclusion

Main Takeaways

1. Routing consistency is critical: MoE routing primarily tracks query semantics, not response behaviors. Preserving native routing patterns is an essential architectural consideration when adapting representation engineering to MoE models.

2. RARE provides a viable solution: By projecting perturbations onto the null space of the router matrix and correcting downstream routing drift, RARE enables effective representation-level control while preserving the model's native expert allocation.

3. AffineGaussian is the recommended estimator: Second-order class-specific geometry consistently outperforms first-order and shared-covariance estimators across all three steering scenarios.

4. Broad generalization: RARE achieves notable performance improvements across all 18 model–scenario combinations, demonstrating strong transfer across architectures and behavioral objectives.

Future Directions

• Investigating why DeepSeek (with always-active shared experts) shows relatively better RepE performance, potentially revealing architecture-level design insights.
• Exploring whether routing consistency can be enforced more efficiently (e.g., through architectural modifications rather than runtime corrections).
• Extending the router-agnostic framework to other conditional computation paradigms beyond MoE.
• Developing better secondary-property preservation (e.g., locality) alongside primary steering efficacy.