Summary of Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning

Summary (Overview)

• Novel architecture (Mobius-v0): Decouples knowledge storage (FFN) from reasoning computation (Self-Attention) by creating a globally shared knowledge-vector Memory accessible to all Reasoners, departing from the layer-wise binding in standard Transformer architectures.
• Superior data efficiency: A 7B Mobius model trained from scratch achieves the same MMLU score as a 7B Transformer baseline using only 62.6% of the baseline's training data (1.6× data efficiency).
• Inference speedup: Continually pre-trained from Qwen3.5-35B, Intern-S2-Mobius achieves nearly 4× end-to-end inference speedup over the Transformer baseline while maintaining or exceeding downstream performance.
• Two key mechanisms: Backward Residual Connection (all layers access all knowledge) and Dynamic Latent Reasoning (iterative latent refinement within few layers, producing shorter, higher-density reasoning chains).
• Broad benchmark improvements: Outperforms Qwen3.5-35B on general benchmarks (67.88 vs 65.05 avg) and dramatically on scientific tasks (52.14 vs 18.20 avg), including Biology-Instructions (51.40 vs 3.77).

---

Introduction and Theoretical Foundation

Background and Motivation

The paper identifies two dominant research directions in foundation model development, both hitting limits:

1. Scaling approach: Increasing parameters, data, and reasoning chains to enhance capability. However, the Scaling Law has plateaued, and long chains of thought (CoT) produce verbose, tangential output regardless of problem difficulty—a trait associated with insufficient intelligence.

2. Efficiency approach: Reducing architectural complexity (e.g., linear attention, SSM, GDN) to lower overhead. However, efficiency gains come at the cost of sacrificing substantial model capability.

Theoretical Foundation

The paper builds on the conventional role assignment within Transformer architectures:

• FFN → knowledge storage
• Hidden States → information transmission
• Self-Attention → compositional reasoning

Key limitation identified: Due to the hierarchical structure of Transformer, Self-Attention at each layer primarily processes knowledge from the preceding layer. Although residual connections enable shallow hidden states to access deep-layer knowledge, deep-layer Self-Attention cannot access shallow-layer knowledge—information transfer in the reverse direction is impossible without generating additional tokens (CoT).

The Mobius Solution

The proposed architecture achieves two "innate talents" through knowledge-reasoning decoupling:

1. Backward Residual Connection: By sharing a single oversized knowledge repository across all layers, every reasoning operator gains access to all knowledge in the model. This indirect realization avoids the infrastructure-unfriendly computation graphs of direct backward residuals.

2. Dynamic Latent Reasoning: Instead of token-mediated reasoning, Mobius iterates and refines continuous latent vectors against the full knowledge repository within just a few layers. This internalizes deliberation (trial-and-error, refinement) into continuous vector optimization.

---

Methodology

Architecture Design

The Mobius architecture comprises:

• Memory (FFN): A globally shared knowledge-vector database, constructed via horizontal concatenation of FFN layers, preserving the Key-Value correspondence of knowledge vectors.
• Reasoners (Self-Attn): Multiple reasoning operators that iteratively query the Memory for required knowledge-vectors.

Information flow: Hidden states act as both cache and carrier. Reasoners repeatedly query the Memory, and knowledge is transmitted back to reasoning operators. At larger parameter scales, a block-wise partitioning technique analogous to MoE is employed for sparse activation.

Training Configurations

---

Empirical Validation / Results

Data Efficiency (Training from Scratch)

As shown in Figure 2, Mobius achieves significantly higher MMLU scores than Transformer across all training stages. Using Transformer's score at 1TB as baseline:

Mobius attains the same score with only 0.626× data — that is, 1.6× data efficiency of Transformer.

Hypothesized mechanism: Under Transformer, the same knowledge may be redundantly stored across multiple layers. The decoupled architecture achieves superior compression rates, requiring substantially less training data.

Benchmark Performance (Continual Pretraining)

Table 1: Performance comparison across general and scientific benchmarks (higher score in bold):

Inference Efficiency

• Throughput: Aggregated across multiple benchmarks, Mobius achieves substantially higher request throughput than Transformer (Figure 3).
• Output length: Mobius produces markedly shorter reasoning chains (Figure 4), with the efficiency gain primarily stemming from reduced CoT length.

Case Study: Reasoning Trace Comparison

Table 2: Step-aligned comparison on a linear-algebra multiple-choice question (both select Option C correctly):

Key insight: The shortened CoT arises from eliminating redundant derivation and verification steps, not from hacking the problem. In this example, $v_1 = (1,0)$ and $v_2 = (2,0)$ are linearly dependent since $2v_1 - v_2 = 0$, making Statement 1 False; a linearly independent spanning set of $k$ vectors forms a basis, so $\dim(V) = k$, making Statement 2 True → Option C.

---

Theoretical and Practical Implications

Relationship to Mainstream Research

Latent Reasoning:

• Extends continuous-thought approaches (COCONUT, CODI, SoftCoT) by making latent reasoning a native architectural capability rather than an external mechanism.
• Builds on looped language models (Universal Transformers, Looped Transformers) but performs each update over fewer layers while retaining full Memory access, enabling higher update frequency.
• Extends pause-token and diffusion-based methods through recurrent latent refinement without explicit placeholders or diffusion steps.

Efficient Reasoning:

• Compared to Speculative Decoding (which traverses knowledge once before prediction), Mobius performs multiple rounds of knowledge traversal and multi-token iteration internally.
• Unlike Concise CoT methods that risk curtailing deliberation, Mobius achieves shorter responses through latent thought iteration without sacrificing reasoning quality.

Architecture Design:

• Unlike attention modifications (sparse, linear, SSM) that reduce interaction cost, Mobius increases information density per attention operation.
• Extends residual design beyond forward propagation toward bidirectional knowledge access without explicit residual pathway expansion.

Highlighted Future Applications

1. Self-Evolving: Knowledge-reasoning decoupling supports unbounded knowledge expansion and cross-domain reasoning generalization, enabling continual learning without catastrophic forgetting.

2. World Model: Native latent reasoning offers a stronger prior for continuous-space modeling, potentially avoiding the petabyte-scale parameter requirements of Transformer-based world models.

3. Scientific Discovery: Internalized deliberation in continuous latent space may develop "scientific intuition," while flattened knowledge enables compositional generalization across domains.

4. Hardware-Software Co-Design: Knowledge-reasoning separation enables keeping only reasoning parameters in GPU memory while storing knowledge on SSD, with on-demand retrieval of high-priority knowledge.

---

Conclusion

The paper introduces Mobius, a foundation model architecture that decouples knowledge storage (FFN) from reasoning computation (Self-Attention) through a globally shared knowledge-vector Memory. This design delivers:

• 1.6× data efficiency in training from scratch
• ~4× end-to-end inference speedup with maintained or improved performance in continual pretraining
• Dramatically shorter reasoning chains (e.g., 516 vs 2,364 tokens on the same problem) without sacrificing accuracy

The architecture's two innate talents—Backward Residual Connection and Dynamic Latent Reasoning—enable more flexible knowledge access and higher-density information transfer, reducing reliance on verbose token-mediated chains of thought.

Future Directions

The authors acknowledge open questions:

• Mechanisms: The precise reasons for Mobius's data efficiency and shorter CoT remain not fully established.
• Validation: Real-world self-evolution, world modeling, and scientific discovery scenarios require joint optimization of data, infrastructure, and training algorithms.
• Scaling: Whether the benefits persist at larger scales and with more diverse modalities remains to be verified.
• Hardware integration: Realizing the hardware-software co-design vision (SSD-resident knowledge) requires substantial engineering effort.