Summary

• Scaling Zero RL to 1T parameters: This paper presents the first successful application of "zero RL" (reinforcement learning with verifiable rewards without human-annotated data) to a trillion-parameter model (Ling-2.5-1T-Base, a 1T-parameter MoE with 63B activated parameters), demonstrating that massive scale fundamentally changes the training dynamics and capability ceiling.
• Stable and efficient training pipeline: The authors introduce a minimalist multi-stage pipeline (First-Stage RL → Self-Distillation → Second-Stage RL → Third-Stage RL) with key algorithmic optimizations including clipped importance sampling, training-inference ratio correction, and mixed-precision control, enabling stable training at unprecedented scale.
• Empirical validation of the "bitter lesson": Scaling to 1T parameters yields vastly superior sample efficiency and performance ceilings compared to the 104B model, and the model spontaneously develops advanced cognitive behaviors—anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety—rendering hand-crafted heuristics redundant.
• Two-phase training dynamics: The training process progresses sequentially through an initial "discovery" phase (expanding the reasoning boundary, evidenced by rising pass@1024) followed by a "sharpening" phase (refining the policy within the established boundary, evidenced by continued pass@1 improvement).
• Competitive performance with superior CoT quality: Ring-2.5-1T-Zero achieves competitive results on seven challenging mathematical benchmarks (e.g., 93.5% on AIME 2024 after Second-Stage RL) while producing reasoning traces that are more comprehensible, reproducible, and token-efficient than frontier models.

---

Introduction and Theoretical Foundation

Background and Motivation

Chain-of-thought (CoT) reasoning has emerged as a critical scaling dimension for large language models (LLMs). Recent work has demonstrated that reinforcement learning with verifiable rewards (RLVR) can effectively enhance reasoning abilities. The paradigm of "zero RL" (Guo et al., 2025) initiates RL directly from a pretrained base model, bypassing supervised CoT data entirely, proving that powerful reasoning can be cultivated purely through trial-and-error.

However, due to computational constraints, existing studies are largely restricted to small models. The central question remains: how do training dynamics and emergent capabilities of zero RL evolve when applied to a trillion-parameter model?

Key Challenges Identified

The authors identify three critical limitations of naive zero RL scaling:

1. Poor readability: Reasoning traces lack logical formatting and clear structure.
2. Token redundancy: Standard algorithms like GRPO introduce an implicit length bias, causing uncontrolled length growth and inefficient inference.
3. Lack of adaptive reasoning depth: Standard pipelines produce a single-mode model constrained by a fixed response budget.

Theoretical Foundation

The work is grounded in the "bitter lesson" of AI research: general methods that leverage computation and scale ultimately overtake human-engineered heuristics. The authors hypothesize that at sufficient scale, hand-crafted heuristics for reasoning (structured formatting, self-verification, parallel exploration) become redundant as the model spontaneously discovers these strategies.

---

Methodology

Multi-Stage Training Pipeline

The training pipeline consists of four distinct phases:

1. First Stage RL: Reasoning Elicitation

Uses a clipped importance-sampling policy gradient (CISPO-style) to amplify low-probability reasoning tokens. The objective is:

$$J(\theta) = \mathbb{E}_{q \sim \mathcal{D}, \{o_i\}_{i=1}^{G} \sim \pi_S} \left[ \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \text{sg}(\hat{\rho}_{i,t}) \cdot \hat{A}_{i,t} \cdot \log \pi_{\theta}^{M}(o_{i,t} | q, o_{i,<t}) \right]$$

where:

• $q$ is the input question from dataset $\mathcal{D}$
• $\{o_i\}_{i=1}^{G}$ are $G$ rollout responses from the inference engine $\pi_S$
• $\hat{A}_{i,t}$ is the advantage estimate (GRPO-style group-normalized rewards)
• $\text{sg}(\cdot)$ is the stop-gradient operator
• $\hat{\rho}_{i,t}$ is the clipped importance ratio

The importance ratio uses training-engine logits as the numerator to correct for numerical discrepancies:

$$\rho_{i,t} = \frac{\pi_{\theta}^{M}(o_{i,t} | q, o_{i,<t})}{\pi_{\theta_{old}}^{S}(o_{i,t} | q, o_{i,<t})}, \quad \hat{\rho}_{i,t} = \text{clip}(\rho_{i,t}, \epsilon_{low}, \epsilon_{high})$$

A KL divergence penalty against a frozen reference model stabilizes training:

$$\mathcal{L}_{KL}(\theta) = \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} D_{KL}\left( \pi_{\theta}^{M}(\cdot | q, o_{i,<t}) \| \pi_{ref}(\cdot | q, o_{i,<t}) \right)$$

The combined first-stage objective is:

$$\mathcal{L}_{\text{Ring-2.5-1T-Zero-I}}(\theta) = -J(\theta) + \beta \cdot \mathcal{L}_{KL}(\theta)$$

A token-level loss (no normalization by $|o_i|$) encourages longer responses. The response window expands from 4k to 64k tokens in a curriculum manner.

2. Self-Distillation: Compression and Stabilization

To address verbosity and the training-inference gap, the authors:

1. Sample multiple rollouts from the first-stage expert policy
2. Select the shortest correct reasoning trace
3. Prompt the model to self-evaluate and filter redundant segments
4. Fine-tune the base model via supervised learning:

$$\mathcal{L}_{\text{self-distillation}}(\theta) = -\mathbb{E}_{q \sim \mathcal{D}, o \sim \pi_{expert}} \left[ \sum_{t=1}^{|o|} \log \pi_{\theta}(o_t | q, o_{<t}) \right]$$

3. Second Stage RL: Sustained Optimization

Switches to a sample-level loss (normalized by $1/|o_i|$) to control length growth, and removes the KL penalty:

$$\mathcal{L}_{\text{Ring-2.5-1T-Zero-II}}(\theta) = -\mathbb{E}_{q \sim \mathcal{D}, \{o_i\}_{i=1}^{G} \sim \pi} \left[ \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \text{sg}(\hat{\rho}_{i,t}) \cdot \hat{A}_{i,t} \cdot \log \pi_{\theta}(o_{i,t} | q, o_{i,<t}) \right]$$

4. Third Stage RL: Adaptive Reasoning Depth

Introduces tier-based training with three difficulty tiers $\mathcal{T} = \{\mathcal{T}_l, \mathcal{T}_m, \mathcal{T}_h\}$, each with a specific maximum token length and system prompt $p_k$:

$$\mathcal{L}_{\text{Ring-2.5-1T-Zero-III}}(\theta) = -\sum_{k \in \{l,m,h\}} \mathbb{E}_{q \sim \mathcal{D}_k, \{o_i\}_{i=1}^{G} \sim \pi(\cdot|p_k, q)} \left[ \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \text{sg}(\hat{\rho}_{i,t}) \cdot \hat{A}_{i,t} \cdot \log \pi_{\theta}(o_{i,t} | p_k, q, o_{i,<t}) \right]$$

Infrastructure Optimizations

1. Mixed-precision control: Compute attention softmax and LM head in FP32 (sensitive to exponentiation errors), keeping the main body in BF16.
2. Context parallelism optimization: All-to-all CP for MLA layers (reduces communication volume via low-rank KV compression) and AllGather for Lightning Attention layers (broadcasts fixed-size KV states).

Reward Design

$$r_i = r_{acc,i} + r_{format,i}$$

where $r_{format,i} \in \{0, 1\}$ checks structural compliance (thinking/response/answer tags) and $r_{acc,i} \in \{0, 1\}$ measures correctness. Early stages use rule-based verification; later stages use LLM-as-a-Judge (Qwen3-Next-80B-A3B-Instruct).

---

Empirical Validation / Results

Main Results

Table 1: Main results on mathematical reasoning benchmarks (pass@1 accuracy, %)

CoT Quality Evaluation

Comprehensibility (LLM-as-a-Judge pairwise comparisons): Our model achieves dominant win rates against GLM-5.1, Kimi-k2.6, MiniMax-M2.7, and Qwen3.5-397B (win rates of 64-78 out of 90 problems).

Reproducibility (knowledge distillation into weaker models with only 100K samples):

Efficiency: On mutually solved AIME problems, our model uses an average of only 6,368 tokens vs. 14,115-17,220 for baselines (less than half).

Key Ablation Findings

1. RL algorithm comparison: CISPO and DAPO accelerate learning but suffer instability; GSPO maintains entropy but provides limited length growth. The authors' approach balances speed and stability.
2. KL penalty: Removing it causes catastrophic failure (log-probability gap diverges, entropy collapses, reward crashes).
3. Ratio correction: Using training-engine logits as numerator prevents collapse (baseline collapses at ~800 steps; IcePop fails at ~2700 steps; our approach remains stable).
4. Format reward: Format B (double-closed tags with EOS requirement) prevents degenerate length growth vs. Format A.
5. Length inertia: The model inflates token usage for already-solved problems, demonstrating a "lazy shortcut" behavior.
6. Hyperparameters: Learning rate is robust in the tested range; G=16 rollouts balances variance reduction and wall-clock time; token-level loss promotes length growth while sample-level keeps length flat.

---

Theoretical and Practical Implications

The "Bitter Lesson" Validated

The paper provides strong empirical evidence for the "bitter lesson" at scale:

1. Scaling dictates capability ceiling: The 1T model exhibits vastly superior sample efficiency and reaches significantly higher performance bounds than the 104B model. After 3,600 steps, Ring-2.5-1T-Zero-I achieves 89.06% on AIME 2024 vs. 71.72% for the flash model even after 5,200 steps.

2. Discovery vs. sharpening as sequential stages: Pass@1024 increases during early training but plateaus, while pass@1 continues climbing. This proves RL first expands the reasoning boundary (discovery) then refines the policy within it (sharpening).

3. Spontaneous emergence of advanced cognitive strategies: Without any human-annotated data or hand-crafted heuristics, the 1T model autonomously converges on:

   • Anthropomorphism: Simulated frustration, slacking, self-praise (e.g., "brain fart", "genius idea")
   • Structured formatting: Natural step-by-step organization (Step 1:, Step 2:, etc.)
   • Self-verification: Cross-checking derivations against known formulas
   • Parallel reasoning: Tree-of-thought search within a single rollout
   • Context anxiety: Strategic panic near token limits, prioritizing structural completeness over rigor

Practical Implications

• Training efficiency: The minimalist pipeline (only a few critical modifications) achieves stable training at 1T scale, avoiding heavy engineering.
• Adaptive inference: Tier-based training enables dynamic compute allocation—users can control reasoning depth via system prompts (Low/Medium/High modes).
• CoT quality framework: The proposed three-dimensional evaluation (comprehensibility, reproducibility, efficiency) provides diagnostic signals beyond final-answer accuracy.

Fundamental Limits Identified

• Training data distribution: Real-world data has a long-tail difficulty distribution (67.6% of problems solvable within 4k tokens), but mimicking this natural frequency wastes compute. A dynamic curriculum is required.
• Model capacity: Scale dictates both performance ceiling and learning speed.
• Context window: Longer reasoning chains unlock complex derivations; 64k is a hardware-imposed limit.
• Pretrained priors: Zero RL can only bootstrap from knowledge embedded during pretraining—RL sharpens within this boundary but cannot transcend it.

---

Conclusion

This paper demonstrates that zero RL can be successfully scaled to a trillion-parameter model, achieving competitive performance on challenging mathematical benchmarks while producing high-quality reasoning traces. The key contributions are:

1. A stable, minimalist training pipeline requiring only minor algorithmic and system improvements (clipped importance sampling, training-inference ratio correction, mixed-precision control) to train Ring-2.5-1T-Zero.

2. A CoT quality evaluation framework spanning comprehensibility, reproducibility, and efficiency, going beyond final-answer accuracy.

3. Empirical validation of the bitter lesson at scale: Scaling unlocks higher sample efficiency and performance ceilings while enabling the autonomous emergence of advanced cognitive strategies, rendering hand-crafted designs redundant.

Future Directions

• Expanding the context window beyond 64k to unlock new levels of mathematical capability
• Jointly optimizing reasoning quality and token efficiency within a single, unified RL objective
• Exploring whether even larger models or more diverse training data can push the reasoning boundary further
• Investigating the "context anxiety" behavior as a potential failure mode to address

The authors hope their experimental details and observed emergent phenomena provide the community with deeper insights into scaling behaviors at the 1-trillion scale.