# Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

> Scaling zero RL to a trillion parameters unlocks emergent reasoning behaviors and superior performance, rendering hand-crafted heuristics redundant.

- **Source:** [arXiv](https://arxiv.org/abs/2607.12395)
- **Published:** 2026-07-17
- **Permalink:** https://picx.dev/p/WcqeEb

## Summary

## Summary

- **Scaling Zero RL to 1T parameters**: This paper presents the first successful application of "zero RL" (reinforcement learning with verifiable rewards without human-annotated data) to a trillion-parameter model (Ling-2.5-1T-Base, a 1T-parameter MoE with 63B activated parameters), demonstrating that massive scale fundamentally changes the training dynamics and capability ceiling.
- **Stable and efficient training pipeline**: The authors introduce a minimalist multi-stage pipeline (First-Stage RL → Self-Distillation → Second-Stage RL → Third-Stage RL) with key algorithmic optimizations including clipped importance sampling, training-inference ratio correction, and mixed-precision control, enabling stable training at unprecedented scale.
- **Empirical validation of the "bitter lesson"**: Scaling to 1T parameters yields vastly superior sample efficiency and performance ceilings compared to the 104B model, and the model spontaneously develops advanced cognitive behaviors—anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety—rendering hand-crafted heuristics redundant.
- **Two-phase training dynamics**: The training process progresses sequentially through an initial "discovery" phase (expanding the reasoning boundary, evidenced by rising pass@1024) followed by a "sharpening" phase (refining the policy within the established boundary, evidenced by continued pass@1 improvement).
- **Competitive performance with superior CoT quality**: Ring-2.5-1T-Zero achieves competitive results on seven challenging mathematical benchmarks (e.g., 93.5% on AIME 2024 after Second-Stage RL) while producing reasoning traces that are more comprehensible, reproducible, and token-efficient than frontier models.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Chain-of-thought (CoT) reasoning has emerged as a critical scaling dimension for large language models (LLMs). Recent work has demonstrated that **reinforcement learning with verifiable rewards (RLVR)** can effectively enhance reasoning abilities. The paradigm of **"zero RL"** (Guo et al., 2025) initiates RL directly from a pretrained base model, bypassing supervised CoT data entirely, proving that powerful reasoning can be cultivated purely through trial-and-error.

However, due to computational constraints, existing studies are largely restricted to small models. The central question remains: **how do training dynamics and emergent capabilities of zero RL evolve when applied to a trillion-parameter model?**

### Key Challenges Identified

The authors identify three critical limitations of naive zero RL scaling:

1. **Poor readability**: Reasoning traces lack logical formatting and clear structure.
2. **Token redundancy**: Standard algorithms like GRPO introduce an implicit length bias, causing uncontrolled length growth and inefficient inference.
3. **Lack of adaptive reasoning depth**: Standard pipelines produce a single-mode model constrained by a fixed response budget.

### Theoretical Foundation

The work is grounded in the **"bitter lesson"** of AI research: general methods that leverage computation and scale ultimately overtake human-engineered heuristics. The authors hypothesize that at sufficient scale, hand-crafted heuristics for reasoning (structured formatting, self-verification, parallel exploration) become redundant as the model spontaneously discovers these strategies.

---

## Methodology

### Multi-Stage Training Pipeline

The training pipeline consists of four distinct phases:

#### 1. First Stage RL: Reasoning Elicitation

Uses a **clipped importance-sampling policy gradient** (CISPO-style) to amplify low-probability reasoning tokens. The objective is:

$$J(\theta) = \mathbb{E}_{q \sim \mathcal{D}, \{o_i\}_{i=1}^{G} \sim \pi_S} \left[ \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \text{sg}(\hat{\rho}_{i,t}) \cdot \hat{A}_{i,t} \cdot \log \pi_{\theta}^{M}(o_{i,t} | q, o_{i,<t}) \right]$$

where:
- $q$ is the input question from dataset $\mathcal{D}$
- $\{o_i\}_{i=1}^{G}$ are $G$ rollout responses from the inference engine $\pi_S$
- $\hat{A}_{i,t}$ is the advantage estimate (GRPO-style group-normalized rewards)
- $\text{sg}(\cdot)$ is the stop-gradient operator
- $\hat{\rho}_{i,t}$ is the clipped importance ratio

The importance ratio uses **training-engine logits** as the numerator to correct for numerical discrepancies:

$$\rho_{i,t} = \frac{\pi_{\theta}^{M}(o_{i,t} | q, o_{i,<t})}{\pi_{\theta_{old}}^{S}(o_{i,t} | q, o_{i,<t})}, \quad \hat{\rho}_{i,t} = \text{clip}(\rho_{i,t}, \epsilon_{low}, \epsilon_{high})$$

A KL divergence penalty against a frozen reference model stabilizes training:

$$\mathcal{L}_{KL}(\theta) = \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} D_{KL}\left( \pi_{\theta}^{M}(\cdot | q, o_{i,<t}) \| \pi_{ref}(\cdot | q, o_{i,<t}) \right)$$

The combined first-stage objective is:

$$\mathcal{L}_{\text{Ring-2.5-1T-Zero-I}}(\theta) = -J(\theta) + \beta \cdot \mathcal{L}_{KL}(\theta)$$

A **token-level loss** (no normalization by $|o_i|$) encourages longer responses. The response window expands from 4k to 64k tokens in a curriculum manner.

#### 2. Self-Distillation: Compression and Stabilization

To address verbosity and the training-inference gap, the authors:
1. Sample multiple rollouts from the first-stage expert policy
2. Select the **shortest correct reasoning trace**
3. Prompt the model to self-evaluate and filter redundant segments
4. Fine-tune the base model via supervised learning:

$$\mathcal{L}_{\text{self-distillation}}(\theta) = -\mathbb{E}_{q \sim \mathcal{D}, o \sim \pi_{expert}} \left[ \sum_{t=1}^{|o|} \log \pi_{\theta}(o_t | q, o_{<t}) \right]$$

#### 3. Second Stage RL: Sustained Optimization

Switches to a **sample-level loss** (normalized by $1/|o_i|$) to control length growth, and removes the KL penalty:

$$\mathcal{L}_{\text{Ring-2.5-1T-Zero-II}}(\theta) = -\mathbb{E}_{q \sim \mathcal{D}, \{o_i\}_{i=1}^{G} \sim \pi} \left[ \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \text{sg}(\hat{\rho}_{i,t}) \cdot \hat{A}_{i,t} \cdot \log \pi_{\theta}(o_{i,t} | q, o_{i,<t}) \right]$$

#### 4. Third Stage RL: Adaptive Reasoning Depth

Introduces **tier-based training** with three difficulty tiers $\mathcal{T} = \{\mathcal{T}_l, \mathcal{T}_m, \mathcal{T}_h\}$, each with a specific maximum token length and system prompt $p_k$:

$$\mathcal{L}_{\text{Ring-2.5-1T-Zero-III}}(\theta) = -\sum_{k \in \{l,m,h\}} \mathbb{E}_{q \sim \mathcal{D}_k, \{o_i\}_{i=1}^{G} \sim \pi(\cdot|p_k, q)} \left[ \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \text{sg}(\hat{\rho}_{i,t}) \cdot \hat{A}_{i,t} \cdot \log \pi_{\theta}(o_{i,t} | p_k, q, o_{i,<t}) \right]$$

### Infrastructure Optimizations

1. **Mixed-precision control**: Compute attention softmax and LM head in FP32 (sensitive to exponentiation errors), keeping the main body in BF16.
2. **Context parallelism optimization**: All-to-all CP for MLA layers (reduces communication volume via low-rank KV compression) and AllGather for Lightning Attention layers (broadcasts fixed-size KV states).

### Reward Design

$$r_i = r_{acc,i} + r_{format,i}$$

where $r_{format,i} \in \{0, 1\}$ checks structural compliance (thinking/response/answer tags) and $r_{acc,i} \in \{0, 1\}$ measures correctness. Early stages use rule-based verification; later stages use LLM-as-a-Judge (Qwen3-Next-80B-A3B-Instruct).

---

## Empirical Validation / Results

### Main Results

**Table 1: Main results on mathematical reasoning benchmarks (pass@1 accuracy, %)**

| Model | AIME 2024 | AIME 2025 | AIME 2026 | HMMT Feb. 2025 | HMMT Nov. 2025 | HMMT Feb. 2026 | IMOAnswer Bench |
|---|---|---|---|---|---|---|---|
| **Frontier Models** | | | | | | | |
| GLM-5.1 | - | - | 95.3 | - | 94.0 | 82.6 | 83.8 |
| DS-V4-Pro Max | - | - | 94.6 | - | 94.4 | 95.2 | 89.8 |
| Qwen3.7-Plus | - | - | 97.0 | - | 95.0 | 92.9 | 86.0 |
| GPT-5.5 | - | - | 98.3 | - | 96.5 | 96.7 | 91.4 |
| **Zero RL (Ours)** | | | | | | | |
| Ring-2.5-flash-Zero (First Stage RL) | 71.2 | 63.5 | 65.3 | 55.2 | 54.8 | 50.3 | - |
| Ring-2.5-1T-Zero (First Stage RL) | 89.1 | 83.3 | 84.2 | 76.7 | 75.8 | 66.2 | 59.3 |
| Ring-2.5-1T-Zero (Self Distillation) | 92.3 | 87.3 | 88.1 | 81.9 | 79.9 | 71.2 | 63.8 |
| Ring-2.5-1T-Zero (Second Stage RL) | 93.5 | 91.6 | 92.5 | 87.4 | 87.1 | 78.1 | 72.7 |
| Ring-2.5-1T-Zero (Second Stage RL, Yarn=2) | 94.1 | 92.3 | 93.2 | 90.6 | 90.8 | 81.0 | 75.5 |
| Ring-2.5-1T-Zero (Third Stage RL, High) | 93.2 | 91.0 | 91.4 | 86.3 | 86.4 | 78.4 | 72.7 |

### CoT Quality Evaluation

**Comprehensibility** (LLM-as-a-Judge pairwise comparisons): Our model achieves dominant win rates against GLM-5.1, Kimi-k2.6, MiniMax-M2.7, and Qwen3.5-397B (win rates of 64-78 out of 90 problems).

**Reproducibility** (knowledge distillation into weaker models with only 100K samples):

| Student Model | Base | DeepSeek-R1-Distill | Ring-Zero-Distill |
|---|---|---|---|
| Qwen-32B | 5.2 | 72.6 | **78.4** |
| Llama-70B | 26.2 | 70.0 | **74.5** |

**Efficiency**: On mutually solved AIME problems, our model uses an average of only **6,368 tokens** vs. 14,115-17,220 for baselines (less than half).

### Key Ablation Findings

1. **RL algorithm comparison**: CISPO and DAPO accelerate learning but suffer instability; GSPO maintains entropy but provides limited length growth. The authors' approach balances speed and stability.
2. **KL penalty**: Removing it causes catastrophic failure (log-probability gap diverges, entropy collapses, reward crashes).
3. **Ratio correction**: Using training-engine logits as numerator prevents collapse (baseline collapses at ~800 steps; IcePop fails at ~2700 steps; our approach remains stable).
4. **Format reward**: Format B (double-closed tags with EOS requirement) prevents degenerate length growth vs. Format A.
5. **Length inertia**: The model inflates token usage for already-solved problems, demonstrating a "lazy shortcut" behavior.
6. **Hyperparameters**: Learning rate is robust in the tested range; G=16 rollouts balances variance reduction and wall-clock time; token-level loss promotes length growth while sample-level keeps length flat.

---

## Theoretical and Practical Implications

### The "Bitter Lesson" Validated

The paper provides strong empirical evidence for the "bitter lesson" at scale:

1. **Scaling dictates capability ceiling**: The 1T model exhibits vastly superior sample efficiency and reaches significantly higher performance bounds than the 104B model. After 3,600 steps, Ring-2.5-1T-Zero-I achieves 89.06% on AIME 2024 vs. 71.72% for the flash model even after 5,200 steps.

2. **Discovery vs. sharpening as sequential stages**: Pass@1024 increases during early training but plateaus, while pass@1 continues climbing. This proves RL first expands the reasoning boundary (discovery) then refines the policy within it (sharpening).

3. **Spontaneous emergence of advanced cognitive strategies**: Without any human-annotated data or hand-crafted heuristics, the 1T model autonomously converges on:
   - **Anthropomorphism**: Simulated frustration, slacking, self-praise (e.g., "brain fart", "genius idea")
   - **Structured formatting**: Natural step-by-step organization (Step 1:, Step 2:, etc.)
   - **Self-verification**: Cross-checking derivations against known formulas
   - **Parallel reasoning**: Tree-of-thought search within a single rollout
   - **Context anxiety**: Strategic panic near token limits, prioritizing structural completeness over rigor

### Practical Implications

- **Training efficiency**: The minimalist pipeline (only a few critical modifications) achieves stable training at 1T scale, avoiding heavy engineering.
- **Adaptive inference**: Tier-based training enables dynamic compute allocation—users can control reasoning depth via system prompts (Low/Medium/High modes).
- **CoT quality framework**: The proposed three-dimensional evaluation (comprehensibility, reproducibility, efficiency) provides diagnostic signals beyond final-answer accuracy.

### Fundamental Limits Identified

- **Training data distribution**: Real-world data has a long-tail difficulty distribution (67.6% of problems solvable within 4k tokens), but mimicking this natural frequency wastes compute. A dynamic curriculum is required.
- **Model capacity**: Scale dictates both performance ceiling and learning speed.
- **Context window**: Longer reasoning chains unlock complex derivations; 64k is a hardware-imposed limit.
- **Pretrained priors**: Zero RL can only bootstrap from knowledge embedded during pretraining—RL sharpens within this boundary but cannot transcend it.

---

## Conclusion

This paper demonstrates that zero RL can be successfully scaled to a trillion-parameter model, achieving competitive performance on challenging mathematical benchmarks while producing high-quality reasoning traces. The key contributions are:

1. **A stable, minimalist training pipeline** requiring only minor algorithmic and system improvements (clipped importance sampling, training-inference ratio correction, mixed-precision control) to train Ring-2.5-1T-Zero.

2. **A CoT quality evaluation framework** spanning comprehensibility, reproducibility, and efficiency, going beyond final-answer accuracy.

3. **Empirical validation of the bitter lesson at scale**: Scaling unlocks higher sample efficiency and performance ceilings while enabling the autonomous emergence of advanced cognitive strategies, rendering hand-crafted designs redundant.

### Future Directions

- Expanding the context window beyond 64k to unlock new levels of mathematical capability
- Jointly optimizing reasoning quality and token efficiency within a single, unified RL objective
- Exploring whether even larger models or more diverse training data can push the reasoning boundary further
- Investigating the "context anxiety" behavior as a potential failure mode to address

The authors hope their experimental details and observed emergent phenomena provide the community with deeper insights into scaling behaviors at the 1-trillion scale.

---

_Markdown view of https://picx.dev/p/WcqeEb, served by PicX — AI-generated visual whiteboard summaries of research papers._