Summary (Overview)

• Core Problem: The paper identifies a "zero-advantage trap" in Reinforcement Learning with Verifiable Rewards (RLVR), where approximately 88% of a large 78K-problem open-source math corpus provides no learning signal for a fixed target policy (gpt-oss-120b) because problems are either solved in all rollouts or failed in all rollouts.
• Key Insight: Hard problems (those with 0/8 success) should not be discarded—they serve as specifications for what reasoning capabilities to target, since their verified solutions expose "reasoning blueprints" (essential operations and their compositions).
• Proposed Method: MagicBeanStalk (MBS), a top-down decomposition and bottom-up recursive data synthesis algorithm that grows hard seed problems into families of substantively new, policy- and budget-calibrated training problems.
• Main Results: MBS improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4 Flash from 72.4 to 77.9 (+5.4) across 8 math benchmarks; in finance, 340 seeds yield 14K+ problems improving Qwen3 by +6.9 (SecQue) and +2.4 (FinanceBench).
• Key Comparison: Direct prompting a teacher to generate variations performs no better than training on source seeds, while MBS achieves +8.6 points—demonstrating that structured construction, not mere prompting, is essential.

---

Introduction and Theoretical Foundation

Background and Motivation

Reinforcement learning with verifiable rewards (RLVR) has become central to improving reasoning models. However, as policies strengthen, scaling RLVR increasingly becomes a data problem requiring continual supply of verifiable problems. Existing approaches include:

• Collected corpora: Human-authored math/coding problems (Chen et al., 2025; Luo et al., 2025)
• Procedural environments: Hand-engineered systems generating multiple problems per environment (Stojanovski et al., 2025; Lacombe et al., 2025)
• Surrogate task synthesis: Mining tasks from Internet data (Lu et al., 2026)

The authors argue that more data does not necessarily ensure useful learning signal for a target policy and training budget.

Theoretical Foundation: The Zero-Advantage Trap

The paper formalizes dataset effectiveness through the empirical success rate and group-relative advantage:

$$\bar{r}_{\pi,L}(p) = \frac{1}{N} \sum_{j=1}^{N} r_j, \quad \hat{A}_i(p) = r_i(p) - \bar{r}_{\pi,L}(p) \tag{1}$$

where $r_i = R(p, a_i) \in \{0, 1\}$ is the binary verifier score, $\pi$ is the target policy, $L$ is the rollout budget, and $N$ responses are sampled.

Key definitions:

• Zero-advantage trap: When all responses receive the same reward ($\hat{A}_i = 0$ for every response), providing no within-problem advantage signal.
• Goldilocks band: Problems satisfying $0 < \bar{r}_{\pi,L}(p) < 1$, yielding a nonzero group-relative learning signal.
• Difficulty is relative: A problem's usefulness depends on the tuple $(p, \pi, L)$—the problem, target policy, and rollout budget.

Three Design Principles

1. Hard problems are specifications: Verified solutions of hard problems identify reasoning operations and compositions ("reasoning blueprints") to target through synthesis.
2. Synthesis must do what filtering cannot: Synthetic problems should be substantively new while preserving the hard seed's reasoning operations and their composition.
3. Usefulness is relative: Generated problems must be calibrated to both the target policy and training budget—yielding mixed rollouts with at least one complete correct response within budget.

---

Methodology

MagicBeanStalk (MBS) Algorithm

MBS operates in three stages, each guided by the design principles:

Stage 1: Rejection-Sample Teacher Traces

Given verified problems $(p, a^*)$, iterate through a teacher pool $\mathcal{T} = \{T_1, T_2, \ldots, T_M\}$:

• Draw up to $N_t$ reasoning traces and answers: $(\tau^{(i)}, a) \sim T(p)$
• Accept the first trace satisfying $\text{verified}(\tau, a^*) = 1$
• Problems with no verified trace after $|\mathcal{T}| \cdot N_t$ attempts are discarded

Stage 2: Top-Down Problem Decomposition

Treat each verified $(p, \tau, a^*)$ as the root of a decomposition tree $G_p$:

$$\{(p_k, \tau_k, a^*_k)\}_{k=1}^{K} \sim T(p, \tau, a^*)$$

The teacher emits $K \leq 3$ easier self-contained subproblems. Each subproblem undergoes an effectivity test: conditioning on subproblems as in-context worked examples must raise accuracy for at least one evaluated student:

$$\exists S \in S_n: \text{acc}_{\text{cond}}(S, p | \{p_k, \tau_k, a^*_k\}_k) > \text{acc}(S, p)$$

Decompositions that fail are re-prompted up to $N_d = 3$ retries; recursion proceeds to depth $D = 3$.

Stage 3: Bottom-Up Synthesis of Calibrated Problems

• Leaf synthesis: Generate harder variants of leaf problems: $\{(\tilde{p}_\ell^{(s)}, \tilde{\tau}_\ell^{(s)}, \tilde{a}_\ell^{*\,(s)})\}_{s=1}^{N_\ell} \sim T(p_\ell, \tau_\ell, a^*_\ell)$, discarding variants exceeding accuracy threshold $\alpha_{\text{leaf}} = 0.8$ on the weakest student.
• Recursive composition: Synthesize parent problems from synthetic children:

$$(\tilde{p}, \tilde{\tau}, \tilde{a}^*) \sim T\left(\{(\tilde{p}_k, \tilde{\tau}_k, \tilde{a}^*_k)\}_{k=1}^{K}; B(G_n)\right)$$

where $B(G_n)$ is the original parent–children configuration used as a demonstration pattern. Candidates exceeding $\alpha_{\text{parent}} = 0.625$ accuracy are discarded, with up to $N_c = 10$ trials.

• Document grounding (for finance): Retrieve top-$K_d$ unseen documents by embedding similarity and re-ground the synthetic tree on new documents.

Training Setup

---

Empirical Validation / Results

Zero-Advantage Trap Characterization

For gpt-oss-120b at 32K rollout budget on ~78K problems:

• 88% fall in the zero-advantage trap (either solved in all rollouts or failed in all)
• Only 12.2% fall in the Goldilocks band
• Changing the target policy reclassifies 32.3% of problems
• Increasing response limit from 16K to 64K reduces hard problems from 33.1% to 10.6%

Main Math Results (16K output tokens)

Qualified Yield from Hard Seeds

Inference Budget Evaluation

On MathArena Apex-Shortlist, MBS improves over base by +11.0, +13.3, +10.9 points at 16K, 24K, and 32K respectively.

Finance Results

Recovery on Original Hard Seeds

Data Quality Assessment

Three frontier judges (GPT-5.6, Opus 4.8, Gemini 3.1 Pro) assessed synthetic problems:

• Math: Only 8.0% unanimously rejected; 89.3% verified by at least one judge
• Finance: 0% rejected; 100% verified by at least one judge

Ablation Studies

---

Theoretical and Practical Implications

Theoretical Implications

1. Redefining dataset value: The paper demonstrates that corpus size alone does not determine RLVR effectiveness. The value of a problem is inherently relative to the tuple $(p, \pi, L)$—challenging the assumption that larger corpora are always better.

2. The paradox of hard problems: Problems with zero observed success provide no direct learning signal yet serve as crucial specifications of frontier capabilities. This reframes hard problems from "useless data" to "blueprints for synthesis."

3. Decomposition as a scaling mechanism: The effectivity test provides a principled way to validate that subproblems genuinely scaffold learning, connecting to pedagogical theory within the RLVR framework.

Practical Implications

1. Data synthesis guidelines: The three design principles offer actionable guidance for any RLVR data synthesis system, applicable beyond MBS itself.

2. Resource efficiency: MBS transforms 340 expert finance questions into 14K+ training problems—demonstrating dramatic data amplification in annotation-expensive domains.

3. Cross-policy transfer: MBS data trained on one policy (gpt-oss-120b) transfers effectively to another (DeepSeek v4 Flash) without regeneration, suggesting the synthesized problems capture generalizable reasoning structure rather than policy-specific artifacts.

4. Budget robustness: MBS-trained policies maintain gains even beyond the training response budget (up to 32K when trained at 16K), unlike baseline methods that degrade.

---

Conclusion

Main Takeaways

1. The zero-advantage trap is pervasive: ~88% of a large open-source math corpus provides no learning signal for a fixed policy and budget.
2. Hard problems are specifications: Their verified solutions reveal reasoning blueprints that should guide synthesis.
3. Direct prompting is insufficient: Teacher-based seed synthesis produces shallow variants and below one qualified problem per seed.
4. MagicBeanStalk escapes the trap: Top-down decomposition + bottom-up recursive synthesis converts 74.7% of hard seeds into qualified training problems (vs. 43.8% for direct synthesis), achieving +8.6 points on gpt-oss-120b and +5.4 on DeepSeek v4 Flash across 8 math benchmarks.

Future Directions

• Scaling to other domains: The MBS framework could extend beyond math and finance to code, science, and other verifiable reasoning domains.
• Synthesis cost optimization: The paper notes its focus is on data effectiveness rather than minimizing synthesis cost—future work could optimize the construction pipeline.
• Algorithmic integration: MBS is complementary to algorithmic improvements (dynamic sampling, scaffolded RL); combining both could yield further gains.
• Teacher pool dynamics: As frontier models evolve, the teacher pool composition and its impact on blueprint consistency warrant further study.

"Hard problems are not merely training examples; they can be specifications of capabilities at the frontier of the current policy."