Summary (Overview)

• DataFlex-RL is an evaluation platform for comparing reinforcement learning with verifiable rewards (RLVR) data policies under a fixed GRPO training recipe, isolating the data-policy component from the surrounding training stack.
• Primary finding: On Qwen2.5-7B-base with 12 matched seeds across 13 configurations, uniform GRPO improves accuracy by +7.76 points, but none of the 8 selection/reweighting methods shows a paired 95% CI excluding zero relative to uniform sampling, and none of the 3 adaptive mixtures improves over a fixed equal mixture.
• Evaluation sensitivity: A math-heavy 6-benchmark summary and the domain-balanced 12-benchmark summary yield negatively correlated rankings ($\rho = -0.33$) on the same runs, showing that omitting a domain (logic) can change the apparent winner.
• Cross-model check: A corrected 12-seed Llama-3.1-8B-base extension finds no common winner across the expanded method set; the base-model scale sweep (1.5B–14B) shows no monotone advantage for any method.
• Platform contribution: 591 runs with matched seeds, shared rollout/verification/optimization stack, calibration smoke tests, and a three-axis taxonomy (selection, reweighting, mixture adaptation) for future fair comparisons.

---

Introduction and Theoretical Foundation

Background

RLVR post-trains reasoning models using verifiable rewards. In Group Relative Policy Optimization (GRPO), each update:

1. Samples prompts
2. Generates $K$ responses per prompt
3. Scores them with a verifier
4. Constructs group-relative advantages

This makes the training distribution an online choice: a data policy decides which prompts receive rollout compute, how domains are mixed, which groups survive filtering, and how strongly responses contribute to the update.

Motivation

Prompts solved by all or none of the rollouts yield degenerate groups, while prompts near the decision boundary provide informative comparisons. This has motivated interventions such as:

• Solve-rate filtering (DAPO)
• High-variance down-sampling (PODS)
• Probability-based Advantage Reweighting
• Prioritized experience replay
• Adaptive domain curricula (DUMP, teacher-student curriculum learning)

The Core Problem

Existing evidence does not establish that these methods provide reproducible gains because:

• Data-processing methods are often bundled with changes to base model, prompt template, verifier, rollout budget, and optimization settings
• The same signal (e.g., advantage magnitude) can be used as a continuous weight or a hard selection rule
• On-policy signals are noisy and endogenous to training
• Training-seed variation and measurement noise can match or exceed reported gaps
• The evaluation summary (which benchmarks, how weighted) can change which method appears best

Theoretical Foundation: The Data-Policy Taxonomy

DataFlex-RL separates what a policy changes from the signal it uses:

• Selection: decides which generated responses enter the update
• Reweighting: changes continuous loss contribution of responses
• Mixture adaptation: changes which domains supply future prompts

---

Methodology

Three Intervention Families

Selection — removes responses from the update via a mask $m_{gk} \in \{0, 1\}$:

$$\mathcal{L}_{\mathrm{sel}} = \frac{1}{GK} \sum_{g=1}^{G} \sum_{k=1}^{K} \frac{m_{gk}}{L_{gk}} \sum_{\ell=1}^{L_{gk}} \ell_{gk\ell}^{\mathrm{GRPO}} \tag{1}$$

Reweighting — keeps responses but changes token weights $w_{gk\ell} \ge 0$:

$$\mathcal{L}_{\mathrm{rew}} = \frac{1}{GK} \sum_{g=1}^{G} \sum_{k=1}^{K} \frac{1}{L_{gk}} \sum_{\ell=1}^{L_{gk}} w_{gk\ell} \ell_{gk\ell}^{\mathrm{GRPO}} \tag{2}$$

Mixture adaptation — changes domain sampling probabilities $p_t(d)$:

$$\mathcal{L}_{\mathrm{mix},t}(\theta) = \sum_{d=1}^{D} p_{t}(d) \mathbb{E}_{\substack{x\sim \mathcal{D}_{d},\\ y_{1:K}\sim \pi_{\theta_{\mathrm{old}}}(\cdot|x)}}\left[\frac{1}{K}\sum_{k=1}^{K}\frac{1}{L_{k}}\sum_{\ell=1}^{L_{k}}\ell_{k\ell}^{\mathrm{GRPO}}(\theta)\right] \tag{3}$$

Key Signals

• Mean absolute advantage: $a_{gk} = \frac{1}{L_{gk}} \sum_{\ell=1}^{L_{gk}} |A_{gk\ell}|$
• Group solve rate: $q_{g} = \frac{1}{K} \sum_{k=1}^{K} \mathbb{1}[r_{gk} > 0.5]$
• Token probability $\pi_{gk\ell}$ and efficiency $r_{gk}/L_{gk}$

Implemented Configurations (Table 1)

Experimental Setup

• Corpus: 15,000 prompts (5,000 each of math, logic, science)
• Training: verl v0.5+, GRPO with $K=5$ rollouts, KL coefficient $10^{-3}$, 300 optimizer steps
• Evaluation: 12 benchmarks — 5 math (MATH-500, AIME-2024, OlympiadBench, Minerva-Math, GSM8K), 4 logic (Knights&Knaves, 2 BBH tasks, ZebraLogic), 3 science (MMLU-Pro Chem/Phys, GPQA-Diamond)
• Overall score: average of MATH, LOGIC, SCIENCE domain means (equal domain weight)
• Seeds: primary matrix uses 12 matched seeds; breadth grid uses 3
• Total runs: 591 (156-run primary block)

---

Empirical Validation / Results

Training Headroom (Table 2)

Primary Result: Selection and Reweighting (Table 4)

None of the 8 paired intervals excludes zero. The nine policy means span only 0.97 point, about one-eighth of the GRPO gain.

Primary Result: Mixture Adaptation (Table 5)

All point estimates are positive but every interval includes zero; means span only 0.61 point.

Evaluation Sensitivity (Table 8)

Math-Heavy-6 ranks topk/diffband first, while DB-12 ranks gfpo/difffilter first — the rankings are negatively correlated. Methods exhibit domain trade-offs (e.g., diffband has highest LOGIC mean 59.0 but lowest SCIENCE mean 44.2).

Base-Model Scale Sweep (Table 7)

Signs vary across models with no common pattern — no monotone advantage for any method.

---

Theoretical and Practical Implications

Theoretical Implications

1. Signals vs. interventions: The study confirms that the same signal (e.g., mean absolute advantage $a_{gk}$) can be deployed as either a selection rule (topk) or a reweighting scheme (per, softmax) — yet neither family shows reproducible gains. This suggests the signal itself, not the intervention form, may be the limiting factor.

2. On-policy endogeneity: The score attached to a prompt changes as the policy learns, making utility signals both noisy and endogenous. The null result may reflect this fundamental challenge rather than poor implementation of individual methods.

3. Evaluation-domain coverage matters: The negative correlation ($\rho = -0.33$) between Math-Heavy-6 and DB-12 rankings demonstrates that evaluation design is not a neutral measurement choice — it can invert conclusions.

Practical Implications

1. Default to uniform sampling: Under the tested recipes (300 steps, $K=5$, KL $10^{-3}$), practitioners should not expect data-policy interventions to add reproducible gains over uniform GRPO.

2. Matched-seed reporting: The study demonstrates that 3-seed comparisons can produce misleading rankings (e.g., topk leads in 3 seeds but falls below baseline with 12 seeds). At least 12 matched seeds are needed for reliable inference.

3. Evaluation coverage: Omitting a training domain (e.g., logic) from evaluation can change the apparent winner. Domain-balanced evaluation is essential for fair comparison.

4. Calibration smoke tests: Training and evaluation format mismatches can produce invalid scores without obvious training failures (as seen in the earlier Llama values that silently scored unsupported math tasks as zero).

Scope of Conclusions

The authors emphasize a scoped conclusion: the null result does not establish equivalence or rule out gains under:

• Longer training (beyond 300 steps)
• Other hyperparameters
• Different policy families
• Different base models with more headroom

---

Conclusion

Main Takeaways

1. No reproducible improvement: On Qwen2.5-7B-base, none of the 8 selection/reweighting methods or 3 adaptive mixtures shows a paired interval excluding zero relative to their controls, despite GRPO itself delivering a substantial +7.76-point gain.

2. Evaluation sensitivity is real: Benchmark coverage can change the apparent winner (negative correlation between math-heavy and domain-balanced summaries), while summaries retaining all 12 benchmarks largely agree.

3. Platform value: DataFlex-RL provides a reusable protocol — shared rollout/verification/optimization stack, matched seeds, intervention taxonomy, and run-level records — for future fair comparisons of RLVR data policies.

Future Directions

• Testing longer training horizons (e.g., 1000 steps)
• Exploring other policy families and signals
• Investigating whether gains emerge at different KL coefficients or rollout budgets
• Expanding base-model coverage to architectures with greater headroom
• Using the released 591-run records and reconstruction scripts for independent audits and new method evaluations

Final scoped conclusion: "Under the controlled recipes tested here, the evaluated selection, reweighting, and mixture policies do not provide a reproducible improvement over uniform training at the achieved precision."