# DataFlex-RL: An Evaluation Platform for RLVR Data Policies

> Under controlled GRPO training, none of 8 selection/reweighting or 3 adaptive-mixture data policies reliably beat uniform sampling, though GRPO itself boosts accuracy by 7.76 points.

- **Source:** [arXiv](https://arxiv.org/abs/2609.06107)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/CtLsZp
- **Whiteboard:** https://picx.dev/p/CtLsZp/image

## Summary

## Summary (Overview)

- **DataFlex-RL** is an evaluation platform for comparing reinforcement learning with verifiable rewards (RLVR) data policies under a fixed GRPO training recipe, isolating the data-policy component from the surrounding training stack.
- **Primary finding**: On Qwen2.5-7B-base with 12 matched seeds across 13 configurations, uniform GRPO improves accuracy by **+7.76 points**, but **none of the 8 selection/reweighting methods** shows a paired 95% CI excluding zero relative to uniform sampling, and **none of the 3 adaptive mixtures** improves over a fixed equal mixture.
- **Evaluation sensitivity**: A math-heavy 6-benchmark summary and the domain-balanced 12-benchmark summary yield **negatively correlated rankings** ($\rho = -0.33$) on the same runs, showing that omitting a domain (logic) can change the apparent winner.
- **Cross-model check**: A corrected 12-seed Llama-3.1-8B-base extension finds no common winner across the expanded method set; the base-model scale sweep (1.5B–14B) shows no monotone advantage for any method.
- **Platform contribution**: 591 runs with matched seeds, shared rollout/verification/optimization stack, calibration smoke tests, and a three-axis taxonomy (selection, reweighting, mixture adaptation) for future fair comparisons.

---

## Introduction and Theoretical Foundation

### Background

RLVR post-trains reasoning models using verifiable rewards. In **Group Relative Policy Optimization (GRPO)**, each update:
1. Samples prompts
2. Generates $K$ responses per prompt
3. Scores them with a verifier
4. Constructs group-relative advantages

This makes the training distribution an **online choice**: a data policy decides which prompts receive rollout compute, how domains are mixed, which groups survive filtering, and how strongly responses contribute to the update.

### Motivation

Prompts solved by all or none of the rollouts yield degenerate groups, while prompts near the decision boundary provide informative comparisons. This has motivated interventions such as:
- **Solve-rate filtering** (DAPO)
- **High-variance down-sampling** (PODS)
- **Probability-based Advantage Reweighting**
- **Prioritized experience replay**
- **Adaptive domain curricula** (DUMP, teacher-student curriculum learning)

### The Core Problem

Existing evidence does not establish that these methods provide **reproducible gains** because:
- Data-processing methods are often bundled with changes to base model, prompt template, verifier, rollout budget, and optimization settings
- The same signal (e.g., advantage magnitude) can be used as a continuous weight or a hard selection rule
- On-policy signals are **noisy and endogenous** to training
- Training-seed variation and measurement noise can match or exceed reported gaps
- The **evaluation summary** (which benchmarks, how weighted) can change which method appears best

### Theoretical Foundation: The Data-Policy Taxonomy

DataFlex-RL separates **what a policy changes** from **the signal it uses**:

- **Selection**: decides which generated responses enter the update
- **Reweighting**: changes continuous loss contribution of responses
- **Mixture adaptation**: changes which domains supply future prompts

---

## Methodology

### Three Intervention Families

**Selection** — removes responses from the update via a mask $m_{gk} \in \{0, 1\}$:

$$
\mathcal{L}_{\mathrm{sel}} = \frac{1}{GK} \sum_{g=1}^{G} \sum_{k=1}^{K} \frac{m_{gk}}{L_{gk}} \sum_{\ell=1}^{L_{gk}} \ell_{gk\ell}^{\mathrm{GRPO}} \tag{1}
$$

**Reweighting** — keeps responses but changes token weights $w_{gk\ell} \ge 0$:

$$
\mathcal{L}_{\mathrm{rew}} = \frac{1}{GK} \sum_{g=1}^{G} \sum_{k=1}^{K} \frac{1}{L_{gk}} \sum_{\ell=1}^{L_{gk}} w_{gk\ell} \ell_{gk\ell}^{\mathrm{GRPO}} \tag{2}
$$

**Mixture adaptation** — changes domain sampling probabilities $p_t(d)$:

$$
\mathcal{L}_{\mathrm{mix},t}(\theta) = \sum_{d=1}^{D} p_{t}(d) \mathbb{E}_{\substack{x\sim \mathcal{D}_{d},\\ y_{1:K}\sim \pi_{\theta_{\mathrm{old}}}(\cdot|x)}}\left[\frac{1}{K}\sum_{k=1}^{K}\frac{1}{L_{k}}\sum_{\ell=1}^{L_{k}}\ell_{k\ell}^{\mathrm{GRPO}}(\theta)\right] \tag{3}
$$

### Key Signals

- **Mean absolute advantage**: $a_{gk} = \frac{1}{L_{gk}} \sum_{\ell=1}^{L_{gk}} |A_{gk\ell}|$
- **Group solve rate**: $q_{g} = \frac{1}{K} \sum_{k=1}^{K} \mathbb{1}[r_{gk} > 0.5]$
- **Token probability** $\pi_{gk\ell}$ and **efficiency** $r_{gk}/L_{gk}$

### Implemented Configurations (Table 1)

| Family | Method | Signal | Intervention |
|---|---|---|---|
| Baseline | `baseline` | none | $w_{i\ell} = 1$ for all tokens |
| Selection | `difffilter` | $q_g$ | retain group iff $0.2 < q_g < 0.8$ |
| Selection | `maxvar` | rewards | retain subset maximizing reward variance |
| Selection | `gfpo` | $r_i/L_i$ | retain top 3 of 5 responses |
| Selection | `topk` | $a_i$ | retain top 50% of responses |
| Reweighting | `ar` | $\pi_{i\ell}$ | $w_{i\ell} \propto 0.5\pi_{i\ell} + 0.5$ |
| Reweighting | `per` | $a_i$ | $w_i \propto (a_i + \epsilon)^{0.5}$ |
| Reweighting | `softmax` | $a_i$ | $w_i \propto \exp(a_i/T)$, $T=1$ |
| Reweighting | `diffband` | $r_i$ | 2× weight between quartiles |
| Mixture | `static` | ignored | $p_t(d) = 1/3$ |
| Mixture | `reward_gap` | $\bar{r}_d$ | favors lagging domains |
| Mixture | `dump_ucb` | $\bar{a}_d, n_d$ | UCB-style exploration |
| Mixture | `tscl` | $s_d$ | favors domains with learning progress |

### Experimental Setup

- **Corpus**: 15,000 prompts (5,000 each of math, logic, science)
- **Training**: verl v0.5+, GRPO with $K=5$ rollouts, KL coefficient $10^{-3}$, 300 optimizer steps
- **Evaluation**: 12 benchmarks — 5 math (MATH-500, AIME-2024, OlympiadBench, Minerva-Math, GSM8K), 4 logic (Knights&Knaves, 2 BBH tasks, ZebraLogic), 3 science (MMLU-Pro Chem/Phys, GPQA-Diamond)
- **Overall score**: average of MATH, LOGIC, SCIENCE domain means (equal domain weight)
- **Seeds**: primary matrix uses 12 matched seeds; breadth grid uses 3
- **Total runs**: 591 (156-run primary block)

---

## Empirical Validation / Results

### Training Headroom (Table 2)

| Base Model | Seeds | Untrained | Uniform GRPO | Δ | 95% CI |
|---|---|---|---|---|---|
| Qwen2.5-7B-base | 12 | 42.01 | 49.77 | **+7.76** | [7.28, 8.25] |
| Llama-3.1-8B-base | 12 | 10.87 | 21.12 | **+10.25** | [7.75, 12.33] |

### Primary Result: Selection and Reweighting (Table 4)

| Method | Mean Score | Paired Δ | 95% CI |
|---|---|---|---|
| baseline | 49.77 | — | — |
| difffilter | 49.85 | +0.08 | [-0.67, 0.82] |
| maxvar | 49.19 | -0.58 | [-1.39, 0.23] |
| gfpo | 48.88 | -0.90 | [-1.87, 0.08] |
| topk | 49.39 | -0.38 | [-1.14, 0.38] |
| ar | 49.50 | -0.28 | [-1.15, 0.60] |
| per | 48.92 | -0.86 | [-3.28, 1.57] |
| softmax | 49.54 | -0.23 | [-1.44, 0.98] |
| diffband | 49.75 | -0.02 | [-0.61, 0.57] |

**None of the 8 paired intervals excludes zero.** The nine policy means span only **0.97 point**, about one-eighth of the GRPO gain.

### Primary Result: Mixture Adaptation (Table 5)

| Method | Mean Score | Paired Δ | 95% CI |
|---|---|---|---|
| static | 49.00 | — | — |
| reward_gap | 49.46 | +0.45 | [-0.09, 1.00] |
| dump_ucb | 49.61 | +0.61 | [-0.02, 1.25] |
| tscl | 49.16 | +0.16 | [-0.61, 0.93] |

All point estimates are positive but **every interval includes zero**; means span only 0.61 point.

### Evaluation Sensitivity (Table 8)

| Evaluation Summary | Rank ρ vs DB-12 | Method Spread |
|---|---|---|
| DB-12 (domain-balanced) | 1.00 | 3.31 |
| Macro-12 | 0.88 | 2.79 |
| Item-12 | 0.88 | 2.17 |
| Math-Heavy-6 | **-0.33** | 0.90 |

Math-Heavy-6 ranks `topk`/`diffband` first, while DB-12 ranks `gfpo`/`difffilter` first — the rankings are **negatively correlated**. Methods exhibit domain trade-offs (e.g., `diffband` has highest LOGIC mean 59.0 but lowest SCIENCE mean 44.2).

### Base-Model Scale Sweep (Table 7)

| Base Model | Uniform baseline | difffilter Δ | maxvar Δ | softmax Δ | tscl Δ |
|---|---|---|---|---|---|
| Qwen2.5-1.5B | 28.14 | -0.83 | -1.18 | -0.31 | -0.10 |
| Qwen2.5-3B | 37.31 | -0.79 | -0.55 | -0.53 | +0.23 |
| Qwen2.5-7B | 49.77 | +0.08 | -0.58 | -0.24 | +0.16 |
| Qwen2.5-14B | 59.38 | +1.20 | -0.84 | -0.24 | +1.43 |
| Llama-3.2-3B | 13.16 | -0.36 | -1.69 | +0.29 | +0.22 |

Signs vary across models with no common pattern — no monotone advantage for any method.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Signals vs. interventions**: The study confirms that the same signal (e.g., mean absolute advantage $a_{gk}$) can be deployed as either a selection rule (`topk`) or a reweighting scheme (`per`, `softmax`) — yet neither family shows reproducible gains. This suggests the **signal itself**, not the intervention form, may be the limiting factor.

2. **On-policy endogeneity**: The score attached to a prompt changes as the policy learns, making utility signals both noisy and endogenous. The null result may reflect this fundamental challenge rather than poor implementation of individual methods.

3. **Evaluation-domain coverage matters**: The negative correlation ($\rho = -0.33$) between Math-Heavy-6 and DB-12 rankings demonstrates that evaluation design is not a neutral measurement choice — it can invert conclusions.

### Practical Implications

1. **Default to uniform sampling**: Under the tested recipes (300 steps, $K=5$, KL $10^{-3}$), practitioners should not expect data-policy interventions to add reproducible gains over uniform GRPO.

2. **Matched-seed reporting**: The study demonstrates that 3-seed comparisons can produce misleading rankings (e.g., `topk` leads in 3 seeds but falls below baseline with 12 seeds). At least 12 matched seeds are needed for reliable inference.

3. **Evaluation coverage**: Omitting a training domain (e.g., logic) from evaluation can change the apparent winner. Domain-balanced evaluation is essential for fair comparison.

4. **Calibration smoke tests**: Training and evaluation format mismatches can produce invalid scores without obvious training failures (as seen in the earlier Llama values that silently scored unsupported math tasks as zero).

### Scope of Conclusions

The authors emphasize a **scoped conclusion**: the null result does not establish equivalence or rule out gains under:
- Longer training (beyond 300 steps)
- Other hyperparameters
- Different policy families
- Different base models with more headroom

---

## Conclusion

### Main Takeaways

1. **No reproducible improvement**: On Qwen2.5-7B-base, none of the 8 selection/reweighting methods or 3 adaptive mixtures shows a paired interval excluding zero relative to their controls, despite GRPO itself delivering a substantial +7.76-point gain.

2. **Evaluation sensitivity is real**: Benchmark coverage can change the apparent winner (negative correlation between math-heavy and domain-balanced summaries), while summaries retaining all 12 benchmarks largely agree.

3. **Platform value**: DataFlex-RL provides a reusable protocol — shared rollout/verification/optimization stack, matched seeds, intervention taxonomy, and run-level records — for future fair comparisons of RLVR data policies.

### Future Directions

- Testing longer training horizons (e.g., 1000 steps)
- Exploring other policy families and signals
- Investigating whether gains emerge at different KL coefficients or rollout budgets
- Expanding base-model coverage to architectures with greater headroom
- Using the released 591-run records and reconstruction scripts for independent audits and new method evaluations

> **Final scoped conclusion**: "Under the controlled recipes tested here, the evaluated selection, reweighting, and mixture policies do not provide a reproducible improvement over uniform training at the achieved precision."

---

_Markdown view of https://picx.dev/p/CtLsZp, served by PicX — AI-generated visual whiteboard summaries of research papers._