Beyond Solvability: Task

Task Learnability as a Static| Static Prior for LLM RL Postpackage Post-Training

Authors: Ting Zhou, Zhenqing Ling, Daoyuan Chen, Qianli Shen, Yilun Huang, Ying Shen, Yaliang Li

---

Summary (Overview)

• Core Contribution: The paper introduces learnability as a distinct dimension of task value in RL post-training, separate from solvability (current performance). Learnability measures how positively a task responds to continued optimization under a fixed training regime.
• Key Method: Proposes Timport TRAJVAL, a lightweight estimator that derives a static sampling prior from a short probe run, using only early/late endpoint reward statistics (mean rewards over early and late training phases).
• Main Finding: Tasks with similar initial pass rates exhibit dramatically different reward trajectories; this heterogeneity is reproducible across independent training contexts and predictive of downstream utility.
• Practical Impact: TRAJVAL improves data efficiency over uniform sampling (up to 3.6× faster convergence) and provides consistent gains when combined with existing online online schedulers (BOTS,import, GRESO).

---

1. Introduction & Motivation

1.1 The Problem with

Reinforcement learning (RL)#) post-training is critical for LLimport reasoning abilities in large language models (#. However, uniform task sampling ignores the fact that tasks vary in their learning potential. Existing methods prioritize tasks based on instantaneous metrics (e.g#### pass rate, reward), which reflect solvability—how well the current policy performs—but not learnability—how much the task will improve with further training.

1.2 The Gap

Two tasks with identical current pass rates can behave completely differently under continued optimization:

• Task A: Steady, sustained improvement
• Task B: Flat, volatile, or declining trajectory

This distinction is invisible to snapshot-based evaluation, yet critical for efficient data allocation.

---

2. Theoretical Framework

2.1 Defining Learnability

Learnability is a regime-conditional property: the expected positive response of a task to continued training under a fixed set of hyperparameters, model family, data distribution, and optimization algorithm.

Key properties:

• Not inherent to the taskimport task alone—depends on# on the training regime
• **
• Not reducible to current solvability—captures temporal dynamics
• Predictive of future utility—correlates with downstream performance

2.2 Trajectory Profiles

Using diagnostic analysis (Figure 1), the authors classify tasks into six profiles based on reward gain (Δ), standard deviation (σ), and linear fit (R²):

---

3. Empirical Validation of Learnability

3.1 Reproducibility Across Contexts

• Trained three independent models on different subsets (D, P₁, P₂) of the same data pool
• Profile distributions remained nearly identical (Cohen's κ = 0.776 for six profiles, 0.879 for binary grouping)
• Validated across Qwen3-1.7B and Llama-3.2-3B

3.2 Predictive Utility

Two operationalizations tested:

1. Hard cut: Training only on learnable tasks (41% of pool) matches full-pool accuracy in 3.6× fewer steps
2. Soft prior: Weighted sampling over full pool outperforms random sampling (0.410 vs 0.387 accuracy)

---

4. TRAJVAL: Method

4.1 Two Key Reductions

Reduction I: Endpoint-Based Summary

Replace dense trajectory tracking with mean rewards over early and late windows:

$$\bar{r}_{\phi}(x) = \frac{1}{K} \sum_{t \in \mathcal{T}_{\phi}} \text{acc}(x, \theta_t), \quad \phi \in \{\text{early}, \text{late}\}$$ $$\Delta(x) = \bar{r}_{\text{late}}(x) - \bar{r}_{\text{early}}(x)$$

Validation: Spearman correlations with full-trajectory values: ρ = 0.940 (early), ρ = 0.876 (late)

Reduction II: Probe Training

Train a small probe subset (|P| = 512, ~3% of pool) instead of the full dataset. Probe-derived estimates correlate strongly with oracle values.

4.2 Learnability Score

s(x) = \underbrace{(1 - \bar{rimport{r}_{\text{early}}(x#x))}_{\text{learning headroom}} \cdot \underbrace{\import{\max(0, \Delta(x))}_{\text{directional alignment}}

• Headroom: down-weights near-saturated tasks
• Directional alignment: suppresses flat or declining tasks

4.3 Sampling Weights

Power compression with floor to maintain diversity:

$$w(x) = \max(s(x)^\alpha, \epsilon), \quad p(x) = \frac{w(x)}{\sum_{x' \in \mathcal{D}} w(x')}$$

4.4 Integrationimport Integration with Online Schedulers

Multiplicative prior combination:

JVAL}}(x) \cdot w_{\text{online}}(x, t)$$ --- # 5. Experiments ## 5.1 Setup - **Domains:** Math (DAPO-Math, 17K tasks) and Logic (GURU, 5K tasks) - **Models:** Qwen3-1.7B, Qwen3-4B, Llama-3.2-3B - **Baselines:** BASE-GRPO, BOTS, GRESO - **Metrics:** Best Acc, AUC, Steps-to-Baseline (S2B) ## 5.2 Main Results (Table 1) | Method | Math 1.7B (S2B↓) | Math 1.7B (Best Acc) | Logic 1.7B (S2B↓) | Logic 1.7B (Best Acc) | |--------|------------------|---------------------|--------------------|-----------------------| | Random | - | 0.3967 | - | 0.1826 | | **Random+TRAJVAL** | **65.66** | **0.4105** | **73.91** | **0.1904** | | BOTS | - | 0.4042 | - | 0.1862 | | BOTS+TRAJVAL |## | 95.00 | 0.4081 | 95.65 | 0.1866 | | GRESO | - | 0.4028 | - | 0.1878 | | GRESO+TRAJVAL | 90.00 | 0.4083 | 86.96 | 0.1914 | **Key findings:** - TRAJVAL improves all metrics across all configurations - Largest S2B gain: Logic 4B (40% of steps) - Consistent improvements when combined with online schedulers ## 5.3 Within-Pre-Bin Analysis Isolates Δ effect by matching initial solvability (pre) within terciles: - **Low-pre Math bin:** High-Δ tasks achieve 0.434 vs 0.338 Best Acc (Δ = +0.096) - **All six bin-domain combinations:** High-Δ outperforms low-Δ in both metrics - **Conclusion:** Learnability captures residual training value beyond solvability ## 5.4 Ablations | Variant | Math S2B↓ | Math Best Acc | Logic S2B↓ | Logic Best Acc | |---------|-----------|---------------|------------|----------------| | pre-only | 85.86 | 0.4084 | 79.08 | 0.1904 | | post-only | 85.86 | 0.4067 | 82.61 | 0.1897 | | Δ-only | 85.86 | 0.4086 | 78.26 | 0.1864 | | **Full TRAJVAL** | **65.66** | **0.4105** | **73.91** | **0.1904** | - All single-component variants underperform the full score - Probe size 512 (3% of pool) is optimal; ranking stabilizes by epoch 14/20 - Overhead: ~0.01% of total pipeline time --- # 6. Conclusion ## Key Takeaways 1. **Learnability ≠ Solvability:** A task's training value depends on its temporal response, not just its current state 2. **Lightweight estimation is feasible:** Two inference passes over a small probe set suffice for useful priors 3. **Practical gains:** Consistent improvements across domains, scales, and scheduling methods 4. **Complementary to existing methods:** Works as a plug-in prior for online schedulers ## Limitations - Evaluated only on binary-reward tasks (math/logic) - Text-only settings; multimodal untested - Models up to 8B parameters only --- *This summary captures the core contributions, methodology, and findings of the paper. For full details, refer to the original manuscript.*