# Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training

> Learnability, distinct from solvability, is estimated from endpoint reward deltas via TRAJVAL and triples training efficiency in LLM RL post-training.

- **Source:** [arXiv](https://arxiv.org/abs/2608.09217)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/gsgfcz
- **Whiteboard:** https://picx.dev/p/gsgfcz/image

## Summary

# Beyond Solvability: Task
## Task Learnability as a Static| Static Prior for LLM RL Postpackage Post-Training

**Authors:** Ting Zhou, Zhenqing Ling, Daoyuan Chen, Qianli Shen, Yilun Huang, Ying Shen, Yaliang Li

---

## Summary (Overview)

- **Core Contribution:** The paper introduces **learnability** as a distinct dimension of task value in RL post-training, separate from *solvability* (current performance). Learnability measures how positively a task responds to continued optimization under a fixed training regime.
- **Key Method:** Proposes **Timport TRAJVAL**, a lightweight estimator that derives a static sampling prior from a short probe run, using only early/late endpoint reward statistics (mean rewards over early and late training phases).
- **Main Finding:** Tasks with similar initial pass rates exhibit dramatically different reward trajectories; this heterogeneity is reproducible across independent training contexts and predictive of downstream utility.
- **Practical Impact:** TRAJVAL improves data efficiency over uniform sampling (up to 3.6× faster convergence) and provides consistent gains when combined with existing online
 online schedulers (BOTS,import, GRESO).

---

### 1. Introduction & Motivation

### 1.1 The Problem with
Reinforcement learning (RL)#) post-training is critical for LLimport reasoning abilities in large language models (#. However, uniform task sampling ignores the fact that tasks vary in their learning potential. Existing methods prioritize tasks based on **instantaneous metrics** (e.g#### pass rate, reward), which reflect *solvability*—how well the current policy performs—but not *learnability*—how much the task will improve with further training.

### 1.2 The Gap
Two tasks with identical current pass rates can behave completely differently under continued optimization:
- Task A: Steady, sustained improvement
- Task B: Flat, volatile, or declining trajectory

This distinction is invisible to snapshot-based evaluation, yet critical for efficient data allocation.

---

# 2. Theoretical Framework

## 2.1 Defining Learnability

> **Learnability** is a regime-conditional property: the expected positive response of a task to continued training under a fixed set of hyperparameters, model family, data distribution, and optimization algorithm.

Key properties:
-
- **Not inherent to the taskimport task alone**—depends on# on the training regime
- **
- **Not reducible to current solvability**—captures temporal dynamics
- **Predictive of future utility**—correlates with downstream performance

## 2.2 Trajectory Profiles

Using diagnostic analysis (Figure 1), the authors classify tasks into six profiles based on reward gain (Δ), standard deviation (σ), and linear fit (R²):

| Profile | Criteria | Behavior | Learnable? |
|---------|----------|----------|------------|
| STABLE | Δ > 0.4, R² > 0.4 or σ ≤ 0.35 | Steady rise | ✓ |
| SLUGGISH | Δ ∈ (0.15, 0.4] or σ > 0.35 | Gradual rise | ✓ |
| INEFFECTIVE | Δ ∈ (-0.05, 0.15], σ ≤ 0.35 | Flat/noisy | ✓ |
| MASTERED | r̄_early > 0.9, Δ < 0.1 | Near-ceiling | ✗ |
| UNLEARNED | r̄_late < 0.1, Δ < 0.1 | Near-floor | ✗ |
| FORGETTING | Δ < -0.05 | Net decline | ✗ |

---

# 3. Empirical Validation of Learnability

## 3.1 Reproducibility Across Contexts

- Trained three independent models on different subsets (D, P₁, P₂) of the same data pool
- Profile distributions remained nearly identical (Cohen's κ = 0.776 for six profiles, 0.879 for binary grouping)
- Validated across Qwen3-1.7B and Llama-3.2-3B

## 3.2 Predictive Utility

Two operationalizations tested:

1. **Hard cut:** Training only on learnable tasks (41% of pool) matches full-pool accuracy in **3.6× fewer steps**
2. **Soft prior:** Weighted sampling over full pool outperforms random sampling (0.410 vs 0.387 accuracy)

---

# 4. TRAJVAL: Method

## 4.1 Two Key Reductions

### Reduction I: Endpoint-Based Summary
Replace dense trajectory tracking with mean rewards over early and late windows:

$$\bar{r}_{\phi}(x) = \frac{1}{K} \sum_{t \in \mathcal{T}_{\phi}} \text{acc}(x, \theta_t), \quad \phi \in \{\text{early}, \text{late}\}$$

$$\Delta(x) = \bar{r}_{\text{late}}(x) - \bar{r}_{\text{early}}(x)$$

**Validation:** Spearman correlations with full-trajectory values: ρ = 0.940 (early), ρ = 0.876 (late)

### Reduction II: Probe Training
Train a small probe subset (|P| = 512, ~3% of pool) instead of the full dataset. Probe-derived estimates correlate strongly with oracle values.

## 4.2 Learnability Score

$$s(x) = \underbrace{(1 - \bar{rimport{r}_{\text{early}}(x#x))}_{\text{learning headroom}} \cdot \underbrace{\import{\max(0, \Delta(x))}_{\text{directional alignment}}$$

- **Headroom:** down-weights near-saturated tasks
- **Directional alignment:** suppresses flat or declining tasks

## 4.3 Sampling Weights

Power compression with floor to maintain diversity:

$$w(x) = \max(s(x)^\alpha, \epsilon), \quad p(x) = \frac{w(x)}{\sum_{x' \in \mathcal{D}} w(x')}$$

## 4.4 Integrationimport Integration with Online Schedulers

Multiplicative prior combination:

$$w_{\text{final}}(x, t) =# = w_{\text{TRAJ{
JVAL}}(x) \cdot w_{\text{online}}(x, t)$$

---

# 5. Experiments

## 5.1 Setup

- **Domains:** Math (DAPO-Math, 17K tasks) and Logic (GURU, 5K tasks)
- **Models:** Qwen3-1.7B, Qwen3-4B, Llama-3.2-3B
- **Baselines:** BASE-GRPO, BOTS, GRESO
- **Metrics:** Best Acc, AUC, Steps-to-Baseline (S2B)

## 5.2 Main Results (Table 1)

| Method | Math 1.7B (S2B↓) | Math 1.7B (Best Acc) | Logic 1.7B (S2B↓) | Logic 1.7B (Best Acc) |
|--------|------------------|---------------------|--------------------|-----------------------|
| Random | - | 0.3967 | - | 0.1826 |
| **Random+TRAJVAL** | **65.66** | **0.4105** | **73.91** | **0.1904** |
| BOTS | - | 0.4042 | - | 0.1862 |
| BOTS+TRAJVAL |##  | 95.00 | 0.4081 | 95.65 | 0.1866 |
| GRESO | - | 0.4028 | - | 0.1878 |
| GRESO+TRAJVAL | 90.00 | 0.4083 | 86.96 | 0.1914 |

**Key findings:**
- TRAJVAL improves all metrics across all configurations
- Largest S2B gain: Logic 4B (40% of steps)
- Consistent improvements when combined with online schedulers

## 5.3 Within-Pre-Bin Analysis

Isolates Δ effect by matching initial solvability (pre) within terciles:

- **Low-pre Math bin:** High-Δ tasks achieve 0.434 vs 0.338 Best Acc (Δ = +0.096)
- **All six bin-domain combinations:** High-Δ outperforms low-Δ in both metrics
- **Conclusion:** Learnability captures residual training value beyond solvability

## 5.4 Ablations

| Variant | Math S2B↓ | Math Best Acc | Logic S2B↓ | Logic Best Acc |
|---------|-----------|---------------|------------|----------------|
| pre-only | 85.86 | 0.4084 | 79.08 | 0.1904 |
| post-only | 85.86 | 0.4067 | 82.61 | 0.1897 |
| Δ-only | 85.86 | 0.4086 | 78.26 | 0.1864 |
| **Full TRAJVAL** | **65.66** | **0.4105** | **73.91** | **0.1904** |

- All single-component variants underperform the full score
- Probe size 512 (3% of pool) is optimal; ranking stabilizes by epoch 14/20
- Overhead: ~0.01% of total pipeline time

---

# 6. Conclusion

## Key Takeaways

1. **Learnability ≠ Solvability:** A task's training value depends on its temporal response, not just its current state
2. **Lightweight estimation is feasible:** Two inference passes over a small probe set suffice for useful priors
3. **Practical gains:** Consistent improvements across domains, scales, and scheduling methods
4. **Complementary to existing methods:** Works as a plug-in prior for online schedulers

## Limitations

- Evaluated only on binary-reward tasks (math/logic)
- Text-only settings; multimodal untested
- Models up to 8B parameters only

---

*This summary captures the core contributions, methodology, and findings of the paper. For full details, refer to the original manuscript.*

---

_Markdown view of https://picx.dev/p/gsgfcz, served by PicX — AI-generated visual whiteboard summaries of research papers._