CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards

Summary (Overview)

• Core contribution: The paper introduces Circuit Reasoning Score (CRS), a training-free, model-internal data selection signal for Reinforcement Learning with Verifiable Rewards (RLVR), computed from 46 reasoning-sensitive attention heads identified via contrastive ablation on a frozen base model.
• Counter-intuitive finding: Against the natural hypothesis that higher reasoning-circuit engagement produces better training data, CRS-bottom selection (lowest-engagement decile) improves over random selection on three medium-difficulty benchmarks (GSM8K, Olympiad-Bench, Minerva Math) by +2.0, +1.6, and +2.9 percentage points respectively on Qwen2.5-Math-7B.
• Regime-dependent effects: RLVR data selection effects are modulated by three factors—pool diversity, model capacity, and evaluation difficulty—rather than reducible to a static ranking of problem quality.
• Key baselines validated: CRS-bottom's advantage is not reducible to difficulty filtering, reward-trajectory selection, arbitrary head signals, data volume, or CRS magnitude.
• Training reward does not predict generalization: The correlation between training reward and downstream accuracy is r = -0.18, with the highest-reward condition underperforming random selection on AIME.

Introduction and Theoretical Foundation

Background and Motivation

Reinforcement Learning with Verifiable Rewards (RLVR) is sensitive to which problems a model trains on, yet existing selection criteria—difficulty filtering, hand-curation, reward-trajectory scoring—treat data value as an intrinsic property of problems, independent of the model that will learn from them. The authors argue that a selection signal reflecting the relationship between a problem and the model's current reasoning state could provide more principled guidance.

Theoretical Basis

The paper builds on three research threads:

1. RLVR for mathematical reasoning: Methods like GRPO (Shao et al., 2024) and DeepSeek-R1 use within-group reward variance to train chain-of-thought reasoning. A key property is that prompts whose completions all receive identical rewards provide no gradient signal, making prompt selection central.

2. Data selection for RLVR: Prior work uses reward-trajectory alignment (LIMR, Li et al., 2025), difficulty filtering (Lin et al., 2024), or dataset scaling (Yu et al., 2025). These methods are model-agnostic.

3. Circuit-based data selection: Mechanistic interpretability has identified task-specific heads for induction (Olsson et al., 2022), factual recall (Meng et al., 2022), and reasoning (Elhage et al., 2021). Prior work applied circuit signals to SFT data selection via attention-pattern variance, but the authors extend this to RLVR with contrastive ablation and test both CRS directions.

Central Hypothesis (Tested and Rejected)

If certain attention heads specialize in mathematical reasoning, then how strongly a problem engages those heads should reflect the problem's relationship to the model's reasoning capacity.

The paper systematically tests whether higher engagement (CRS-top) produces better training data. The results show this intuition does not hold.

Methodology

Stage 1: Identifying Reasoning-Sensitive Heads

The base model is Qwen2.5-Math-7B with 784 attention heads $(28 \times 28, d_h = 128)$. To isolate heads specifically important for reasoning (not just general language modeling), the authors use contrastive ablation:

• 50 reasoning probes (Olympiad-style) vs. 50 trivial probes (single-step arithmetic), held out from all training and evaluation data
• Compare LM loss increases on reasoning vs. trivial probes when each head is zero-ablated
• Excluding one global-bottleneck outlier (L0H3, score $4.8\times$ the runner-up), the final set $H_r$ contains 46 heads spanning 21 of 28 layers (bootstrap Jaccard = 0.91)

Stage 2: Computing CRS

For each problem $x$ with $n$ tokens, a single forward pass records each reasoning head's output activation at every position. The score takes the maximum activation across positions:

$$\text{CRS}(x) = \max_{i \in H_r} \max_{j \leq n} \|a_{i,j}(x)\|$$

Taking the maximum captures peak reasoning engagement rather than diluting signal across boiler-plate tokens.

Length Residualization

Since length is a $2.8\times$ stronger difficulty predictor than CRS ($R^2 = 4.36\%$ vs. $1.57\%$), the authors residualize via OLS on log $n$:

$$\text{CRS}_{\text{resid}}(x) = \text{CRS}(x) - \hat{\beta}_0 - \hat{\beta}_1 \log n$$

Training Setup

• Algorithm: GRPO via TRL v1.3.0, following DeepSeek-AI (2025) defaults, $G = 16$ completions per problem
• Models: Qwen2.5-Math-7B and Qwen2.5-Math-1.5B
• Pool: 16,478 problems (Tier 1 diverse pool); 10% subsets (≈1,648 problems) for main comparisons
• Baselines: Random, CRS-top, CRS-middle, CRS-bottom, RandHeads (46 random heads), LIMR (reward-trajectory), Difficulty-filtering, Full-data (17K problems)

Benchmarks

Empirical Validation / Results

Main Results (7B, Tier 1 Diverse Pool)

The paper reports 7 conditions × 5 benchmarks (mean ± std across seeds). Key findings:

• CRS-bottom outperforms Random on GSM8K (+2.0 pp), Olympiad-Bench (+1.6 pp), and Minerva Math (+2.9 pp)
• CRS-top shows no separable advantage over Random on any benchmark; its margin is not distinguishable from CRS-middle
• On MATH-500, no Tier 1 condition separates from Random, consistent with a ceiling effect at 66.8% base accuracy
• On AIME (hardest tier), CRS-top exceeds LIMR by +1.7 pp, while LIMR trails Random by -1.6 pp

CRS Is Not a Difficulty Proxy

A natural concern is that CRS simply recapitulates problem difficulty. The authors test this directly:

The regression line (Spearman $\rho = 0.125$, $R^2 = 1.57\%$) is barely visible against the spread. CRS and difficulty rank problems in nearly independent orders.

Token length explains 4.36% of CRS variance—a 2.8× stronger predictor than base-model success rate.

Training Reward Does Not Predict Generalization

The highest-reward condition (LIMR, reaching 0.775 training reward) underperforms Random on AIME. Across all runs:

$$r = -0.18 \text{ (correlation between training reward and downstream accuracy)}$$

Training reward measures how well the model solves its training problems; it does not measure whether the model has learned transferable skills.

Scale Effects (1.5B Results)

At 1.5B scale, the effect reverses: CRS-bottom trails CRS-top by 3.0 pp on MATH-500 (vs. -0.6 pp at 7B). The authors explain this via reward-variance mechanics:

• At 7B, both CRS-top and CRS-bottom problems lie within the regime of useful reward variance, but CRS-bottom provides more novel reasoning patterns when successes occur
• At 1.5B, CRS-bottom problems fall below the threshold where the weaker model can generate correct completions, yielding predominantly zero-advantage steps

Pool Diversity Modulates Effects

• Diverse pool: CRS stratification separates grade-school arithmetic from Olympiad reasoning, and training outcomes diverge
• Homogeneous pool: CRS stratification produces subsets that differ in identity but converge in outcome, consistent with adaptive-environment work showing narrow distributions lose learning signal

Control Experiments

The paper includes extensive controls:

• RandHeads (46 random heads): Tests whether the specific 46-head identity matters; the identified heads provide a more consistent signal
• Post-training ablation: Six layer-matched random head sets approach the question causally; zero-ablating the CRS heads removes the CRS-bottom advantage but is equally destructive to the untrained base model, indicating a pathway already present in the base model
• Cross-model transfer ($\mathrm{xmod}$): Data selected by 7B's CRS, used to train 1.5B, showing CRS subsets can be reused across moderate scale shifts

Cost Asymmetry

Difficulty filtering required 65,912 generations, against a single forward pass per problem for CRS—a substantial computational advantage.

Theoretical and Practical Implications

Theoretical Implications

1. Data quality is not problem-intrinsic: The training value of a problem for RLVR depends on the model-data regime, not just the problem itself.

2. Selection direction is regime-dependent: The optimal CRS direction shifts continuously with model capacity—CRS-bottom at 7B, CRS-top at 1.5B. This predicts a continuous shift with intermediate-scale models, left for future verification.

3. Exploration vs. exploitation: CRS-bottom forces exploration of reasoning patterns outside the model's existing repertoire, and the resulting skills transfer effectively to evaluation benchmarks. This echoes weak-to-strong filtering results.

4. Reward optimization is not capability optimization: The negative correlation between training reward and downstream accuracy (r = -0.18) suggests that optimizing for training reward can be actively harmful for generalization.

Practical Implications

1. For practitioners in comparable settings: On a diverse pool with a 7B-scale model, CRS-bottom improved medium-difficulty reasoning in the reported runs.

2. Computational efficiency: CRS requires only a single forward pass per problem, compared to thousands of generations for reward-trajectory methods.

3. Selection should be matched to target: No single direction dominates all evaluation regimes—practitioners should match selection direction to their target benchmark difficulty.

4. Caution on intermediate-difficulty optimization: Selection criteria optimized for intermediate difficulty do not transfer to the hardest evaluation tier (AIME).

Conclusion

Within the Qwen2.5-Math settings tested, the training value of a problem for RLVR does not appear to be an intrinsic property of the problem. CRS serves less as a universal selector and more as a probe that exposes how selection effects depend on context:

• Appears on a diverse pool but not detectable on a curated one
• Favours low-engagement data at 7B but high-engagement data at 1.5B
• These interactions are invisible to criteria that treat data quality as problem-intrinsic

Limitations and Future Directions

• All experiments use the Qwen2.5-Math family at 7B and 1.5B, on mathematical data only; generalization to other architectures and verifiable-reward domains is untested
• CRS is correlational: activation magnitudes at heads identified via ablation, presented as a selection-level predictive signal rather than a verified reasoning circuit
• The prediction that optimal CRS direction shifts continuously with model capacity awaits verification with intermediate-scale models

Practitioners should not optimize data selection for training reward: in RLVR, the conditions that look worst during training may generalize best.