# CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

> CodeLift's behavior-balanced test weighting and snapshot slicing deliver denser, better-calibrated rewards, improving SWE-bench Verified pass@1 by 3.67 points over binary reward baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2610.17224)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/McXEhQ
- **Whiteboard:** https://picx.dev/p/McXEhQ/image

## Summary

## Summary (Overview)

- **CodeLift** is a reinforcement learning method for software engineering (SWE) agents that dynamically calibrates pass-rate rewards by treating the test suite as an adaptive curriculum, addressing three identified imbalances in raw pass-rate rewards: multiplicity, progress-regression, and difficulty imbalances.
- The method introduces **behavior balancing**, which groups tests and patches by their outcome patterns and weights test groups by the fraction of patch groups they "kill" (fail), naturally reducing the influence of trivial all-passing tests while emphasizing challenging ones.
- CodeLift also introduces **snapshot slicing**, which applies the same scoring rule to intermediate repository states at self-test checkpoints, enabling credit assignment for useful intermediate actions even when later edits introduce regressions.
- Experiments on SWE-bench Verified and SWE-bench Pro (Python) show consistent improvements over binary and enhanced pass-rate baselines across pass@1/2/4/8, with gains ranging from +2.75 to +7.40 percentage points on Verified.
- Analyses demonstrate that CodeLift provides denser learning signals (75.1% active rollout groups vs. Binary's 47.9%), more balanced progress-regression rewards (1.08 ratio vs. PassRate's 3.71), and an adaptive curriculum where test weights dynamically shift as the policy improves.

## Introduction and Theoretical Foundation

### Background

Large language model (LLM) agents have made significant progress in software engineering (SWE), with reinforcement learning with verifiable rewards (RLVR) serving as a key contributor. In typical SWE agentic tasks, agents navigate repositories, edit code, and run tests, with final patches verified against hidden test suites.

### The Binary Reward Problem

The standard recipe uses a pass-all-test binary verifier:

$$
J_{\mathrm{success}} (\theta) = \mathbb{E}_{x, \tau} [ R_{\mathrm{Binary}} (s_{\mathrm{final}} (\tau)) ], \qquad R_{\mathrm{Binary}} (s) = \prod_{t = 1}^{m_{x}} o_{t} (s),\tag{1}
$$

where $o_t(s) \in \{0, 1\}$ is the result of test $t$. This binary reward is too sparse to distinguish patches that partially resolve issues from those that break existing functionality. When all patches in a rollout group fail, there is no learning signal from relative advantage.

### Pass-Rate as a Dense Alternative

The natural dense alternative is test pass-rate:

$$
R_{\text{PassRate}} (s) = \frac{1}{m_{x}} \sum_{t = 1}^{m_{x}} o_{t} (s).\tag{2}
$$

However, the paper identifies **three test-weighting imbalances** that miscalibrate pass-rate rewards:

1. **Multiplicity imbalance**: Multiple tests may check the same functionality and fail together. A patch with higher pass rate might fix only a few heavily-tested functionalities while breaking many lightly-tested ones, creating reward bias.

2. **Progress-regression imbalance**: Most repository tests are pass-to-pass (P2P) regression tests protecting existing functionality, while only a few fail-to-pass (F2P) tests check requested progress. Under pass-rate rewards, penalties for P2P regressions can outweigh rewards for F2P progress, discouraging agents from making changes.

3. **Difficulty imbalance**: Pass rate assigns equal weight to tests of different difficulty. When easy tests dominate, a patch can get high reward while still failing challenging tests.

## Methodology

### Behavior Balancing

For each issue $x$, CodeLift evaluates $n \geq 2$ final patches on its $m_x$ golden tests, forming a matrix $M \in \{0,1\}^{n \times m_x}$. It then:

1. Groups identical rows into patch groups $\mathcal{G}_{\text{patch}}$ and identical columns into test groups $\mathcal{G}_{\text{test}}$
2. Computes each test group's failure frequency and normalized weight:

$$
d_{g} = \frac{1}{|\mathcal{G}_{\text{patch}}|} \sum_{p \in \mathcal{G}_{\text{patch}}} (1 - Y_{gp}), \quad w_{g} = \frac{d_{g}}{\sum_{h \in \mathcal{G}_{\text{test}}} d_{h}}.\tag{3}
$$

where $\sum_g w_g = 1$, or $w_g = 0$ if all $d_g = 0$. This ensures:
- All-passing tests receive zero weight (they cannot distinguish patch quality)
- Tests passed by fewer patch groups receive more weight
- The weights adapt dynamically as the policy improves, creating an **adaptive curriculum**

### Snapshot Slicing

CodeLift uses self-test actions as checkpoints to cut rollouts into token intervals. For each slice $i$ with cumulative endpoint state $s_i^j$, it computes the state score:

$$
S(s) = \sum_{g \in \mathcal{G}_{\mathrm{test}}} \frac{w_{g}}{|g|} \sum_{t \in g} o_{t}(s).\tag{4}
$$

The snapshot correction and slice advantage are:

$$
\Delta_{i}^{j} = S(s_{i}^{j}) - r_{\mathrm{final}}^{j}, \qquad A_{i}^{j} = \Delta_{i}^{j} + A_{\mathrm{final}}^{j}.\tag{5}
$$

### Credit Assignment

Using a leave-one-out (LOO) estimator:

$$
b_{j} = \frac{1}{n - 1} \sum_{j' \neq j} r_{\mathrm{final}}^{j'}, \qquad A_{\mathrm{final}}^{j} = r_{\mathrm{final}}^{j} - b_{j}, \qquad A_{i}^{j} = S(s_{i}^{j}) - b_{j}.\tag{6}
$$

This compares each endpoint against the mean of other $n-1$ rollouts' final scores, giving positive credit when it exceeds that reference. The full algorithm is summarized in Algorithm 1.

## Empirical Validation / Results

### Main Results

**Table 1: Main results on SWE benchmarks (pass@k %)**

| Method | Verified pass@1 | Verified pass@2 | Verified pass@4 | Verified pass@8 | Pro pass@1 | Pro pass@2 | Pro pass@4 | Pro pass@8 |
|--------|-----------------|-----------------|-----------------|-----------------|------------|------------|------------|------------|
| Base | 26.00 | 35.14 | 43.52 | 50.60 | 12.17 | 18.96 | 27.00 | 35.71 |
| Binary | 34.63 | 43.04 | 50.36 | 56.80 | 16.54 | 25.00 | 34.16 | 41.73 |
| PR-Harmonic | 34.45 | 43.76 | 51.62 | 58.00 | 16.92 | 24.70 | 32.56 | 38.72 |
| PR-VeRPO | 34.45 | 42.41 | 49.43 | 55.40 | 19.50 | 28.24 | 36.62 | 42.86 |
| PR-Process | 35.55 | 44.11 | 51.60 | 58.40 | 19.36 | 27.48 | 35.79 | 42.48 |
| **CodeLift** | **38.30** | **47.53** | **55.84** | **62.80** | **20.16** | **28.71** | **36.84** | **43.23** |
| Δ vs. Binary | +3.67 | +4.49 | +5.48 | +6.00 | +3.62 | +3.71 | +2.68 | +1.50 |

CodeLift consistently outperforms all baselines across all benchmarks and sampling budgets.

### Ablation Study

**Table 2: Ablation for CodeLift's components**

| Method | Verified pass@1 | Verified pass@4 | Pro pass@1 | Pro pass@4 |
|--------|-----------------|-----------------|------------|------------|
| CodeLift | 38.30 | 55.84 | 20.16 | 36.84 |
| -Δ snapshot | -2.47 | -4.47 | -0.28 | -0.15 |
| CL-Outcome | 35.83 | 51.37 | 19.88 | 36.69 |
| -Δ behavior | -1.20 | -1.01 | -3.34 | -2.53 |
| Binary | 34.63 | 50.36 | 16.54 | 34.16 |

Both components contribute meaningfully: behavior-balanced scoring has larger impact on Pro, while snapshot slicing has larger impact on Verified.

### Robustness Analysis

**Table 3: Robustness across training settings (Qwen3-Coder on SWE-rebench)**

| Method | Adv. | Verified pass@1 | Verified pass@4 | Pro pass@1 | Pro pass@4 |
|--------|------|-----------------|-----------------|------------|------------|
| Base | - | 44.13 | 57.80 | 34.59 | 49.94 |
| Binary | RLOO | 47.70 | 63.20 | 34.30 | 50.88 |
| Binary | GRPO | 47.05 | 61.36 | 34.77 | 50.38 |
| CodeLift | RLOO | 48.30 | 65.40 | 35.95 | 51.76 |
| CodeLift | GRPO | 48.80 | 65.80 | 35.76 | 51.40 |

CodeLift outperforms Binary under both RLOO and GRPO advantage estimators.

## Theoretical and Practical Implications

### Dense and Stable Learning Signals

- CodeLift maintains a mean of **75.1% active rollout groups** (with non-zero advantages) throughout training, compared to Binary's 47.9%.
- Raw PassRate collapses quickly during training, while CodeLift maintains stable performance (35.8–36.2% after step 180 vs. Binary's 30.4–35.0%).

### Balanced Progress-Regression Preferences

- Under raw PassRate, the mean penalty for introducing regressions is **3.71×** the mean reward for making progress, encouraging conservative behavior (58.3% of positive advantages come from patches without pass-rate changes).
- CodeLift achieves a **1.08 ratio** (near parity), with 81.6% of positive advantages coming from patches that make real progress.

### Adaptive Curriculum

- CodeLift assigns zero weight to all-passing test groups, dynamically redistributing weight to tests that fewer patch groups pass.
- As the policy improves, easy test groups join the all-pass group without occupying weight space, while hard groups' average weights keep rising.
- This curriculum emerges naturally from existing final-patch test outcomes, requiring no extra rollout generation (only ~2.35 additional CPU-based test executions per rollout for snapshot slicing).

## Conclusion

CodeLift combines **behavior balancing** and **snapshot slicing** to provide dense, well-calibrated rewards for SWE agents. The method:

1. Groups tests and patches by outcome behavior to reduce pass-rate imbalances
2. Naturally introduces an adaptive curriculum from existing final-patch outcomes
3. Applies the same scoring rule to intermediate patches, assigning credit for progress even when later edits introduce regressions

Experimental results on SWE-bench Verified and SWE-bench Pro (Python) demonstrate consistent advantages over binary and enhanced pass-rate baselines. The analysis confirms denser learning signals, more balanced rewards, and test weights that adapt to the changing policy.

**Limitations**: The method is validated on Python repositories only; performance in other languages remains to be evaluated. Snapshot slicing requires additional test executions (~2.35 non-final patches per rollout), though these run on CPUs rather than GPUs.

**Future directions** could include extending CodeLift to other programming languages and repository environments, and further refining the credit assignment mechanism for even finer-grained learning signals.

---

_Markdown view of https://picx.dev/p/McXEhQ, served by PicX — AI-generated visual whiteboard summaries of research papers._