Full text not available for this paper
Summary (Overview)
- Problem Identification: The paper identifies the Critique-to-Action Handoff Gap—a phenomenon where long-horizon coding agents can observe a failure, generate a plausible critique, yet continue along the same incorrect trajectory. The critique remains "behaviorally inert" rather than inducing effective corrective change.
- Framework Introduction: RIPPLE (Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit) is a recovery-oriented training framework that internalizes process-level guidance into a single self-recovering policy, decomposing recovery into three functional stages: reflection triggering, evidence-grounded diagnosis, and corrective action.
- Key Innovation: A prefix-only teacher provides evidence-grounded recovery supervision without access to future actions or terminal outcomes, combined with stage-specific credit assignment where local validity signals gate trajectory advantages so intermediate recovery decisions don't indiscriminately inherit terminal outcomes.
- Empirical Results: Across four repository-level software-engineering benchmarks (SWE-bench Verified, Pro, Multilingual, SWE-Compass), RIPPLE improves the base policy by an arithmetic mean of 4.8 score points with no auxiliary critic at deployment.
- Mechanism Validation: Controlled ablations isolate the benefits of local, stage-aligned credit, and matched-prefix interventions demonstrate that critique semantics functionally affect subsequent correction, with recovery gains concentrating where the prefix contains sufficient diagnostic evidence and the repository state remains reversible.
Introduction and Theoretical Foundation
The Critique-to-Action Handoff Gap
Large language model-based coding agents explore repositories, edit files, execute tests, and revise behavior using environmental feedback. However, observing feedback does not necessarily imply using it effectively. The paper formalizes the Critique-to-Action Handoff Gap as:
"An evidence-grounded critique remains behaviorally inert instead of inducing an effective corrective change."
This gap exposes a mismatch between the functional structure of recovery and existing training objectives:
- Critique imitation supervises what the model says about a failure without establishing whether subsequent actions implement its diagnosis.
- Terminal rewards supervise trajectory success but cannot isolate whether reflection was triggered appropriately, the diagnosis was correct, or the corrective action was effective.
- Process supervision methods generally assess intermediate reasoning or redistribute trajectory outcomes rather than separately optimize the functional stages of recovery.
Theoretical Foundation
The paper decomposes the -th recovery episode into three functionally distinct decision spans:
where:
- contains decision tokens that initiate reflection (when recovery should begin)
- contains the resulting diagnosis (why the trajectory deviated)
- contains subsequent corrective actions (how to revise behavior)
Addressing the handoff gap requires jointly learning when to initiate recovery, which diagnosis the evidence supports, and how it should alter subsequent behavior.
Methodology
3.1 Prefix-Only Recovery Teacher
A lightweight generative model serves as a training-time recovery teacher. Given only the trajectory prefix , the teacher produces:
where:
- : indicates whether a recoverable deviation is observable
- : confidence of this judgment
- : evidence-grounded reference critique identifying error type, relevant evidence, invalidated assumption, and recovery direction
The teacher-estimated recovery onset is:
where is a confidence threshold and is the candidate region for the -th recovery event. Crucially, teacher outputs are regenerated from prefixes only—they do not condition on future actions, final patches, or terminal verifier outcomes.
3.2 Policy Initialization
The initial policy is constructed from two trajectory sets:
- : task trajectories teaching repository exploration, test interpretation, code modification, and patch verification
- : recovery trajectories inserting teacher-generated critiques at recoverable positions
3.3 Recovery-Oriented Reinforcement Learning
Trajectory-Level Advantage
Following GRPO, the trajectory-level advantage is:
where is the terminal reward, is the within-group reward standard deviation, and is a numerical stabilizer.
Three Process Signals
1. Trigger Signal — measures whether reflection occurred near the teacher-estimated boundary:
where is the position of the -th self-critique and is a tolerance window in interaction steps.
2. Diagnosis Signal — evaluates critique quality across four dimensions:
where is the diagnostic-consistency threshold and requires all cited evidence to be observable in the current prefix.
3. Corrective-Action Signal — uses training-time snapshot tests (hidden from the agent):
where and are passing-test sets before and after the corrective-action span. The signal is:
Stage-Specific Credit Assignment
The shaped advantage for recovery stage is:
where and .
Key insight: A valid recovery decision inherits positive credit from successful trajectories without inheriting negative credit from unsuccessful ones. Conversely, invalid decisions inherit negative credit from failed trajectories but cannot benefit merely from appearing in successful ones.
Final Objective
with the clipped token-level surrogate:
where and .
Empirical Validation / Results
RQ1: End-to-End Effectiveness
| Method | Ext. calls | Act. tok. | p95 lat. | SWE-bench Verified | SWE-bench Pro | SWE-bench Multilingual | SWE-Compass |
|---|---|---|---|---|---|---|---|
| Qwen3.5-27B Base | 0 | 1.00× | 1.00× | 74.6 | 50.8 | 68.8 | 24.3 |
| Base, longer single trajectory | 0 | 1.16× | 1.21× | 74.7 | 51.7 | 69.3 | 25.0 |
| Base, two restarts (split budget) | 0 | 1.17× | 1.24× | 75.0 | 53.2 | 70.1 | 26.3 |
| Prompted periodic reflection | 0 | 1.18× | 1.24× | 74.5 | 52.6 | 69.7 | 25.6 |
| Recovery-SFT (π_sft) | 0 | 1.19× | 1.26× | 75.1 | 53.8 | 70.7 | 26.9 |
| Outcome-only GRPO | 0 | 1.06× | 1.09× | 75.0 | 54.5 | 71.1 | 27.5 |
| Pivotal-aware credit | 0 | 1.09× | 1.13× | 75.2 | 55.3 | 71.8 | 28.6 |
| MURPHY-style retrospective | 0 | 1.09× | 1.13× | 75.2 | 55.7 | 72.3 | 28.9 |
| ICRL-style internalization | 0 | 1.15× | 1.18× | 75.4 | 56.0 | 72.7 | 29.4 |
| RIPPLE | 0 | 1.17× | 1.23× | 75.9 | 57.3 | 73.6 | 30.7 |
| Online same teacher | 4.8/task | 1.00× | 1.57× | 76.0 | 57.9 | 73.8 | 31.0 |
| Online strong critic | 10.9/task | 1.00× | 2.61× | 76.4 | 58.4 | 74.4 | 31.6 |
Key findings:
- RIPPLE improves over the base policy by 1.3 points on Verified, 6.5 on Pro, 4.8 on Multilingual, and 6.4 on SWE-Compass (arithmetic mean: 4.8 points)
- Gains track recovery opportunities: recoverable deviations occur in 17.8% of trajectories on Verified vs. 43.6% on Pro
- On Pro, gains increase from 1.8 points (Easy) to 11.9 points (Extreme difficulty quartile)
- RIPPLE repairs 8.8% of base-policy failures while damaging only 2.3% of solutions
- RIPPLE stays within 0.6 points of the online same-teacher reference while eliminating auxiliary deployment calls
RQ2: Locality and Stage Alignment
Seven credit-assignment variants were compared with identical rollouts and process labels:
| Credit Rule | Pro Resolved@1 | Full-Loop Recovery | Positive-Credit Contamination |
|---|---|---|---|
| Outcome-only | 54.5 | 45.1% | 38.4% |
| Trajectory-wide | 54.8 | 45.1% | — |
| Event-local generic | 55.6 | 50.4% | 24.6% |
| Stage/event (stage-aware, event-wide) | 56.1 | 53.1% | — |
| Random block | 54.9 | 45.8% | — |
| Stage shuffle | 55.2 | 48.4% | — |
| RIPPLE | 57.3 | 59.6% | 11.2% |
Leave-one-stage-out results:
- Removing diagnosis credit: Diagnosis Accuracy drops from 81.5%→69.8%, Action Alignment from 74.8%→64.3%
- Removing action credit: Pro drops from 57.3→54.0, Full-Loop Recovery from 59.6%→43.5% (largest end-to-end contribution)
RQ3: Functional Recovery Mechanisms
Stage-wise behavior on fixed Recovery Challenge Set:
| Method | Trig. F1 | Diag. | Align. | Func. | Full |
|---|---|---|---|---|---|
| Periodic reflection | 0.43 | 55.8 | 48.1 | 42.5 | 31.4 |
| Recovery-SFT | 0.56 | 64.9 | 54.8 | 49.4 | 38.1 |
| Outcome-only | 0.60 | 66.9 | 58.6 | 53.0 | 41.6 |
| ICRL-style | 0.65 | 73.2 | 64.4 | 60.5 | 50.8 |
| RIPPLE | 0.79 | 81.5 | 74.8 | 70.2 | 59.6 |
- RIPPLE achieves 84.0% Grounded Evidence vs. 81.5% Diagnosis Accuracy, distinguishing citing observed failures from identifying causes
- False-trigger rate on non-deviating prefixes: 8.9% (RIPPLE) vs. 71.5% (periodic reflection)
Matched-prefix intervention (holding policy, prefix, snapshot, budget, and seeds fixed):
| Critique Condition | K-step Recovery | P2F Regression |
|---|---|---|
| Full critique | 62.9% | 2.4% |
| Remove critique | 39.8% | 6.8% |
| Generic critique | 42.0% | 6.1% |
| Wrong-direction critique | 25.6% | 18.9% |
Recoverability boundary: Among 248 prefixes with sufficient evidence and reversible state, RIPPLE improves resolution by 20.4 points. Gains drop to 5.8 when evidence is sufficient but state isn't reversible, 4.7 when evidence is insufficient but state is reversible, and 1.7 when neither holds.
Theoretical and Practical Implications
Theoretical Contributions
-
Recovery as a structured policy: The paper reframes self-correction not as a monolithic capability but as three functionally distinct decision stages (trigger, diagnose, correct), each requiring separate credit assignment.
-
Local validity gating: The advantage-shaping mechanism in Equation 11 demonstrates that intermediate decisions should not indiscriminately inherit terminal outcomes—a theoretical departure from standard credit broadcasting approaches.
-
Prefix-only supervision: The finding that prefix-only teacher inputs retain much of the boundary-detection ability (F1 0.84 vs. 0.86 offline, 0.90 full-trajectory oracle) supports the viability of decision-time-available supervision.
Practical Implications
- Deployment efficiency: RIPPLE removes the need for auxiliary critics at inference (0 external calls) while achieving performance within 0.6 points of online-critic references that require 4.8–10.9 additional model calls per task.
- Token/latency overhead: Only 1.17× actor tokens and 1.23× p95 latency relative to base policy—a modest cost for substantial performance gains.
- Recovery boundary conditions: The framework works best when (a) the prefix contains sufficient diagnostic evidence and (b) the repository state remains reversible within the remaining budget—important for practitioners deploying such agents.
Conclusion
RIPPLE addresses the Critique-to-Action Handoff Gap by internalizing prefix-only recovery guidance into a single self-recovering policy. The framework's core contributions are:
- Stage-specific credit distinguishing reflection triggering, diagnosis, and corrective action while retaining terminal task supervision
- Local validity gates preventing ineffective recovery decisions from inheriting positive trajectory credit while protecting useful corrections from negative credit attributable to unrelated failures
- Arithmetic mean improvement of 4.8 score points across four software-engineering benchmarks without an online critic
Future directions identified by the authors:
- Improved exploration in uniform-reward groups (where sign gating provides no learning signal)
- Direct estimation of local recovery effects across evolving repositories, programming languages, and deployment environments
- Jointly learning when to reflect, what to diagnose, and how to act as an internal policy capability
Limitations acknowledged:
- Evaluation limited to Qwen3.5-27B in one coding scaffold
- Process labels are approximate; offline candidate nomination uses complete trajectories despite prefix-only teacher inputs
- Verified benchmark gain's confidence interval crosses zero
- Sign gating cannot overcome missing evidence or irreversible states within a fixed budget
Related papers
- BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
BuggyMLE generates 6,600+ bug-injected ML repositories and a structural complexity metric (CCI) predicting solve rates with R²=0.951, enabling difficulty-graded benchmarks where a 9B model rivals frontier agents.
- POSEIDON: Scaling Training Environments for Scientific Agents from Code Repositories
POSEIDON's automated pipeline generates 97,659 verifiable scientific-agent training environments from 7,110 repositories, and fine-tuning on them consistently boosts model performance across six benchmarks.
- Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal
Looped transformers solve sequential algorithmic tasks untied models of equal effective depth cannot, by executing one algorithm step per iteration without intermediate supervision.