Full text not available for this paper

Summary (Overview)

  • Problem Identification: The paper identifies the Critique-to-Action Handoff Gap—a phenomenon where long-horizon coding agents can observe a failure, generate a plausible critique, yet continue along the same incorrect trajectory. The critique remains "behaviorally inert" rather than inducing effective corrective change.
  • Framework Introduction: RIPPLE (Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit) is a recovery-oriented training framework that internalizes process-level guidance into a single self-recovering policy, decomposing recovery into three functional stages: reflection triggering, evidence-grounded diagnosis, and corrective action.
  • Key Innovation: A prefix-only teacher provides evidence-grounded recovery supervision without access to future actions or terminal outcomes, combined with stage-specific credit assignment where local validity signals gate trajectory advantages so intermediate recovery decisions don't indiscriminately inherit terminal outcomes.
  • Empirical Results: Across four repository-level software-engineering benchmarks (SWE-bench Verified, Pro, Multilingual, SWE-Compass), RIPPLE improves the base policy by an arithmetic mean of 4.8 score points with no auxiliary critic at deployment.
  • Mechanism Validation: Controlled ablations isolate the benefits of local, stage-aligned credit, and matched-prefix interventions demonstrate that critique semantics functionally affect subsequent correction, with recovery gains concentrating where the prefix contains sufficient diagnostic evidence and the repository state remains reversible.

Introduction and Theoretical Foundation

The Critique-to-Action Handoff Gap

Large language model-based coding agents explore repositories, edit files, execute tests, and revise behavior using environmental feedback. However, observing feedback does not necessarily imply using it effectively. The paper formalizes the Critique-to-Action Handoff Gap as:

"An evidence-grounded critique remains behaviorally inert instead of inducing an effective corrective change."

This gap exposes a mismatch between the functional structure of recovery and existing training objectives:

  • Critique imitation supervises what the model says about a failure without establishing whether subsequent actions implement its diagnosis.
  • Terminal rewards supervise trajectory success but cannot isolate whether reflection was triggered appropriately, the diagnosis was correct, or the corrective action was effective.
  • Process supervision methods generally assess intermediate reasoning or redistribute trajectory outcomes rather than separately optimize the functional stages of recovery.

Theoretical Foundation

The paper decomposes the kk-th recovery episode into three functionally distinct decision spans:

ek=(Tktrig,Tkcrit,Tkca)e_k = \left( T^{\text{trig}}_k, T^{\text{crit}}_k, T^{\text{ca}}_k \right)

where:

  • TktrigT^{\text{trig}}_k contains decision tokens that initiate reflection (when recovery should begin)
  • TkcritT^{\text{crit}}_k contains the resulting diagnosis (why the trajectory deviated)
  • TkcaT^{\text{ca}}_k contains subsequent corrective actions (how to revise behavior)

Addressing the handoff gap requires jointly learning when to initiate recovery, which diagnosis the evidence supports, and how it should alter subsequent behavior.


Methodology

3.1 Prefix-Only Recovery Teacher

A lightweight generative model gϕg_\phi serves as a training-time recovery teacher. Given only the trajectory prefix hth_t, the teacher produces:

gϕ(ht)=(qt,αt,ctref)g_\phi(h_t) = \left( q_t, \alpha_t, c^{\text{ref}}_t \right)

where:

  • qt∈{0,1}q_t \in \{0, 1\}: indicates whether a recoverable deviation is observable
  • αt∈[0,1]\alpha_t \in [0, 1]: confidence of this judgment
  • ctrefc^{\text{ref}}_t: evidence-grounded reference critique identifying error type, relevant evidence, invalidated assumption, and recovery direction

The teacher-estimated recovery onset is:

b^k=min⁡{t∈Wk∣qt=1,αt≥γ}\hat{b}_k = \min\{ t \in W_k \mid q_t = 1, \alpha_t \geq \gamma \}

where γ\gamma is a confidence threshold and WkW_k is the candidate region for the kk-th recovery event. Crucially, teacher outputs are regenerated from prefixes only—they do not condition on future actions, final patches, or terminal verifier outcomes.

3.2 Policy Initialization

The initial policy is constructed from two trajectory sets:

Linit=LSFT(Dtask)+λrecLSFT(Drec)\mathcal{L}_{\text{init}} = \mathcal{L}_{\text{SFT}}(\mathcal{D}_{\text{task}}) + \lambda_{\text{rec}} \mathcal{L}_{\text{SFT}}(\mathcal{D}_{\text{rec}})
  • Dtask\mathcal{D}_{\text{task}}: task trajectories teaching repository exploration, test interpretation, code modification, and patch verification
  • Drec\mathcal{D}_{\text{rec}}: recovery trajectories inserting teacher-generated critiques at recoverable positions

3.3 Recovery-Oriented Reinforcement Learning

Trajectory-Level Advantage

Following GRPO, the trajectory-level advantage is:

A^i=Ri−μRσR+ϵ,μR=1G∑j=1GRj\hat{A}_i = \frac{R_i - \mu_R}{\sigma_R + \epsilon}, \quad \mu_R = \frac{1}{G}\sum_{j=1}^{G} R_j

where RiR_i is the terminal reward, σR\sigma_R is the within-group reward standard deviation, and ϵ\epsilon is a numerical stabilizer.

Three Process Signals

1. Trigger Signal — measures whether reflection occurred near the teacher-estimated boundary:

hktrig=1[∣pk−b^k∣≤w]h^{\text{trig}}_k = \mathbb{1}\left[ |p_k - \hat{b}_k| \leq w \right]

where pkp_k is the position of the kk-th self-critique and ww is a tolerance window in interaction steps.

2. Diagnosis Signal — evaluates critique quality across four dimensions:

skcrit=λtypesktype+λeviskevi+λassumpskassump+λdirskdirs^{\text{crit}}_k = \lambda_{\text{type}} s^{\text{type}}_k + \lambda_{\text{evi}} s^{\text{evi}}_k + \lambda_{\text{assump}} s^{\text{assump}}_k + \lambda_{\text{dir}} s^{\text{dir}}_k hkcrit=1[skcrit≥η∧Grounded(ck,hpk)]h^{\text{crit}}_k = \mathbb{1}\left[ s^{\text{crit}}_k \geq \eta \land \text{Grounded}(c_k, h_{p_k}) \right]

where η\eta is the diagnostic-consistency threshold and Grounded(⋅)\text{Grounded}(\cdot) requires all cited evidence to be observable in the current prefix.

3. Corrective-Action Signal — uses training-time snapshot tests (hidden from the agent):

F2Pk=Pk+∖Pk−,P2Fk=Pk−∖Pk+F2P_k = P^+_k \setminus P^-_k, \quad P2F_k = P^-_k \setminus P^+_k

where Pk−P^-_k and Pk+P^+_k are passing-test sets before and after the corrective-action span. The signal is:

hkca=1[Attr(ck,Akca,F2Pk,P2Fk)=effective]h^{\text{ca}}_k = \mathbb{1}\left[ \text{Attr}(c_k, A^{\text{ca}}_k, F2P_k, P2F_k) = \text{effective} \right]

Stage-Specific Credit Assignment

The shaped advantage for recovery stage m∈{trig,crit,ca}m \in \{\text{trig}, \text{crit}, \text{ca}\} is:

A~i,t=hkm[A^i]++(1−hkm)[A^i]−,t∈Tkm\tilde{A}_{i,t} = h^m_k [\hat{A}_i]^+ + (1 - h^m_k)[\hat{A}_i]^-, \quad t \in T^m_k

where [A^i]+=max⁡(A^i,0)[\hat{A}_i]^+ = \max(\hat{A}_i, 0) and [A^i]−=min⁡(A^i,0)[\hat{A}_i]^- = \min(\hat{A}_i, 0).

Key insight: A valid recovery decision inherits positive credit from successful trajectories without inheriting negative credit from unsuccessful ones. Conversely, invalid decisions inherit negative credit from failed trajectories but cannot benefit merely from appearing in successful ones.

Final Objective

JRIPPLE(θ)=E[1G∑i=1G1∣τi∣∑tℓi,t(θ)]−βDKL\mathcal{J}_{\text{RIPPLE}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G} \frac{1}{|\tau_i|} \sum_{t} \ell_{i,t}(\theta) \right] - \beta D_{\text{KL}}

with the clipped token-level surrogate:

ℓi,t(θ)=min⁡(ρi,tA~i,t,ρˉi,tA~i,t)\ell_{i,t}(\theta) = \min\left( \rho_{i,t} \tilde{A}_{i,t}, \bar{\rho}_{i,t} \tilde{A}_{i,t} \right)

where ρi,t=πθ(ai,t∣hi,t)/πθold(ai,t∣hi,t)\rho_{i,t} = \pi_\theta(a_{i,t}|h_{i,t})/\pi_{\theta_{\text{old}}}(a_{i,t}|h_{i,t}) and ρˉi,t=clip(ρi,t,1−ε,1+ε)\bar{\rho}_{i,t} = \text{clip}(\rho_{i,t}, 1-\varepsilon, 1+\varepsilon).


Empirical Validation / Results

RQ1: End-to-End Effectiveness

MethodExt. callsAct. tok.p95 lat.SWE-bench VerifiedSWE-bench ProSWE-bench MultilingualSWE-Compass
Qwen3.5-27B Base01.00×1.00×74.650.868.824.3
Base, longer single trajectory01.16×1.21×74.751.769.325.0
Base, two restarts (split budget)01.17×1.24×75.053.270.126.3
Prompted periodic reflection01.18×1.24×74.552.669.725.6
Recovery-SFT (π_sft)01.19×1.26×75.153.870.726.9
Outcome-only GRPO01.06×1.09×75.054.571.127.5
Pivotal-aware credit01.09×1.13×75.255.371.828.6
MURPHY-style retrospective01.09×1.13×75.255.772.328.9
ICRL-style internalization01.15×1.18×75.456.072.729.4
RIPPLE01.17×1.23×75.957.373.630.7
Online same teacher4.8/task1.00×1.57×76.057.973.831.0
Online strong critic10.9/task1.00×2.61×76.458.474.431.6

Key findings:

  • RIPPLE improves over the base policy by 1.3 points on Verified, 6.5 on Pro, 4.8 on Multilingual, and 6.4 on SWE-Compass (arithmetic mean: 4.8 points)
  • Gains track recovery opportunities: recoverable deviations occur in 17.8% of trajectories on Verified vs. 43.6% on Pro
  • On Pro, gains increase from 1.8 points (Easy) to 11.9 points (Extreme difficulty quartile)
  • RIPPLE repairs 8.8% of base-policy failures while damaging only 2.3% of solutions
  • RIPPLE stays within 0.6 points of the online same-teacher reference while eliminating auxiliary deployment calls

RQ2: Locality and Stage Alignment

Seven credit-assignment variants were compared with identical rollouts and process labels:

Credit RulePro Resolved@1Full-Loop RecoveryPositive-Credit Contamination
Outcome-only54.545.1%38.4%
Trajectory-wide54.845.1%—
Event-local generic55.650.4%24.6%
Stage/event (stage-aware, event-wide)56.153.1%—
Random block54.945.8%—
Stage shuffle55.248.4%—
RIPPLE57.359.6%11.2%

Leave-one-stage-out results:

  • Removing diagnosis credit: Diagnosis Accuracy drops from 81.5%→69.8%, Action Alignment from 74.8%→64.3%
  • Removing action credit: Pro drops from 57.3→54.0, Full-Loop Recovery from 59.6%→43.5% (largest end-to-end contribution)

RQ3: Functional Recovery Mechanisms

Stage-wise behavior on fixed Recovery Challenge Set:

MethodTrig. F1Diag.Align.Func.Full
Periodic reflection0.4355.848.142.531.4
Recovery-SFT0.5664.954.849.438.1
Outcome-only0.6066.958.653.041.6
ICRL-style0.6573.264.460.550.8
RIPPLE0.7981.574.870.259.6
  • RIPPLE achieves 84.0% Grounded Evidence vs. 81.5% Diagnosis Accuracy, distinguishing citing observed failures from identifying causes
  • False-trigger rate on non-deviating prefixes: 8.9% (RIPPLE) vs. 71.5% (periodic reflection)

Matched-prefix intervention (holding policy, prefix, snapshot, budget, and seeds fixed):

Critique ConditionK-step RecoveryP2F Regression
Full critique62.9%2.4%
Remove critique39.8%6.8%
Generic critique42.0%6.1%
Wrong-direction critique25.6%18.9%

Recoverability boundary: Among 248 prefixes with sufficient evidence and reversible state, RIPPLE improves resolution by 20.4 points. Gains drop to 5.8 when evidence is sufficient but state isn't reversible, 4.7 when evidence is insufficient but state is reversible, and 1.7 when neither holds.


Theoretical and Practical Implications

Theoretical Contributions

  1. Recovery as a structured policy: The paper reframes self-correction not as a monolithic capability but as three functionally distinct decision stages (trigger, diagnose, correct), each requiring separate credit assignment.

  2. Local validity gating: The advantage-shaping mechanism in Equation 11 demonstrates that intermediate decisions should not indiscriminately inherit terminal outcomes—a theoretical departure from standard credit broadcasting approaches.

  3. Prefix-only supervision: The finding that prefix-only teacher inputs retain much of the boundary-detection ability (F1 0.84 vs. 0.86 offline, 0.90 full-trajectory oracle) supports the viability of decision-time-available supervision.

Practical Implications

  • Deployment efficiency: RIPPLE removes the need for auxiliary critics at inference (0 external calls) while achieving performance within 0.6 points of online-critic references that require 4.8–10.9 additional model calls per task.
  • Token/latency overhead: Only 1.17× actor tokens and 1.23× p95 latency relative to base policy—a modest cost for substantial performance gains.
  • Recovery boundary conditions: The framework works best when (a) the prefix contains sufficient diagnostic evidence and (b) the repository state remains reversible within the remaining budget—important for practitioners deploying such agents.

Conclusion

RIPPLE addresses the Critique-to-Action Handoff Gap by internalizing prefix-only recovery guidance into a single self-recovering policy. The framework's core contributions are:

  1. Stage-specific credit distinguishing reflection triggering, diagnosis, and corrective action while retaining terminal task supervision
  2. Local validity gates preventing ineffective recovery decisions from inheriting positive trajectory credit while protecting useful corrections from negative credit attributable to unrelated failures
  3. Arithmetic mean improvement of 4.8 score points across four software-engineering benchmarks without an online critic

Future directions identified by the authors:

  • Improved exploration in uniform-reward groups (where sign gating provides no learning signal)
  • Direct estimation of local recovery effects across evolving repositories, programming languages, and deployment environments
  • Jointly learning when to reflect, what to diagnose, and how to act as an internal policy capability

Limitations acknowledged:

  • Evaluation limited to Qwen3.5-27B in one coding scaffold
  • Process labels are approximate; offline candidate nomination uses complete trajectories despite prefix-only teacher inputs
  • Verified benchmark gain's confidence interval crosses zero
  • Sign gating cannot overcome missing evidence or irreversible states within a fixed budget

Related papers