Full text not available for this paper

Summary (Overview)

  • GUARD (Grader-free UED with Adaptive Rule Distillation) is a closed-loop framework for reference-free continual evolution of agent harnesses (prompts, skills, tool rules, memory) around a frozen base LLM, without weight updates or external graders.
  • The key insight is decomposing tasks into pre-execution self-checklists and guiding harness updates via paired directional regret, which separates newly broken behaviors (Δ_hurt) from unresolved capability gaps (Δ_gap).
  • GUARD uses GRPO-inspired parallel candidate pooling with critic-free group-relative fitness for selection, and a Soft-HURT promotion gate that accepts edits only when gains outweigh regressions, guaranteeing non-regression on gated task buffers.
  • Across benchmarks (PAST-Bench, EvoHarnessBench, Drift Gauntlet, professional suites), GUARD achieves 19/19 non-regressive promotions, zero post-block forgetting under hidden drift, and 54.5% token reduction per task, with monotonic accuracy gains.
  • Three theoretical guarantees are provided: non-regressive gating with concentration bounds (Theorem 1), removal of the 1/ε passive-discovery penalty via dense checklist grading (Theorem 2), and strict regression dominance in priority ranking (Proposition 1).

Introduction and Theoretical Foundation

Background and Motivation

Modern language model agents pair a frozen base model with an editable harness of prompts, skills, tool interfaces, memory, and control logic. Recursive self-improvement (RSI) adapts this harness from experience without weight updates—a practical approach since frontier models sit behind black-box APIs and open-world tasks rarely provide ground-truth rewards.

Core problem: Self-editing without external graders often backfires. Free-form reflection and scalar scores conflate two distinct failure signals:

  1. Edit-induced regressions (behaviors the latest edit newly broke)
  2. Unresolved capability gaps (things the agent still hasn't solved)

Mixing these signals causes three failure modes:

  • Local overfitting (over-triggering): broad rules written for recent failures break tasks that already worked
  • Catastrophic forgetting: shifts in workflow/topic/domain overwrite earlier skills
  • Memory pollution: unverified additions accumulate stale or conflicting rules that bloat the prompt

Theoretical Foundation

The framework draws on two ideas from reinforcement learning:

  1. Curriculum regret from Unsupervised Environment Design (UED) [7, 14, 15, 35] — prioritizing high-regret levels, transferred here to directional criterion ranking over task checklists
  2. Group Relative Policy Optimization (GRPO) [37, 3] — critic-free group-relative scoring, applied to non-parametric harness optimization without weight updates

Problem formulation: An agent A=(πθ,Wt,g)A = (\pi_\theta, W_{t,g}) pairs a frozen policy πθ\pi_\theta with an editable harness state Wt,gW_{t,g} at round tt. The reference-free evolver updates:

Wt,g=E(Wt−1,g,Ht,g),t=1,…,T≤TmaxW_{t,g} = \mathcal{E}\left(W_{t-1,g}, \mathcal{H}_{t,g}\right), \quad t = 1, \ldots, T \leq T_{\text{max}}

The goal is maximizing μg(WT,g)=Eq∈Qg,τ∼πθ(⋅∣q,WT,g)[V∗(q,τ)]\mu_g(W_{T,g}) = \mathbb{E}_{q \in Q_g, \tau \sim \pi_\theta(\cdot|q, W_{T,g})}[V^*(q, \tau)] where V∗V^* is the unobserved task evaluator.

Reference-free means neither the solver, self-reviewer, nor distiller accesses held-out answers, test cases, hidden verifiers, oracle skills, or rubrics—but the framework still leverages observable structure: task prompt distributions, tool declarations, interaction trajectories, and criterion-level self-review judgments.


Methodology

GUARD instantiates the evolver as a four-stage closed loop:

Stage 1: Reference-Free Self-Review via Task Checklists (§4.1)

Pre-execution checklist: The self-reviewer reads the task prompt, tool declarations, and schema once at Turn 0 to derive a fixed checklist C(q)\mathcal{C}(q) of atomic binary predicates across two classes:

  • Execution-verifiable predicates: clean sandbox execution, valid entity IDs/columns, idempotent mutations (checked deterministically on an ephemeral sandbox copy)
  • Semantic predicates: honoring prompt constraints, recipient, format, safety rules (graded individually from prompt, output, and trace)

Two-layer grading: Trajectory τ\tau receives criterion score S(τ)S(\tau) and strict score Sstrict(τ)S_{\text{strict}}(\tau):

S(τ)=100⋅1∣Cgr(τ)∣∑c∈Cgr(τ)r(τ,c)S(\tau) = 100 \cdot \frac{1}{|\mathcal{C}_{\text{gr}}(\tau)|} \sum_{c \in \mathcal{C}_{\text{gr}}(\tau)} r(\tau, c) Sstrict(τ)={0on blocking sandbox error100if all graded criteria passS(τ)/2otherwiseS_{\text{strict}}(\tau) = \begin{cases} 0 & \text{on blocking sandbox error} \\ 100 & \text{if all graded criteria pass} \\ S(\tau)/2 & \text{otherwise} \end{cases}

This exploits generation–verification asymmetry: checking one atomic predicate after execution requires localized attention, while long-horizon generation couples many constraints.

Stage 2: Paired Regret Ranking — Analyze (§4.2)

Definition 1 (Criterion obedience and paired regret): For trajectories Tc(W)\mathcal{T}_c(W) grading criterion cc under harness WW:

obey(c;W)=1∣Tc(W)∣∑τ∈Tc(W)r(τ,c)∈[0,1]\text{obey}(c; W) = \frac{1}{|\mathcal{T}_c(W)|} \sum_{\tau \in \mathcal{T}_c(W)} r(\tau, c) \in [0, 1]

Comparing the two most recent incumbents Wt−2W_{t-2} and Wt−1W_{t-1}:

Δhurt,t(c)=max⁡(0,obey(c;Wt−2)−obey(c;Wt−1))\Delta_{\text{hurt},t}(c) = \max\left(0, \text{obey}(c; W_{t-2}) - \text{obey}(c; W_{t-1})\right) Δgap,t(c)=1−obey(c;Wt−1)\Delta_{\text{gap},t}(c) = 1 - \text{obey}(c; W_{t-1}) Priorityt(c)=Δhurt,t(c)+Δgap,t(c)∈[0,2]\text{Priority}_t(c) = \Delta_{\text{hurt},t}(c) + \Delta_{\text{gap},t}(c) \in [0, 2]

Criteria are partitioned into three diagnostic lists: Regressed (Δ_hurt > 0, wrap offending rules in [iff] preconditions), Unsolved Gap (Δ_gap > 0), and Mastered (Priority = 0, pinned against overwriting).

Stage 3: Group-Relative Candidate Pooling — Propose (§4.3)

Inspired by GRPO, GUARD samples M≥2M \geq 2 candidate harnesses in parallel, scores them on the same task batch, and ranks by critic-free group-standardized fitness:

Rm=Perfm−β∣Qm−∣+γ∣Qm+∣,β>γ≥0R_m = \text{Perf}_m - \beta|Q_m^-| + \gamma|Q_m^+|, \quad \beta > \gamma \geq 0 A^m=Rm−μRσR,σR=1M∑m=1M(Rm−μR)2+10−6\hat{A}_m = \frac{R_m - \mu_R}{\sigma_R}, \quad \sigma_R = \sqrt{\frac{1}{M}\sum_{m=1}^{M}(R_m - \mu_R)^2 + 10^{-6}}

ContrastEvolve contrasts the best (m+m^+) and worst (m−m^-) candidates to update strategy prompts:

Pt+1,g={st+1+}⌊M/2⌋∪{st+1−}M−⌊M/2⌋\mathcal{P}_{t+1,g} = \{s^+_{t+1}\}^{\lfloor M/2\rfloor} \cup \{s^-_{t+1}\}^{M - \lfloor M/2\rfloor}

Stage 4: Soft-HURT Promotion Gate — Gate (§4.4)

The group-averaged regression penalty:

Hurt(W~,W)=1∣Qg∣∑q∈Q−(W~,W)(Sq(W)−Sq(W~))\text{Hurt}(\tilde{W}, W) = \frac{1}{|Q_g|} \sum_{q \in Q^-(\tilde{W}, W)} \left(S_q(W) - S_q(\tilde{W})\right)

Soft-HURT rule: Promote candidate W~\tilde{W} over Wt−1W_{t-1} only when:

Sˉ(W~)−Sˉ(Wt−1)>max⁡(Hurt(W~,Wt−1),μmin⁡I[∣Q−(W~,Wt−1)∣>0])\bar{S}(\tilde{W}) - \bar{S}(W_{t-1}) > \max\left(\text{Hurt}(\tilde{W}, W_{t-1}), \mu_{\min}\mathbb{I}[|Q^-(\tilde{W}, W_{t-1})| > 0]\right)

When a candidate fails solely due to isolated regressions (1≤∣Q−∣≤hrep1 \leq |Q^-| \leq h_{\text{rep}}), GUARD triggers at most one in-round [iff] scope-guard repair. Rejected edits are logged to a rejection ledger Lg\mathcal{L}_g so bad changes are not repeated.

Theoretical Guarantees (§4.5)

Theorem 1 (Non-Regressive Gating and Concentration): Under the gate of (12):

  • (i) No downward drift: Sˉ(WT)=Sˉ(W0)+∑t=1TΔt≥Sˉ(W0)\bar{S}(W_T) = \bar{S}(W_0) + \sum_{t=1}^{T}\Delta_t \geq \bar{S}(W_0)
  • (ii) Regressions are paid for: ∑q∈Qt−(Sq(Wt−1)−Sq(Wt))<∣Qg∣Δt\sum_{q \in Q_t^-}\left(S_q(W_{t-1}) - S_q(W_t)\right) < |Q_g|\Delta_t
  • (iii) Concentration: with probability ≥ 1−α, μ(WT)≥μ(W0)+∑t=1TΔt−2σ2log⁡(2neval∣G∣/α)∣Qg∣Nrep\mu(W_T) \geq \mu(W_0) + \sum_{t=1}^{T}\Delta_t - 2\sigma\sqrt{\frac{2\log(2n_{\text{eval}}|G|/\alpha)}{|Q_g|N_{\text{rep}}}}
  • (iv) Factored speedup from per-group gating

Theorem 2 (Targeted Repair vs. Passive Discovery): Passive learning requires Ω(log⁡J/ϵ)\Omega(\log J / \epsilon) rounds to discover JJ rare invariants (each surfacing with probability pj≤ϵp_j \leq \epsilon), while dense checklist grading achieves E[Ttarget]≤Jρbmin⁡+1+Hbmin⁡−1ρ\mathbb{E}[T_{\text{target}}] \leq \frac{J}{\rho b_{\min}} + \frac{1 + H_{b_{\min}-1}}{\rho}, removing the 1/ϵ1/\epsilon dependence.

Proposition 1 (Strict Regression Dominance): In the single-rollout binary case, a fresh regression (1→0, priority 2) outranks every non-regressed criterion (priority at most 1). Priority t(c)=0t(c) = 0 iff obey(c;Wt−1)=1\text{obey}(c; W_{t-1}) = 1 and Δhurt,t(c)=0\Delta_{\text{hurt},t}(c) = 0 — no regressed or failed criterion is ever pinned.


Empirical Validation / Results

PAST-Bench: Cross-Session Memory Write Path (Table 1)

ArmTaskProc.Mech.Upd.W.Prec.RecallRet.Poll.↓Evolve
No-Evolve (W₀)0.3720.4270.4540.2780.0000.0000.9900.000
Hermes+0.8520.8080.8400.7910.7240.8541.0000.167
RRSI0.8090.7140.7250.9010.4380.8110.9740.500
GEPA0.8580.8150.8370.7770.7190.8580.9960.167
Meta-Harness0.8120.6770.7390.9030.4790.8140.9980.500
GUARD (Ours)0.8540.8300.8780.9240.7760.8560.9990.1670.436

GUARD leads procedural task (83.0%), procedural mechanism (90.0%), and update (90.1%) at 0.000 procedural pollution, with 0/48 family losses on W.Prec. or Poll. across all baselines.

EvoHarnessBench: Continual Evolution (Table 2)

MethodACCSolvedBWT ScoreBWT PassFWTTokens/Task↓Calls↓
No-Evolve (W₀)71.91%48.84%−6.00 pp−5.13 pp–131,03214.47
MemToolAgent68.00%51.16%−4.02 pp+2.56 pp−4.98 pp82,42615.84
RRSI74.28%51.16%−0.45 pp0.00 pp+1.79 pp84,51913.19
GEPA70.69%48.84%−3.97 pp−2.56 pp+13.61 pp85,15617.05
Meta-Harness74.52%51.16%+5.83 pp+5.13 pp−13.21 pp78,70514.19
GUARD (Ours)80.80%58.14%+0.85 pp+2.56 pp0.00 pp59,651 (−54.5%)11.21 (−22.5%)

GUARD is the only arm with no negative backward or forward transfer, achieves 0/43 budget exhaustions (vs. 6–10 on baselines), and solves 9.75 tasks per 1M tokens (1.50× the best baseline).

Drift Gauntlet: Continual RSI Under Hidden Drift (Table 3)

MethodFinal MeanΔ_K¯ΔPlast. PForgetting FStab. SBlock Drops↓
No-Evolve (W₀)69.92%0.000.000.000.000.000/12
Hermes+70.31%+0.40+1.09+2.67−5.59−2.665/12
RRSI67.92%−2.00−0.47+4.49−8.11−6.028/12
GEPA72.40%+2.48+2.18+2.80−1.09−0.353/12
Meta-Harness71.21%+1.30+0.15+3.28−0.07−2.514/12
GUARD (Ours)73.22%+3.30+2.71+4.550.00−0.930/12

GUARD is the only arm whose stream-mean accuracy never decreases across all 12 block transitions, achieving both top plasticity (+4.55 pp) and zero post-block forgetting.

Self-Reviewer Calibration and Gate Fidelity (Table 4)

  • Layer-1 sandbox checks curb LLM-only optimism: 22.2% of execution-broken traces pass 1-layer review vs. 0.0% under 2-layer
  • 19/19 execution-gated promotions satisfy Δ_test ≥ 0 (15 strictly > 0, 4 ties)
  • Removing Analyze drops macro ACC by −7.33 pp; removing Soft-HURT drops it by −10.35 pp
  • Parallel M=2 pooling (81.08% ACC) beats matched-budget sequential search (78.85%, p=0.0267) with 64% fewer rejected gate rounds

Theoretical and Practical Implications

Theoretical Contributions

  1. Non-regressive gating guarantee: Theorem 1 proves self-review scores never decrease on gated task buffers, with concentration bounds that scale as Ω(σ2log⁡(neval∣G∣/α)/Δ2)\Omega(\sigma^2 \log(n_{\text{eval}}|G|/\alpha)/\Delta^2) samples

  2. Removal of passive-discovery penalty: Theorem 2 shows dense per-criterion grading eliminates the 1/ϵ1/\epsilon dependence on rare task invariants that plagues passive end-task learners

  3. Strict regression dominance: Proposition 1 proves paired priority uniquely satisfies both exact pinning (no regressed/failed criterion is pinned) and strict dominance (fresh regressions outrank static gaps), unlike frequency ranking or signed differences alone

Practical Implications

  • Deployment efficiency: GUARD cuts tokens per task by 54.5% and MCP calls by 22.5% while improving accuracy—the compact playbook (2,315 chars vs. 4,538–10,013 on baselines) prevents budget exhaustion
  • No catastrophic forgetting: By keying playbook sections to pre-execution observable tool/schema signatures and gating with Soft-HURT, GUARD achieves zero post-block forgetting across all four drift streams
  • Inspectability: Unlike weight-space RL, GUARD produces inspectable conditional rules with explicit [iff] preconditions, aiding debugging and trust
  • Teacher–student applicability: GUARD works both in strictly self-referential single-model settings (Gemini 4 for all roles) and cross-model teacher–student distillation pipelines

Conclusion

GUARD addresses the central challenge of continual harness RSI without external graders by treating the text harness as a non-parametric policy component updated via regret-guided, group-relative evolution:

  • Analyze separates edit-induced regressions (Δ_hurt) from unresolved gaps (Δ_gap) while pinning mastered criteria
  • Propose selects among parallel candidate edits via critic-free group-standardized fitness and contrasts best/worst candidates to rewrite proposal prompts
  • Gate (Soft-HURT) promotes only net-positive updates with formal non-regression guarantees

Across four benchmark settings, GUARD improves upon or matches the base agent in held-out accuracy, achieves 19/19 non-regressive execution-gated promotions, sustains monotonic pass@1 gains with zero post-block forgetting under hidden stream drift, and cuts hosted tokens per task by 54.5%.

Future directions suggested by the work include: extending the framework to more complex harness structures (code-level control logic), exploring the interplay between signature routing and replay-buffer design, and investigating whether the theoretical guarantees can be strengthened to finite-sample certificates at benchmark task counts.

Related papers