# GUARD: Stabilizing Recursive Self-Improvement with Regret Guidance

> GUARD achieves non-regressive agent harness self-improvement without external graders by separating edit-induced regressions from capability gaps via paired regret, guaranteeing monotonic accuracy gains.

- **Source:** [arXiv](https://arxiv.org/abs/2611.23476)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/hKdOOe
- **Whiteboard:** https://picx.dev/p/hKdOOe/image

## Summary

## Summary (Overview)

- **GUARD (Grader-free UED with Adaptive Rule Distillation)** is a closed-loop framework for reference-free continual evolution of agent harnesses (prompts, skills, tool rules, memory) around a frozen base LLM, without weight updates or external graders.
- The key insight is decomposing tasks into **pre-execution self-checklists** and guiding harness updates via **paired directional regret**, which separates newly broken behaviors (Δ_hurt) from unresolved capability gaps (Δ_gap).
- GUARD uses **GRPO-inspired parallel candidate pooling** with critic-free group-relative fitness for selection, and a **Soft-HURT promotion gate** that accepts edits only when gains outweigh regressions, guaranteeing non-regression on gated task buffers.
- Across benchmarks (PAST-Bench, EvoHarnessBench, Drift Gauntlet, professional suites), GUARD achieves **19/19 non-regressive promotions**, **zero post-block forgetting** under hidden drift, and **54.5% token reduction** per task, with monotonic accuracy gains.
- Three theoretical guarantees are provided: non-regressive gating with concentration bounds (Theorem 1), removal of the 1/ε passive-discovery penalty via dense checklist grading (Theorem 2), and strict regression dominance in priority ranking (Proposition 1).

---

## Introduction and Theoretical Foundation

### Background and Motivation

Modern language model agents pair a frozen base model with an editable **harness** of prompts, skills, tool interfaces, memory, and control logic. Recursive self-improvement (RSI) adapts this harness from experience without weight updates—a practical approach since frontier models sit behind black-box APIs and open-world tasks rarely provide ground-truth rewards.

**Core problem:** Self-editing without external graders often backfires. Free-form reflection and scalar scores conflate two distinct failure signals:
1. **Edit-induced regressions** (behaviors the latest edit newly broke)
2. **Unresolved capability gaps** (things the agent still hasn't solved)

Mixing these signals causes three failure modes:
- **Local overfitting (over-triggering):** broad rules written for recent failures break tasks that already worked
- **Catastrophic forgetting:** shifts in workflow/topic/domain overwrite earlier skills
- **Memory pollution:** unverified additions accumulate stale or conflicting rules that bloat the prompt

### Theoretical Foundation

The framework draws on two ideas from reinforcement learning:
1. **Curriculum regret from Unsupervised Environment Design (UED)** [7, 14, 15, 35] — prioritizing high-regret levels, transferred here to directional criterion ranking over task checklists
2. **Group Relative Policy Optimization (GRPO)** [37, 3] — critic-free group-relative scoring, applied to non-parametric harness optimization without weight updates

**Problem formulation:** An agent $A = (\pi_\theta, W_{t,g})$ pairs a frozen policy $\pi_\theta$ with an editable harness state $W_{t,g}$ at round $t$. The reference-free evolver updates:

$$W_{t,g} = \mathcal{E}\left(W_{t-1,g}, \mathcal{H}_{t,g}\right), \quad t = 1, \ldots, T \leq T_{\text{max}}$$

The goal is maximizing $\mu_g(W_{T,g}) = \mathbb{E}_{q \in Q_g, \tau \sim \pi_\theta(\cdot|q, W_{T,g})}[V^*(q, \tau)]$ where $V^*$ is the unobserved task evaluator.

**Reference-free** means neither the solver, self-reviewer, nor distiller accesses held-out answers, test cases, hidden verifiers, oracle skills, or rubrics—but the framework still leverages observable structure: task prompt distributions, tool declarations, interaction trajectories, and criterion-level self-review judgments.

---

## Methodology

GUARD instantiates the evolver as a four-stage closed loop:

### Stage 1: Reference-Free Self-Review via Task Checklists (§4.1)

**Pre-execution checklist:** The self-reviewer reads the task prompt, tool declarations, and schema once at Turn 0 to derive a fixed checklist $\mathcal{C}(q)$ of atomic binary predicates across two classes:
- **Execution-verifiable predicates:** clean sandbox execution, valid entity IDs/columns, idempotent mutations (checked deterministically on an ephemeral sandbox copy)
- **Semantic predicates:** honoring prompt constraints, recipient, format, safety rules (graded individually from prompt, output, and trace)

**Two-layer grading:** Trajectory $\tau$ receives criterion score $S(\tau)$ and strict score $S_{\text{strict}}(\tau)$:

$$S(\tau) = 100 \cdot \frac{1}{|\mathcal{C}_{\text{gr}}(\tau)|} \sum_{c \in \mathcal{C}_{\text{gr}}(\tau)} r(\tau, c)$$

$$S_{\text{strict}}(\tau) = \begin{cases} 0 & \text{on blocking sandbox error} \\ 100 & \text{if all graded criteria pass} \\ S(\tau)/2 & \text{otherwise} \end{cases}$$

This exploits **generation–verification asymmetry**: checking one atomic predicate after execution requires localized attention, while long-horizon generation couples many constraints.

### Stage 2: Paired Regret Ranking — Analyze (§4.2)

**Definition 1 (Criterion obedience and paired regret):** For trajectories $\mathcal{T}_c(W)$ grading criterion $c$ under harness $W$:

$$\text{obey}(c; W) = \frac{1}{|\mathcal{T}_c(W)|} \sum_{\tau \in \mathcal{T}_c(W)} r(\tau, c) \in [0, 1]$$

Comparing the two most recent incumbents $W_{t-2}$ and $W_{t-1}$:

$$\Delta_{\text{hurt},t}(c) = \max\left(0, \text{obey}(c; W_{t-2}) - \text{obey}(c; W_{t-1})\right)$$
$$\Delta_{\text{gap},t}(c) = 1 - \text{obey}(c; W_{t-1})$$
$$\text{Priority}_t(c) = \Delta_{\text{hurt},t}(c) + \Delta_{\text{gap},t}(c) \in [0, 2]$$

Criteria are partitioned into three diagnostic lists: **Regressed** (Δ_hurt > 0, wrap offending rules in [iff] preconditions), **Unsolved Gap** (Δ_gap > 0), and **Mastered** (Priority = 0, pinned against overwriting).

### Stage 3: Group-Relative Candidate Pooling — Propose (§4.3)

Inspired by GRPO, GUARD samples $M \geq 2$ candidate harnesses in parallel, scores them on the same task batch, and ranks by **critic-free group-standardized fitness**:

$$R_m = \text{Perf}_m - \beta|Q_m^-| + \gamma|Q_m^+|, \quad \beta > \gamma \geq 0$$
$$\hat{A}_m = \frac{R_m - \mu_R}{\sigma_R}, \quad \sigma_R = \sqrt{\frac{1}{M}\sum_{m=1}^{M}(R_m - \mu_R)^2 + 10^{-6}}$$

**ContrastEvolve** contrasts the best ($m^+$) and worst ($m^-$) candidates to update strategy prompts:

$$\mathcal{P}_{t+1,g} = \{s^+_{t+1}\}^{\lfloor M/2\rfloor} \cup \{s^-_{t+1}\}^{M - \lfloor M/2\rfloor}$$

### Stage 4: Soft-HURT Promotion Gate — Gate (§4.4)

The group-averaged regression penalty:

$$\text{Hurt}(\tilde{W}, W) = \frac{1}{|Q_g|} \sum_{q \in Q^-(\tilde{W}, W)} \left(S_q(W) - S_q(\tilde{W})\right)$$

**Soft-HURT rule:** Promote candidate $\tilde{W}$ over $W_{t-1}$ only when:

$$\bar{S}(\tilde{W}) - \bar{S}(W_{t-1}) > \max\left(\text{Hurt}(\tilde{W}, W_{t-1}), \mu_{\min}\mathbb{I}[|Q^-(\tilde{W}, W_{t-1})| > 0]\right)$$

When a candidate fails solely due to isolated regressions ($1 \leq |Q^-| \leq h_{\text{rep}}$), GUARD triggers at most one in-round **[iff] scope-guard repair**. Rejected edits are logged to a **rejection ledger** $\mathcal{L}_g$ so bad changes are not repeated.

### Theoretical Guarantees (§4.5)

**Theorem 1 (Non-Regressive Gating and Concentration):** Under the gate of (12):
- (i) No downward drift: $\bar{S}(W_T) = \bar{S}(W_0) + \sum_{t=1}^{T}\Delta_t \geq \bar{S}(W_0)$
- (ii) Regressions are paid for: $\sum_{q \in Q_t^-}\left(S_q(W_{t-1}) - S_q(W_t)\right) < |Q_g|\Delta_t$
- (iii) Concentration: with probability ≥ 1−α, $\mu(W_T) \geq \mu(W_0) + \sum_{t=1}^{T}\Delta_t - 2\sigma\sqrt{\frac{2\log(2n_{\text{eval}}|G|/\alpha)}{|Q_g|N_{\text{rep}}}}$
- (iv) Factored speedup from per-group gating

**Theorem 2 (Targeted Repair vs. Passive Discovery):** Passive learning requires $\Omega(\log J / \epsilon)$ rounds to discover $J$ rare invariants (each surfacing with probability $p_j \leq \epsilon$), while dense checklist grading achieves $\mathbb{E}[T_{\text{target}}] \leq \frac{J}{\rho b_{\min}} + \frac{1 + H_{b_{\min}-1}}{\rho}$, removing the $1/\epsilon$ dependence.

**Proposition 1 (Strict Regression Dominance):** In the single-rollout binary case, a fresh regression (1→0, priority 2) outranks every non-regressed criterion (priority at most 1). Priority $t(c) = 0$ iff $\text{obey}(c; W_{t-1}) = 1$ and $\Delta_{\text{hurt},t}(c) = 0$ — no regressed or failed criterion is ever pinned.

---

## Empirical Validation / Results

### PAST-Bench: Cross-Session Memory Write Path (Table 1)

| Arm | Task | Proc. | Mech. | Upd. | W.Prec. | Recall | Ret. | Poll.↓ | Evolve |
|-----|------|-------|-------|-------|-------|--------|-------|--------|--------|
| No-Evolve (W₀) | 0.372 | 0.427 | 0.454 | 0.278 | 0.000 | 0.000 | 0.990 | 0.000 |
| Hermes+ | 0.852 | 0.808 | 0.840 | 0.791 | 0.724 | 0.854 | 1.000 | 0.167 |
| RRSI | 0.809 | 0.714 | 0.725 | 0.901 | 0.438 | 0.811 | 0.974 | 0.500 |
| GEPA | 0.858 | 0.815 | 0.837 | 0.777 | 0.719 | 0.858 | 0.996 | 0.167 |
| Meta-Harness | 0.812 | 0.677 | 0.739 | 0.903 | 0.479 | 0.814 | 0.998 | 0.500 |
| **GUARD (Ours)** | **0.854** | **0.830** | **0.878** | **0.924** | **0.776** | **0.856** | **0.999** | **0.167** | **0.436** |

GUARD leads procedural task (83.0%), procedural mechanism (90.0%), and update (90.1%) at **0.000 procedural pollution**, with 0/48 family losses on W.Prec. or Poll. across all baselines.

### EvoHarnessBench: Continual Evolution (Table 2)

| Method | ACC | Solved | BWT Score | BWT Pass | FWT | Tokens/Task↓ | Calls↓ |
|--------|-----|--------|-----------|----------|-----|--------------|--------|
| No-Evolve (W₀) | 71.91% | 48.84% | −6.00 pp | −5.13 pp | – | 131,032 | 14.47 |
| MemToolAgent | 68.00% | 51.16% | −4.02 pp | +2.56 pp | −4.98 pp | 82,426 | 15.84 |
| RRSI | 74.28% | 51.16% | −0.45 pp | 0.00 pp | +1.79 pp | 84,519 | 13.19 |
| GEPA | 70.69% | 48.84% | −3.97 pp | −2.56 pp | +13.61 pp | 85,156 | 17.05 |
| Meta-Harness | 74.52% | 51.16% | +5.83 pp | +5.13 pp | −13.21 pp | 78,705 | 14.19 |
| **GUARD (Ours)** | **80.80%** | **58.14%** | **+0.85 pp** | **+2.56 pp** | **0.00 pp** | **59,651 (−54.5%)** | **11.21 (−22.5%)** |

GUARD is the only arm with no negative backward or forward transfer, achieves 0/43 budget exhaustions (vs. 6–10 on baselines), and solves 9.75 tasks per 1M tokens (1.50× the best baseline).

### Drift Gauntlet: Continual RSI Under Hidden Drift (Table 3)

| Method | Final Mean | Δ_K | ¯Δ | Plast. P | Forgetting F | Stab. S | Block Drops↓ |
|--------|-----------|-----|-----|---------|--------------|---------|--------------|
| No-Evolve (W₀) | 69.92% | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0/12 |
| Hermes+ | 70.31% | +0.40 | +1.09 | +2.67 | −5.59 | −2.66 | 5/12 |
| RRSI | 67.92% | −2.00 | −0.47 | +4.49 | −8.11 | −6.02 | 8/12 |
| GEPA | 72.40% | +2.48 | +2.18 | +2.80 | −1.09 | −0.35 | 3/12 |
| Meta-Harness | 71.21% | +1.30 | +0.15 | +3.28 | −0.07 | −2.51 | 4/12 |
| **GUARD (Ours)** | **73.22%** | **+3.30** | **+2.71** | **+4.55** | **0.00** | **−0.93** | **0/12** |

GUARD is the **only arm whose stream-mean accuracy never decreases** across all 12 block transitions, achieving both top plasticity (+4.55 pp) and zero post-block forgetting.

### Self-Reviewer Calibration and Gate Fidelity (Table 4)

- **Layer-1 sandbox checks** curb LLM-only optimism: 22.2% of execution-broken traces pass 1-layer review vs. 0.0% under 2-layer
- **19/19 execution-gated promotions** satisfy Δ_test ≥ 0 (15 strictly > 0, 4 ties)
- Removing Analyze drops macro ACC by −7.33 pp; removing Soft-HURT drops it by −10.35 pp
- Parallel M=2 pooling (81.08% ACC) beats matched-budget sequential search (78.85%, p=0.0267) with 64% fewer rejected gate rounds

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Non-regressive gating guarantee:** Theorem 1 proves self-review scores never decrease on gated task buffers, with concentration bounds that scale as $\Omega(\sigma^2 \log(n_{\text{eval}}|G|/\alpha)/\Delta^2)$ samples

2. **Removal of passive-discovery penalty:** Theorem 2 shows dense per-criterion grading eliminates the $1/\epsilon$ dependence on rare task invariants that plagues passive end-task learners

3. **Strict regression dominance:** Proposition 1 proves paired priority uniquely satisfies both exact pinning (no regressed/failed criterion is pinned) and strict dominance (fresh regressions outrank static gaps), unlike frequency ranking or signed differences alone

### Practical Implications

- **Deployment efficiency:** GUARD cuts tokens per task by 54.5% and MCP calls by 22.5% while improving accuracy—the compact playbook (2,315 chars vs. 4,538–10,013 on baselines) prevents budget exhaustion
- **No catastrophic forgetting:** By keying playbook sections to pre-execution observable tool/schema signatures and gating with Soft-HURT, GUARD achieves zero post-block forgetting across all four drift streams
- **Inspectability:** Unlike weight-space RL, GUARD produces inspectable conditional rules with explicit [iff] preconditions, aiding debugging and trust
- **Teacher–student applicability:** GUARD works both in strictly self-referential single-model settings (Gemini 4 for all roles) and cross-model teacher–student distillation pipelines

---

## Conclusion

GUARD addresses the central challenge of continual harness RSI without external graders by treating the text harness as a non-parametric policy component updated via regret-guided, group-relative evolution:

- **Analyze** separates edit-induced regressions (Δ_hurt) from unresolved gaps (Δ_gap) while pinning mastered criteria
- **Propose** selects among parallel candidate edits via critic-free group-standardized fitness and contrasts best/worst candidates to rewrite proposal prompts
- **Gate (Soft-HURT)** promotes only net-positive updates with formal non-regression guarantees

Across four benchmark settings, GUARD improves upon or matches the base agent in held-out accuracy, achieves 19/19 non-regressive execution-gated promotions, sustains monotonic pass@1 gains with zero post-block forgetting under hidden stream drift, and cuts hosted tokens per task by 54.5%.

**Future directions** suggested by the work include: extending the framework to more complex harness structures (code-level control logic), exploring the interplay between signature routing and replay-buffer design, and investigating whether the theoretical guarantees can be strengthened to finite-sample certificates at benchmark task counts.

---

_Markdown view of https://picx.dev/p/hKdOOe, served by PicX — AI-generated visual whiteboard summaries of research papers._
