PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

Authors: Shuhan Xue, Zixin Ding, Yichen Shen, Yinjie Wang, Zhenfei Yin, Yingcheng Wu, Yuxin Chen, Mengdi Wang, Ling Yang

Summary (Overview)

• New benchmark for self-evolution: PAST-Bench is a performance-attribution benchmark for personal AI agents, testing whether retained experience actually improves future behavior across sessions. It spans 26 task-family scenarios and 204 episodes across four capabilities: Memory, Procedural Reuse, Information Gathering, and Update.

• Matched persistence controls: Each episode is evaluated under paired conditions (persistence-on vs. persistence-off), isolating the contribution of retained experience from base-model and runtime effects. The benchmark reports both task-score gaps (Δ) and mechanism-evidence scores (Mech) to separate outcome gains from intended pathway support.

• Diagnostic findings: Across seven base models and four agent frameworks, improvement from persistence is real but uneven. Agents with the same headline gain can differ markedly in mechanism evidence (e.g., Hermes vs. nanobot: 0.64 vs. 0.57 at the same Δ = +0.13).

• Hermes+ framework: A new agent framework extending Hermes with five targeted runtime interventions (Plan, Render, Route, Gate, Close) that raises the average gain from retained experience (Δ: +0.13 → +0.15, Mech: 0.64 → 0.73), with super-additive composition on Update (Δ = +0.24).

Introduction and Theoretical Foundation

Background and Motivation

Recursive self-improvement (RSI) concerns an AI system's ability to use its own operational experience to improve future capabilities. While stronger forms of RSI may involve modifying model parameters or architectures, a more immediate operational layer exists in personal AI agents. These agents persist across sessions, accumulating files, memories, skills, and histories over time.

The paper introduces the concept of online self-evolution: "a personal agent changes its future behavior by reusing experience accumulated during prior interactions, without model retraining, prompt optimization, or long-context adaptation." This is the basic loop that must be closed before stronger forms of recursive improvement can be built: identifying useful experience, preserving it, retrieving it when relevant, applying it correctly, and revising it when outdated.

The core evaluation problem is performance attribution: if later-session performance improves, the gain might come from retained experience—or from the base model, runtime, prompt, retrieval shortcuts, task difficulty, or scoring noise. Existing benchmarks cannot make this distinction.

Key contribution of PAST-Bench is the trajectory-level evaluation with matched controls: earlier episodes give the agent a chance to save reusable experience, later episodes test reuse, and matched control episodes strip persistence so gains can be read against a no-persistence baseline.

Methodology

Benchmark Construction

PAST-Bench targets four core capabilities of online self-evolution:

• Memory (5 families / 41 episodes): Declarative pathway for casual user-facing facts (preferences, constraints, policies). Tests one-shot lookup-and-apply with trigger wording removed.
• Procedural Reuse (8 families / 64 episodes): Imperative pathway for multi-step technical workflows (SOPs, playbooks, build pipelines). Order errors and wrong-tool substitutions are graded as failures.
• Information Gathering (6 families / 48 episodes): The relevant artifact is pre-seeded; the test is whether the agent proactively retrieves under noisy context.
• Update (7 families / 51 episodes): Can a second write override a first one without leaking the first? Tests fact correction, rule migration, exception expiry, and SOP patching.

Evaluation Pipeline

Episodes play one of four roles: cold (first-contact), learning (deposit target), evaluation (probe reuse), and control (check for shortcuts or stale reuse). Each evaluation episode is graded under a matched ablation.

The primary metric per family is:

$$\Delta_f = S_f^{\mathrm{w/evolve}} - S_f^{\mathrm{w/o-evolve}}$$

where $S_f$ is the within-family mean of the per-episode task score $s_e \in [0,1]$. The capability-level Δ is the macro-average over families. Control episodes set bounds for shortcut, surface-memorization, stale-reuse, and wrong-mechanism explanations.

Hermes+ Framework

Five targeted interventions, one per stage of the agent loop:

Empirical Validation / Results

Main Results (Table 2: Hermes, varying base models)

Subscripts give each capability's signed share of the row's total absolute movement; per-row absolute values sum to 100%.

Fixed-Model Agent Comparison (Table 3: MiniMax-M2.7)

Single-Mechanism Ablations (Table 4: MiniMax-M2.7)

Single mechanisms support their target diagnoses: E2 gives the best Memory score (0.80), E3 the largest Procedural Δ (+0.10), E4 the largest Info Δ (+0.17), and E5 the strongest single-mechanism Update Δ (+0.16). The full Hermes+ has a super-additive composition on Update (+0.24, well above any single mechanism).

Cross-Model Transfer (Table 6)

Theoretical and Practical Implications

• Performance attribution matters: The paper shows that a single headline Δ can hide fundamentally different persistence paths. Agents with identical task-score gains can differ in whether the gain is supported by evidence of the intended save-retrieve-update pathway.

• Capability-specific self-evolution: Improvements are not uniform; different base models concentrate gains on different capabilities (e.g., Kimi K2.6 on Memory 49%, GLM-5.1 on Update 46%). The capability the model already excels at is where retained experience helps most.

• Mechanism diagnosis is feasible: The five failure categories (unconditioned plans, loose memory storage, unsaved procedures, skipped retrieval, stale evidence) map one-to-one onto loop-stage interventions, demonstrating that trace-level diagnosis can guide framework design.

• Caveats on composition: The +0.02 Overall Δ difference between Hermes and Hermes+ is smaller than run-to-run variation ($0.13 \pm 0.04$ vs. $0.15 \pm 0.06$), so the clear mean shift is on Update (+0.12 → +0.24). Hermes+ is a diagnostic scaffold, not a universal improvement.

Conclusion

PAST-Bench provides the first benchmark that:

1. Pairs persistence-on/off evaluations within task families to isolate retained-experience contributions
2. Reports mechanism evidence alongside task scores to separate outcome gains from intended pathway
3. Supports both model-side and framework-side isolation (the only benchmark supporting all four methodology axes: retained experience, model comparison, framework comparison, trajectory diagnosis)

Hermes+ raises the reported mean Overall Δ from +0.13 to +0.15 and Mech from 0.64 to 0.73, with its clearest gain on Update. The effect is not uniform across capabilities or base models; the paper treats Hermes+ as a diagnostic scaffold rather than a universal improvement.

Future directions include: human-authored scenarios, longer task sequences, cross-family transfer, stronger forms of recursive improvement (tool-use strategies, long-horizon plans), counterfactual interventions for causal attribution, and adaptive routing of experience across memory, skills, and session history.