Memory Reward Inflation in Self-Improving LLM Agents

Summary (Overview)

• Identifies a novel failure mode ("Echo Gap") in label-free, self-improving LLM memory agents: when LLMs self-grade their own stored episodes, incorrect memories receive inflated reward scores, and this inflation compounds through retrieval and reuse rather than averaging out.
• Formalizes the Error-Independence Assumption (EIA) as a necessary condition for correcting reward inflation: a verifier must both track truth (high Corr(V, U)) and decorrelate its errors from the original self-grade bias (low |Corr(ν, b)|). The recoverable payoff is proven to be a closed-form function of exactly these two quantities (Proposition 2).
• Proves two amplification theorems: Theorem 1 shows inflation amplifies multiplicatively through both a retrieval channel (softmax ranking, up to $e^{b/T}$) and a trust channel (persists even under similarity-only retrieval), while Theorem 2 demonstrates the write-back loop drives the bank to a corrupted attractor exceeding one-shot corruption.
• Demonstrates that stronger parametric re-graders fail to correct the bank: across model families (GPT-5.4 family, ensembles), capability alone does not provide the required error decorrelation, whereas a retrieval-based information verifier satisfies EIA and recovers most of the oracle de-inflation payoff (+0.59 vs +0.75).
• Introduces LUCID (Leniency-corrected Utility Calibration via Independent Debiasing), an answer-free de-inflation algorithm that raises BIRD text-to-SQL execution accuracy to 56.9%, outperforming both a Memento-style self-graded agent (54.0%) and a memory-less control (52.4%).

Introduction and Theoretical Foundation

Background and Motivation

Large Language Model (LLM) agents increasingly improve behavior from past experience without updating model parameters. This improvement comes from an external memory system where the agent:

1. Stores past episodes (query, answer, utility score)
2. Assigns each episode a utility score
3. Retrieves episodes judged relevant for similar future tasks
4. Writes new episodes back into the bank

While the LLM remains parametrically fixed, future behavior changes through which memories are selected and trusted. This creates a memory decision process where retrieval acts as an implicit, non-parametric policy.

The Core Problem: Proxy-Reward Failure

In real-world deployment, ground-truth labels are unavailable when memories are written. The practical substitute is LLM grading—the agent or a related judge assigns utility scores. This creates the Echo Gap: a memory-specific form of reward hacking where:

• Incorrect episodes receive inflated rewards
• The agent preferentially reuses the very mistakes it is most confident in
• Errors compound through memory reuse rather than averaging out
• The confirming judge's errors remain correlated with the original self-grading bias

Formal Framework

A memory bank contains episodes:

$$m_i = (q_i, a_i, Q_i) \tag{1}$$

where $q_i$ is the task, $a_i$ is the response, and $Q_i$ is the stored utility score. Each memory carries three signals:

$$r_i, \quad V_i, \quad U_i \tag{3}$$

• $r_i$: LLM grade (self-grade at write time)
• $V_i$: verifier score (alternative audit channel)
• $U_i$: ground-truth utility (post-hoc measurement only)

The self-grade bias and verifier error are:

$$b_i = r_i - U_i, \qquad \nu_i = V_i - U_i \tag{4}$$

The Echo Gap Conditions

The Echo Gap has two defining components. First, self-grades inflate wrong memories:

$$\mathbb{E}[b_i \mid U_i = 0] > 0 \tag{5}$$

Second, inflation is operationally amplified by reuse among wrong memories:

$$\operatorname{Cov}(b_i, n_i \mid U_i = 0) > 0 \tag{6}$$

where $n_i$ is the reuse count of memory $m_i$.

Methodology

Bank-Level Experiments (Factual QA)

The authors constructed factual memory banks where:

• Claude Haiku 4.5 served as the answering and self-grading agent
• Cross-vendor banks used GPT-5.4-mini and frontier GPT-5.4 for reproduction
• Ground truth was used only post-hoc to measure whether self-grades matched correctness

Verifier Evaluation

Six base verifiers were tested across three classes:

1. Same-model baselines: self-consistency (resampling), adversarial same-model judge
2. Cross-vendor parametric re-graders: GPT-5.4-nano, GPT-5.4-mini, frontier GPT-5.4
3. Retrieval-based information verifier: conditions on external evidence

Ensembles (mean/majority/capability-weighted) and a gold-fit optimal aggregator were also tested.

End-to-End BIRD Experiment

A faithful Memento-style memory agent was evaluated on the full BIRD text-to-SQL development set (1,534 questions):

• Retrieval: top-k=4 by SimCSE cosine similarity (same encoder as Memento)
• Planner conditions on retrieved episodes, executor produces SQL
• Self-grader scores without access to reference answers
• Three arms: no-memory control, naive self-graded memory, LUCID

LUCID's Answer-Free De-Inflation Signal

The detector reads only deployment-available evidence across three channels:

1. Execution: query errors, times out, or returns non-deterministic results
2. Degeneracy: empty or all-NULL results for questions expecting answers
3. Literal grounding: filters on entity-like literals not in the question (fingerprint of copied values from wrong memories)

Empirical Validation / Results

Self-Grading Inflates Wrong Memories Across Model Families

Table 2: Leniency = Pr[self = correct | U = 0]. Self-grade inflation persists across model families and capability levels.

EIA Experiment: Parametric Re-Graders Fail

Table 3: Only the retrieval-based information verifier satisfies EIA on both axes.

De-Inflation Payoff on the Bank

Table 4: EIA predicts actual correction payoff. The retrieval-based verifier recovers most of the oracle ceiling.

LUCID vs. Generic Memory Filtering

Table 7: LUCID dominates generic memory filtering at every budget; the gain comes from which memories are demoted, not how many.

End-to-End BIRD Results

Table 5: LUCID consistently outperforms both baselines. Paired 95% CIs for LUCID−naive exclude zero (+1.9 [+0.2, +3.6] in seed 0; +3.9 [+2.1, +5.6] in seed 1).

Dynamical Model Validation

Measured parameters on BIRD: $(\kappa, \ell, s, e_0) = (0.38, 0.76, 0.90, 0.32)$

• Memory-to-behavior coupling: $\kappa \approx 0.38$ (positive within every difficulty stratum)
• Predicted corrupted attractor: $p^\star = 0.45$
• Observed trusted-bank corruption: 0.42 (within 0.03)
• Static no-loop account underpredicts: 0.28
• Compounding ratio: 1.6× the one-shot value

Theoretical and Practical Implications

Theoretical Contributions

Theorem 1 (Echo-Gap amplification) proves inflation amplifies through two channels:

• Retrieval channel (score-ranked softmax retrieval, $\pi(i) \propto e^{Q_i/T}$):

$$1 \le A_{\mathrm{ret}} = \frac{e^{b/T}(W + Re^{1/T})}{We^{b/T} + Re^{1/T}} \le e^{b/T} \tag{8}$$

with $A_{\mathrm{ret}} \to e^{b/T}$ in the sparse-error limit $W \ll Re^{1/T}$.

• Trust channel (similarity-only retrieval, where $A_{\mathrm{ret}} = 1$):

$$A_{\mathrm{tr}} = \frac{\tau(b)}{\tau(0)} > 1 \tag{9}$$

Theorem 2 (Corrupted attractor) shows the write-back loop has a fixed point $p^\star$ solving:

$$\kappa(\ell - s)p^2 + (e_0(\ell - s) + s - \kappa\ell)p - e_0\ell = 0$$

which strictly exceeds the one-shot corruption $p_0^\star = \frac{e_0\ell}{e_0\ell + (1-e_0)s}$ when $\kappa > 0$.

Proposition 2 (EIA necessity) proves that for de-inflation $Q_i' = Q_i - \alpha(Q_i - V_i)$, the corrected inflation variance is:

$$\operatorname{Var}(b') = \bigl(1 - \alpha(1 - \beta)\bigr)^2\operatorname{Var}(b) + \alpha^2\operatorname{Var}(\eta) \tag{11}$$

If the verifier echoes the self-grade ($\beta \ge 1$), demotion can only re-inject inflation. The recoverable payoff:

$$\operatorname{Var}(b) - \min_\alpha \operatorname{Var}(b') = \frac{(1-\beta)^2\operatorname{Var}(b)^2}{(1-\beta)^2\operatorname{Var}(b) + \operatorname{Var}(\eta)} \tag{12}$$

is strictly increasing as the error decorrelates ($\beta \to 0$).

Practical Implications

1. Label-free self-improvement is a reward-design problem: Without gold labels, the stored score is a proxy reward, and its integrity determines whether memory improves or degrades behavior.

2. Capability alone is insufficient: Stronger parametric re-graders improve average truth-tracking but preserve the bias that distorted the bank. Judge ensembles inherit shared blind spots (correlated errors, Corr(νᵢ, νⱼ) = +0.69).

3. De-correlated evidence channels are essential: Retrieval-based factual checking, execution-based tests, and invariant checks provide corrective signals because their failure modes differ from the agent's own self-grade.

4. Precision governs de-inflation: De-inflation is beneficial when detector precision satisfies:

$$\rho > \rho^\star := \frac{h_r}{g_r + h_r} \tag{19}$$

Low recall is acceptable; what matters is that flagged episodes are wrong often enough.

Conclusion

The Echo Gap reveals a fundamental risk in label-free self-improving LLM agents: when stored scores are self-graded, wrong answers can receive inflated utility, and when that inflation couples to reuse, a feedback loop emerges where overvalued errors are repeatedly surfaced as useful precedents. The authors proved that correcting this requires the Error-Independence Assumption—a signal that both tracks truth and avoids echoing the self-grade bias—and demonstrated that answer-free de-inflation (LUCID) provides consistent gains on BIRD text-to-SQL.

The broader lesson is that label-free memory improvement requires evidence whose failure mode differs from the agent's own. Safer self-improving agents should:

• Separate memory writing from memory trust
• Audit stored episodes with decorrelated, answer-free evidence
• De-inflate memories that influence future behavior

Future directions include designing constructive decorrelation mechanisms (the paper shows $W \perp Z \mid U$ is a sufficient condition), extending to other domains (code agents, math/proof, tool use), and understanding when agents cross the stability threshold $\kappa^\star(\ell, s)$ beyond which corruption collapses the benign regime.