Summary (Overview)

• Problem formulation: The paper introduces long-horizon memorization—a continual learning setting where language models sequentially learn 100 query-answer tasks via supervised fine-tuning (SFT) without retaining earlier examples or receiving task identifiers at inference.
• Key hypothesis: Mechanisms addressing complementary sources of forgetting will be more effective when composed than any individual mechanism alone—organized along two design dimensions: anchors (data, function, weight) and low-rank allocation rules (shared vs. merged LoRA).
• Main result: The best method—combining all three anchors (generative replay, self-distillation, importance-based regularization) with merged LoRA—raises average final retention from 1.2% (naive fine-tuning) to 34.9%, a 28-fold improvement, and consistently ranks among the top 3 methods across all three datasets.
• Factorial analysis: The data anchor (replay) and merged LoRA provide the largest main effects and exhibit a super-additive interaction on all three datasets—meaning their combined benefit far exceeds the sum of their individual gains.
• Methodological contributions: Three new 100-task memorization datasets (Symbol-QA, LLM-QA, Real-QA), task-level successive halving (TSH) for efficient combinatorial search, and a $2^4$ factorial experimental design for measuring interaction effects.

---

Introduction and Theoretical Foundation

Background and Motivation

Language models that continuously update their parameters must remember previously learned information despite subsequent updates. The paper distinguishes this from prompting/retrieval approaches, which supply information at inference time but leave it outside the model's parameters. The authors ask whether repeated updates can build and preserve memory within the model itself.

The setting is domain-incremental continual learning (Van de Ven and Tolias, 2019): each of $T=100$ tasks contains query-answer pairs; the model learns tasks sequentially without raw examples from earlier tasks and receives no task identifier at inference. The core challenge is catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999).

Theoretical Foundation

The authors organize the design space of continual learning mechanisms along two dimensions:

1. Anchors — specify what prior information each update should preserve:

   • Data anchor: generative replay of pseudo-sequences from a frozen copy of the previous model (Shin et al., 2017)
   • Function anchor: self-distillation on current-task inputs, preserving earlier model outputs (Li and Hoiem, 2017)
   • Weight anchor: importance-based regularization (EWC, SI) constraining parameter updates (Kirkpatrick et al., 2017; Zenke et al., 2017)

2. Low-rank allocation rules — determine where successive LoRA updates are retained (Hu et al., 2022):

   • Shared LoRA: reuses the same adapter across all tasks
   • Merged LoRA: folds each update into the dense weights, then initializes a new adapter—adapting ReLoRA's merge-and-reinitialize pattern (Lialin et al., 2024)

The central hypothesis: "mechanisms addressing complementary sources of forgetting will retain associations more effectively when composed."

---

Methodology

Problem Setup

The current-task SFT loss is:

$$\mathcal{L}_{\mathrm{SFT}}^{t}(\Theta) = -\mathbb{E}_{(x,y) \sim \mathcal{D}_t}\left[\log p_{\Theta}(x, y)\right] \tag{1}$$

The full objective adds three anchor-based retention terms:

$$\Theta_t = \underset{\Theta}{\arg\min}\; \mathcal{L}_{\mathrm{SFT}}^{t}(\Theta) + \mathcal{R}_D^t(\Theta) + \mathcal{R}_F^t(\Theta) + \mathcal{R}_W^t(\Theta) \tag{2}$$

Three Anchors

Data anchor (generative replay) trains on replayed sequences from a distribution $Q_{t-1}$ over earlier task sequences:

$$\mathcal{R}_D^t(\Theta) = \mathbb{E}_{z \sim Q_{t-1}}[\ell_D(\Theta, z)] \tag{3}$$

The frozen previous model generates complete pseudo-sequences from a task-agnostic replay token (300 sequences per task). Replay loss weight and generation temperature are tuned.

Function anchor (self-distillation) constrains the current model's predictions on current-task inputs relative to a reference distribution $q_{t-1}$:

$$\mathcal{R}_F^t(\Theta) = \mathbb{E}_{x \sim \mu_t}\left[d(q_{t-1}(\cdot | x), p_{\Theta}(\cdot | x))\right] \tag{4}$$

Following Learning without Forgetting, the previous model defines the reference distribution; applied only to current-task data.

Weight anchor (importance regularization) constrains parameter updates with importance-weighted quadratic penalty:

$$\mathcal{R}_W^t(\Theta) = \frac{1}{2}\left(\vartheta - \vartheta_{t-1}^{\star}\right)^{\top} H_{t-1}\left(\vartheta - \vartheta_{t-1}^{\star}\right), \qquad H_{t-1} \succeq 0 \tag{5}$$

Implemented via EWC and Online EWC (diagonal Fisher information) and SI (path-integrated importance).

Low-Rank Allocation

For a pretrained matrix $W_0$, LoRA is specified as $\rho B A$ with $A \in \mathbb{R}^{r \times d_{\mathrm{in}}}$, $B \in \mathbb{R}^{d_{\mathrm{out}} \times r}$, $\rho = \alpha_{\mathrm{LoRA}}/r$. The allocation rules are:

$$W_t = \begin{cases} W_0 + \rho B_t A_t, & \text{shared LoRA}, \\ W_{t-1} + \rho B_t A_t, & \text{merged LoRA}. \end{cases} \tag{6}$$

Shared LoRA continues optimizing the same $A, B$ across all tasks; merged LoRA assigns each task a new pair, folds the update into dense weights after each task, and reinitializes. Both maintain constant retained state size as tasks increase.

Evaluation Metrics

Temporal accuracy matrix $M_{i,j}$ measures accuracy on task $j$ after learning task $i$. Three metrics:

$$\mathrm{Final} = \frac{1}{T}\sum_{j=1}^{T} M_{T,j}, \quad \mathrm{Diag} = \frac{1}{T}\sum_{j=1}^{T} M_{j,j}, \quad \mathrm{Forget} = \frac{1}{T-1}\sum_{j=1}^{T-1}\left(\max_{j \leq i \leq T} M_{i,j} - M_{T,j}\right) \tag{8}$$

Three Memorization Datasets

Real-QA filters out items the model answers correctly in any of five sampled completions.

Task-Level Successive Halving (TSH)

TSH ranks configurations by mean final retention at increasing task horizons:

$$F_r(a) = \frac{1}{|\mathcal{S}|}\sum_{s \in \mathcal{S}} \frac{1}{r}\sum_{j=1}^{r} M_{r,j}^{a,s} \tag{9}$$

The initial candidate set is:

$$\mathcal{A}_1 = \{\varnothing, \text{onlineEWC}, \text{SI}\} \times \{\varnothing, \text{SD}_1, \text{SD}_2\} \times \{\varnothing, \text{Replay}_1, \ldots, \text{Replay}_4\} \times \{\text{sharedLoRA}, \text{mergedLoRA}\}$$

This yields $n_1 = 3 \times 3 \times 5 \times 2 = 90$ configurations. TSH retains the top 45 after 10 tasks, top 23 after 20 tasks, and top 10 after 50 tasks.

---

Empirical Validation / Results

Compositions Outperform Standalone Mechanisms

After 100 tasks, the strongest standalone mechanism retains only 4.2% (Symbol-QA), 7.5% (LLM-QA), and 12.5% (Real-QA). The strongest compositions reach 23.2%, 41.8%, and 54.8% on the three datasets.

The best method—combining all three anchors with merged LoRA—is the only composition ranking among the top 3 in all datasets, achieving 34.9% average final retention (from 1.2% naive, a 28-fold improvement).

Table 1: Main and Interaction Effects from $2^4$ Factorial Analysis (percentage points)

| Dataset | \multicolumn{4}{c|}{Main effects} | \multicolumn{6}{c|}{Two-way} | |---|---|---|---|---|---|---|---|---|---| | | SI | SD | R | M | SI×SD | SI×R | SI×M | SD×R | SD×M | R×M | | SYMBOL-QA | +0.3 | +5.7 | +9.5 | +5.9 | -0.3 | +0.7 | -3.0 | -0.2 | +0.6 | +3.6 | | LLM-QA | +5.8 | +5.0 | +18.5 | +14.9 | +1.6 | +2.9 | -0.1 | -3.5 | +1.7 | +9.4 | | REAL-QA | +6.3 | +3.7 | +19.3 | +20.5 | +1.3 | +0.1 | +0.2 | -10.3 | +1.7 | +11.7 |

| Dataset | \multicolumn{4}{c|}{Three-way} | Four-way | |---|---|---|---|---| | | SI×SD×R | SI×SD×M | SI×R×M | SD×R×M | SI×SD×R×M | | SYMBOL-QA | +0.1 | -1.8 | -0.8 | -1.3 | +0.2 | | LLM-QA | -1.3 | -0.1 | +0.2 | -2.6 | -0.2 | | REAL-QA | -1.9 | +0.5 | -0.6 | -4.9 | -0.6 |

Bold marks statistically significant entries ($p < 0.05$). Replay and merged LoRA have the largest main effects and a positive interaction in all three datasets.

Key Findings

1. Super-additive replay × merged LoRA interaction: The standalone gains from replay and merged LoRA sum to only 3.9, 7.7, and 13.9 points (Symbol-QA, LLM-QA, Real-QA), but combining them improves retention over naive fine-tuning by 15.6, 31.0, and 46.9 points—far exceeding the sum of individual contributions.

2. Replay and merged LoRA are the largest sources of improvement: Replay raises retention by 9.5–19.3 points; merged LoRA by 5.9–20.5 points.

3. SI's interaction with merged LoRA is negative on Symbol-QA: Merged LoRA replaces LoRA factors after each task while SI carries forward importance values tied to previous factors—a structural mismatch that matters more on arbitrary Symbol-QA mappings than on natural-language data.

4. SD has positive main effects everywhere but smaller benefits when replay is already present (negative SD×R interaction on LLM-QA and Real-QA).

5. Memory half-life extension: Naive fine-tuning has a half-life of 1–2 tasks; the strongest composition extends this to 19, 32, and 32 tasks on the three datasets. However, all curves still decline—composition delays forgetting rather than preventing it.

6. O-LoRA and OSRM (growing-state methods) do not consistently improve retention; O-LoRA preserves more general capability on held-out benchmarks (GSM8K, MATH, MGSM, MMLU-Redux) but with only small retention gains.

---

Theoretical and Practical Implications

Theoretical Implications

1. Complementary mechanisms compose super-additively: The data anchor and merged LoRA address different sources of forgetting—one re-exposes the model to prior associations, the other prevents interference by isolating each task's update in a fresh low-rank subspace. Their interaction shows that continual learning methods should be designed as compositions rather than standalone interventions.

2. Replay specificity: The data anchor's effectiveness depends on its soft-label nature (using frozen model outputs), whereas the function anchor is restricted to current-task inputs—suggesting that coverage of prior task inputs is the critical factor.

3. Allocation rules shape anchor efficacy: Merged LoRA provides a "fresh workspace" per task that prevents cross-task LoRA-space interference; this interacts positively with replay but risks misplacing weight-anchor importance estimates (SI×M negative interaction).

Practical Implications

1. Memory life-extension: The best method achieves a 28-fold improvement in final retention, enabling practical long-horizon continual memorization within model parameters rather than relying on external retrieval.

2. Constant-state methods suffice: Merged LoRA matches or outperforms growing-state methods (O-LoRA, OSRM), making it memory-efficient for deployment.

3. The "open challenge" of general capability: Even the best methods lose substantial accuracy on general benchmarks (GSM8K, MATH, MMLU-Redux) after 100 tasks—preserving general abilities while memorizing associations remains unsolved.

4. Hyperparameter sensitivity: TSH enabled efficient search over the 90-configuration space, but the best composition varies by dataset, suggesting dataset-specific tuning is needed.

---

Conclusion

Summary of Contributions

1. Formulation: Long-horizon memorization as a distinct continual learning problem for language models.
2. Design space organization: Anchors (data, function, weight) × low-rank allocation rules (shared, merged).
3. New resources: Three 100-task memorization datasets, task-level successive halving (TSH), and a factorial evaluation framework.
4. Empirical demonstration: Composing all three anchors with merged LoRA improves retention from 1.2% to 34.9% (28×), with replay and merged LoRA as the largest and most synergistic components.

Key Takeaway

"Effective long-horizon continual memorization depends on finding the right combination of complementary mechanisms."

Limitations and Future Directions

• Memorization, not generalization: Evaluation tests recall on trained queries; paraphrased queries may fail—future work should test generalization to new query formulations.
• General capability preservation: Methods still lose substantial accuracy on general benchmarks; future work should address joint memorization + capability retention.
• Residual forgetting: All curves decline with memory age; composition changes the timescale of forgetting but cannot prevent eventual loss of old memories.

The authors suggest that future work should explore even longer horizons, additional anchor types, and methods that explicitly balance memorization with general capability preservation.