Summary

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Authors: Borui Kang, Jinrui Gu, Junhan Lv, Wenbin Li, Lei Wang, Yang Gao (Nanjing University; University of Wollongong)

---

Summary (Overview)

• New Paradigm: The paper introduces Harness Continual Learning (HCL), a novel continual learning paradigm where the "harness"—prompts, memories, tools, skills, and routing rules—evolves around a frozen foundation model, rather than updating model parameters.
• Harness-Level Forgetting: The authors formalize harness-level forgetting, a new type of catastrophic forgetting where updates to one harness component can disrupt previously reliable behavior in other components, even with a frozen model.
• Guarded Evolution Framework: They propose a "guarded harness evolution" mechanism with a Continual Optimizer that proposes candidate harnesses and a Continual Evaluator that commits changes only after verifying current improvement, historical retention, and validity.
• Empirical Validation: Across textual reasoning, multimodal perception, and open-world interaction tasks, HCL demonstrates capability accumulation and failure recovery, achieving relative gains exceeding 10% over baselines. The framework provides explicit control over the stability-plasticity trade-off.
• Key Finding: More permissive updates do not necessarily yield a stronger final harness; a moderate historical-loss tolerance (b=1) achieved the highest final average performance (63.46%) compared to both stricter (b=0: 61.25%) and looser (b=∞: 60.13%) settings.

---

Introduction and Theoretical Foundation

Background and Motivation

Traditional continual learning is model-centric, treating model parameters as the state that changes with sequential experience. However, modern AI agents adapt not only through model weights but also through an external harness—the persistent runtime contents that determine how a frozen foundation model receives information, retrieves experience, and acts.

"Prompts, memories, tool and skill specifications, and routing policies can persist and evolve across interactions even when the foundation model remains frozen."

The authors identify a critical problem: harness components are coupled in execution. A memory update can change evidence retrieved for an earlier query; a skill revision can alter tool use; a routing edit can break a previously successful workflow. This creates harness-level forgetting—a degradation of earlier behavior caused by harness updates—which extends the classical stability-plasticity problem from model state to harness state.

Key Definitions

The paper formalizes the HCL problem as follows. Let $H_n$ be the deployed harness at interaction step $n$, and $\theta$ the frozen foundation model parameters. The harness transforms raw interaction $\mathbf{u}_n$ into outcome $\mathbf{y}_n$:

$$\mathbf{e}_n = \left(\mathbf{u}_n, \mathbf{i}_n, \mathbf{z}_n, \mathbf{y}_n, \mathbf{f}_n\right)$$

where $\mathbf{i}_n$ is the structured interaction, $\mathbf{z}_n$ the execution context, and $\mathbf{f}_n$ post-execution feedback.

The Continual Optimizer proposes a candidate harness:

$$\widetilde{H}_{n+1} = \mathcal{O}_{F_\theta}\left(H_n, \mathbf{e}_n\right)$$

The deployed harness evolves based on commitment decision $G_n \in \{0,1\}$:

$$H_{n+1} = \begin{cases} \widetilde{H}_{n+1}, & G_n = 1, \\ H_n, & G_n = 0. \end{cases}$$

---

Methodology

Harness State Architecture

The HCL harness state is organized into four jointly versioned components:

$$H_n = \left(I_n, M_n, C_n, R_n\right)$$

Table 1: Components of the HCL Harness

1. Task Interface

Transforms raw input into structured representation:

$$\mathbf{i}_n = I_n(\mathbf{u}_n) = (\mathbf{x}_n, \mathbf{g}_n, \mathbf{k}_n)$$

where $\mathbf{x}_n$ is the available input, $\mathbf{g}_n$ the task objective, and $\mathbf{k}_n$ records constraints.

2. Experience Memory

Dual-form memory combining:

• Raw Memory $M_n^{\text{raw}}$: concrete interactions (input, output, feedback)
• Abstract Memory $M_n^{\text{abs}}$: LLM-summarized guidance and recurring patterns

$$M_n = \left(M_n^{\text{raw}}, M_n^{\text{abs}}\right)$$

3. Capability Map

$$C_n = \left(C_n^{\text{outer}}, C_n^{\text{inner}}\right)$$

• Outer capabilities: external APIs, retrieval services, environment actions
• Inner capabilities: reusable skills abstracted from Abstract Memory

4. Adaptive Router

Connects all components to execution:

$$\mathbf{z}_n = R_n\left(\mathbf{i}_n, M_n, C_n\right)$$

Guarded Harness Evolution

The framework separates update generation from state commitment through a proposal–evaluation–commitment process.

Continual Optimizer

Generates candidate harnesses from post-execution feedback, proposing up to $K$ alternatives per component in a sequential strategy.

Continual Evaluator

Three checks determine commitment:

Current Improvement:

$$\Delta_n = P\left(\widetilde{H}_{n+1}, V_n\right) - P\left(H_n, V_n\right)$$

Accepted when $\Delta_n \geq \delta_n$.

Historical Retention: Counts previously solved anchors that fail under candidate:

$$D_n = \sum_{a \in A_n} \mathbb{1}\left[q(H_n, a) = 1 \wedge q(\widetilde{H}_{n+1}, a) = 0\right]$$

Accepted when $D_n \leq B_n$.

Validity Check: $v_{n,\ell}\left(\widetilde{H}_{n+1}\right) \in \{0,1\}$ for each check $\ell$.

Commitment Decision:

$$G_n^{(k)} = \mathbb{1}\left[(\Delta_n^{(k)} \geq \delta_n) \wedge (D_n^{(k)} \leq B_n) \wedge \left(\forall \ell, v_{n,\ell}(\widetilde{H}_{n+1}^{(k)}) = 1\right)\right]$$

Two configurations are studied: Stability-HCL ($B_n = 0$, strict retention) and Plasticity-HCL ($B_n = \infty$, permissive updates).

---

Empirical Validation / Results

Experimental Setup

Open-World Interaction

ALFWorld Results:

Minecraft: HCL completed all 50 tasks while the Static Harness plateaued at 15. HCL used only 83 environment actions versus 88 (MemRL) and 91 (MemP), demonstrating superior execution efficiency.

Controlled Streams

Textual Reasoning:

Multimodal Perception:

Stability-Plasticity Trade-off

Varying only the historical-loss tolerance $B_n$:

Component Ablation

---

Theoretical and Practical Implications

Theoretical Contributions

1. New Learning Paradigm: HCL reframes continual learning as operating on harness state rather than model parameters, extending the field's scope to agentic AI systems where the model is frozen but the surrounding infrastructure evolves.

2. Formalization of Harness-Level Forgetting: The paper provides a rigorous definition and measurement framework for harness-level forgetting, establishing it as a distinct phenomenon from classical catastrophic forgetting.

3. Unified Framework: HCL conceptually unifies multiple model-centric continual learning principles (replay, representation learning, architecture-based, optimization-based) within a single system-level formulation.

Practical Implications

1. Efficient Adaptation: HCL enables capability acquisition without expensive model fine-tuning, making it practical for large-scale deployment with frozen foundation models.

2. Controlled Trade-offs: The explicit $B_n$ parameter gives practitioners direct control over the stability-plasticity trade-off, allowing task-specific tuning.

3. Failure Recovery: The framework supports recovery from failures through its memory and capability revision mechanisms.

4. Scalability: Demonstrated across model families (Qwen, DeepSeek) and scales (4B-27B), suggesting generalizability.

---

Conclusion

Key Takeaway: Harness Continual Learning represents a paradigm shift in continual learning—moving the learning object from model parameters to the harness state surrounding frozen foundation models.

Main Findings:

• Harness evolution can accumulate capabilities and recover from failures without updating model parameters
• Harness-level forgetting is measurable and controllable through explicit retention constraints
• The stability-plasticity trade-off can be adjusted explicitly via the historical-loss tolerance parameter $B_n$
• More permissive updates do not necessarily produce stronger final harnesses

Future Directions:

• Efficient retention evaluation methods
• Harness-content consolidation techniques
• Evaluation over longer interaction streams
• Application to production agentic AI systems

The authors hope HCL "provides a foundation for addressing these challenges and encourages broader research on reliable agent continual learning."