# Online Experiential Learning for Language Models > Online Experiential Learning enables LLMs to continuously improve from deployment experience without rewards by extracting and consolidating experiential knowledge. - **Source:** [arXiv](https://arxiv.org/abs/2603.16856) - **Published:** 2026-03-19 - **Permalink:** https://picx.dev/p/vJjlgH - **Whiteboard:** https://picx.dev/p/vJjlgH/image ## Summary # Online Experiential Learning for Language Models ## Summary (Overview) * **Key Contribution**: Introduces **Online Experiential Learning (OEL)**, a reward-free framework that enables Large Language Models (LLMs) to continuously improve from their own deployment experience, forming an online learning loop. * **Core Mechanism**: Operates in two iterable stages: (1) **Extraction** of transferable experiential knowledge from user-side interaction trajectories, and (2) **Consolidation** of this knowledge into model parameters via **on-policy context distillation**. * **Main Findings**: OEL achieves consistent performance improvements across successive iterations on text-based games, enhances token efficiency (shorter responses), and preserves out-of-distribution performance better than off-policy alternatives. * **Critical Insights**: Extracted experiential knowledge is significantly more effective than raw trajectories, and **on-policy consistency** between the knowledge source and the policy model is essential for effective learning. * **Practical Benefit**: The framework requires **no human annotations, reward models, or server-side access to user environments**, making it scalable for real-world deployment. ## Introduction and Theoretical Foundation The prevailing paradigm for improving LLMs relies on **offline training** with human annotations or simulated environments (e.g., SFT, RL). This creates a bottleneck: once deployed, the model's rich stream of real-world interaction experience is discarded. The authors advocate for a shift to **online learning**, where models continuously improve from test-time experience accumulated during deployment. **Key Challenges**: The server side cannot access user-side environments, and real-world interactions typically provide only unstructured **textual feedback** (e.g., outcome descriptions), not scalar rewards. Standard RL algorithms cannot consume such signals directly. **Theoretical Insight**: OEL addresses these challenges by converting textual environment feedback into **experiential knowledge** that can be extracted, accumulated, and internalized. The process is entirely **reward-free**. ## Methodology OEL operates in an iterative loop between the **User Side** (deployment) and the **Server Side** (training). ### 1. Extract Experiential Knowledge from User Trajectories On the user side, the model $\pi_{\theta}$ interacts with an environment $E$, collecting a set of $n$ multi-turn trajectories $\mathcal{T} = \{ \tau_1, \tau_2, ..., \tau_n \}$. Each trajectory $\tau_i = (f_1^i, a_1^i, f_2^i, a_2^i, ...)$ is an alternating sequence of model actions ($a$) and textual environment feedback ($f$). A language model $\pi_{\text{extract}}$ (default: $\pi_{\text{extract}} = \pi_{\theta}$) extracts transferable knowledge from these trajectories in an **accumulative fashion**. Formally, let $e_i$ denote the accumulated experiential knowledge after processing trajectory $\tau_i$, with $e_0 = \emptyset$. The recursive extraction and accumulation process is defined for $i = 1,...,n$ as: $$ e'_i \sim \pi_{\text{extract}}(\cdot | \tau_i, e_{i-1}) $$ $$ e_i = [e_{i-1}; e'_i] $$ where $[e_{i-1}; e'_i]$ denotes concatenation. This process is repeated with different random seeds to produce a set of accumulated experiential knowledge $\mathcal{C} = \{e_1, e_2, ..., e_K\}$. ### 2. Consolidate Experiential Knowledge into Model Weights The server side consolidates the knowledge $\mathcal{C}$ into the model parameters via **on-policy context distillation**. **Training Data Construction**: From a separate set of user-collected trajectories $\mathcal{T}'$, all **partial rollout prefixes** $x_i^j = (f_1^i, a_1^i, ..., f_{j-1}^i, a_{j-1}^i, f_j^i)$ are extracted, forming a dataset $\mathcal{D} = \{x_i^j\}$. **Training Objective**: The student model $\pi_{\theta}$ generates a response $y$ conditioned only on prefix $x$. It is optimized to match the output of a **knowledge-conditioned teacher** $\pi_{\text{teacher}}$ (typically the frozen initial $\pi_{\theta}$) via token-level reverse KL divergence: $$ \mathcal{L}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, e \sim \mathcal{C}, y \sim \pi_{\theta}(\cdot|x)} \left[ \frac{1}{|y|} \sum_{t=1}^{|y|} D_{\text{KL}}\big( \pi_{\theta}(\cdot | x, y_{