# Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

> Long-context training shifts models from internalizing knowledge in weights to relying on context, degrading performance when context is absent or misleading.

- **Source:** [arXiv](https://arxiv.org/abs/2608.12218)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/oAAZ43
- **Whiteboard:** https://picx.dev/p/oAAZ43/image

## Summary

## Summary (Overview)

- **Core hypothesis**: The paper proposes the **Information Abundance Paradox**, which posits that when task-relevant information is abundantly available in the training context, models reduce their incentive to encode that information parametrically, shifting their learning mode from *internalization* (storing knowledge in weights) toward *contextualization* (relying on in-context evidence).
- **Key finding in pretraining**: Increasing the training context window yields an **inverted-U performance curve** across model scales (20M–750M parameters)—performance on SuperGLUE, MCQA, and language modeling improves up to an intermediate optimum (around 2048 tokens for SuperGLUE/MCQA, 8192 for language modeling) before declining.
- **Key finding in supervised fine-tuning**: Training with more task-relevant context improves performance when supporting context is present at test time, but **reduces robustness** when context is absent or misleading—a behavioral signature the authors call **"context addiction."**
- **Mechanistic evidence**: Longer informative contexts shift gradient pressure from feed-forward networks (FFNs, associated with parametric knowledge) toward self-attention (SA) modules, and causally increase reliance on context tokens during inference.
- **Theoretical formalization**: The authors prove (Proposition 4.2) that larger context windows monotonically reduce the minimum task information that must be stored in weights to achieve a given risk threshold.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Large language models (LLMs) are increasingly trained with long context windows spanning documents, codebases, and interaction histories. The implicit assumption driving this scaling is that longer contexts expose models to richer evidence and will only help. The authors challenge this view by pointing to a **motivating observation**: long-context variants of models like Phi-3 (128K) and OLMo 3 (65K) consistently underperform their short-context counterparts (4K and 8K) in few-shot and zero-shot settings, despite being trained on more data.

### Two Modes of Learning

Training a language model via next-token cross-entropy can be viewed as compression, where regularities in the corpus are distilled into parameters. This gives rise to **two sources of predictive power**:

1. **Parametric knowledge**: Information stored in the model's weights (internalization)
2. **Contextual information**: Evidence supplied in the input context (contextualization)

These two channels offer alternative routes to prediction, and their relative use is shaped by training dynamics and context informativeness.

### The Information Abundance Paradox

> **The Information Abundance Paradox**: When task-relevant information is made available through the training context, the model can reduce loss by using that information directly rather than by encoding it in its parameters. Consequently, this can shift the model's mode of learning away from parametric internalization and toward contextualization.

The authors define **context addiction** as the observable behavioral manifestation: a model trained with informative context performs well when useful context is available but deteriorates when that context is absent or misleading.

---

## Methodology

### Experiment 1: Pretraining with Varying Context Length

- **Models**: Llama-2 architecture at four scales (20M, 55M, 259M, 750M parameters) with RoPE positional encoding
- **Data**: 10B tokens from Project Gutenberg, retaining documents ≥ 65536 tokens
- **Training**: Context window swept over $W \in \{512, 1024, \ldots, 32768\}$ (powers of two), with fixed token budget (~1.05M tokens per batch, 9537 optimization steps for all variants)
- **Evaluation**: Zero-shot assessment on three testbeds:
  - Language modeling suite (LAMBADA, WikiSPAN, Penn Treebank)
  - SuperGLUE
  - Closed-book MCQA (ARC, CommonsenseQA, PIQA)

### Experiment 2: Supervised Fine-Tuning with Varying Context Informativeness

- **Models**: Qwen3 (0.6B, 1.7B, 4B, 8B, 14B) with LoRA adapters
- **Task**: MMLU-Pro domains (Health, Economics, Law, Psychology)
- **Setup**: Fixed context budget of $n = 8$ documents; vary number of target-domain documents $k \in \{0, 4, 8\}$, with the remaining $8 - k$ drawn from other domains
- **Evaluation**: Three test-time conditions—(i) supporting context, (ii) conflicting context, (iii) no context

### Theoretical Framework

The authors formalize the tradeoff using an information-theoretic framework. Let $\tau \sim P_\tau$ be a latent task variable, $X^{(k)}$ an input of context size $k$, and $W$ the learned weights. The risk is defined as:

$$\mathcal{R}_k(\Pi, q_k) = \mathbb{E}[\ell(Y, \hat{Y})]$$

where the expectation is over $(X^{(k)}, Y) \sim P_\tau^{(k)}$, $W \sim \Pi(\cdot \mid \tau)$, and $\hat{Y} \sim q_k(\cdot \mid X^{(k)}, W)$.

**Definition 4.1 (Parametric information frontier)**:

$$\mathcal{I}_k(\rho) = \inf_{\Pi, q_k} I(W; \tau) \quad \text{such that} \quad \mathcal{R}_k(\Pi, q_k) \leq \rho$$

This measures the minimum task information that must be stored in the weights to attain risk at most $\rho$ with context size $k$.

**Proposition 4.2 (Monotonicity of the parametric information frontier)**: If $X^{(k)} = T_{k,m}(X^{(m)})$ almost surely for $k < m$ (nested context windows), then:

$$\mathcal{I}_m(\rho) \leq \mathcal{I}_k(\rho) \quad \text{for all } \rho$$

*Proof sketch*: A predictor with access to $X^{(m)}$ can simulate any $k$-context predictor by applying the projection map $T_{k,m}$, making every feasible solution for the $k$-context problem feasible for the $m$-context problem with the same weight channel.

---

## Empirical Validation / Results

### Pretraining Results

As shown in Figure 2, performance follows an **inverted-U pattern**:
- SuperGLUE and MCQA peak around **2048 tokens**
- Language modeling improves up to **8192 tokens**
- Beyond these optima, further context extension progressively erodes gains
- The pattern persists across all four model scales (20M to 750M), indicating that greater capacity does not eliminate the degradation

### Supervised Fine-Tuning Results

Figure 3 shows that increasing target-domain documents from $k = 0$ to $k = 8$:
- **Improves** performance with supporting context
- **Reduces** no-context accuracy significantly
- **Increases** vulnerability to conflicting context

This pattern holds across all model sizes (0.6B–14B) and domains.

### Mechanistic Findings

**Solution Complexity (§5.1)**: In synthetic tasks (bitwise operations, string operations, mod10 arithmetic, Caesar cipher), context addiction emerges selectively. Tasks with growing supporting–conflicting gaps (bitwise, string) also show **decreasing average training gradient norms**, defined as:

$$G_k = \frac{1}{T} \sum_{t=1}^{T} \| \nabla_\theta \ell(\theta_t^{(k)}; B_t) \|_2$$

This suggests context addiction arises when demonstrations provide a **lower-complexity optimization path**.

**Module-Level Gradient Allocation (§5.2)**: The FFN-to-SA gradient norm ratio decreases with:
- Increasing task-relevant context in SFT ($k = 8$ vs. no-context baseline)
- Increasing context window in pretraining

Module-restricted fine-tuning provides causal evidence: **FFN-only tuning** improves no-context robustness, while **SA-only tuning** strengthens supporting-context performance but increases sensitivity to conflicting context.

**Token-Level Attention Allocation (§5.3)**: Models trained with task-relevant context ($k = 8$) allocate significantly more attention mass to context tokens at test time, concentrated in middle layers consistent with their role in contextual integration.

---

## Theoretical and Practical Implications

### Theoretical Significance

The Information Abundance Paradox reframes long-context scaling as a **tradeoff between two learning modes** rather than a simple data-delivery problem. The theoretical result (Proposition 4.2) provides a rigorous foundation: longer contexts *can* reduce parametric information storage requirements, but this comes at the cost of context-dependence at inference time.

### Practical Implications

1. **Context window is not a neutral scaling axis**: Training with longer contexts fundamentally alters what models internalize, not just how much data they see.
2. **Application-dependent tradeoffs**: The shift toward contextualization may be beneficial in retrieval-augmented settings where context is always available, but harmful in scenarios where context may be absent or unreliable.
3. **Data-centric solutions are insufficient**: The findings suggest that merely providing more high-quality long-context data will not solve the problem, since the issue is structural to how context shapes learning.
4. **Architectural interventions**: The module-level findings suggest that targeted training (e.g., FFN-focused updates) could preserve parametric robustness, while SA-focused training could enhance contextual grounding.

---

## Conclusion

The paper demonstrates that **long-context processing is a powerful capability but not a neutral scaling axis for language models**. The key takeaways are:

1. Increasing train-time context can shift learning from parametric internalization toward contextualization, governed by the Information Abundance Paradox
2. This shift manifests as inverted-U performance curves in pretraining and context addiction in fine-tuning
3. The mechanism involves reduced gradient pressure on FFNs (parametric knowledge) and increased pressure on attention modules (contextual processing)
4. The tradeoff is theoretically grounded in the monotonicity of the parametric information frontier

### Future Directions

- Testing at larger scales (beyond 750M parameters in pretraining) to determine how inflection points shift with model size and data scale
- Developing training strategies that preserve parametric robustness while leveraging long-context capabilities
- Exploring whether curriculum approaches or mixed-context training can mitigate context addiction

**The takeaway is not that long-context processing is undesirable, but that it should be understood through its role in mediating the tradeoff between context use and context-independent competence.**

---

_Markdown view of https://picx.dev/p/oAAZ43, served by PicX — AI-generated visual whiteboard summaries of research papers._