Visual Summary | Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

Summary (Overview)

Core hypothesis: The paper proposes the Information Abundance Paradox, which posits that when task-relevant information is abundantly available in the training context, models reduce their incentive to encode that information parametrically, shifting their learning mode from internalization (storing knowledge in weights) toward contextualization (relying on in-context evidence).
Key finding in pretraining: Increasing the training context window yields an inverted-U performance curve across model scales (20M–750M parameters)—performance on SuperGLUE, MCQA, and language modeling improves up to an intermediate optimum (around 2048 tokens for SuperGLUE/MCQA, 8192 for language modeling) before declining.
Key finding in supervised fine-tuning: Training with more task-relevant context improves performance when supporting context is present at test time, but reduces robustness when context is absent or misleading—a behavioral signature the authors call "context addiction."
Mechanistic evidence: Longer informative contexts shift gradient pressure from feed-forward networks (FFNs, associated with parametric knowledge) toward self-attention (SA) modules, and causally increase reliance on context tokens during inference.
Theoretical formalization: The authors prove (Proposition 4.2) that larger context windows monotonically reduce the minimum task information that must be stored in weights to achieve a given risk threshold.

Introduction and Theoretical Foundation

Background and Motivation

Large language models (LLMs) are increasingly trained with long context windows spanning documents, codebases, and interaction histories. The implicit assumption driving this scaling is that longer contexts expose models to richer evidence and will only help. The authors challenge this view by pointing to a motivating observation: long-context variants of models like Phi-3 (128K) and OLMo 3 (65K) consistently underperform their short-context counterparts (4K and 8K) in few-shot and zero-shot settings, despite being trained on more data.

Two Modes of Learning

Training a language model via next-token cross-entropy can be viewed as compression, where regularities in the corpus are distilled into parameters. This gives rise to two sources of predictive power:

Parametric knowledge: Information stored in the model's weights (internalization)
Contextual information: Evidence supplied in the input context (contextualization)

These two channels offer alternative routes to prediction, and their relative use is shaped by training dynamics and context informativeness.

The Information Abundance Paradox

The Information Abundance Paradox: When task-relevant information is made available through the training context, the model can reduce loss by using that information directly rather than by encoding it in its parameters. Consequently, this can shift the model's mode of learning away from parametric internalization and toward contextualization.

The authors define context addiction as the observable behavioral manifestation: a model trained with informative context performs well when useful context is available but deteriorates when that context is absent or misleading.

Methodology

Experiment 1: Pretraining with Varying Context Length

Models: Llama-2 architecture at four scales (20M, 55M, 259M, 750M parameters) with RoPE positional encoding
Data: 10B tokens from Project Gutenberg, retaining documents ≥ 65536 tokens
Training: Context window swept over $W \in \{512, 1024, \ldots, 32768\}$ (powers of two), with fixed token budget (~1.05M tokens per batch, 9537 optimization steps for all variants)
Evaluation: Zero-shot assessment on three testbeds:
- Language modeling suite (LAMBADA, WikiSPAN, Penn Treebank)
- SuperGLUE
- Closed-book MCQA (ARC, CommonsenseQA, PIQA)

Experiment 2: Supervised Fine-Tuning with Varying Context Informativeness

Models: Qwen3 (0.6B, 1.7B, 4B, 8B, 14B) with LoRA adapters
Task: MMLU-Pro domains (Health, Economics, Law, Psychology)
Setup: Fixed context budget of $n = 8$ documents; vary number of target-domain documents $k \in \{0, 4, 8\}$ , with the remaining $8 - k$ drawn from other domains
Evaluation: Three test-time conditions—(i) supporting context, (ii) conflicting context, (iii) no context

Theoretical Framework

The authors formalize the tradeoff using an information-theoretic framework. Let $\tau \sim P_\tau$ be a latent task variable, $X^{(k)}$ an input of context size $k$ , and $W$ the learned weights. The risk is defined as:

\mathcal{R}_k(\Pi, q_k) = \mathbb{E}[\ell(Y, \hat{Y})]

where the expectation is over $(X^{(k)}, Y) \sim P_\tau^{(k)}$ , $W \sim \Pi(\cdot \mid \tau)$ , and $\hat{Y} \sim q_k(\cdot \mid X^{(k)}, W)$ .

Definition 4.1 (Parametric information frontier):

\mathcal{I}_k(\rho) = \inf_{\Pi, q_k} I(W; \tau) \quad \text{such that} \quad \mathcal{R}_k(\Pi, q_k) \leq \rho

This measures the minimum task information that must be stored in the weights to attain risk at most $\rho$ with context size $k$ .

Proposition 4.2 (Monotonicity of the parametric information frontier): If $X^{(k)} = T_{k,m}(X^{(m)})$ almost surely for $k < m$ (nested context windows), then:

\mathcal{I}_m(\rho) \leq \mathcal{I}_k(\rho) \quad \text{for all } \rho

Proof sketch: A predictor with access to $X^{(m)}$ can simulate any $k$ -context predictor by applying the projection map $T_{k,m}$ , making every feasible solution for the $k$ -context problem feasible for the $m$ -context problem with the same weight channel.

Empirical Validation / Results

Pretraining Results

As shown in Figure 2, performance follows an inverted-U pattern:

SuperGLUE and MCQA peak around 2048 tokens
Language modeling improves up to 8192 tokens
Beyond these optima, further context extension progressively erodes gains
The pattern persists across all four model scales (20M to 750M), indicating that greater capacity does not eliminate the degradation

Supervised Fine-Tuning Results

Figure 3 shows that increasing target-domain documents from $k = 0$ to $k = 8$ :

Improves performance with supporting context
Reduces no-context accuracy significantly
Increases vulnerability to conflicting context

This pattern holds across all model sizes (0.6B–14B) and domains.

Mechanistic Findings

Solution Complexity (§5.1): In synthetic tasks (bitwise operations, string operations, mod10 arithmetic, Caesar cipher), context addiction emerges selectively. Tasks with growing supporting–conflicting gaps (bitwise, string) also show decreasing average training gradient norms, defined as:

G_k = \frac{1}{T} \sum_{t=1}^{T} \| \nabla_\theta \ell(\theta_t^{(k)}; B_t) \|_2

This suggests context addiction arises when demonstrations provide a lower-complexity optimization path.

Module-Level Gradient Allocation (§5.2): The FFN-to-SA gradient norm ratio decreases with:

Increasing task-relevant context in SFT ( $k = 8$ vs. no-context baseline)
Increasing context window in pretraining

Module-restricted fine-tuning provides causal evidence: FFN-only tuning improves no-context robustness, while SA-only tuning strengthens supporting-context performance but increases sensitivity to conflicting context.

Token-Level Attention Allocation (§5.3): Models trained with task-relevant context ( $k = 8$ ) allocate significantly more attention mass to context tokens at test time, concentrated in middle layers consistent with their role in contextual integration.

Theoretical and Practical Implications

Theoretical Significance

The Information Abundance Paradox reframes long-context scaling as a tradeoff between two learning modes rather than a simple data-delivery problem. The theoretical result (Proposition 4.2) provides a rigorous foundation: longer contexts can reduce parametric information storage requirements, but this comes at the cost of context-dependence at inference time.

Practical Implications

Context window is not a neutral scaling axis: Training with longer contexts fundamentally alters what models internalize, not just how much data they see.
Application-dependent tradeoffs: The shift toward contextualization may be beneficial in retrieval-augmented settings where context is always available, but harmful in scenarios where context may be absent or unreliable.
Data-centric solutions are insufficient: The findings suggest that merely providing more high-quality long-context data will not solve the problem, since the issue is structural to how context shapes learning.
Architectural interventions: The module-level findings suggest that targeted training (e.g., FFN-focused updates) could preserve parametric robustness, while SA-focused training could enhance contextual grounding.

Conclusion

The paper demonstrates that long-context processing is a powerful capability but not a neutral scaling axis for language models. The key takeaways are:

Increasing train-time context can shift learning from parametric internalization toward contextualization, governed by the Information Abundance Paradox
This shift manifests as inverted-U performance curves in pretraining and context addiction in fine-tuning
The mechanism involves reduced gradient pressure on FFNs (parametric knowledge) and increased pressure on attention modules (contextual processing)
The tradeoff is theoretically grounded in the monotonicity of the parametric information frontier

Future Directions

Testing at larger scales (beyond 750M parameters in pretraining) to determine how inflection points shift with model size and data scale
Developing training strategies that preserve parametric robustness while leveraging long-context capabilities
Exploring whether curriculum approaches or mixed-context training can mitigate context addiction

The takeaway is not that long-context processing is undesirable, but that it should be understood through its role in mediating the tradeoff between context use and context-independent competence.