Summary (Overview)

• Core finding: In retrieval tasks, context length itself is nearly free—neutral filler costs almost nothing even at 32K tokens—but competition from similar-format statements causes dramatic accuracy collapse (0.11–0.21 accuracy vs. 0.97–1.00 with neutral filler).
• Key mechanism: The attention mass left on the evidence ($a^*$) predicts the loss far better than length: $\log(1/a^*)$ explains 35–58% of variance in loss across 126,473 trials, versus only 2–8% for $\log L$.
• Critical methodological contribution: The masked-filler protocol from Du et al. [18]—the primary evidence for "length alone hurts"—fails a visible control: masking costs 46 accuracy points at 32K tokens while the identical filler left visible costs zero.
• Practical implication: For retrieval-augmented generation, deduplication and conflict resolution matter more than context truncation; the effective context budget is the number of competing statements, not token count.
• Open resources: All code and 126K trial-level measurements released at github.com/yliusci/length-is-cheap and huggingface.co/datasets/yliusci/length-is-cheap.

Introduction and Theoretical Foundation

The paper addresses the phenomenon of context rot—the degradation of retrieval accuracy as inputs grow—which has been attributed primarily to length itself. The authors challenge this explanation through a theoretical lens grounded in softmax attention mechanics.

The key theoretical foundation is the softmax attention weight on the evidence:

$$a^{*} = \frac{e^{s^{*}}}{e^{s^{*}} + \sum_{j \neq *} e^{s_j}}$$

where $s^{*}$ is the attention score of the evidence and $s_j$ are scores of all other keys. The critical insight: a thousand dissimilar tokens with low $s_j$ add little to the denominator, but a handful of near-copies of the evidence can halve $a^{*}$. This predicts that context rot is a problem of competition, not size.

The authors contrast two competing explanations:

1. Length hypothesis (Du et al. [18]): Length alone hurts, even when filler is masked from attention
2. Competition hypothesis (this paper): Degradation tracks how much attention competitors draw away from the evidence

Methodology

Task design: A key-value retrieval task where each context hides one evidence statement: "The secret code of the river is 4821." The query asks for the secret code, and the loss $\Delta_c = \log(p_{\text{short}}/p_c)$ is measured in nats.

Four conditions (holding length and evidence position fixed):

Models tested: Qwen2.5-0.5B/1.5B-Instruct, SmolLM2-1.7B-Instruct, OLMo-2-0425-1B-Instruct, Llama-3.2-3B, at lengths $L \in \{512, 1024, 2048\}$ and up to 32,768 tokens.

Attention measurement: Eight (layer, head) pairs selected in short contexts that place most attention on evidence tokens; $a^*$ is the attention these heads send from the final query token to evidence tokens.

Replication of Du et al. [18]: Masked-filler protocol on 1,000 MMLU questions with 3,750–30,000 masked tokens.

Empirical Validation / Results

Length is cheap: On Qwen2.5-1.5B, median loss stays ≤0.006 nats from 8K–32K tokens with neutral filler; accuracy stays 0.96–1.00.

Competition is not: Same positions filled with competing statements drive accuracy to 0.11–0.21. At 32K tokens on Qwen2.5-1.5B: neutral costs 0.004 nats, masking costs 1.20 nats, competing costs 4.76 nats.

Conflicts concentrate the loss:

Attention predicts the loss (Table 4):

Masking fails as a length control: MMLU accuracy drops 11–18 points at 30K masked tokens (replicating Du et al.), but in retrieval the same masking costs 46 accuracy points at 32K while visible filler costs zero. The masked condition creates positional gaps unseen in training, not a pure length effect.

Theoretical and Practical Implications

Theoretical implications:

• Softmax dilution theory [62, 63, 89] predicts that degradation should track attention mass on evidence—this paper provides the first trial-by-trial confirmation on measured attention in pretrained models
• The threshold behavior ($a^* \approx 0.35$ on Qwen2.5-1.5B, with slopes > 1) indicates a nonlinear readout: models tolerate moderate attention loss, then fail abruptly
• Attention sharpening methods [58, 59, 90–100] should help most when they keep $a^*$ above the threshold

Practical implications:

• Curate, don't truncate: Deduplicating and resolving conflicting statements before they reach the model matters more than shortening context
• Measure attention, not just accuracy: $a^*$ requires only a forward pass and can serve as a diagnostic for any open model and retrieval benchmark
• Report visible controls: Any claim about length effects should include a visible neutral-text control at the same positions; masked conditions are not equivalent to irrelevant tokens

Conclusion

The paper demonstrates that in retrieval, long contexts fail not because they are long but because of competition for attention. The attention mass left on evidence ($a^*$) is a powerful predictor of failure, far exceeding length in explanatory power. The authors reframe context rot as a competition problem, with practical guidance for context engineering: resolve conflicts and deduplicate before they reach the model. Future work should explore causal interventions on attention, extend beyond 3B parameters, and investigate whether length effects observed in reasoning tasks (e.g., MMLU with whitespace padding) share the same mechanism or constitute a distinct phenomenon.