LEAPS: Long-Context Evaluation of Adaptive Pattern Learning and State Tracking

Summary (Overview)

• New benchmark suite: LEAPS introduces 12 novel datasets (6 for state tracking, 6 for pattern learning) designed to evaluate two understudied long-context capabilities beyond traditional recall-based benchmarks.
• Key finding on model differentiation: Models that perform near-perfectly on recall tasks show significant and varied degradation on state tracking and pattern learning, revealing that these capabilities are distinct axes of long-context performance.
• Length degradation: Model performance drops by up to 80% from 8K to 128K input tokens on LEAPS tasks, with statistically significant degradation on state tracking and pattern learning (but not recall) as context length increases.
• Thinking models underperform: Post-trained "thinking" models often perform worse than their base counterparts, exhausting generation budgets on reasoning traces or making oversimplifying assumptions.
• Contamination-resistant design: LEAPS uses synthetic generation and real-derived data that can be regenerated or sampled past model knowledge cutoffs, scaling to millions of input tokens.

---

Introduction and Theoretical Foundation

Background

Long-context language models (LCLMs) power applications requiring comprehension of large amounts of context, including coding agents, personal assistants, and forecasting systems. These applications draw on three distinct long-context capabilities:

1. Recall: The answer is a fact present in the context (e.g., needle-in-a-haystack retrieval).
2. State tracking: The outcome is computed by performing transformations to a specified state.
3. Pattern learning: Prediction made by sampling from a distribution inferred from many observations.

Key Theoretical Distinction

The authors formalize the two understudied capabilities:

State tracking is defined as computing the state $q_T$ after transformations $M = \{m_1, \ldots, m_T\}$ on an initial state $q_0$. Interactive environments can be formulated as semiautomata with finite states $Q$ modified by transformations $M$.

Pattern learning is defined as approximating a latent distribution $p_\theta(\cdot \mid x)$ from observations $\{(x_i, y_i)\}_{i=1}^{n}$ by predicting $y'$ for a new $x'$.

Motivation

Existing benchmarks (Needle-In-A-Haystack, RULER) primarily test recall. Recent benchmarks (GraphWalks, MRCR, LongBench v2) broaden coverage but do not systematically study how model capabilities are apportioned across recall, state tracking, and pattern learning. Notably:

"As input length increases, state-tracking tasks remain perfectly solvable and require a constant-sized memory that does not depend on the trajectory length if done efficiently, whereas the ceiling of performance on pattern learning tasks increases as more data is available."

---

Methodology

Benchmark Design Desiderata

1. Contamination robustness: Data generated after model knowledge cutoff or sampled past it
2. Scalability: Extends to millions of input tokens
3. Reliable metrics: Standardized between 0 and 1 (higher is better)
4. Naive baselines: For contextualizing model performance
5. Cheap inference costs

State Tracking Datasets

Metrics: F1 score (Cups), 1 − normalized L1 distance (Coins, Orderbook), exact match (ML Dev, Code Tracking, Commit Re-ordering).

Pattern Learning Datasets

Metrics: 1 − Hellinger distance (RPS), exact match (Chess, Kaggle, Social Media), NDCG@20 (Review ranking), 1 − Brier score (Forecast).

Evaluation Setup

• 30 models: API models (GPT 5.6 Sol/Terra, Claude Opus 4.8/Sonnet 5, Gemini 3.5 Flash) and open-weight models (Llama 3.1, ProLong, GPT-OSS, Qwen 3/3.5, Gemma 4, Nemotron Nano, Kimi Linear)
• Context lengths: 8K to 512K tokens (measured in Llama 3 tokenizer)
• Generation budgets: Base models get sufficient budget; instruction-tuned +256 tokens; thinking models +16K tokens
• Models instructed not to rely on code for state tracking to test native representations

---

Empirical Validation / Results

State Tracking and Pattern Learning are Unsaturated

API models at 512K input length: Despite saturating recall tasks, frontier models show substantial variation on LEAPS. Many are comparable to or worse than naive baselines on Cups, Coins, and Orderbook.

Open-weight models at 128K: Even models above 90% on recall tasks perform similar to or worse than naive baselines on LEAPS.

Degradation with Longer Context

Performance relative to each model's 8K baseline shows:

• Recall: Relatively consistent across context lengths
• State tracking: Smoothly declines
• Pattern learning: Some initial improvement (strong models leverage additional data) before declining

"Model performance drops by up to 80% from 8K to 128K on LEAPS."

Thinking Models Underperform Base Counterparts

Across Qwen3.5, Nemotron, Gemma, and Kimi Linear families, base models generally outperform post-trained counterparts by up to 1 standard deviation (in z-score normalized terms).

Correlation Analysis

• Pattern learning tasks (Chess, RPS, Forecast) correlate highly with each other
• Market prediction tasks (Kaggle, Social Media) show low correlation with all other tasks
• State tracking tasks correlate moderately with each other (except Cups)
• State tracking and pattern learning capabilities appear intertwined (e.g., ML Dev and Code Tracking correlate with Social Media, Forecast, Ranking, Chess, RPS)

Generation Budget Analysis

Performance may improve from B=8K to B=16K thinking tokens but plateaus beyond 16K; some settings even degrade (e.g., Chess at 128K). More thinking alone is insufficient.

Qualitative Failure Modes

---

Theoretical and Practical Implications

Theoretical Contributions

1. Distinct capability axes: LEAPS empirically demonstrates that recall, state tracking, and pattern learning are separable long-context capabilities that require different evaluation.

2. Scaling behavior differs: State tracking has constant memory requirements regardless of trajectory length (if done efficiently), while pattern learning has an increasing performance ceiling with more data—yet models fail to exploit this.

3. Capability intertwinement: The correlation between state tracking and pattern learning tasks suggests shared underlying mechanisms that merit further investigation.

Practical Implications

1. Benchmark design: LEAPS provides contamination-resistant, scalable evaluation methodology applicable to future benchmark development.

2. Model development guidance: The finding that thinking models underperform suggests current post-training techniques (optimized for reasoning) may compromise long-context capabilities.

3. Application readiness: For agentic applications requiring state tracking (coding agents) or pattern learning (forecasting, personalization), current models are far from production-ready at long context lengths.

---

Conclusion

LEAPS introduces a comprehensive benchmark for evaluating two critical yet understudied long-context capabilities: state tracking and pattern learning. Key takeaways:

1. Even frontier models struggle substantially more on LEAPS than on recall-oriented benchmarks, which remain largely saturated.
2. Performance degrades significantly with increasing input length—up to 80% from 8K to 128K tokens.
3. Thinking models often underperform their base counterparts, suggesting improvements needed in long-context post-training.
4. LEAPS distinguishes models along two new axes of long-context capability, complementing existing recall-based evaluations.

Future directions: The authors hope LEAPS guides development of the next generation of LCLMs, particularly for long-horizon agentic applications. The benchmark's contamination resistance and scalability make it suitable for ongoing model evaluation as LCLMs continue to improve.