# LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking

> LEAPS reveals that frontier models, despite saturating recall benchmarks, degrade up to 80% on state tracking and pattern learning tasks at long context, with thinking models underperforming base counterparts.

- **Source:** [arXiv](https://arxiv.org/abs/2610.25919)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/RO8Twi
- **Whiteboard:** https://picx.dev/p/RO8Twi/image

## Summary

# LEAPS: Long-Context Evaluation of Adaptive Pattern Learning and State Tracking

## Summary (Overview)

- **New benchmark suite**: LEAPS introduces 12 novel datasets (6 for state tracking, 6 for pattern learning) designed to evaluate two understudied long-context capabilities beyond traditional recall-based benchmarks.
- **Key finding on model differentiation**: Models that perform near-perfectly on recall tasks show significant and varied degradation on state tracking and pattern learning, revealing that these capabilities are distinct axes of long-context performance.
- **Length degradation**: Model performance drops by up to 80% from 8K to 128K input tokens on LEAPS tasks, with statistically significant degradation on state tracking and pattern learning (but not recall) as context length increases.
- **Thinking models underperform**: Post-trained "thinking" models often perform worse than their base counterparts, exhausting generation budgets on reasoning traces or making oversimplifying assumptions.
- **Contamination-resistant design**: LEAPS uses synthetic generation and real-derived data that can be regenerated or sampled past model knowledge cutoffs, scaling to millions of input tokens.

---

## Introduction and Theoretical Foundation

### Background

Long-context language models (LCLMs) power applications requiring comprehension of large amounts of context, including coding agents, personal assistants, and forecasting systems. These applications draw on **three distinct long-context capabilities**:

1. **Recall**: The answer is a fact present in the context (e.g., needle-in-a-haystack retrieval).
2. **State tracking**: The outcome is computed by performing transformations to a specified state.
3. **Pattern learning**: Prediction made by sampling from a distribution inferred from many observations.

### Key Theoretical Distinction

The authors formalize the two understudied capabilities:

**State tracking** is defined as computing the state $q_T$ after transformations $M = \{m_1, \ldots, m_T\}$ on an initial state $q_0$. Interactive environments can be formulated as semiautomata with finite states $Q$ modified by transformations $M$.

**Pattern learning** is defined as approximating a latent distribution $p_\theta(\cdot \mid x)$ from observations $\{(x_i, y_i)\}_{i=1}^{n}$ by predicting $y'$ for a new $x'$.

### Motivation

Existing benchmarks (Needle-In-A-Haystack, RULER) primarily test recall. Recent benchmarks (GraphWalks, MRCR, LongBench v2) broaden coverage but do not systematically study how model capabilities are apportioned across recall, state tracking, and pattern learning. Notably:

> "As input length increases, state-tracking tasks remain perfectly solvable and require a constant-sized memory that does not depend on the trajectory length if done efficiently, whereas the ceiling of performance on pattern learning tasks increases as more data is available."

---

## Methodology

### Benchmark Design Desiderata

1. **Contamination robustness**: Data generated after model knowledge cutoff or sampled past it
2. **Scalability**: Extends to millions of input tokens
3. **Reliable metrics**: Standardized between 0 and 1 (higher is better)
4. **Naive baselines**: For contextualizing model performance
5. **Cheap inference costs**

### State Tracking Datasets

| Dataset | Description | Source |
|---------|-------------|--------|
| Cup swapping | Track positions of balls as cups are swapped | Synthetic |
| Coins | Track count of piles as coins are added, removed, swapped | Synthetic |
| Orderbook | Track buy/sell orders in multi-actor market | Synthetic |
| ML Dev | Follow parallel model developments, generate commands | Synthetic |
| Commit re-ordering | Identify pairs of conflicting commits | Real |
| Code tracking | Track code files across commits and changes | Real |

**Metrics**: F1 score (Cups), 1 − normalized L1 distance (Coins, Orderbook), exact match (ML Dev, Code Tracking, Commit Re-ordering).

### Pattern Learning Datasets

| Dataset | Description | Source |
|---------|-------------|--------|
| Rock-paper-scissors | Learn behaviors of players with random strategies | Synthetic |
| Review ranking | Learn preferences of area chairs ranking papers | Synthetic |
| Kaggle | Learn relationships between abstract features and market movements | Real |
| Social media | Learn relationships between textual features and market movements | Real |
| Chess | Learn behaviors of top chess players, predict moves | Real |
| Forecast | Forecast real-world events using news articles | Real |

**Metrics**: 1 − Hellinger distance (RPS), exact match (Chess, Kaggle, Social Media), NDCG@20 (Review ranking), 1 − Brier score (Forecast).

### Evaluation Setup

- **30 models**: API models (GPT 5.6 Sol/Terra, Claude Opus 4.8/Sonnet 5, Gemini 3.5 Flash) and open-weight models (Llama 3.1, ProLong, GPT-OSS, Qwen 3/3.5, Gemma 4, Nemotron Nano, Kimi Linear)
- **Context lengths**: 8K to 512K tokens (measured in Llama 3 tokenizer)
- **Generation budgets**: Base models get sufficient budget; instruction-tuned +256 tokens; thinking models +16K tokens
- Models instructed not to rely on code for state tracking to test native representations

---

## Empirical Validation / Results

### State Tracking and Pattern Learning are Unsaturated

**API models at 512K input length**: Despite saturating recall tasks, frontier models show substantial variation on LEAPS. Many are comparable to or worse than naive baselines on Cups, Coins, and Orderbook.

**Open-weight models at 128K**: Even models above 90% on recall tasks perform similar to or worse than naive baselines on LEAPS.

### Degradation with Longer Context

Performance relative to each model's 8K baseline shows:

- **Recall**: Relatively consistent across context lengths
- **State tracking**: Smoothly declines
- **Pattern learning**: Some initial improvement (strong models leverage additional data) before declining

> "Model performance drops by up to 80% from 8K to 128K on LEAPS."

### Thinking Models Underperform Base Counterparts

Across Qwen3.5, Nemotron, Gemma, and Kimi Linear families, base models generally outperform post-trained counterparts by up to 1 standard deviation (in z-score normalized terms).

### Correlation Analysis

- Pattern learning tasks (Chess, RPS, Forecast) correlate highly with each other
- Market prediction tasks (Kaggle, Social Media) show low correlation with all other tasks
- State tracking tasks correlate moderately with each other (except Cups)
- State tracking and pattern learning capabilities appear **intertwined** (e.g., ML Dev and Code Tracking correlate with Social Media, Forecast, Ranking, Chess, RPS)

### Generation Budget Analysis

Performance may improve from B=8K to B=16K thinking tokens but **plateaus** beyond 16K; some settings even degrade (e.g., Chess at 128K). More thinking alone is insufficient.

### Qualitative Failure Modes

| Model | Task | Failure |
|-------|------|---------|
| Qwen 3.5 30B | Cups | Exhausts generation budget while manually tracking swaps |
| GPT-OSS-120B | Kaggle | Assumes spurious pattern (responder = last value) from 2 examples, ignores rest of input |

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Distinct capability axes**: LEAPS empirically demonstrates that recall, state tracking, and pattern learning are separable long-context capabilities that require different evaluation.

2. **Scaling behavior differs**: State tracking has constant memory requirements regardless of trajectory length (if done efficiently), while pattern learning has an increasing performance ceiling with more data—yet models fail to exploit this.

3. **Capability intertwinement**: The correlation between state tracking and pattern learning tasks suggests shared underlying mechanisms that merit further investigation.

### Practical Implications

1. **Benchmark design**: LEAPS provides contamination-resistant, scalable evaluation methodology applicable to future benchmark development.

2. **Model development guidance**: The finding that thinking models underperform suggests current post-training techniques (optimized for reasoning) may compromise long-context capabilities.

3. **Application readiness**: For agentic applications requiring state tracking (coding agents) or pattern learning (forecasting, personalization), current models are far from production-ready at long context lengths.

---

## Conclusion

LEAPS introduces a comprehensive benchmark for evaluating two critical yet understudied long-context capabilities: state tracking and pattern learning. Key takeaways:

1. Even frontier models struggle substantially more on LEAPS than on recall-oriented benchmarks, which remain largely saturated.
2. Performance degrades significantly with increasing input length—up to 80% from 8K to 128K tokens.
3. Thinking models often underperform their base counterparts, suggesting improvements needed in long-context post-training.
4. LEAPS distinguishes models along two new axes of long-context capability, complementing existing recall-based evaluations.

**Future directions**: The authors hope LEAPS guides development of the next generation of LCLMs, particularly for long-horizon agentic applications. The benchmark's contamination resistance and scalability make it suitable for ongoing model evaluation as LCLMs continue to improve.

---

_Markdown view of https://picx.dev/p/RO8Twi, served by PicX — AI-generated visual whiteboard summaries of research papers._