# ATLAS: All-round Testing of Long-context Abilities across Scales

> ATLAS reveals long-context model quality is a capability- and length-dependent profile, not a single score, with rankings reshuffling substantially between 128K and 1M scopes.

- **Source:** [arXiv](https://arxiv.org/abs/2605.28079)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/kb1U9l
- **Whiteboard:** https://picx.dev/p/kb1U9l/image

## Summary

# ATLAS: All-round Testing of Long-context Abilities across Scales

## Summary

- **ATLAS** is a benchmarking framework for long-context language models that evaluates capabilities across **eight dimensions** (3 foundational + 5 application) and **eight length slices** (8K to 1M tokens), using **6,438 instances** and testing **26 models**.
- The framework introduces **length-aware AUC scoring**, which integrates performance over a geometric length grid rather than measuring at a single point, and **ATLAScore**, a harmonic mean of category aggregates that penalizes imbalanced capability profiles.
- **Key empirical findings**: Rankings reshuffle substantially between 128K and 1M scopes (7 models move ≥2 ranks); degradation is capability-specific (Retrieval/QA decay most, Code is most independent); foundational and application layers capture non-redundant signals (R²=0.61 at 128K).
- **Uncertainty propagation** is handled end-to-end via the delta method, with Monte Carlo validation, providing confidence intervals for all reported scores.
- **Component selection** follows three criteria (length extensibility, deterministic scoring, cross-model discrimination) and is validated via leave-one-out ablations (Spearman ρ ≥ 0.96).

## Introduction and Theoretical Foundation

### The Problem
Long-context language models advertise context windows up to millions of tokens, but existing evaluations suffer from three limitations:
- **Single-point measurement**: Reporting scores at one length (e.g., 128K) hides degradation curves.
- **Narrow task coverage**: Synthetic probes (needle-in-a-haystack) test only retrieval; real workloads require reasoning, aggregation, and code understanding.
- **Lack of uncertainty quantification**: Most benchmarks report point estimates without confidence intervals.

### Design Principles
ATLAS addresses these gaps through three properties:
1. **Capability decomposition** – failures can be attributed to specific dimensions rather than averaged away.
2. **Fixed length grid** – degradation is measured, not assumed.
3. **Compact instance budget** – the full protocol remains repeatable for model iteration.

The framework is **component-agnostic**: individual benchmarks are replaceable instantiations that satisfy explicit selection criteria. The layered 3+5 taxonomy separates **foundational** capabilities (retrieval, aggregation, multi-step reasoning) from **application** workloads (QA, ICL, code, memory, holistic assessment), recognizing that these can disagree—a model may retrieve evidence reliably yet fail on downstream use.

## Methodology

### Capability Taxonomy
The taxonomy organizes evaluation into two layers:
- **Foundational layer** (3 dimensions): Retrieval (MRCR-8 needle), Aggregation (OOLong-Synth), Reasoning (GraphWalks Extend)
- **Application layer** (5 dimensions): QA (LOFT-Text Retrieval Extend), ICL (Helmet-ICL Extend), Code (LongCodeBench), Memory (AMemBench-ACU), Holistic assessment (LongBench-v2 / AA-LCR)

The **holistic assessment** dimension is scored separately because its tasks cannot be meaningfully length-sliced without distorting the task itself.

### Length-Aware Scoring

The benchmark evaluates each dimension over a fixed set of context lengths:
$$\mathcal {L} = \{8 \mathrm{K}, 16 \mathrm{K}, 32 \mathrm{K}, 64 \mathrm{K}, 128 \mathrm{K}, \dots, 1 \mathrm{M}\}$$

For a reporting scope $L^{\star} \in \{128\mathrm{K}, 1\mathrm{M}\}$, the normalized trapezoidal AUC score is:
$$\mathrm{AUC} _ {d} (L ^ {\star}) = \frac {\sum_ {i = 0} ^ {n - 1} \Delta_ {i} [ s _ {d} (\ell_ {i}) + s _ {d} (\ell_ {i + 1}) ] / 2}{\ell_ {n} - \ell_ {0}}$$
where $\Delta_i = \ell_{i+1} - \ell_i$ and $s_d(\ell) \in [0, 100]$ is the normalized score at length $\ell$.

The geometric spacing means the trapezoidal rule naturally weights longer-range intervals more heavily, where degradation is most consequential.

### Category Aggregates and ATLAScore

Three category aggregates are computed:
- **B** (foundational): mean of AUC scores over base dimensions
- **C** (application): mean of AUC scores over length-sliced application dimensions  
- **S** (holistic): mean of original-length scores

The overall score is the harmonic mean:
$$\text{ATLAScore} (L ^ {\star}) = 3 \Big (B (L ^ {\star}) ^ {- 1} + C (L ^ {\star}) ^ {- 1} + S (L ^ {\star}) ^ {- 1} \Big) ^ {- 1}$$

The harmonic mean prevents a model from receiving a high score by excelling in only one category.

### Component Selection Criteria

Each dimension is filled by a component satisfying:
1. **Length extensibility** – controllable context lengths
2. **Deterministic scoring** – no judge dependence
3. **Cross-model discrimination** – separates models

Open-ended summarization is excluded because n-gram overlap metrics are weak proxies and model-based judging introduces judge-version dependence.

### Uncertainty Propagation

- Subset-level confidence intervals use task-appropriate estimators with cluster correction.
- AUC variance propagates linearly (weighted average over independent slices).
- The nonlinear harmonic ATLAScore uses the **delta method** for variance propagation.
- Monte Carlo validation is provided in Appendix F.

## Empirical Validation / Results

### Evaluation Setup
26 proprietary and open-weight models were evaluated using each provider's recommended inference settings. Middle truncation is applied when a model's advertised context length falls below a target slice.

### Key Findings

#### 1. Length-Aware Scoring Changes Rankings
- **20 of 26 models** shift position when scope extends from 128K to 1M; **7 move by ≥2 ranks**.
- Gemini-3.1-Pro-Preview leads at 128K; Claude-Opus-4.6 leads at 1M.
- GPT-5.2 drops from 4th to 8th (loss of >15 points) due to steeper degradation.
- Mean relative decay is 24.3%, ranging from 8.5% (Claude-Opus-4.6) to 60.5% (GLM-4.7).
- 95% confidence intervals are 1–2 points, well below the 5–16 point gaps driving reshuffling.

#### 2. Degradation Is Capability-Specific
- **Retrieval and QA** are most decay-prone: multiple models lose >40% of their 128K score at 1M, with extremes >70%.
- **Code** shows lower, more uniform decay, suggesting a different bottleneck.
- GPT-5.2 retains strong ICL (97.96 at 1M) but drops sharply on Retrieval (46.4% decay) and QA (59.2% decay).
- **Reasoning models** (with explicit CoT traces) show systematically lower decay: 11 of 15 have decay <25%, versus non-reasoning models spanning 19.8%–60.5%.

#### 3. Layered Taxonomy Captures Non-Redundant Signals
- At 128K: foundational vs. application aggregates have R²=0.61, Spearman ρ=0.74.
- At 1M: R²=0.73, ρ=0.88.
- **15 of 26 models** shift ≥4 positions between layer rankings at 128K (max shift: 12).
- Kimi-K2.6 ranks 18th on foundational but 6th on application; GPT-5.2 ranks 3rd foundational but 12th application.

#### 4. Component Validation
- Mean pairwise Spearman correlation across 7 length-sliced dimensions: ρ̄=0.64.
- Code is most independent: correlations with Retrieval (ρ=0.07) and Reasoning (ρ=0.17) are not statistically significant.
- Leave-one-dimension-out preserves full ranking: Spearman ρ ≥ 0.97 at 128K, ρ ≥ 0.96 at 1M.

### Table: ATLAS Capability Dimensions and Components

| Layer | Dimension | Component | Range | Metric | Instances |
|-------|-----------|-----------|-------|--------|-----------|
| Foundational | Retrieval | MRCR-8 needle | 8K–1M | EM | 792 |
| Foundational | Aggregation | OOLong-Synth | 8K–1M | Answer-level | 800 |
| Foundational | Reasoning | GraphWalks Extend | 8K–1M | F1 | 800 |
| Application | QA | LOFT-Text Retrieval Extend | 8K–1M | MRecall@K | 800 |
| Application | ICL | Helmet-ICL Extend | 8K–1M | Acc | 800 |
| Application | Code | LongCodeBench | 32K–1M | Acc / Pass@1 | 1,043 |
| Application | Memory | AMemBench-ACU | 8K–1M | QPEM | 800 |
| Application | Holistic | LongBench-v2 / AA-LCR | varied | Acc | 603 |
| **Total** | | | | | **6,438** |

## Theoretical and Practical Implications

### Theoretical Contributions
1. **Formal framework for length-aware evaluation**: AUC over geometric length grids is a principled alternative to single-point measurement.
2. **Layered taxonomy with empirical validation**: The 3+5 structure is shown to capture non-redundant signals, not just redundant composite measures.
3. **Uncertainty-aware scoring pipeline**: The delta method for harmonic means provides rigorous confidence intervals.

### Practical Implications
1. **Reporting standard**: Public comparisons should state both the evaluated length scope and the capability profile behind the aggregate score.
2. **Model selection guidance**: Deployment decisions should consider dimension-specific strengths:
   - RAG systems should not treat high ICL scores as substitutes for retrieval robustness.
   - Repository assistants should not over-weight general QA if Code is weak.
   - Demonstration-heavy workflows may prefer high ICL even without top retrieval.
3. **Development protocol**: ATLAS-Lite (128K-only) is provided for fast screening but should not replace full ATLAScore reporting.

## Conclusion

ATLAS establishes that long-context quality is a **profile over both capability dimensions and context lengths**, not a single score at an advertised window size. The empirical results across 26 models demonstrate that:
- Rankings reshuffle substantially between 128K and 1M scopes
- Degradation is highly capability-specific
- Foundational strength explains only part of application performance

The framework supports a stricter reporting norm for long-context models, emphasizing transparency over single-number comparisons.

### Limitations and Future Work
- **Current scope**: English-only; holistic assessment uses original benchmark lengths; AMemBench-ACU relies on model-generated transcripts; LongCodeBench begins at 32K; open-ended summarization omitted.
- **Contamination risk**: Public components require periodic refresh.
- **Future directions**: Multilingual and open-ended generation coverage, stronger refresh mechanisms, and analyses of training choices for ultra-long robustness.

### Ethical Considerations
- ATLAS promotes transparency rather than a single universal ranking.
- Reports both aggregate scores and capability-level profiles to reduce overclaiming risk.
- Uses public or constructed components; avoids private user data.
- Documents generation bias in model-generated transcripts.
- Recommends reporting length scope, component scores, and limitations alongside any headline ATLAScore.

---

_Markdown view of https://picx.dev/p/kb1U9l, served by PicX — AI-generated visual whiteboard summaries of research papers._