ATLAS: All-round Testing of Long-context Abilities across Scales

Summary

• ATLAS is a benchmarking framework for long-context language models that evaluates capabilities across eight dimensions (3 foundational + 5 application) and eight length slices (8K to 1M tokens), using 6,438 instances and testing 26 models.
• The framework introduces length-aware AUC scoring, which integrates performance over a geometric length grid rather than measuring at a single point, and ATLAScore, a harmonic mean of category aggregates that penalizes imbalanced capability profiles.
• Key empirical findings: Rankings reshuffle substantially between 128K and 1M scopes (7 models move ≥2 ranks); degradation is capability-specific (Retrieval/QA decay most, Code is most independent); foundational and application layers capture non-redundant signals (R²=0.61 at 128K).
• Uncertainty propagation is handled end-to-end via the delta method, with Monte Carlo validation, providing confidence intervals for all reported scores.
• Component selection follows three criteria (length extensibility, deterministic scoring, cross-model discrimination) and is validated via leave-one-out ablations (Spearman ρ ≥ 0.96).

Introduction and Theoretical Foundation

The Problem

Long-context language models advertise context windows up to millions of tokens, but existing evaluations suffer from three limitations:

• Single-point measurement: Reporting scores at one length (e.g., 128K) hides degradation curves.
• Narrow task coverage: Synthetic probes (needle-in-a-haystack) test only retrieval; real workloads require reasoning, aggregation, and code understanding.
• Lack of uncertainty quantification: Most benchmarks report point estimates without confidence intervals.

Design Principles

ATLAS addresses these gaps through three properties:

1. Capability decomposition – failures can be attributed to specific dimensions rather than averaged away.
2. Fixed length grid – degradation is measured, not assumed.
3. Compact instance budget – the full protocol remains repeatable for model iteration.

The framework is component-agnostic: individual benchmarks are replaceable instantiations that satisfy explicit selection criteria. The layered 3+5 taxonomy separates foundational capabilities (retrieval, aggregation, multi-step reasoning) from application workloads (QA, ICL, code, memory, holistic assessment), recognizing that these can disagree—a model may retrieve evidence reliably yet fail on downstream use.

Methodology

Capability Taxonomy

The taxonomy organizes evaluation into two layers:

• Foundational layer (3 dimensions): Retrieval (MRCR-8 needle), Aggregation (OOLong-Synth), Reasoning (GraphWalks Extend)
• Application layer (5 dimensions): QA (LOFT-Text Retrieval Extend), ICL (Helmet-ICL Extend), Code (LongCodeBench), Memory (AMemBench-ACU), Holistic assessment (LongBench-v2 / AA-LCR)

The holistic assessment dimension is scored separately because its tasks cannot be meaningfully length-sliced without distorting the task itself.

Length-Aware Scoring

The benchmark evaluates each dimension over a fixed set of context lengths:

$$\mathcal {L} = \{8 \mathrm{K}, 16 \mathrm{K}, 32 \mathrm{K}, 64 \mathrm{K}, 128 \mathrm{K}, \dots, 1 \mathrm{M}\}$$

For a reporting scope $L^{\star} \in \{128\mathrm{K}, 1\mathrm{M}\}$, the normalized trapezoidal AUC score is:

$$\mathrm{AUC} _ {d} (L ^ {\star}) = \frac {\sum_ {i = 0} ^ {n - 1} \Delta_ {i} [ s _ {d} (\ell_ {i}) + s _ {d} (\ell_ {i + 1}) ] / 2}{\ell_ {n} - \ell_ {0}}$$

where $\Delta_i = \ell_{i+1} - \ell_i$ and $s_d(\ell) \in [0, 100]$ is the normalized score at length $\ell$.

The geometric spacing means the trapezoidal rule naturally weights longer-range intervals more heavily, where degradation is most consequential.

Category Aggregates and ATLAScore

Three category aggregates are computed:

• B (foundational): mean of AUC scores over base dimensions
• C (application): mean of AUC scores over length-sliced application dimensions
• S (holistic): mean of original-length scores

The overall score is the harmonic mean:

$$\text{ATLAScore} (L ^ {\star}) = 3 \Big (B (L ^ {\star}) ^ {- 1} + C (L ^ {\star}) ^ {- 1} + S (L ^ {\star}) ^ {- 1} \Big) ^ {- 1}$$

The harmonic mean prevents a model from receiving a high score by excelling in only one category.

Component Selection Criteria

Each dimension is filled by a component satisfying:

1. Length extensibility – controllable context lengths
2. Deterministic scoring – no judge dependence
3. Cross-model discrimination – separates models

Open-ended summarization is excluded because n-gram overlap metrics are weak proxies and model-based judging introduces judge-version dependence.

Uncertainty Propagation

• Subset-level confidence intervals use task-appropriate estimators with cluster correction.
• AUC variance propagates linearly (weighted average over independent slices).
• The nonlinear harmonic ATLAScore uses the delta method for variance propagation.
• Monte Carlo validation is provided in Appendix F.

Empirical Validation / Results

Evaluation Setup

26 proprietary and open-weight models were evaluated using each provider's recommended inference settings. Middle truncation is applied when a model's advertised context length falls below a target slice.

Key Findings

1. Length-Aware Scoring Changes Rankings

• 20 of 26 models shift position when scope extends from 128K to 1M; 7 move by ≥2 ranks.
• Gemini-3.1-Pro-Preview leads at 128K; Claude-Opus-4.6 leads at 1M.
• GPT-5.2 drops from 4th to 8th (loss of >15 points) due to steeper degradation.
• Mean relative decay is 24.3%, ranging from 8.5% (Claude-Opus-4.6) to 60.5% (GLM-4.7).
• 95% confidence intervals are 1–2 points, well below the 5–16 point gaps driving reshuffling.

2. Degradation Is Capability-Specific

• Retrieval and QA are most decay-prone: multiple models lose >40% of their 128K score at 1M, with extremes >70%.
• Code shows lower, more uniform decay, suggesting a different bottleneck.
• GPT-5.2 retains strong ICL (97.96 at 1M) but drops sharply on Retrieval (46.4% decay) and QA (59.2% decay).
• Reasoning models (with explicit CoT traces) show systematically lower decay: 11 of 15 have decay <25%, versus non-reasoning models spanning 19.8%–60.5%.

3. Layered Taxonomy Captures Non-Redundant Signals

• At 128K: foundational vs. application aggregates have R²=0.61, Spearman ρ=0.74.
• At 1M: R²=0.73, ρ=0.88.
• 15 of 26 models shift ≥4 positions between layer rankings at 128K (max shift: 12).
• Kimi-K2.6 ranks 18th on foundational but 6th on application; GPT-5.2 ranks 3rd foundational but 12th application.

4. Component Validation

• Mean pairwise Spearman correlation across 7 length-sliced dimensions: ρ̄=0.64.
• Code is most independent: correlations with Retrieval (ρ=0.07) and Reasoning (ρ=0.17) are not statistically significant.
• Leave-one-dimension-out preserves full ranking: Spearman ρ ≥ 0.97 at 128K, ρ ≥ 0.96 at 1M.

Table: ATLAS Capability Dimensions and Components

Theoretical and Practical Implications

Theoretical Contributions

1. Formal framework for length-aware evaluation: AUC over geometric length grids is a principled alternative to single-point measurement.
2. Layered taxonomy with empirical validation: The 3+5 structure is shown to capture non-redundant signals, not just redundant composite measures.
3. Uncertainty-aware scoring pipeline: The delta method for harmonic means provides rigorous confidence intervals.

Practical Implications

1. Reporting standard: Public comparisons should state both the evaluated length scope and the capability profile behind the aggregate score.
2. Model selection guidance: Deployment decisions should consider dimension-specific strengths:
   • RAG systems should not treat high ICL scores as substitutes for retrieval robustness.
   • Repository assistants should not over-weight general QA if Code is weak.
   • Demonstration-heavy workflows may prefer high ICL even without top retrieval.
3. Development protocol: ATLAS-Lite (128K-only) is provided for fast screening but should not replace full ATLAScore reporting.

Conclusion

ATLAS establishes that long-context quality is a profile over both capability dimensions and context lengths, not a single score at an advertised window size. The empirical results across 26 models demonstrate that:

• Rankings reshuffle substantially between 128K and 1M scopes
• Degradation is highly capability-specific
• Foundational strength explains only part of application performance

The framework supports a stricter reporting norm for long-context models, emphasizing transparency over single-number comparisons.

Limitations and Future Work

• Current scope: English-only; holistic assessment uses original benchmark lengths; AMemBench-ACU relies on model-generated transcripts; LongCodeBench begins at 32K; open-ended summarization omitted.
• Contamination risk: Public components require periodic refresh.
• Future directions: Multilingual and open-ended generation coverage, stronger refresh mechanisms, and analyses of training choices for ultra-long robustness.

Ethical Considerations

• ATLAS promotes transparency rather than a single universal ranking.
• Reports both aggregate scores and capability-level profiles to reduce overclaiming risk.
• Uses public or constructed components; avoids private user data.
• Documents generation bias in model-generated transcripts.
• Recommends reporting length scope, component scores, and limitations alongside any headline ATLAScore.