Back to News

METR: AI R&D-Relevant Capabilities Improving Rapidly, Harder to Measure

#ai-capabilities#evaluation#metr#ai-rd

METR published an analysis of capability evaluations for public models up to Opus 5.5 and GPT-6 Astra, finding that AI R&D-relevant capabilities are improving rapidly and becoming harder to measure. The report highlights significant uncertainty about which capabilities are necessary for drastic acceleration and how quickly they are improving.

Coverage timeline

  1. METR

    Contents Overview 1 Background 2 Methodology 3 Results 4 Robustness Checks 5 Discussion 6 Acknowledgments Appendices A Difficulty Estimation and Baselining B Time Horizon Diagnostics C MCI Modelling Methodology D Limitations of IRT E MCI Robustness Checks F Estimating Lifecycle Cost for Tasks G Data Integrity and Reward Hacking H Changes in Capability Growth Rates I Inference Scaling J Swarm Experiments K Al Zimmermann's Programming Contests L Budget NanoGPT M Making METR tasks N Earthborne Rangers and Continual Learning O Qualitative and External Evidence on Models’ Strengths and Limitations Overview This post presents evidence from our capability evaluations of public models relevant to AI R&D acceleration, up to Opus 5.5 and GPT-6 Astra. We remain significantly uncertain about central questions, including (1) what capabilities are in practice necessary for drastic acceleration, and (2) how quickly relevant model capabilities are improving. We think that current public evidence is in

Related stories

  • METR report: AI capabilities grow exponentially, but models lag in designing AI evaluations

    The AI evaluation nonprofit METR has updated its time horizon plot to include judgment-heavy tasks, finding that some AI capabilities continue to improve exponentially. However, the report also finds that models lag in designing AI evaluations themselves, highlighting a gap in meta-cognitive abilities.

    1 reports · 1 sources
  • METR: Anthropic models improving twice as fast, AI use drives ~80% of acceleration

    A METR report, based on an 11-week internal assessment at Anthropic, finds model capability progress roughly doubled since October 2025, with AI-assisted research driving about 80% of the acceleration. The report also reveals three unreleased internal models (Model 2, 3, 4), with the strongest exceeding Opus 5.5 by about three months of progress. Anthropic redacted much of the most informative evidence from the public version, citing confidentiality concerns.

    6 reports · 5 sources47 points · 6 comments
  • OpenAI launches GPT-6.2 Astra and Sol, cuts Astra API prices 20%

    OpenAI introduced GPT-6.2 Astra, a frontier model for complex professional work, and GPT-6.2 Sol, an efficient model for everyday tasks, available in ChatGPT Work, Codex, and the API. Astra's API pricing is $8 per million input tokens and $40 per million output tokens, a 20% cut from GPT-6 Astra, while Sol retains its predecessor's pricing. Benchmarks show Astra trailing only Claude Fable 5.2 in one index and tied in another, though a critical review cites usability frustrations and poor long-task performance.

    7 reports · 4 sources
  • MIT: GPT-6 Astra edits Qwen weights, boosting MATH-500 from 7.9% to 48.1%

    MIT CSAIL researchers demonstrated that GPT-6 Astra can directly read and modify the weights of Qwen3.5-0.8B, improving its MATH-500 score from 7.9% to 48.1% by altering only 0.016% of parameters, without training or gradient descent. The team named this capability Weight-Space Programming. However, spaCy author Matthew Honnibal publicly challenged the paper, arguing that the experimental conditions across the three claimed abilities differ significantly and that the presentation is misleading.

    6 reports · 5 sources2949 points · 892 comments

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe