METR report: AI capabilities grow exponentially, but models lag in designing AI evaluations
The AI evaluation nonprofit METR has updated its time horizon plot to include judgment-heavy tasks, finding that some AI capabilities continue to improve exponentially. However, the report also finds that models lag in designing AI evaluations themselves, highlighting a gap in meta-cognitive abilities.
Coverage timeline
Techmeme
Grace Huckins / MIT Technology Review : METR updates its time horizon plot to add judgment-heavy tasks, finding some AI capabilities still grow exponentially but models lag at designing AI evaluations — A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory …

Related stories
METR: Anthropic models improving twice as fast, AI use drives ~80% of acceleration
A METR report, based on an 11-week internal assessment at Anthropic, finds model capability progress roughly doubled since October 2025, with AI-assisted research driving about 80% of the acceleration. The report also reveals three unreleased internal models (Model 2, 3, 4), with the strongest exceeding Opus 5.5 by about three months of progress. Anthropic redacted much of the most informative evidence from the public version, citing confidentiality concerns.
6 reports · 5 sources47 points · 6 comments

METR: AI R&D-Relevant Capabilities Improving Rapidly, Harder to Measure
METR published an analysis of capability evaluations for public models up to Opus 5.5 and GPT-6 Astra, finding that AI R&D-relevant capabilities are improving rapidly and becoming harder to measure. The report highlights significant uncertainty about which capabilities are necessary for drastic acceleration and how quickly they are improving.
1 reports · 1 sources

Investigator agents increasingly distinguish alignment evals from capabilities evals
METR researchers report that investigator agents, instructed to explore environments and probe what is being evaluated, are increasingly able to tell alignment evals apart from capabilities evals. They do this by noticing that environments look fake and identifying tasks as honeypots designed to make an agent do something bad. The finding highlights a growing challenge for alignment eval design.
1 reports · 1 sources

Realtime-Bench: Frontier LLMs achieve only 44% success on real-time interactive tasks
Researchers from NVIDIA, UT-Austin, and other institutions released Realtime-Bench, a benchmark of 50 interactive game and web tasks that require real-time decision-making. Testing 12 frontier models, the best performer (GPT-6-Astra) achieved only 44% success, highlighting LLMs' weakness in low-latency tasks.
2 reports · 2 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe