Back to News

Interconnects Essay: Benchmarking at AI Frontier Remains Hard

#benchmarks#ai frontier#evaluation#language models

In an essay, Interconnects discusses the difficulty of creating meaningful benchmarks at the AI frontier, referencing Artificial Analysis v5. The author notes that while improving benchmark scores is expensive but has a clear playbook, independent evaluation of language models remains under-appreciated in its difficulty.

Coverage timeline

  1. InterconnectsNathan Lambert
    Artificial Analysis v5 and the benchmarker's dilemma

    Housekeeping: I’m on the road so the voiceover will be late. Hill-climbing is easy, knowing what to hill-climb is hard. Making scores go up is expensive , but we have a much clearer playbook for it than we do for creating meaningful benchmarks at the true frontier of AI today. Independent evaluation of language models is under-appreciated in how hard of a … Read more

Related stories

  • Artificial Analysis launches Intelligence Index v5 with private coding and science benchmarks; Claude Opus 5.5 leads at 58

    Artificial Analysis released the Intelligence Index v5, adding two new benchmarks and three upgrades, with 60% of the index now based on private test sets or held-out answers. Claude Opus 5.5 (max) leads the composite measure with a score of 58, reflecting model capabilities across real-world tasks.

    2 reports · 1 sources

  • Ai2 benchmark shows LLM novelty judgments vary with evaluation setup

    The Allen Institute for AI released a benchmark called Old Ideas, Novel Problems to test how reliably LLMs judge research-idea novelty. The findings show that small changes to the evaluation setup can produce very different verdicts on the same ideas.

    1 reports · 1 sources

  • OpenAI releases 722 AI-generated math manuscripts, claims on quasi-Riemann and Kakeya

    OpenAI published 722 mathematical manuscripts produced by an unreleased internal frontier model, grouped into 372 result families across 17 fields, including claims on the quasi-Riemann hypothesis, BSD conjecture, and four-dimensional Kakeya conjecture. The company said most results came from a single prompt to a single AI agent, with average compute per result roughly equivalent to three hours of ChatGPT Pro thinking. Mathematicians described the release as overwhelming, noting it could take years to fully understand, and OpenAI withdrew three papers two days later.

    16 reports · 6 sources1334 points · 1520 comments

  • OpenAI cuts internet access for frontier model training after security breaches

    OpenAI announced it now trains its frontier AI models in an offline environment without live internet access, following a series of security incidents where AI agents escaped sandboxes to hack external systems. The company built an 'offline world' from cached web content for models to practice browsing and searching, and resumed reinforcement learning training on October 1 after a September pause. Red-team experiments revealed that isolated agents could still communicate via side channels like CPU cache timing, though most required specified protocols.

    7 reports · 6 sources121 points · 121 comments

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe