Prime Intellect Audits Public Evaluations, Finds Broken Graders and Reward Hacks
Prime Intellect published findings from auditing popular evaluations reported in model cards, uncovering broken graders, ambiguous tasks, and reward hacks that affect how results should be interpreted. The organization also open-sourced fixes for these issues.
Coverage timeline
Prime Intellect
While running popular evaluations reported in model cards, we found broken graders, ambiguous tasks and reward hacks that change how results should be read. We share what we found and open-source our fixes.
Related stories
Anthropic Reports Unintended Model Actions in Evaluations and Internal Use
Anthropic published a research post examining unintended model actions occurring in its evaluations and internal use, highlighting safety-relevant findings. The post discusses potential risks and implications for AI safety practices.
1 reports · 1 sources
Prime Intellect releases prime-rl v0.10.0 with IPO loss as default
Prime Intellect released prime-rl v0.10.0 and verifiers v0.4.0. The update introduces the new RL algorithm IPO as the default, designed for async RL stability, along with performance improvements and new model support.
1 reports · 1 sources
Artificial Analysis launches Intelligence Index v5 with private coding and science benchmarks; Claude Opus 5.5 leads at 58
Artificial Analysis released the Intelligence Index v5, adding two new benchmarks and three upgrades, with 60% of the index now based on private test sets or held-out answers. Claude Opus 5.5 (max) leads the composite measure with a score of 58, reflecting model capabilities across real-world tasks.
2 reports · 1 sources
OpenAI claims disproof of Exponential Time Hypothesis, plus math and physics results
OpenAI announced three new discoveries in mathematics, computer science, and physics, including a claimed disproof of the Exponential Time Hypothesis (ETH), a conjecture widely believed for nearly 30 years. The results cover local smoothing for the Euclidean wave equation, the Kakeya conjecture in four dimensions, and a rigorous mathematical existence proof for four-dimensional quantum field theories including quantum chromodynamics. External experts guided the results, and materials were released on GitHub for verification.
5 reports · 5 sources1153 points · 701 comments
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe