Back to News

UniPat AI's PaperBenchX: GPT-6 Astra full reproduction rate only 13.98% on 93 tasks

#ai-benchmark#paper-reproduction#math-ai#gpt-6

UniPat AI's PaperBenchX benchmark evaluated 10 model configurations on 93 real paper replication tasks, finding that the strongest model, GPT-6 Astra, achieved only a 13.98% full reproduction rate. This comes amid debate over AI math benchmarks, with 25 Fields Medalists including Tao and Deng Yu signing an open letter warning that treating math problem-solving as a benchmark is harmful to mathematics and the community.

Coverage timeline

  1. 量子位量子位

    ChatGPT踢到铁板了!能破解千禧数学难题,但论文复现率低至13.98%? 允中 2026-10-08 17:28:44 来源: 量子位 PaperBenchX为代表的基准或许能更好地衡量AI的科研实力 允中 发自 凹非寺 量子位 | 公众号QbitAI OpenAI最近真是频频往数学界扔重磅炸弹。 上个月,OpenAI宣布其AI模型仅用88小时,便攻克了困扰人类百年的千禧年七大顶级数学难题之一——N-S方程;昨天,又放出722篇数学手稿,千禧难题又有仨。 但在UniPat AI发布的PaperBenchX测评中,面对93个真实论文复现任务,表现最强的GPT-6 Astra,完整复现率却只有13.98%。 △10组受测配置在93个复现任务上总体与分阶段表现 随橙想! 当前模型已经能取得破解千禧年数学难题这样的伟大突破,但很少能完成一次可靠的端到端科学复现。 (正经脸)这引发了一些思考:在广泛的AI for Science领域中,我们要以什么标准判断「做对了科学」?本着严谨踏实的科研态度,迈向AI数学家乃至General AI Scientist的第一步到底是什么?我们又该如何形成一套可验证、可比较的标准,真正衡量AI在科学研究上的进步? 该用什么尺子衡量AI在科学里的进步? 此时此刻,让我们再细品陶哲轩、邓煜等25位菲尔兹奖得主联名发布的公开信。 他们肯定了大语言模型的数学能力急剧提升,已经能解决数学领域的重大难题,但也提出了更关键的问题: AI公司把解决数学问题当作基准测试 ,这对数学这门科学、对数学共同体都是有害的。 所以,这不是一封反对AI的信,它问的是一个更朴素的问题: 衡量AI在科学里的进步,到底该用什么尺子? 信里还有一句分量更重的话:把解决数学问题当作基准测试的错位问题,只是更广泛的错位问题的一部分,这种错位同样影响着其他科学与创意职业,乃至整个社会。 的确,上面提出的问题从数学领域扩展到自然科学乃至社会领域,只会更难回答。 数学至少还有最后一道防线:Lean。证明对不对,机器可以逐步检查,答案的真假有兜底。 可放到其他科学领域里,一个电磁仿真的反射系数、一条能带、一个收敛后的带隙,都没有Lean。一个和论文对得上的数字,背后可能是错误的物理模型、没收敛的仿真,或者碰巧凑出合理值的无效后处理。 所以,在衡量AI自主提出问题、设计实验、做出新发现的Ge

Related stories

  • SenseTime-led study: GPT-6-Astra scores near-perfect on benchmarks but fails real-robot tasks

    A research report by SenseTime, Daxia Robotics, Ropedia, NTU Singapore, and the University of Tokyo analyzes GPT-6-Astra's embodied intelligence across 12 simulation benchmarks and 8 real-robot platforms. The model achieves near-perfect scores on traditional benchmarks (e.g., 100% on LIBERO-Long++, 99.3% on LIBERO-Pro) but fails on real-robot tasks, such as taking over 4 hours and $55 to place a pen in a holder, and failing 11 insertion attempts. The report concludes GPT-6-Astra has largely crossed the embodied intelligence threshold but remains far from practical deployment.

    1 reports · 1 sources
  • GPT-6 Astra solves last unsolved FrontierMath Tier 4 problem; Epoch AI declares benchmark saturated

    OpenAI's GPT-6 Astra has solved the last unsolved FrontierMath Tier 4 problem, leading Epoch AI to declare the benchmark saturated. All Tier 4 problems have now been successfully solved at least once by AI, with OpenAI reporting a 97.6% score for Astra. The benchmark, launched in November 2024 to resist AI saturation, saw top scores around 5% at its Tier 4 debut in July 2025.

    1 reports · 1 sources
  • Artificial Analysis: Gemini 4 Argon matches GPT-6 Astra on intelligence, cuts hallucination to 15%

    Artificial Analysis reported that Google's Gemini 4 Argon (high) matches GPT-6 Astra (max) on its Intelligence Index, with a 15% hallucination rate versus 51% for Astra. The model also costs 60% of Astra's per-task price under discounted pricing.

    1 reports · 1 sources
  • OpenAI launches GPT-6.1 Sol, nearly matching Astra at one-fifth price

    OpenAI announced GPT-6.1 Sol at DevDay, an upgrade to GPT-6 Sol that nearly matches GPT-6 Astra on agentic coding, computer use, and professional work, at one-fifth of Astra's standard API prices. The model is priced at $2 per million input tokens and $10 per million output tokens, matching GPT-6 Sol and Claude Sonnet 5.5, and is available in ChatGPT Work, Codex, and the API. OpenAI also introduced a $500-per-month Pro 500 plan with Astra Ultrafast access and the always-on agent 'dots'.

    4 reports · 3 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe