SLDBench v2: Frontier Models Degrade When Planning Scaling-Law Experiments
Researchers from CUHK-Shenzhen, HKU, Tsinghua, and PKU released SLDBench v2, an ICLR 2026 benchmark where AI must autonomously choose scaling-law experiments from a priced catalog with real compute costs. All 8 frontier models degraded when selecting experiments, with prediction errors increasing 1.5 to 4.7 times. Best performer Claude Opus 5.5 used only 4% of full data cost yet surpassed classical power-law fitting accuracy.
Coverage timeline
机器之心ScienceAI
作者丨论文团队 编辑丨ScienceAI 近来， AI for AI （用 AI 研发 AI）备受关注，但相关工作大多停留在 小规模实验 ：少量 GPU、数小时即可得出结论，例如训练算法设计、数据筛选方法对比。真正决定大模型能力走向的 预训练量级实验 ，单次就要消耗数百乃至上千 GPU 小时，AI 还很少参与其中的规划与决策。 人类研究者同样无法大量开展这类实验，通行的做法是借助 Scaling Law ：先完成一批小规模实验，总结出性能随规模变化的规律，再推算大规模实验的结果。先做哪些实验、做到多大规模、何时停止，本身就是 对大规模实验的规划 。 而 Scaling Law 的公式长期依赖研究者 手工设计 ，换到新场景往往需要大量试错。发表于 ICLR 2026 的 SLDBench 尝试让 AI 自己发现公式：在 5,000 多个真实训练实验上，AI 提出的规律，外推精度超过了人类专家此前提出的公式。 但在 SLDBench 中，实验结果事先全部给定， 做哪些实验，仍由人来决定 。 如果这一步也交给 AI 呢？ 近日，来自香港中文大学（深圳）、香港大学、清华大学和北京大学的研究团队发布了 SLDBench v2 ：AI 面对一份由真实训练记录构成的 标价实验目录 ，每选择一个实验，就要支付当年实际消耗的算力，最后用总结出的规律预测从未见过的更大规模实验。 评测给出的答案是： 目前还难以做到。 8 个前沿模型一旦需要自己选择实验，成绩全部下滑，预测误差扩大到原来的 1.5 至 4.7 倍。不过，表现最好的 Claude Opus 5.5 仅用全量实验 4% 的成本，预测精度就超过了经典幂律公式使用全部数据拟合的结果。 论文标题：SLDBench v2: Active Scaling Law Discovery 论文地址：https://arxiv.org/abs/2610.16297 项目主页：https://lingbei-meng.github.io/SLDBench-v2/ 数据与代码：https://github.com/lingbei-meng/SLDBench-v2 Active 设定：实验由 AI 自主选择 SLDBench 的设定相当于「开卷考试」：所有实验结果直接交给 AI，由它写出公式并预测更大规模的实验，论文称之为 Passive（被动）设定
Related stories
UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code
Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.
2 reports · 1 sources
Fudan, Tencent Hunyuan, Tsinghua Release ExplorationBench to Test AI Rule Discovery
Fudan University, Tencent Hunyuan, and Tsinghua University released ExplorationBench, a benchmark measuring how AI systems discover rules in unfamiliar environments. It includes two altered 'alien worlds'—the programming language AlienCode and the logic proof system AlienLogic—where models must learn rules through trial and error. Results showed that without environment feedback, scores remained low (max 11.0%), while autonomous exploration led to high scores (up to 89.0%).
Anthropic: Small AI models may self-replicate on IoT devices within months
Anthropic researchers report that small AI models may be months away from self-replicating on commercial IoT devices like smart ovens and off-brand Android TV boxes. In tests, a 9B model ran inference on 8 of 9 hardware types, and GLM-5.2 autonomously spread an 8B model to all 8 nodes in 4 of 5 attempts. The researchers predict models under 30B could achieve end-to-end replication by the end of 2026.
Ai2 benchmark shows LLM novelty judgments vary with evaluation setup
The Allen Institute for AI released a benchmark called Old Ideas, Novel Problems to test how reliably LLMs judge research-idea novelty. The findings show that small changes to the evaluation setup can produce very different verdicts on the same ideas.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe