ChemCost benchmark evaluates LLM agents on chemical procurement cost reasoning
Researchers from CMU, Notre Dame, UNC Chapel Hill, and Pitt introduced ChemCost, a benchmark for multi-step chemical procurement cost estimation by AI agents, accepted at NeurIPS 2026 Datasets and Benchmarks Track. The benchmark includes 1,427 reactions, 2,261 chemicals, and 230,775 supplier quotes, requiring agents to perform name alignment, quote retrieval, packaging selection, unit conversion, and cost aggregation. Experiments show that even the strongest frontier model agents achieve only 50.6% accuracy within a 25% error tolerance on clean inputs, with performance dropping further on realistic text and OCR inputs.
Coverage timeline
机器之心新闻资讯
让目前最强的大模型智能体去估算一个化学反应的原料采购成本，误差控制在 25% 以内an的成功率只有一半左右；一旦输入换成真实文献里常见的自然语言段落和 OCR 扫描文本，这个数字还会进一步大幅下滑。 近日，来自卡内基梅隆大学（CMU）、圣母大学、北卡罗来纳大学教堂山分校和匹兹堡大学的研究团队提出了多步化学采购成本估算基准 ChemCost，将科学智能体的评测从高层的路线规划，推进到真实工业工作流中的定量成本推理。该工作已被 NeurIPS 2026 Datasets and Benchmarks Track 接收。 现有科学智能体已经能够调用领域工具规划化学合成路线，但相关评测大多停留在精选案例演示、专家主观评估或 LLM-as-judge 打分，缺乏精确、无需裁判（judge-free）的客观真值。ChemCost 构建了一个涵盖 1,427 个可评估化学反应、2,261 种化学品和 230,775 条供应商包装报价的冻结数据库，要求智能体从反应描述出发，自主完成化学名称对齐、供应商报价检索、合规包装选择、计量单位换算和最终成本汇总。实验表明，即便是最强的前沿模型智能体，在干净输入下的成本预测准确率（25% 误差容忍度内）也仅为 50.6%。 本文的共同第一作者是卡内基梅隆大学的 Yuyang Wu 和圣母大学的 Yue Huang，通讯作者是卡内基梅隆大学的Jian Ma 教授、Olexandr Isayev 教授，以及圣母大学的 Xiangliang Zhang 教授。 论文标题：Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning 论文链接：https://arxiv.org/abs/2605.07251 引言 大型语言模型作为工具调用智能体已经展现出强大的能力，在软件工程（如 SWE-bench）和网页交互（如 WebArena）等通用任务上取得了显著进展。在化学领域，ChemCrow 和 Coscientist 等系统也证明了大模型可以辅助规划合成路线、调用外部专业工具。 然而，一旦落到真实实验室或工业场景，智能体往往难以胜任这样一类工作流：既要与结构化数据库交互，又要满足严格的领域约束，还要给出精确的定量决策。化学采购成本估算正是其中极具代表性的一项任务。在化工生
Related stories
UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code
Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.
2 reports · 1 sources
WorldMemArena benchmark evaluates multimodal agent memory via action-world interaction
Researchers from UCSB, Stanford, ETH Zurich, Johns Hopkins, CMU, and J.P. Morgan Chase released WorldMemArena, a multimodal agent memory benchmark accepted at NeurIPS 2026. It evaluates memory through an observation-action-environment feedback loop, breaking long-term memory into write, maintain, retrieve, and use stages to pinpoint where agents fail. The benchmark includes multimodal cues, memory lifecycle tracking, and world changes after actions, addressing limitations of static recall or final QA evaluations.
Tsinghua introduces SRHarness and MechBench for agentic symbolic regression and mechanism discovery
Tsinghua University researchers introduced SRHarness, a runtime framework for agentic symbolic regression, and MechBench, a benchmark for mechanism discovery, aiming to improve AI's ability to derive formulas from data and understand underlying physical mechanisms. Experiments show SRHarness achieves 72.97% symbol recovery with DeepSeek-v4-flash-0731, far exceeding Codex's 20.72%, while MechBench reveals that even GPT-5.6-sol-powered Codex recovers mechanisms at only 13.75% on the core set, highlighting a significant gap between pattern discovery and mechanism understanding.
Fudan, Tencent Hunyuan, Tsinghua Release ExplorationBench to Test AI Rule Discovery
Fudan University, Tencent Hunyuan, and Tsinghua University released ExplorationBench, a benchmark measuring how AI systems discover rules in unfamiliar environments. It includes two altered 'alien worlds'—the programming language AlienCode and the logic proof system AlienLogic—where models must learn rules through trial and error. Results showed that without environment feedback, scores remained low (max 11.0%), while autonomous exploration led to high scores (up to 89.0%).
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe