Back to News

NeoCognition's ApprenticeBench: Fable 5.1 and GPT-6 Astra beat human testers in simulated accounting

#ai-agents#benchmark#enterprise-software#accounting

NeoCognition released ApprenticeBench, a benchmark that places AI agents in a simulated construction company to perform real accounting tasks using Odoo. Fable 5.1 and GPT-6 Astra achieved pass rates of 72% and 68%, respectively, surpassing the best human tester's 51%. The benchmark highlights a 72% vs 36% performance gap between Fable 5.1 and Opus 5, despite similar scores on other benchmarks.

Coverage timeline

  1. 机器之心机器之心
    Fable 5.1超越人类却贵了2.5倍，NeoCognition给Agent进公司算了笔账

    大模型榜单越来越多，但榜单上的几分差距，到了真实工作中究竟意味着什么，并不容易判断。 在 Artificial Analysis（AA）、Terminal-Bench 4.0 和 CursorBench 4.0 等评测中，Fable 5.1 和 Opus 5 的成绩只相差几分。可如果让它们进入同一家公司，使用同样的软件、学习同样的业务，再连续处理同一批账单，结果却变成了 72% 对 36%。 完成的任务数，差了一倍。 这个结果来自 NeoCognition 最新发布的 ApprenticeBench。与常见的单项能力测试不同，它把 Agent 放进一家模拟建筑公司，让它像新员工一样入职：阅读员工手册和历史资料，使用真实企业软件，并根据主管反馈逐渐摸清公司的业务规则。 ApprenticeBench 主页链接：https://apprenticebench.com/ Agent 没有针对这份工作接受专门训练。它需要一边工作，一边从公司的历史数据和带教过程中学习。在这项评测中，Fable 5.1 和 GPT-6 Astra 分别取得 72% 和 68% 的通过率，也超过了本次表现最好的人类测试者的 51%。 Fable 5.1 和 Astra 出现明显的能力跃升，超过本次人类测试者。图源：ApprenticeBench 博客（https://neocognition.io/blog/apprentice-bench/） 这也带出了一个更大的问题：如果 Agent 能自己熟悉软件、学会业务，今天需要 FDE（前线部署工程师）为每家客户做的部署和适配，未来能不能交给 Agent 自己完成？ 一份看似普通的会计工作，为什么能拉开这么大的差距？对正在训练下一代模型的团队，这些差距又意味着什么？ 公司的规矩，进公司才知道 会计学知识可以提前学习，但是一家公司具体怎么做账，最近改了什么政策，任何通用模型都不可能，也不应该知道。一个新员工入职后要补的课，Agent 一样要补。 这便是 ApprenticeBench 的核心设计思路：Agent 进入一家模拟建筑公司，使用和人类员工同样的软件（Odoo）处理应付账款。入职时，它拿到员工手册、软件教程和六个月的历史账单，随后连续处理 100 张新账单。在第一个月里，主管会逐单指导，后来只在月底提供稀疏反馈。 从翻阅历史记录，到逐步独立处理

Related stories

  • Huawei Noah's Ark Lab and UCL unveil Memento 3, beating 25 ARC-AGI-3 games with 56% fewer actions

    Huawei Noah's Ark Lab and UCL introduced Memento 3, an agent system that completes all 25 games and 183 levels in the ARC-AGI-3 public set using 7,518 actions versus 17,135 for humans, achieving a perfect RHAE score of 100. The system, based on Claude Opus 5 without weight updates or game-specific fine-tuning, uses model-based recursive self-improvement to learn rules, write code, and reflect on errors. In Atari Pong, it developed a control strategy and then won three consecutive rounds 21:0 without further LLM inference requests.

    1 reports · 1 sources

  • Anthropic releases Opus 5.5, claims strongest model tested, 40% cheaper than Opus 5

    Anthropic released Opus 5.5 on September 22, 2026, describing it as the strongest-performing model it has tested to date. The company says Opus 5.5 matches Fable 5.1 on most tasks while costing about 40% less to run than Opus 5, with pricing at $4 per million input tokens and $20 per million output tokens.

    2 reports · 2 sources

  • Anthropic unveils Claude Fable 5.1 and Mythos 5.1, cutting costs up to 45%

    Anthropic released Claude Fable 5.1, generally available, and Mythos 5.1, for trusted partners in cybersecurity and life sciences. Fable 5.1 claims new standards in coding, knowledge work, and long-running problem-solving, with reduced token costs and fewer false-positive restrictions. Pricing remains at $10/$50 per million tokens, but cache read costs drop 75% to $0.25, cutting overall costs by 25% typically and up to 45% for agentic workloads. These are Anthropic's first models to watermark text outputs, with a detection API for eligible groups under EU law.

    12 reports · 7 sources1409 points · 1377 comments

  • OpenAI rolls out GPT-6 Astra broadly, priced at $10/$50 per million tokens

    OpenAI began rolling out GPT-6 Astra on September 3, 2026, initially to a limited set of organizations and Daybreak cybersecurity platform customers, then expanding to all Pro, Enterprise, and Business Premium users and the API by September 5. The model is priced at $10 per million input tokens and $50 per million output tokens, matching Anthropic's Claude Fable 5.1, and scores 99.9% on ARC-AGI 3 with a custom harness. CEO Sam Altman apologized for the 'messy' initial rollout that locked out some paying users, while the company touted the model as a 'generational leap' and the start of 'the AGI era.'

    4 reports · 4 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe