Back to News

Einsia AI's Visual Cipher Bench: Top GPT-6 Astra Scores 26.7%, Three Models Zero

#benchmark#visual-reasoning#llm-agents#einsia-ai

Einsia AI's Navers Lab released Visual Cipher Bench, a benchmark for long-horizon visual reasoning with 40 original puzzles and 67 black-and-white line drawings, where clues exist only in images and no feedback is given. Across 8 models and 3,480 attempts, the best configuration (GPT-6 Astra max) scored 26.7%, while DeepSeek V4.1 Flash, Qwen 3.8 Max, and Kimi K3.1 Pro scored zero; a one-sentence rule hint raised overall accuracy from 12.2% to 51.1%.

Coverage timeline

  1. 机器之心机器之心

    修 bug、写功能、跑长任务…… 今天的 Agent 几乎都能做。因为要求有文字说明,做错了有测试报错。 那如果规则只画在图里呢? 一张黑白线稿,几个编号,几条箭头。没有题干,没有选项,做对做错也没人告诉你。 所有线索都在图里,很难转成文字,只能反复看图、提假设、再回到图里验证。这就是 长程视觉推理 。 8 个前沿模型、3480 次尝试: 最强的 GPT-6 Astra(max)只答对 26.7%,DeepSeek V4.1 Flash、Qwen 3.8 Max、Kimi K3.1 Pro 三个模型一题没对。 更戏剧的是:只要用一句话点明规则,总体准确率立刻从 12.2% 跳到 51.1%。 近日 Einsia AI 旗下 Navers Lab 发布 Agent 视觉推理基准 Visual Cipher Bench 。 40 道原创视觉谜题,67 张黑白线稿,它考的是长程视觉推理,正好卡在现有评测的空白上: 视觉评测大多是单轮问答,题目问什么写得清清楚楚,推理链很短 Agent 评测几乎都在文本和代码里,就算用到图像,也能靠试错拿到环境反馈 Visual Cipher Bench 两头都不给:线索只在图里,对错没人告诉你 40 道题共用同一句指令:看图,把最终答案写进文件。 哪些图形是样例、哪些是待解的输入、答案是单词还是数字都得自己看。 40 道题分为四类,每类 10 道: 空间操作与模拟 :折叠、旋转、投影之后长什么样 逻辑约束与规划 :从图里读出约束,走通唯一方案 规则归纳 :从几个样例里悟出这道题独有的规则 知识解码 :认出摩斯码、盲文、文字游戏这类 “暗号” 论文题目:Visual Cipher Bench: Benchmarking LLM Agents on Long-Horizon Visual Reasoning 项目主页:https://lab.einsia.ai/visual-cipher-bench/ arXiv:https://arxiv.org/abs/2611.16861 GitHub:https://github.com/Einsia/Visual-Cipher-Bench 成绩单:榜首不到三成,三个模型 0 分 8 个模型、29 种推理强度配置,每种配置跑三遍。按各模型的最佳配置排: GPT-6 Astra(max):26.7% ,

Related stories

  • Realtime-Bench: Frontier LLMs achieve only 44% success on real-time interactive tasks

    Researchers from NVIDIA, UT-Austin, and other institutions released Realtime-Bench, a benchmark of 50 interactive game and web tasks that require real-time decision-making. Testing 12 frontier models, the best performer (GPT-6-Astra) achieved only 44% success, highlighting LLMs' weakness in low-latency tasks.

    2 reports · 2 sources
  • OpenAI releases GPT-6.1 Terra with computer-use focus, API price halved

    OpenAI announced GPT-6.1 Terra, a fast, efficient model emphasizing visual reasoning and computer use, priced at $1 per million input tokens and $5 per million output tokens. The standard API price is 50% lower than GPT-5.6 Terra and GPT-6.1 Sol, and the model is available in ChatGPT Work, Codex, and via API (gpt-6.1-terra) for Plus, Pro, Enterprise, and Edu users. In the Agents' Last Exam benchmark, Terra scored 15 points higher than GPT-5.6 Terra at medium reasoning intensity, with about one-third the cost, and matched GPT-6 Astra's performance at one-tenth the token price.

    4 reports · 4 sources
  • OpenAI launches GPT-6 Cyber cybersecurity model, cuts out-of-bounds incidents to under 1%

    OpenAI introduced GPT-6 Cyber, a cybersecurity model designed to find exploitable vulnerabilities and develop fixes while staying within authorized scope. During testing, the model went out of bounds less than 1% of the time, down from about 5% with GPT-5.6 Cyber, and outperformed GPT-6 Astra on cyber benchmarks. The model will first launch to a limited set of customers in OpenAI's Daybreak Red program and has found over 81 novel vulnerabilities in Firefox.

    5 reports · 4 sources61 points · 7 comments
  • UCLA and Adobe open-source Tev-27B long-horizon agent model, matching GPT-6 Astra and Claude Opus 5 on ALE-CM

    Researchers from UCLA and Adobe Research released a paper and open-sourced Tev-27B, a long-horizon agent model trained on their new Tev-Gym benchmark with dense, verifiable, progressive rewards via progress potential reward shaping (PPRS). On the Agents' Last Exam computation and mathematics subset (ALE-CM), Tev-27B achieves Pass@5 of 0.28, matching GPT-6 Astra and Claude Opus 5; the same recipe applied to Tev-320B makes it the strongest open-source model on Terminal-Bench 4.0's computation and mathematics subset.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe