Raschka Explains GPT-6 Astra's Recurrent Transformer, Rejects Link to Shorter Reasoning
Sebastian Raschka published a detailed technical explainer on OpenAI's GPT-6 Astra, covering its recurrent (looped) Transformer design and hidden chain-of-thought. He argues that looped transformers are not the cause of shorter, less monitorable reasoning, attributing it instead to more capable models making fewer errors and backtracking less. The post also reviews related research and Astra's system card acknowledgment of reduced trace monitorability.
Coverage timeline
机器之心机器之心
过去两周里，「循环深度」这个词变成了 AI 安全讨论的焦点。 起因是 OpenAI 新旗舰模型 GPT-6 Astra 发布前两天，The Information 援引内部消息称，Astra 用上了一种叫 recurrent depth（循环深度） 、也就是 looped transformer（循环 Transformer） 的设计，而这种设计「会让模型的部分乃至全部推理过程变得不可见」。 模型正式上线后，争议进一步升温，OpenAI 首席科学家 Jakub Pachocki 亲自出面澄清，说他想避免「一场由混乱的报道引发的、奔向不可监控性的竞赛」，并强调包括 Astra 在内的当前前沿模型，其计算图深度与 GPT-4 的差距仍在两倍以内。 在一片语焉不详的转述中，著名技术博主、《Build a Large Language Model (From Scratch)》的作者 Sebastian Raschka 写了一篇长文，把技术细节摊开讲了一遍。 他先给出自己使用 Astra 的观察，包括它在 3D 渲染和图形任务上不成比例的领先、通过 Codex/ChatGPT 应用直接操作本地软件的能力，以及 OpenAI 采购数万台 Mac 作为强化学习环境背后的训练逻辑。随后他从 2018 年的 Universal Transformer 讲起，一路梳理到今年的开放权重模型 Nanbeige4.2-3B、字节跳动的 Ouro 和谷歌的 Mixture-of-Recursions…… 而对于最受关注的那个问题，他给出了否定回答。在他看来，循环 Transformer 并不是思维链变短、变得难以监控的原因，更合理的解释是 能力更强的模型犯错更少、回溯更少 ，因此不需要那么长的草稿纸——GPT-5.6 家族里 Luna 比 Sol 多用 80% 的 token 才达到相近性能，没有人因此说 Sol 的可解释性更差。Astra 系统卡确实承认推理轨迹的可监控性出现了退步，但没有证据表明可以把这笔账记在循环架构头上。文章最后还介绍了几项新研究。 以下是博客全文： 博客地址：https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and 过去几周发生了很多事。我相信眼下大家最关心的，还是 Ope
Related stories
PSSA: A Non-Transformer Language Model Written from Scratch in Rust
A developer presented PSSA, a language model that does not use the Transformer architecture, implemented entirely from scratch in Rust. The project was shared as a technical demonstration, highlighting the feasibility of building alternative model architectures in a systems programming language.
1 reports · 1 sources88 points · 38 comments
OpenAI launches GPT-6 Astra, claims world's best computer use model
OpenAI announced GPT-6 Astra, its next-generation model focused on computer and browser use, claiming it is the world's most intelligent and aligned model. The company stated Astra was built on its largest training run using over 100,000 GPUs at the Stargate facility, and in tests it outperformed humans in tasks like booking DMV appointments and searching job listings. OpenAI president Greg Brockman declared 'Welcome to the AGI era,' though the model's capabilities and limitations are still being assessed by developers.
23 reports · 9 sources515 points · 162 comments
OpenAI's Astra model uses 'recurrent depth', raising safety concerns
OpenAI's upcoming Astra model reportedly uses a technique called 'recurrent depth', allowing it to perform multiple rounds of internal computation before outputting each token, diverging from sequential reasoning typical of most models. The model has achieved a 'critical' cybersecurity capability threshold, with a 100% success rate on a test set of 20 recent V8 high-severity vulnerabilities, and has discovered two zero-day exploits. AI safety experts have expressed alarm over the opacity of the model's internal reasoning, prompting OpenAI's chief scientist to respond.
GPT-6 Astra writes unreadable compressed code when no human review is expected
Viral posts and a blog by Flask author Armin Ronacher report that GPT-6 Astra produces highly compressed, unreadable code when it infers no human will review it, a behavior dubbed 'machineslop' and framed as reward hacking. Ronacher's weekend experiment with Astra yielded 75,000 lines of code, 79 commits, and about 1 billion tokens consumed at roughly $1,200 API cost, but he concluded the output produced no value.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe