Colibrì: C-based inference framework runs 744B-param GLM-5.2 on laptops via SSD offloading, no GPU needed
Colibrì, an open-source C-based hierarchical inference framework, has gained 32k GitHub stars for enabling large MoE models to run on laptops with limited RAM by offloading inactive experts to SSD, without requiring a GPU. It supports models from GLM-5.2/5.3, DeepSeek V4 Flash, Qwen, to 975B Inkling and 2.8T Kimi K3, with RAM requirements as low as 16GB for int4-processed GLM-5.2.
Coverage timeline
量子位量子位
笔记本跑7000亿参数GLM！无GPU也行? SSD当显存用火爆GitHub 田, 晏林 2026-09-26 17:01:00 来源： 量子位 GitHub现在最火热的大模型开源小蜂鸟Colibrì是个啥？ 闻乐 发自 凹非寺 量子位 | 公众号 QbitAI 25GB笔记本硬跑744B GLM-5.2，32GB内存挑战2.8T Kimi K3—— 甚至都不用GPU 。 GitHub现在最火热的大模型开源小蜂鸟 Colibrì ，纯C实现、零引擎依赖的分层推理框架，已经狂揽 32k Star 。 它最早是冲着GLM-5.2来的。 正常情况下，744B的总参数规模已经让普通消费级电脑望而却步，但Colibrì的办法可以说是相当简单粗暴： 内存装不下，那就别全装进去。 模型里暂时用不到的部分，直接扔在SSD里；等推理真的需要哪个专家，再现场从硬盘把它捞出来。 于是，GLM-5.2经过int4处理后大约372GB的权重，可以在最低16GB、推荐24GB左右RAM的机器上运行，GPU甚至不是必需品。 作者最初验证它的开发机，也只有 12核CPU+25GB RAM 。 现在，Colibrì已经不满足于744B了。 它目前已经覆盖9个模型家族， 从GLM-5.2/5.3、DeepSeek V4 Flash、Qwen，一路支持到了975B的Inkling，以及2.8T参数的Kimi K3 。 虽然后者需要大约1.6TB硬盘空间，但RAM要求只要32GB起步。 744B模型，内存里只放9.9GB Colibrì能这么玩，首先得感谢这两年越来越流行的 MoE架构 。 以GLM-5.2为例。虽然整个模型足足有744B参数，但MoE并不会在生成每个token时把744B参数全部计算一遍。 它会先通过Router判断：这个token该交给哪些“专家”处理？然后只激活其中一小部分。 GLM-5.2总参数744B，但每个token实际激活的参数大约只有40B，这就留下了一个很大的操作空间： 既然绝大多数专家当前根本用不上，为什么非得把它们一直放在昂贵的高速内存里？ 于是Colibrì干脆把模型拆成了 常驻 和 临时调用 两部分。 Attention、Embedding、共享专家这些每次推理都要用到的Dense部分，大约17B参数，int4之后只占约9.9GB，直接常驻RAM。 真正占地方的是
Related stories
Intel Unveils Data-Center GPU D-Series With Up to 480GB LPDDR5X Memory
Intel announced its data-center GPU D-series, featuring up to 480GB of LPDDR5X memory, an unusually large capacity aimed at demanding inference workloads. The memory uses laptop-class LPDDR5X, setting a new GPU memory record for the company.
1 reports · 1 sources
Janas Tool Runs LLMs Larger Than RAM by Streaming Experts from SSD
A Hacker News post introduces Janas, a tool that enables running large language models (LLMs) larger than available RAM by streaming expert modules from SSD storage. This approach allows users to execute models that exceed memory capacity, potentially expanding the range of models deployable on standard hardware.
1 reports · 1 sources220 points · 94 comments
Inferact runs Kimi K3 on 16 Google TPU v7 at 709 tokens/s, 57% faster than GB200
Inferact, a startup founded by the original vLLM team, reported that 16 Google TPU v7 chips ran Kimi K3 at 709 tokens per second, 57% faster than 16 Nvidia GB200 GPUs under identical conditions. The speedup comes from a custom megakernel inference kernel and DeepSeek's DSpark speculative decoding framework; the code has been open-sourced.
1 reports · 1 sources
vLLM-Omni and FastVideo Optimize MiniMax H3 Serving, Generate MP4 in 8.7s on Eight B300 GPUs
A technical post from vLLM-Omni and FastVideo describes system-level optimizations for serving MiniMax H3, a multimodal model that generates video with synchronized audio. The optimizations span the full resident pipeline, including attention and communication, fused DiT operators, parallel VAE decoding, compact output transfer, and parallel MP4 construction. FastVideo's FastH3 reduces 49 DiT forward passes to 4, achieving real-time full MP4 generation in 8.678–8.710 seconds on eight B300 GPUs for a 10.125-second clip.
2 reports · 2 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe