EdgeRock-4B open-weight model cuts KV cache 99.1%, boosts throughput 6.42x
Rock Matrix (a unit of Yandex? no, 岩山科技) together with HKUST and CMU released and open-sourced EdgeRock-4B, an open-weight model that keeps KV cache fixed at 72MiB from 4K to 256K context, a 99.1% reduction versus the source model. On a single 80GB GPU it runs 512 concurrent long-context inferences at 12,124 tokens/s output throughput, 6.42x the source model, while retaining an 82.9% average pass rate on AIME 2024-2026, far above the best bounded-state model at 18.8%. The model supports text and image inference, with weights and code open-sourced.
Coverage timeline
量子位量子位
KV Cache砍掉99%，单卡吞吐飙6倍，港科CMU联手岩山科技开源新模型 一水 2026-10-26 14:45:25 来源： 量子位 一张卡，512路长推理同时跑 允中 发自 凹非寺 量子位 | 公众号 QbitAI 谷歌今年的一篇论文，愣是把存储芯片巨头的股价都给整出动静了。 论文里提到， KV Cache至少能压缩6倍 。 要知道，大模型推理越来越吃内存，KV Cache又是其中一笔不小的开销。 现在谷歌突然把它压下去这么多，美光等存储芯片巨头的股价随之下跌。 结果才过去几个月，又有人对KV Cache下了更狠的一刀。 这次连压缩都不满足了。 上下文从4K一路涨到256K，KV Cache占用愣是没变。 在256K上下文下，始终固定在72MiB。相比源模型，KV Cache直接减少 99.1% 。 更夸张的是，一张80GB显卡同时跑512路长推理，输出吞吐达到12124 tokens/s，是源模型的6.42倍。 而且推理能力也基本保住了，AIME 2024—2026平均一次通过率达到82.9%，而参评的有界状态模型即便做到14B，最好也只有18.8%。 这就是 岩山科技 旗下岩芯矩阵联合港科大、CMU等高校刚刚发布并开源的 EdgeRock-4B 。 它给长推理带来的一个变化是： 推理再长，内存不涨。 模型权重和推理代码目前均已 开源 ，还支持文本和图像推理。 推理越长，为什么越贵？ 现在的模型，动不动就要思考几万token；Agent更是能连续工作几十轮，搜索结果、代码、工具调用记录不断往上下文里塞。 模型越干越起劲，显存却先扛不住了。 问题就出在KV Cache上。 如今大多数模型，都把推理历史存在KV Cache里，每生成一个token，就多存一些。 以Qwen3.5-4B为例，每个token要新增32 KiB的KV Cache。推理到64K，一条序列就是2 GiB；推理到256K，就是8 GiB。 这在推理模型时代尤其要命。 模型越来越爱“深度思考”，动辄输出几万token；Agent一跑就是几十轮工具调用，上下文越滚越长。 推理越长，占的显存越多，一张卡能同时服务的请求就越少。 到了端侧，KV Cache还得跟感知、控制模块争抢本就有限的内存。 业界的办法大致有两类。 一类是给每个token“瘦身”： 多头潜在注意力（MLA）压缩KV，KIVI这
Related stories
HKUST and Huawei propose BalanceRoute to cut MoE LLM serving load-balancing bottleneck
Researchers from HKUST and Huawei's 2012 Lab propose BalanceRoute, an online routing method that models synchronous waiting in MoE LLM serving and converts short-term load prediction into routing decisions. In a 96-GPU test, the predictor-based variant BR-H improves decode throughput by about 15% over the strongest baseline. The work is accepted by NeurIPS 2026 and deployed in the jointly built MaaS system MathBot-Pie.
1 reports · 1 sources
Tsinghua and Qujing Introduce KTransformers-FineTune for CPU-GPU MoE Fine-Tuning
Tsinghua University's MADSys lab and Qujing Technology, with LlamaFactory, introduced KTransformers-FineTune (KT-FT), a CPU-GPU co-computation method for fine-tuning mixture-of-experts (MoE) models. It claims 2.5-21.2x throughput gains over traditional offloading by having the CPU compute expert weights directly instead of transferring them to the GPU. The approach includes a semantic-aware proactive layout engine and dynamic thread scheduling for routing skew, and is fully open-sourced.
Open-source projects unlock hidden Tensor Core throughput on GeForce GPUs, doubling RTX 4090 performance
Open-source developers released projects on GitHub that unlock higher Tensor Core throughput on NVIDIA GeForce GPUs by modifying boot-time configuration, without driver swaps or BIOS flashes. Tests reportedly doubled the FP16/FP32 and BF16/FP32 throughput of an RTX 4090 from about 165 TFLOPS to over 330 TFLOPS, with gains also observed on RTX 3090 Ti, RTX 4070 Ti SUPER, and RTX 2080 Ti. The projects target RTX 20, 30, and 40 series, with some experiments done on Windows 11.
MiniCPM-V 4.7 open-sourced: 35B MoE runs offline on iPhone in 3GB memory
ModelBest and OpenBMB have open-sourced MiniCPM-V 4.7, a multimodal MoE model available in 35B-A3B and 1B versions. The 35B-A3B, the first MoE in the MiniCPM-V series, runs on iOS and Android devices with about 3GB memory, achieving over 13 tokens per second on an iPhone 17 Pro Max offline. The 1B version (1.3B actual parameters) targets OCR, math, and video understanding.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe