Back to News

Random KV Cache Eviction Matches Strong Baselines, Boosts Throughput by 43%

#kv-cache#reasoning-models#inference-optimization#random-attention

Salesforce AI Research and UIUC propose Random Attention, which randomly evicts KV cache entries in reasoning traces instead of using content-based importance scores. It matches strong baselines across four models and six reasoning tasks, and improves inference throughput by 32%–43% over the strongest baseline TriAttention in vLLM serving tests.

Coverage timeline

  1. 机器之心机器之心
    KV Cache不用「精挑细选」？随机删除媲美最强基线，推理吞吐最高再提43%

    大模型越来越会 “想”，也越来越能把 GPU 显存 “想满”。 在数学、科学问答和代码生成等任务中，reasoning model 往往会生成数千乃至数万 token 的长推理链。随着生成持续进行，每个历史 token 对应的 Key-Value 表示都会进入 KV Cache，显存占用随序列长度不断增长。对于长推理部署来说，KV Cache 很快会变成真正的系统瓶颈。 一种直接的解决思路是 KV Cache eviction：给缓存设定固定预算，推理过程中不断判断哪些历史 KV 值得留下，其余永久删除。 过去几年的很多方法，核心都围绕同一个问题展开： 怎样更准确地判断一个 KV 将来还有没有用 ？ 有的方法累计历史 attention，有的观察最近 query 的 attention，有的显式考虑 redundancy，还有工作进一步利用 value magnitude 或 key statistics。虽然打分方式不同，它们共享一个直觉：只要 importance signal 更准，就应该能留下更有价值的 KV。 来自 Salesforce AI Research 和 UIUC 的一项最新研究，却选择从一个近乎 “反算法” 的问题出发： 这些精心设计的 selection signal，本身到底贡献了多少？ 研究团队提出 Random Attention：完整保护输入 Prompt，对后续模型自己生成的 reasoning trace 不计算内容相关的重要性分数，而是在每个 KV head 内独立随机保留。 结果有些反直觉。Random Attention 在四个模型、六个数学、科学与代码推理任务上整体可以媲美论文中表现最强的基线；在主结果表的 60 个 baseline comparison 中，它显著领先 31 个，显著落后只有 1 个。更进一步，在 vLLM 的 32k-token serving 测试中，由于省掉 scoring pass，Random Attention 相比最强基线 TriAttention 的吞吐还能再提高 32%–43%。 但这篇工作真正有意思的，并不只是 “随机方法居然很强”。作者进一步把这个结果拆开，试图回答两个更基础的问题： 过去方法的收益究竟来自 “选得准”，还是来自它们恰好保护了某些关键内容？而 reasoning t

Related stories

  • Apple and CMU propose SSR framework cutting multimodal search agent inference latency by over 90%

    Researchers from Apple and Carnegie Mellon University propose SSR (Selection-based Structured Reasoning), a framework that replaces free-form token-by-token reasoning with selection from predefined reasoning options in multimodal search agents. In experiments across 7 multimodal search benchmarks, SSR reduces per-step inference latency by over 90% while maintaining task success rates comparable to free-form thinking.

    1 reports · 1 sources

  • ByteDance Seed: DeepSeek-V4 accuracy varies every 4 tokens due to chunked KV cache compression

    ByteDance Seed researchers found that DeepSeek-V4's retrieval accuracy fluctuates periodically every 4 tokens, with up to 40.2 percentage points difference in 128K long-context tests. They traced the issue to DeepSeek-V4's chunked KV cache compression, a long-context optimization technique. In a code completion experiment, padding length modulo 4 determined whether the model output the correct answer.

    1 reports · 1 sources

  • vLLM Introduces Tiered KV Cache Offloading Framework

    vLLM's blog details a host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers. This tiered offloading approach aims to reduce recomputation and increase serving capacity.

    1 reports · 1 sources

  • Apple's Internalized Visual Thinking speeds proactive video reasoning 5x

    Apple researchers propose Internalized Visual Thinking (IVT), a post-training framework that internalizes predictive world modeling into model representations. During training, the model learns both text answers and latent representations of future frames, but at inference it skips generating intermediate images, directly answering from current video observations. This removes the overhead of Visual Chain-of-Thought, achieving a 5x speedup in proactive video reasoning tasks such as early-event and next-event prediction.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe