Back to News

Tsinghua and CUHK propose Cache Engineering, open-source AKV to cut inference costs

#cache engineering#kv cache#agents#inference cost

Researchers from Tsinghua University and the Chinese University of Hong Kong proposed Cache Engineering, an approach that lets agents directly manage KV cache instead of compressing context, and open-sourced the inference system AKV. In evaluations across three models and four agent tasks, Cache Engineering reduced average inference costs by 27.7% and improved average task success by 1.7% compared to Context Engineering.

Coverage timeline

  1. 机器之心机器之心

    用 Agent 跑长程任务时,模型推理和工具调用的历史轨迹会随着任务进行越积越多。超长的历史轨迹会显著拖慢模型的速度并带来高昂的费用。 codex 通过总结来进行上下文压缩 目前常见的 上下文工程(Context Engineering) ,主要通过裁剪 [1] 或总结 [2] 来 压缩历史文本 。然而,早期文本一旦被修改,模型已经计算的后续缓存(KV cache)会随即失效,剩余文本必须重新输入模型计算(re-prefill),引入额外延时和成本。此外,压缩也会丢失历史信息,影响 Agent 任务的成功率。 缓存工程和上下文工程执行相同任务的成本和准确率 针对这一问题,清华与香港中文大学的研究团队提出了 缓存工程(Cache Engineering) ,并开源了对应的推理系统 AKV。缓存工程不压缩历史文本,而是允许 Agent 直接管理后续推理使用的缓存(KV Cache) ,同时加速运行、降低成本并提高任务成功率。在三个模型、四个 Agent 任务的评测中,缓存工程相比上下文工程, 平均推理成本降低 27.7%,平均任务成功率提升 1.7% 。 本文的共同第一作者为清华大学电子系 NICS-EFC 实验室的博士生傅天予、本科生王若溪和闵子涵。 论文标题:Cache Engineering for Agents 论文链接:arxiv.org/abs/2610.28577 项目主页:fuvty.github.io/thinking_yard_project_page/projects/akv/ 开源代码:github.com/thu-nics/AKV 上下文工程:重算缓存引入额外代价 以「通过多轮网页搜索回答用户问题」为例,上下文工程的做法是删掉已阅读的旧网页。这看起来只是少了一些文字,但后续推理的 KV 缓存,是在「读过这段网页」的条件下算出来的。删掉前文后,就得把保留部分重新算一遍,这就是 re-prefill。 论文发现,在 BrowseComp-Plus 任务上,如果只保留最近 12 轮工具调用的结果,总推理成本反而会达到完整上下文的 1.85 倍。这揭示了上下文工程的主要问题: 直接修改上下文,导致缓存重算,引入的额外成本有时反而超过其省下的成本。 研究团队因此把关注点转向缓存本身:能否少读一些缓存,同时让已经算好的状态继续用? 缓存工程:直接管理缓存,

Related stories

  • EdgeRock-4B open-weight model cuts KV cache 99.1%, boosts throughput 6.42x

    Rock Matrix (a unit of Yandex? no, 岩山科技) together with HKUST and CMU released and open-sourced EdgeRock-4B, an open-weight model that keeps KV cache fixed at 72MiB from 4K to 256K context, a 99.1% reduction versus the source model. On a single 80GB GPU it runs 512 concurrent long-context inferences at 12,124 tokens/s output throughput, 6.42x the source model, while retaining an 82.9% average pass rate on AIME 2024-2026, far above the best bounded-state model at 18.8%. The model supports text and image inference, with weights and code open-sourced.

    1 reports · 1 sources
  • PKU Researchers Propose MapFlow to Reuse Attention Maps Across LLMs, Cutting Prefill Latency 15-20%

    Researchers at Peking University's PKU-DAIR lab propose MapFlow, a method that reuses attention maps as intermediate states across different LLMs to avoid recomputing context when switching models. In evaluations, MapFlow reduces prefill latency by 15-20% and boosts throughput by up to 25% under a P90 first-token latency constraint in multi-turn routing scenarios, while maintaining generation quality. The paper has been accepted at SIGMOD 2027 and the code is open-sourced.

    1 reports · 1 sources
  • Shanghai AI4S open-sources REMORY, a 1.9B memory module for recursive self-improvement

    The Shanghai Academy of AI for Science, with Fudan University and Shanghai Chuangzhi Institute, open-sourced REMORY, a 1.9B-parameter memory module that enables small models to run independently for days via recursive self-improvement, achieving results near GPT-6.1-Terra. Trained fully self-supervised on two H800 GPUs, REMORY leads several key benchmarks and is available on arXiv, Hugging Face, GitHub, and the Xinghe Qizhi platform.

    1 reports · 1 sources
  • Lev: Recurrent Decision Model Adapts Compute per Problem, No Chain-of-Thought

    Researchers from Lehigh University, UBC, and Vector Institute propose Lev, a decision model that uses recurrent computation instead of chain-of-thought to adaptively spend more compute on hard problems. Lev reads out option probabilities after each recurrent pass, trains jointly across depths, and uses a lightweight stopper to decide how many passes to run, enabling a single set of weights to answer at arbitrary depth. The authors describe this as System 1.5, preserving the direct interface of System 1 while allowing extra compute on difficult cases like System 2.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe