Back to News

VideoMemBench benchmark tests long-video memory in multimodal LLMs; Claude Opus 5 scores 45.7

#video-memory#benchmark#multimodal-llm#long-video

Researchers from the University of Technology Sydney, Chinese University of Hong Kong, Hong Kong University of Science and Technology, and Zhejiang University introduced VideoMemBench, a benchmark evaluating whether multimodal large language models can remember and correctly use earlier information in multi-turn conversations about long videos. The benchmark focuses on two dependency types—episodic and answer—and reports Claude Opus 5 scoring 45.7, indicating significant room for improvement.

Coverage timeline

  1. 机器之心机器之心

    设想我们正看一场篮球比赛,并持续询问大模型有关的画面。对话一开始,我们问:“刚才完成那次突破得分的是谁?” 模型给出了球员的名字。又经过几轮关于其他回合的问答,我们追问:“我们第一次聊到的那名球员,这场比赛最后一共得了多少分?” 这时,模型需要先从早先的回答中找回那名球员,再结合比赛信息确定得分。即使它能看懂最终统计,只要把 “那名球员” 认错,仍然会答非所问。 看电影时也有类似的需求。我们先讨论开头的某个场景,之后又聊到其他角色和情节。几轮问答后,我们指着当前画面中的一个角色问:“这个角色在我们最开始聊到的那个场景里出现过吗?” 此时,模型需要回忆最初讨论的是哪一幕,再判断这个角色是否出现在那一幕。仅仅理解眼前的画面,还不足以回答这个问题。 这样的追问在真实交流中很自然,要求模型把当前问题与先前的对话联系起来:比如前一个例子需要找回早先答案确定的人物,后一个例子需要找回最初讨论的场景。 来自悉尼科技大学、香港中文大学、香港科技大学和浙江大学的研究者联合提出 VideoMemBench ,关注大模型在围绕同一视频持续对话时,能否记住并正确使用前面讨论过的信息。 项目主页(Project):https://videomembench.github.io/ 论文(arXiv):https://arxiv.org/abs/2610.40131 代码(GitHub):https://github.com/Oliiveralien/VideoMemBench 数据集(Dataset):https://huggingface.co/datasets/VideoMemBench/data 论文标题: VideoMemBench: How Well Do MLLMs Remember in Video Conversations? 目前许多视频问答评测采用相互独立的问题:模型依靠视频和眼前这一道题,就能给出回答,未必需要回顾对话历史。VideoMemBench 则把问题组织成有前后联系的多轮对话,让后续回答需要用到之前已经建立的信息,并重点考察两种依赖。 第一种是场景依赖(Episodic Dependency)。当问题回指之前讨论的某一幕时,模型需要通过对话确定所指的片段,再利用画面回答新问题。第二种是答案依赖(Answer Dependency)。问题沿着模型前面识别出的人物、

Related stories

  • PKU Researchers Propose MapFlow to Reuse Attention Maps Across LLMs, Cutting Prefill Latency 15-20%

    Researchers at Peking University's PKU-DAIR lab propose MapFlow, a method that reuses attention maps as intermediate states across different LLMs to avoid recomputing context when switching models. In evaluations, MapFlow reduces prefill latency by 15-20% and boosts throughput by up to 25% under a P90 first-token latency constraint in multi-turn routing scenarios, while maintaining generation quality. The paper has been accepted at SIGMOD 2027 and the code is open-sourced.

    1 reports · 1 sources
  • Meta and academic researchers propose World-T³, a video world model with test-time training for long-term memory

    Researchers from Meta Reality Labs, the University of Illinois Chicago, and Brown University propose World-T³, a camera-controllable video world model that uses test-time training to store historical observations in fast weights, keyed by camera pose, enabling consistent scene recall when revisiting locations. On a long-term revisit benchmark of 120 trajectories, it improves revisit PSNR from 13.52 dB to 18.31 dB and reduces absolute rotation error from 3.344° to 1.075°, while a 4-step distilled model runs at 20.6 FPS on a single NVIDIA B200 GPU at 1280×704 resolution.

    1 reports · 1 sources
  • ToolSpace: Tool-Centric Environment Synthesis Framework Trains 744B Model to Match Claude Opus 5

    Researchers from Renmin University, Fudan, and Shanghai Jiao Tong propose ToolSpace, a tool-centric framework for synthesizing cross-environment training data for long-horizon agents. It extracts tools from execution traces, builds tool composition graphs, and generates verifiable cross-environment tasks via state-aware random walks. Training with ToolSpace data improved a 744B model's composite benchmark score from 44.6 to 56.6, matching Claude Opus 5, and the team also built upgraded benchmarks AutomationBench Pro and τ³-Bench Banking Pro.

    1 reports · 1 sources
  • Sleep framework proposed to help LLMs retain knowledge in continual learning

    A research paper introduces the Sleep framework, a new method designed to help large language models retain knowledge during continual learning. The framework aims to address the challenge of catastrophic forgetting, where models lose previously learned information when trained on new tasks.

    1 reports · 1 sources233 points · 33 comments

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe