Realtime-Bench: LLMs struggle with low-latency interactive tasks
Researchers introduced Realtime-Bench, a benchmark of 50 interactive game and web tasks designed to test AI systems' ability to make decisions in real-time. The benchmark includes 30 web tasks across travel, e-commerce, service, and security categories, and 20 game tasks spanning arcade, puzzle, strategy, and simulation. Initial results indicate that current general-purpose LLMs perform poorly on these low-latency tasks, suggesting significant progress is needed before AI can handle real-time applications like gaming or online auctions.
Coverage timeline
Import AI (Jack Clark)Jack Clark
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Subscribe now LLMs are very bad at low-latency tasks: …Realtime-Bench says it’ll be a while till AI systems can play games or do online auctions for you… Researchers have built Realtime-Bench, “a suite of carefully curated dynamic tasks where delayed actions directly affect success”. The test helps evaluate how well modern, general-purpose AI systems can perform tasks where decisions need to be made in real-time. The benchmark consists of 50 interactive game and web tasks, split across 30 web tasks in four categories (travel, e-commerce, service, and security), and 20 game tasks (arcade, puzzle, strategy, and simulation). Who did the research: The benchmark was built by people at the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Stea
Related stories
Google DeepMind unveils Genie 4 world model with real-time audio and natural-language scene editing
Google DeepMind announced Genie 4, its fourth-generation world model for real-time interactive simulations. It generates audio alongside visuals, allows users to modify scenes via natural language prompts in real time, and produces simulations of about two minutes, up from one minute in Genie 3. Powered by Gemini 3.8 Flash, it is initially available as a research preview to trusted testers, targeting pre-visualization, game prototyping, and robot training.
5 reports · 5 sources
Ai2 benchmark shows LLM novelty judgments vary with evaluation setup
The Allen Institute for AI released a benchmark called Old Ideas, Novel Problems to test how reliably LLMs judge research-idea novelty. The findings show that small changes to the evaluation setup can produce very different verdicts on the same ideas.
1 reports · 1 sources
Mature CNN/RNN and small LLMs deployed in industrial settings
LatePost reports that as frontier models dominate headlines, mature and cheaper CNN/RNN models and small LLMs are being deployed in real-world industrial settings such as logistics sorting centers and manufacturing lines. Examples include J&T Express using AI servers to monitor conveyor faults, an Apple supplier using vision models for hinge inspection, and judicial departments using a 27B Qwen model for archival processing. The article notes that these deployments often use GPUs from Denglin, which sold over 10,000 units in the first four months of the year.
1 reports · 1 sources
Google and collaborators release CLASHarness benchmark for multi-agent conflict in computer use
Researchers from Google, Emory, USC, UF, UChicago, UCSB, and Columbia released CLASHarness, a benchmark and pipeline for evaluating 15 major LLMs on multi-agent conflict scenarios in computer-use tasks. The dataset includes 600 multi-turn executable conflict tasks across 20 domains, with 7,970 4K visual evidence images and 3,284 mandatory checkpoints, and is open-sourced on GitHub.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe