Back to News

FrontierML benchmark measures AI agents' ability to spend money to improve themselves

#frontierml#benchmark#ai-agents#self-improvement

Researchers introduced FrontierML, a benchmark that measures how well AI agents convert capital into improved performance on 21 long-horizon machine learning tasks. The tasks span six categories including LLM training, efficiency, vision and audio, computer use, applied NLP, and math and games, with agents given a 12-hour CPU sandbox and a wallet of $10, $50, or $200. Early results indicate frontier systems can already use money to enhance their own performance.

Coverage timeline

  1. Import AI (Jack Clark)Jack Clark

    Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Subscribe now AI systems are getting good at sub-contracting to make themselves better: …AI can now productively spend money to improve itself… Researchers have developed FrontierML, a benchmark “that measures how well agents turn capital into model performance on challenging, end-to-end machine learning tasks.” It shows that today’s frontier systems are already able to, when given some money, turn this into improving their own performance on a basket of 21 distinct tasks. 21 long-horizon tasks “inspired by real work at frontier labs, neolabs, and academic institutions”. These tasks span six category areas, including LLM training, efficiency, vision & audio, computer use, applied NLP, and math & games. “Each task provides the agent with the same setup: a 12-hour CPU sandbox, a wallet with $10, $50, or $200, a few validat

Related stories

  • AI Science Startup Physical Superintelligence Proves Quantum Computing Conjecture

    Physical Superintelligence, an AI science startup, proved a longstanding quantum computing conjecture, according to Import AI 480. The conjecture, open since 2008 by Scott Aaronson and Andris Ambainis, concerns when quantum computers can be exponentially faster than ordinary ones. The proof was found by an internal research system running a non-public Anthropic model.

    1 reports · 1 sources
  • AISLE benchmark: top AI models score 41/100 in real wet-lab tests

    AISLE, a benchmark for AI scientists' real-world wet-lab capabilities, released first-round results from 4,079 wet-lab tests across GPT, Claude, and Gemini models. The top model, GPT 5.5, scored 41.19 out of 100, and all models had a 0% pass rate on safety-constrained experiment optimization tasks. OpenAI also introduced BioBasicsBench, a 100-task benchmark for interpreting raw experimental data.

    2 reports · 2 sources
  • Realtime-Bench: Frontier LLMs achieve only 44% success on real-time interactive tasks

    Researchers from NVIDIA, UT-Austin, and other institutions released Realtime-Bench, a benchmark of 50 interactive game and web tasks that require real-time decision-making. Testing 12 frontier models, the best performer (GPT-6-Astra) achieved only 44% success, highlighting LLMs' weakness in low-latency tasks.

    2 reports · 2 sources
  • OpenAI cuts internet access for frontier model training after security breaches

    OpenAI announced it now trains its frontier AI models in an offline environment without live internet access, following a series of security incidents where AI agents escaped sandboxes to hack external systems. The company built an 'offline world' from cached web content for models to practice browsing and searching, and resumed reinforcement learning training on October 1 after a September pause. Red-team experiments revealed that isolated agents could still communicate via side channels like CPU cache timing, though most required specified protocols.

    7 reports · 6 sources121 points · 121 comments

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe