Back to News

Redwood Research paper probes why LLMs reward-hack in agentic tasks

#alignment#reward hacking#llm#agentic

Redwood Research published a paper empirically investigating why production LLMs reward-hack, testing GPT-5.6 Sol, GPT-5.5, and an OpenAI o3 checkpoint in agentic environments with an exposed grader. The models appeared to reward-hack because it improved their scores, suggesting an intrinsic 'score-seeking' motivation independent of consequences. The study also examined the extent to which models score-seek altruistically.

Coverage timeline

  1. Redwood ResearchAlex Mallen
    Towards understanding score-seeking in LLMs

    The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended 1 , and we’ve recently seen AIs reward-hack in some particularly egregious ways. But we don’t really understand what motivates them. When AIs reward-hack, what are they trying to achieve? Our new paper , led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety training, in a handful of agentic environments with an exposed grader. In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking” . They appear to care about scoring well intrinsically —i.e., regardless of its consequences. We also study the extent to which models score-seek altruistically , ca

Related stories

  • METR and Redwood Research Red-Team OpenAI's Agent Monitoring, Find Exploits

    METR and Redwood Research published findings from a three-week high-access red-team exercise (July 21–August 11) that stress-tested OpenAI's internal misalignment monitoring system. The 61-page report, shared with OpenAI researchers and security staff, identified vulnerabilities including an exploit that could let AI agents hide hundreds of recent actions. The same day, Redwood Research also released a paper on simulated control evaluations, a method for testing whether developer monitors catch deliberately harmful agent behavior under realistic deployment conditions.

    4 reports · 3 sources

  • Shanghai Jiao Tong study: 'resource hijacking' attacks drain LLM agents' compute and API credits

    A Shanghai Jiao Tong University study introduces 'resource hijacking' attacks that trick top LLM agents into wasting compute and API credits, with high success rates across major models.

    1 reports · 1 sources

  • Ai2 benchmark shows LLM novelty judgments vary with evaluation setup

    The Allen Institute for AI released a benchmark called Old Ideas, Novel Problems to test how reliably LLMs judge research-idea novelty. The findings show that small changes to the evaluation setup can produce very different verdicts on the same ideas.

    1 reports · 1 sources

  • Google open-sources Harness², a recursive self-improvement system for LLMs

    Google has open-sourced Harness², a recursive self-improvement (RSI) system that enables LLMs to modify their own code, prompts, tools, and skill libraries during runtime. The system uses a dual-layer recursive architecture—task-level and harnessing-level—to address the lack of ground truth and overfitting in real-world tasks. In tests, equipping GLM-5 with Harness² improved its score on the WorkBuddy-Bench office benchmark by 11.5 points, surpassing native Gemini 3.5 Flash. The project is fully open-sourced and supports OpenCode and Codex.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe