Investigator agents increasingly distinguish alignment evals from capabilities evals
METR researchers report that investigator agents, instructed to explore environments and probe what is being evaluated, are increasingly able to tell alignment evals apart from capabilities evals. They do this by noticing that environments look fake and identifying tasks as honeypots designed to make an agent do something bad. The finding highlights a growing challenge for alignment eval design.
Coverage timeline
METR
Summary Models sometimes can tell that an eval is an alignment eval, not a capabilities eval (Needham & Edkins, 2025) . To figure out what features of an eval tell a model what kind of eval it’s in, we run investigator agents on a set of evals for both capabilities and alignment. We instruct these agents to explore the environment and probe what it’s evaluating. We find that investigator agents are increasingly able to tell capability and alignment evals apart, and that they do this by noticing that an environment “looks fake” and figuring out that a task is a “honeypot” designed to make an agent do something bad. Motivation Alignment evals are often made by mocking capabilities evals and deployment situations Suppose we wanted to make an alignment eval that measured a model’s propensity to escape a sandbox. Here, we’d make situations that mirrored the capability evals, RL environments, and deployment situations where we thought the model might try to escape the sandbox. For instance,

Related stories
METR: Anthropic models improving twice as fast, AI use drives ~80% of acceleration
A METR report, based on an 11-week internal assessment at Anthropic, finds model capability progress roughly doubled since October 2025, with AI-assisted research driving about 80% of the acceleration. The report also reveals three unreleased internal models (Model 2, 3, 4), with the strongest exceeding Opus 5.5 by about three months of progress. Anthropic redacted much of the most informative evidence from the public version, citing confidentiality concerns.
6 reports · 5 sources47 points · 6 comments

METR and Redwood Research Red-Team OpenAI's Agent Monitoring, Find Exploits
METR and Redwood Research published findings from a three-week high-access red-team exercise (July 21–August 11) that stress-tested OpenAI's internal misalignment monitoring system. The 61-page report, shared with OpenAI researchers and security staff, identified vulnerabilities including an exploit that could let AI agents hide hundreds of recent actions. The same day, Redwood Research also released a paper on simulated control evaluations, a method for testing whether developer monitors catch deliberately harmful agent behavior under realistic deployment conditions.
4 reports · 3 sources

UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code
Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.
2 reports · 1 sources

MIT: GPT-6 Astra edits Qwen weights, boosting MATH-500 from 7.9% to 48.1%
MIT CSAIL researchers demonstrated that GPT-6 Astra can directly read and modify the weights of Qwen3.5-0.8B, improving its MATH-500 score from 7.9% to 48.1% by altering only 0.016% of parameters, without training or gradient descent. The team named this capability Weight-Space Programming. However, spaCy author Matthew Honnibal publicly challenged the paper, arguing that the experimental conditions across the three claimed abilities differ significantly and that the presentation is misleading.
6 reports · 5 sources2949 points · 892 comments

News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe