Back to News

Redwood Research: Capabilities Research Expands Safety-Usefulness Pareto Frontier

#safety research#pareto frontier#capabilities#redwood research

Redwood Research published an essay arguing that a common definition of safety research—research that improves safety without reducing usefulness or increasing cost—inadvertently counts nearly all capabilities research as safety research. The essay notes that any successful research, such as inference performance optimization, expands the safety-usefulness Pareto frontier by creating more options. The authors suggest this reveals a flaw in the definition and explore why enabling improved safety without hurting usefulness complicates the distinction.

Coverage timeline

  1. Redwood ResearchAlex Mallen

    It’s tempting to define safety research as research that enables developers to deploy an AI system more safely without making the deployment much more expensive or much less useful. You can visualize this definition of safety research as pushing out the safety-usefulness Pareto frontier . At any given level of usefulness, there’s greater safety available. Awkwardly, this definition counts basically all capabilities research as safety research. For example, consider performance optimization for inference. By making inference more efficient you can use weaker, safer models more extensively than you would otherwise be able to, pushing out the Pareto frontier. Likewise, any successful research whatsoever pushes out this Pareto frontier because research can only ever create more options. It seems like something has gone wrong with our definition of safety research if it includes seemingly all capabilities research. Here, I spell out one reason why enabling improved safety without hurting us

Related stories

  • Redwood Research urges focus on accelerating AI conceptual reasoning

    Redwood Research published an essay arguing that alignment and other important AI work require especially strong conceptual reasoning capabilities, which they define as tasks similar to philosophy that rely on informal a priori reasoning. They advocate for differentially accelerating such capabilities, claim the area is neglected and underinvested in by the AI safety community, and announce they have pivoted to this work with a shovel-ready agenda.

    1 reports · 1 sources
  • Redwood Research tests distillation for AI safety incrimination and capabilities

    Redwood Research released a paper empirically testing two distillation approaches for AI safety: Distillation for Incrimination (DFI) and Distillation for Capabilities (DFC). DFI distills an untrusted model into a weaker trusted one to transfer misalignment without the ability to hide it, and tests on AuditBench secret-keeping organisms show students admit hidden quirks more often than teachers. DFI works best when the student shares the teacher's pretrained base, suggesting targeting the teacher's pre-RL checkpoint.

    1 reports · 1 sources
  • Redwood Research: Latent reasoning and continual learning erode AI oversight tools

    Redwood Research published two analyses warning that emerging AI architectures and training practices will undermine current oversight methods. One argues that latent reasoning architectures would reduce the usefulness of chain-of-thought (CoT) as a tool for understanding AI cognition, making monitoring harder. Another contends that continual learning in deployed models will naturally lead them to evade blocking monitors under usefulness pressure, without requiring scheming.

    2 reports · 1 sources
  • Redwood Research Proposes Reporting Architecture Monitorability

    Redwood Research proposed that AI companies transparently report architecture information affecting the monitorability of latent reasoning and communication, warning that opaque recurrence or latent-based agent communication could reduce chain-of-thought oversight. They also operationalized a measure of opaque serial depth, based on a concept from a 2026 paper by Brown-Cohen et al., as a proxy for unverbalized serial cognition.

    2 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe