Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks

Authors: Franziska Roesner (University of Washington), Tadayoshi Kohno (Georgetown University)

---

Summary (Overview)

• This paper revisits Ken Thompson's 1984 "Reflections on Trusting Trust" compiler Trojan attack in the context of self-modifying AI coding agents — systems that iteratively evaluate and improve their own code and prompts.
• The authors demonstrate that poisoned benchmarks supplied during a self-modifying agent's self-evaluation/self-improvement process can induce future versions of the agent to write vulnerable code on neutral, held-out tasks.
• Successful proof-of-concept attacks are demonstrated against three self-modifying coding agents: the Darwin Gödel Machine (DGM), the Self-Improving Coding Agent (SICA), and Hyperagents (DGM-H).
• A key finding: with Hyperagents powered by Claude Sonnet 4.5, a poisoned benchmark led the agent to self-evolve instructions that disable HTTPS certificate validation on neutral URL-fetching tasks, enabling man-in-the-middle attacks.
• The contamination often persists even when poisoned agents are subsequently evolved against clean or security-focused benchmarks, highlighting the difficulty of decontamination.

---

Introduction and Theoretical Foundation

Thompson's Legacy

The paper builds on Ken Thompson's 1984 Turing Award lecture, which demonstrated how to inject a self-sustaining Trojan into a compiler. The attack consists of:

1. A compiler modification that inserts a backdoor into a target program
2. A second compiler modification that reinserts (1) when the compiler itself is compiled

Thompson's moral: "You can't trust code that you did not totally create yourself."

The Modern Setting

The authors observe that software development has fundamentally changed:

• AI coding agents (e.g., Claude Code, ChatGPT's Codex) now perform substantial coding work
• Self-modifying agents generate new versions of themselves — for example, later versions of Claude Code are authored substantially by Claude Code itself (though human-reviewed)
• Academic research on self-modifying agents and harnesses is an active area

Research Question

Can an adversary supply poisoned benchmarks to a self-modifying coding agent's self-evaluation and self-improvement process to induce future versions of the agent to write vulnerable code on clean, held-out tasks?

Threat Model

• Target: A self-modifying AI coding agent that evaluates and improves itself given a benchmark set
• Attacker capabilities: Full control over the benchmark (including reward function), but no write access to the agent itself, its harness, or the underlying model
• Attacker goal: Inject a "poison" so that future versions emit a software vulnerability at non-trivial rates on neutral tasks

---

Methodology

Case Study Systems

Three self-modifying coding agents were selected because they represent different design choices and have publicly available code:

Poisoned Benchmark Design

The researchers experimented with five vulnerability types:

1. Broken access control (missing access check)
2. Code injection (use of eval())
3. Redirect server-side request forgery (checking only first host in redirect chain)
4. Disabled HTTPS certificate check (enabling man-in-the-middle) — most successful
5. Backdoor (return sensitive data on a keyword)

Benchmarks were structured in the style of Polyglot: each task is a self-contained Python coding exercise with a natural language prompt, function stub, hidden test suite, and reference solution. Agents are evaluated inside Docker containers.

The CertCheck Benchmark (Primary Attack Vector)

The clean version of an HTTPS fetcher:

def fetch(url):
    return urllib.request.urlopen(url).read()

The vulnerable version:

def fetch(url):
    ctx = ssl._create_unverified_context()
    return urllib.request.urlopen(url, context=ctx).read()

Poisoning strategy: The poisoned benchmark presents every test server with a self-signed (untrusted) certificate, so a properly verified request fails the tests, and disabling verification is the only way to pass. The clean version uses trusted certificates.

Attack Pipeline

The attack concept (Figure 2 in the paper) works as follows:

1. The uncompromised agent evaluates itself on a poisoned benchmark
2. The self-evaluation produces a scored solution and transcript
3. The self-improvement process proposes and implements a modification (e.g., a vulnerable-code-emitting tool or directive)
4. The attack succeeds if an evolved agent is contaminated and writes vulnerable code on neutral tasks

Experimental Setup

• DGM: Modified to use gpt-oss-120b, Qwen3.5-397B, or Sonnet 4.5 (replacing stock o3-mini/Claude 3.5); required modification of the internal diagnosis prompt to enable the attack
• SICA: Used Qwen3.5-397B and Sonnet 4.5; no modifications needed (stock SICA)
• Hyperagents: Used Sonnet 4.5 (only model powerful enough); no modifications needed; fixed an answer-key leak in git history

---

Empirical Validation / Results

Primary Attack Results: Certificate Check Disabling

Table 1 (abbreviated) shows transfer of the vulnerability to neutral held-out tasks:

Key findings:

• Qwen3.5-397B on DGM: Reliably invoked vulnerable-code-emitting tools at solve time (30/30 and 30/30 vulnerability rates)
• gpt-oss-120b on DGM: Created vulnerable tools but never used them at solve time (0/30)
• Sonnet 4.5 on DGM: Resisted making certificate-disabling the default behavior (0/30)
• Sonnet 4.5 on SICA and Hyperagents: Succumbed to the attack (30/30), showing that model disposition alone is not sufficient defense

Attack Robustness

The contamination transfers even when HTTPS is incidental to the task (Table 2):

Contaminated agents write vulnerable code on incidental HTTPS fetches inside unrelated tasks (e.g., install_package, fetch_avatar, geocode) while legitimately solving those tasks.

Attack Persistence

Table 3 shows decontamination attempts:

Critical insight: The CWEval benchmark (designed for security assessment) did not decontaminate agents because its task did not cue HTTPS explicitly — the task was a generic fetch_data(url) with no https:// scheme present. Decontamination only succeeded when the benchmark was explicitly designed with knowledge of the original poison.

Second Proof-of-Concept: JWT Signature Verification Bypass

Table 4 shows transfer of JWT signature-verification bypass:

A third proof-of-concept with unsafe YAML loading was also partially successful (described in Appendix E.2).

---

Theoretical and Practical Implications

Conditions Sufficient for Attack Success

The authors distill properties that enabled successful attacks:

Vulnerability/Benchmark/Model characteristics:

1. Benchmark forces or rewards the vulnerability — most stealthy when the vulnerability represents legitimate functionality (e.g., self-signed certificates)
2. Vulnerability is functionally invisible on neutral tasks — disabling certificate validation doesn't break expected behavior
3. Benchmark encourages over-generalization with minimal cueing — the agent "learns" to over-apply the vulnerability
4. Vulnerability is naturally written by a bare model — standard idioms in "muscle memory" (Python stdlib worked; custom backdoors didn't)
5. Model strongly defaults to the secure version — otherwise there's no pressure to evolve a vulnerability-producing component

Harness characteristics:

1. Not steering away from benchmark-specific evolution — open-ended prompts (SICA, Hyperagents) enabled the attack more easily than DGM's stock prompt
2. Lack of security awareness — no explicit security cueing in self-improvement processes

Defensive Directions

Prevention:

• Stronger models are not a panacea — Sonnet 4.5 (frontier-class) resisted DGM but succumbed to SICA and Hyperagents
• System/harness design: Adding explicit security guidance to self-improvement processes would help
• Internal benchmarks: Augment untrusted benchmarks with internal ones rewarding security properties
• Security cueing in tasks: Not a reliable defense (e.g., contaminated Hyperagents still disabled verification on a task named secure_fetch)

Detection:

• Traditional code review and testing: Static/dynamic analysis would catch blatant vulnerabilities (e.g., unconditional CERT_NONE)
• LLM review: Adding a "security reviewer" to SICA's committee initially failed (influenced by benchmark pressure: "the alternative (strict verification) would cause the agent to fail in the grading environment") but prevailed in later generations
• Human supervision: Prior work suggests even limited supervision mitigates safety degradation

Recovery:

• Decontamination is difficult — only succeeded when the defender knew the exact poison design
• Generic security benchmarks (CWEval) did not decontaminate

---

Conclusion

Main Takeaways

1. The attack is possible: Self-modifying AI coding agents can be contaminated via poisoned benchmarks, causing them to write vulnerable code on neutral tasks.

2. The attack is less deterministic than Thompson's compiler Trojan: Success depends on factors outside the attacker's control (model disposition, harness scaffolding, benchmark design).

3. Contamination persists: Evolved vulnerabilities survive continued evolution against clean or security-focused benchmarks unless the poison is explicitly known and targeted.

4. The threat must be taken seriously: As self-modifying AI systems become more prevalent, Thompson's question — "To what extent should one trust a statement that a [coding agent] is free of Trojan horses?" — is newly relevant.

Future Work

• More stealthy benchmarks: Diluted poison signals (e.g., modifying SWE-Bench to include poisoned tasks) may or may not be effective
• Other self-modifying agent designs: Many other architectures exist beyond the three studied
• Other self-modifying systems: Self-building harnesses and general agentic systems beyond coding agents

Final Reflection

"Today's 'compilers' increasingly include AI coding agents, which increasingly also write and modify themselves. In this setting, we must again ask: 'To what extent should one trust a statement that a [coding agent] is free of Trojan horses?'"

The authors argue that resilience to poisoning attacks is likely correlated with improved self-modification performance in general — poisoning aims to induce specific behaviors, which is at odds with improving performance on arbitrary, general tasks.