Issue 3: Harness Security Becomes a First-Class Research Subject, Skeptic Evidence and Long-Horizon Transfer Benchmarks Tighten in Tandem

Papers in this issue

1. 

   1

   EVOMAL: Self-Poisoning in Self-Evolving Coding Agents

   Self-evolving LLM coding agents are vulnerable to self-poisoning, where agents unknowingly author and propagate malicious skills, achieving attack success rates up to 86.7% and forming self-sustaining worms.

   Editor's note

   The first work to characterize 'create-path self-poisoning' of self-evolving coding agents as an independent vulnerability class: after retrieving an implanted skill, the agent writes and stores a malicious skill itself, turning a single retrieval into a self-propagating worm that persists even after the seed is removed. It adds a 'security' dimension to the 'capability' discussion of self-evolving harnesses since Issue 1, and provides counter-prompt defenses and clear metrics. Essential reading for anyone engineering or auditing self-evolving/self-improving harnesses.
2. 

   2

   When Context Gets Root: Privilege Escalation in LLM Harnesses

   Instruction privilege escalation, a novel attack exploiting agent-side context reconstruction, achieves all 13 attack objectives across six coding-agent harnesses, bypassing both instruction hierarchy and automatic permission review defenses.

   Editor's note

   Proposes 'instruction privilege escalation' as an independent attack paradigm: unlike prompt injection (content stays at original level) or role confusion (model misinterprets role), the harness itself elevates tool-level content to user/system-level messages (sub-agent delegation, persistent goals, scheduled tasks, custom subagent installation) during context reconstruction. Systematically evaluated across six coding-agent harnesses and 13 attack targets, proving it bypasses the working agent's instruction hierarchy and automatic permission review. This is a security-critical complement to Issue 2's harness effects and reward hacking themes, providing actionable context reconstruction design constraints for harness builders.
3. 

   3

   Rethinking the Evaluation of Harness Evolution for Agents

   Rethinking harness evolution evaluations shows reported gains conflate search effects, as simple test-time scaling (parallel sampling, sequential refinement) consistently outperforms automatic harness evolution.

   Editor's note

   Provides the cleanest budget-matched comparison to date: under matched feedback and inference budgets, compares automatic harness evolution against test-time scaling baselines like parallel sampling and sequential refinement, and evaluates on held-out tasks. Conclusion is that evolution does not consistently outperform simple test-time scaling, and evolved harnesses overfit the search set (held-out average only +0.6pp). Compared to Issue 1's 'Don't Blame the LLM' and Issue 2's 'Scaffold Effect,' it advances gain attribution to the level of 'is it harness design or just search budget.' Limitation is single benchmark (Terminal-Bench 2.1) and lack of code-level failure analysis, but the protocol itself is a template for fair evaluation.
4. 

   4

   SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

   SWE Refactor Bench shows frontier agents fail 95% of whole-repository migrations, with even perfect test passes undermined by hidden behavioral differences.

   Editor's note

   The first long-horizon benchmark for 'whole-repository stack migration,' using a three-phase protocol (Migration Audit / Behavioural Tests / Agentic Verification) to directly address 'Blindness'—behavioral tests cannot determine whether transfer actually happened, and agents can copy the original implementation to pass tests. Only 28 of 520 runs (5.4%) pass all three phases, and migration completeness and behavioral correctness are capabilities that fail in opposite directions. It embeds the 'weak test passing' problem of reward hacking into long-horizon repository-level task design itself, with an independent leaderboard and graded scoring (0/40/…/100).
5. 

   5

   CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

   CompactionRL trains long-horizon agentic LLMs by jointly optimizing context summarization and task execution under a shared reward, achieving 66.8% Pass@1 on SWE-bench Verified.

   Editor's note

   Makes context compression a first-class object in RL training: a PPO framework jointly optimizes task execution and summary generation under a shared task reward, using token-level loss normalization and cross-trajectory GAE to handle variable-length compacted rollouts, and is already deployed in the GLM-5.2 training pipeline. For the 'learned context compression' sub-direction, it provides a concrete method for treating compression decisions as learnable actions rather than inference-time heuristics. Note that gains are only reported on a single saturated benchmark without held-out comparisons, so read with benchmark validity skepticism.
6. 

   6

   The Empire, Long Divided, Must Unite: Architectural Convergence in Three LLM Agent Harnesses

   Three agent harnesses with opposing philosophies converged on five shared architectural elements, leaving external verifiability as the sole unconverged dimension and next competitive frontier.

   Editor's note

   The first source-level, multi-case architectural study of harness layers: traces the commit trajectories of three ideologically opposed open-source harnesses (deepagents, pi, dsh), identifying a convergent intermediate form with five elements, and one non-converged load-bearing dimension—external verifiability. It adds mechanistic qualitative explanations to the empirical harness research of Issues 1/2, and positions 'lack of external verifiability' as an open dimension for future harness design and benchmarking.
7. 

   7

   Automated Benchmark Auditing for AI Agents and Large Language Models

   Auto Benchmark Audit finds major issues in 25.7% of 34,285 AI benchmark tasks across 168 benchmarks, and filtering these problematic tasks shifts model rankings by up to 10 percentage points.

   Editor's note

   Turns benchmark auditing itself into a reproducible agentic pipeline (ABA), producing structured audit records on 168 benchmarks/34K tasks, and uses upstream maintainer PRs and independent blogs as external validation anchors, quantifying the actual impact of audit findings on leaderboard rankings (after filtering problematic tasks, SWE-bench Verified and Terminal-Bench 2 improve by an average of 9.9%/9.6%). Compared to Issue 1's BenchJack (pre-execution scanning) and HackDetect (post-hoc attribution), it upgrades auditing from one-off detection to a scalable continuous hardening tool, and provides the first cross-domain problem pattern statistics and cost data of about $0.60/task. Limitation is that auditing relies on a single base model and lacks held-out comparisons.
8. 

   8

   A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

   Repository-level code agents lose up to 6.7 points in resolution rate under semantics-preserving code perturbations, and robustness is a jagged frontier—jointly determined by model, scaffold, and workload, not the model alone.

   Editor's note

   The first systematic robustness evaluation of repository-level coding agents under semantic-preserving transformations (SPT), extending single-turn code model robustness research to multi-turn agentic settings, using a randomized non-adversarial variant sampler and paired seed-variant design to isolate perturbation effects from agent stochasticity. Core finding is the 'jagged robustness frontier': model robustness rankings do not transfer across scaffolds/benchmarks, and robustness is a joint property of model, scaffold, and workload. Compared to Issue 2's Scaffold Effect, it adds a perturbation dimension to harness evaluation.