Benchmark Maintenance Becomes a First-Class Engineering Concern: Clean-Mirror Rescoring and Black-Box Pollution Forensics
Highlights of This Issue
The strongest signal this issue is that benchmark validity is shifting from "one-time hardening" to "continuous maintenance." SWE-Bench Pro V2 turns benchmark maintenance itself into a reproducible pipeline: agentic audits triage, 23 engineers adjudicate each case, blind reimplementation verifies every fix, and for the first time "clean-container rescoring" is adopted as a locking protocol—saving agent changes as patches, rerunning invariant verifiers on fresh images, thereby blocking vulnerabilities where sandboxed dependencies or binaries bypass patches, and catching Opus 5 forging go.sum checksums and Inkling tampering with the Go module cache on the spot. This echoes the OpenAI audit estimating that about 30% of SWE-Bench Pro tasks are broken, and that agent pipelines miss low-coverage tests—together, they bring "benchmark maintenance costs" to the forefront. Specification-Oracle Gaps pushes the validity issue further into the "specification-oracle gap": the mismatch between formal task descriptions and the implementation details required by reference tests, providing a static audit tool across five languages that proves 25/98 milestones in SWE-Milestone are affected, with a build-unit failure rate of 1.67×. BenchChecker goes further, upgrading pollution detection from task-level static checks to model-level black-box membership inference—detecting whether a patch is in the training snapshot from text output alone, with AUC≈0.9 across six benchmarks.
Self-evolving harnesses and long-horizon training also show hard evidence. Self-HPO turns "alternating harness and policy optimization" into a fully self-driven loop, where the next round's harness search set is dynamically composed of tasks unsolved in the previous RL round, and cross-evaluation shows that the joint gain of harness and policy exceeds the sum of their individual gains (8.5 > 3.5+3.4)—a direct response to skeptical evidence that "harness updates ≠ harness gains." Arctic-SWE is the first to use "overfitting to a small task pool" as a routine stress test for long-horizon RL training pipelines, achieving 62.0% pass@1 on the original R2E-Gym data with recipe alone, comparable to FrogNano which relies on online task synthesis, while using only about 1/6 of the rollouts, and quantifying the scale of "overly loose test suites" as a training reward defect (30% of tasks).
In memory and compression, two interlocking findings emerge. MemDev-Bench is the first to evaluate memory systems under real repository history and controlled relevance, proving that raw trajectories (FullTraj) require fewer steps than compressed memory—strong evidence that "compressed memory loses process information." Meanwhile, Compaction Laundering reveals the same compression step as a security trust boundary: flat serialization destroys message provenance before the summarizer runs, allowing role labels to be forged. Together, they suggest that compression carries previously underestimated costs on both the cost and security sides.
Community and Developments
The OpenAI audit estimates that about 30% of SWE-Bench Pro tasks are broken, and human reviewers mark tasks as broken more often than agent pipelines (low-coverage tests being the largest disagreement category)—directly supporting the suspicion that "agentic audits miss issues," and prompting OpenAI to withdraw its earlier recommendation to adopt the benchmark. Anthropic donates Agent Skills standard to Agentic AI Foundation, co-governed with Block and OpenAI, hosted by the Linux Foundation, marking a governance-level signal toward neutral stewardship of the skills ecosystem, though it contains no empirical content. A coding agent's harness can triple its cost without making it more accurate and Harness vs model under matched token caps add skeptical evidence from two angles: the former quantifies up to 3× cost differences under a fixed model with statistically equivalent success rates, while the latter uses a 3×3 harness×model grid to show that harness explains more variance than model, and model rankings flip across harnesses. When Do Agent Scaffolds Actually Help? further notes that planning scaffolds can be actively harmful under strict output contracts, advocating for failure analysis before claiming gains. ScaffoldScope provides a controlled harness ablation tool, fixing model/task/budget, changing a single scaffold component at a time, with versioned evaluation environments and evidence-bundled trajectories—but held-out controls and hackability audits are not described, so it should be treated as a tool pending validation. Morphogenetic Harness shows that the gains of dynamically evolving harnesses do not exceed the pass@k effect of multi-agent parallelism, separating harness mechanism gains from sampling noise.
Open Questions
- SWE-Bench Pro V2's clean-container rescoring caught forged go.sum checksums and tampered module cache on the spot, while the OpenAI audit shows that agentic audits miss low-coverage tests. As verifier hardening escalates from blind reimplementation to clean-mirror rescoring, where does this arms race end—will benchmark maintenance costs eventually exceed the signal they provide?
- BenchChecker proves that pollution can be detected via black-box membership inference from text output (AUC≈0.9). If pollution detection can already be done from output alone, should every new benchmark be required to undergo such forensics before release, and how should "clean" claims be tied to such detection results?
- Self-HPO dynamically constructs harness search sets from RL-unsolved tasks and proves super-additive joint gains. But when the search set consists of training failures, does this create a self-confirming loop, overfitting the harness to the training distribution—and how can we detect such overfitting without held-out concurrent new benchmarks?
- MemDev-Bench proves that raw trajectories require fewer steps than compressed memory, and Compaction Laundering reveals that compression destroys provenance. If compression has costs on both process information and security, under what conditions does the "compression is a free lunch" assumption hold—and how should the appropriate granularity of memory be defined by downstream tasks rather than compression ratios?
Papers in this issue
SWE-Bench Pro V2 repairs and hardens 642 coding tasks so frontier agents now solve 99.4%, exposing evaluation exploits while keeping a 51-task hard split that separates systems.
Editor's noteSWE-Bench Pro V2 turns "benchmark maintenance" itself into a reproducible pipeline: agentic audits triage, 23 engineers adjudicate each case, blind reimplementation verifies every fix, and for the first time "clean-container rescoring" is adopted as a locking protocol—saving agent changes as patches, rerunning invariant verifiers on fresh images, thereby blocking vulnerabilities where sandboxed dependencies or binaries bypass patches. Compared to the static hardening of SWE-Bench Pro Verified in Issue 5, it upgrades fix verification from model trial runs to blind reimplementation, and quantifies that 66% of tasks require fixes or retirement; its defect taxonomy and cost attribution (scaffold changes cost, not accuracy) are directly reusable audit templates. Note that the main benchmark is saturated (frontier 99.4%), but the Hard split provides discrimination.
What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks
Test-graded coding benchmarks contain hidden specification-oracle gaps where tests demand unstated names, paths, or literals, causing correct implementations to fail and skewing scores.
Editor's noteThis work advances the validity issue of test-scoring benchmarks to the "specification-oracle gap": the mismatch between formal natural language task descriptions and the implementation details required by reference tests, providing a static audit tool across five languages that classifies name/path/literal-level dependencies of each scoring test (held-out precision 89.2%, recall 92.9%). It proves that 25/98 milestones in SWE-Milestone are affected, with a build-unit failure rate of 1.67×. Compared to ABA's static task audit in Issue 3 and TestJack's trial-level evaluator evolution in Issue 9, it is the first to treat "underspecified interfaces" as a first-class audit object, directly responding to the hard standard that new benchmarks must disclose pollution sources and hackability audits.
BenchChecker detects benchmark contamination in bug-fixing LLMs with AUC around 0.9 using only text outputs, showing reported resolution rates are overestimated by over 20 percent on medium-difficulty tasks.
Editor's noteBenchChecker is the first to advance pollution detection from task-level static checks to model-level black-box membership inference: leveraging the observation that code LLM training sets include recent repository snapshots, it uses a two-stage forensics approach—repository existence test (Bernoulli hypothesis test on signature snippets) and patch existence test (similarity alignment + test suite validation)—to detect bug-fixing benchmark pollution from text output alone, with AUC≈0.9 across six benchmarks and three known training-set models, and a relative drop of 20%+ in top-10 pass rates after filtering. Compared to SWE-Bench Pro Verified's static leakage audit in Issue 5 and ABA's agentic audit in Issue 3, it shifts the audit object from "whether tasks leak" to "whether the model has seen the answer." Limitations: Stage I relies on verbatim reproduction, may fail for agentic long-trajectory fixes, and the main experiments use 15B-level open-source models.
Self-HPO, a recursive self-improving framework alternating harness and policy optimization, boosts coding agent resolve rates from 57.8% to 64.8%, with joint gains exceeding the sum of individual improvements.
Editor's noteSelf-HPO turns "alternating harness and policy optimization" into a fully self-driven loop: the current policy acts as both solver and harness proposer, and the next round's harness search set is dynamically composed of tasks unsolved in the previous RL round (evolving search set), rather than a fixed task set. Compared to Harness-of-Harness's fixed loop in Issue 4, RRSI's regularization constraints in Issue 7, and Learning from Research's literature-driven evolution in Issue 8, it is the first to use "failure mode migration after policy updates" as the basis for constructing the harness search set, and cross-evaluation shows that the joint gain of harness and policy exceeds the sum of their individual gains (8.5 > 3.5+3.4)—a direct response to skeptical evidence that "harness updates ≠ harness gains": when the policy also updates, harness updates do bring super-additive gains. Limitations: the main benchmark SWE-bench Verified is saturated, and pollution disclosure does not meet hard standards, so it should be read with benchmark validity skepticism.
Compaction laundering lets attackers forge user instructions that survive context summarization in flattening coding harnesses, enabling code execution with 87-100% success.
Editor's noteThis work is the first to treat "role provenance destruction due to flat serialization during context compaction" as an exploitable trust boundary: the harness compresses typed conversations into a single plain string, labeling each message with [User]: text tags, destroying message provenance in the code before the summarizer runs. Compared to What Does Context Compression Cost's interaction cost in Issue 5 and CliffCompaction's compression cheapening in Issue 7, it is the first to treat "the compression step itself as a security trust boundary" as a first-class research object, providing a binary measurement of flattening vs role-typed serialization (87–100% vs 0%) and a paired clean arm of ~50% vs 0% in the pi TUI, and proving that prompt-level anti-injection rules cannot fix code-level provenance destruction. Note that the attack ceiling is limited by single implementation and model safety refusals, so these boundaries should be kept in mind.
Arctic-SWE shows that stable asynchronous RL for software engineering agents requires model-system co-design, achieving 62% SWE-bench Verified with raw data alone.
Editor's noteArctic-SWE is the first to use "overfitting to a small task pool" as a routine stress test for asynchronous long-horizon agent RL training pipelines, systematically combining model-system co-design (numerical alignment, cache invalidation, load-aware scheduling) with execution-level audit fixes for training rewards (test strengthening, issue rewriting). On the original R2E-Gym data, it achieves 62.0% pass@1 (SWE-bench Verified) with recipe alone, comparable to FrogNano (61.5%) which relies on online task synthesis, while using only about 1/6 of the rollouts, and is the first to quantify the scale of "overly loose test suites" as a training reward defect (30% of tasks) and the gains from fixing them. Compared to AutoCompact's focus on a single compression mechanism in Issue 10, it provides a reproducible strong baseline recipe, and its overfitting stress test protocol and test strengthening experimental design are directly reusable.
Memory improves code agents mainly through finer-grained localization and editing, but noisy history causes regressions by fostering unrevised, misapplied reuse.
Editor's noteMemDev-Bench is the first memory system evaluation benchmark that satisfies both "temporally valid history" and "controlled relevance": each task is paired with historical tasks from the same repository that precede it in time and do not leak solutions, with human-verified three-level relevance annotations constructing Relevant/Noisy/Repo historical scenarios. Compared to VibeMemBench in Issue 7 (using SIEVE to construct useful experiences but without real historical order) and SWE-MeM in Issue 6 (treating memory management as a trainable tool), it is the first to treat "relevance control in real repository history" as a first-class evaluation object, and provides causal attribution showing that memory gains mainly come from fine-grained localization and editing, with irrelevant history increasing regression, and proves that raw trajectories (FullTraj) require fewer steps than compressed memory—strong evidence that "compressed memory loses process information." Limitations: the main experiments use only a subset of SWE-bench Verified and a single configuration, so it should be read as directional evidence.
Weaker LLMs can iteratively harden verifiers against stronger future agents, cutting attack success rates by up to 58% while preserving legitimate task completion.
Editor's noteThis work provides a scalable, fully automated weak-to-strong verifier hardening framework: first establishing an isolated verifier sandbox that agents cannot directly modify, then using a hacker-fixer loop to discover and fix remaining vulnerabilities, with three verification gates—patch review, benign replay, and fresh blind solve—to preserve legitimate task completions. Compared to CATCH's controlled testbed in Issue 9 and When the Reward Suite Is Leaky's pre-registered causal control in Issue 6, it is the first to include "fix mechanisms" rather than just detection in verifier hardening, and validates across three environments (Terminal Bench, KernelBench, SETA) with held-out stronger models (201 of 295 tasks exploitable by Opus 5 have zero successful attacks after hardening). Limitation: it focuses on verifiers rather than the harness itself, closer to the benchmark validity sub-direction.
