Summary (Overview)

• Auto Benchmark Audit (ABA) is an agentic framework that systematically audits benchmark tasks for AI agents and LLMs, identifying issues such as hidden environment dependencies, specification gaps, and brittle grading logic.
• The framework was applied to 168 benchmarks across nine domains (34,285 tasks), finding that 25.7% of tasks carry major issues and 15.1% contain minor issues, leaving fewer than 60% of tasks clean.
• Validation through upstream pull requests, independent third-party reports, and expert review confirms high precision, with ABA recovering 66.7% (strict) / 81.0% (partial) recall against maintainer-fixed issues in Terminal-Bench 2.
• Filtering out problematic tasks shifts model rankings and increases average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively.
• The framework is benchmark-agnostic, operates in static and trajectory modes, and releases all findings, per-task evidence, and rubric annotations as a public artifact.

Introduction and Theoretical Foundation

The paper addresses a fundamental problem in AI research: the trustworthiness of measurement instruments (benchmarks). Historically, benchmark quality was maintained through manual effort—for example, Gururangan et al. [2018] exposed annotation artifacts in SNLI/MNLI, and Akrami et al. [2020] showed data redundancy in knowledge graph benchmarks. However, modern benchmarks like SWE-bench and Terminal-Bench operate behind:

• Containerized environments
• Multi-stage evaluation harnesses
• Grading logic dependent on runtime state

The key theoretical insight is that domain experts who author benchmarks are paradoxically the least likely to make their assumptions explicit—they take critical context for granted, leaving setup requirements and task constraints incomplete. Generalist annotation cannot fill this gap, as the problem is fundamentally one of expertise and coverage.

Methodology

Evidence Collector Agent

The pipeline begins with a deterministic manifest schema that handles heterogeneous benchmark configurations:

• Downloads files from GitHub repositories or HuggingFace datasets
• Maps heterogeneous artifacts into a unified representation
• Inlines lightweight data (instruction_text) and isolates heavyweight assets via filesystem references (solution_ref, environment_ref, trajectory_path)
• Onboarding a new benchmark requires only the benchmark's name and repository URL

Auditor Agent

The auditor operates in two modes:

1. Static mode: Reads only the task definition (instruction, tests, evaluation configuration) with no execution observed.
2. Trajectory mode: Additionally reads recorded agent traces and test outputs, enabling identification of runtime issues.

Audit Findings Schema

Each finding carries five fields:

1. category: Three independent axes—Instruction, Environment, and Evaluation
2. severity: 3-class score [0, 1, 2], where 2 = major issue (task unsolvable), 1 = minor issue (resolvable with domain expertise), 0 = no issue
3. evidence: Filesystem paths supporting the claim
4. claim & why it matters: Natural-language description plus downstream failure mode
5. suggested fix: Concrete change to prompt, environment, or tests

Agent Configuration

The framework runs through Claude Code CLI v2.1.96 with Opus 4.7, default tool permissions (file read/write, bash, search), and default system prompt.

Empirical Validation / Results

Overall Findings (Table 1)

Issue Patterns by Domain

• Deterministic evaluation domains are instruction-issue dominant: Math (65% of major issues from instructions, some benchmarks exceeding 90%), Science (69% instruction-based)
• Safety and retrieval domains are evaluation-issue dominant: 51% and 57% of issues respectively, due to complex grading mechanisms (LLM-as-judge, test rubrics)
• Complex runtimes introduce environment vulnerabilities: Agentic, Professional, and Multimodal domains show higher environment-based issue percentages

External Validation (Table 2)

Manual Confirmation (Table 3)

Key Validation Cases

1. Terminal-Bench 2 maintainer fix PR: ABA independently recovered 14/21 issues (strict) and 17/21 (partial) that maintainers fixed without sight of ABA findings
2. SWE-bench Verified retirement: OpenAI retired the benchmark citing two tasks (pylint-dev__pylint-4551 and sympy__sympy-18199) as examples of unreliable grading—both were independently flagged as major issues by ABA

Trajectory Audit Findings

• Trajectory mode flags 8.5% more major tasks and 2.4% less minor tasks than static mode (+6.1% more tasks flagged overall)
• Max-severity agreement between modes ranges from 29% to 63% (average 53%)
• Filtering audit-flagged issues shifts leaderboard rankings: mean lifts of 10.5 percentage points when filtering both major and minor issues, reaching ~12.8 points on Dabstep

Theoretical and Practical Implications

Theoretical Implications

• Benchmarks as living artifacts: The paper argues benchmarks should be treated as revisable artifacts requiring cyclical improvement, analogous to systematic testing in open-source software development
• Limits of human annotation: Demonstrates that manual effort alone cannot catch issues in complex, containerized benchmark environments—the problem is one of expertise and coverage, not scale
• Reproducible measurement: Turns ad hoc critique into a reproducible measurement protocol with a stable finding schema under a single rubric, applicable uniformly across heterogeneous benchmark families

Practical Implications

• Leaderboard distortion: Problematic tasks severely distort capability assessments—filtering them shifts model rankings significantly (9.9% and 9.6% performance increases on SWE-bench Verified and Terminal-Bench 2)
• Benchmark hardening tool: ABA serves as a practical tool to catch what expert eyes miss, suggesting targeted fixes alongside each finding
• Cross-domain applicability: Works across agentic, patch-based, and static-QA benchmarks within a unified protocol
• Public artifact: All findings, per-task evidence, and rubric annotations are released to support future benchmark development

Conclusion

The paper demonstrates that current frontier benchmarks contain a wide range of major-severity issues, calling for cyclically improving benchmarks as part of the standard development process. Similar to how open-source software communities developed systematic testing infrastructure because complexity makes issues inevitable regardless of contributor expertise, AI benchmarks now face the same condition.

Key contributions:

1. An agentic auditing framework combining static and trajectory modes with task-level audit scopes
2. Large-scale benchmark auditing across 168 benchmarks and 34,285 tasks with public artifact release
3. Empirical validation confirming issues through upstream PRs and expert review, including issues the original authors had not identified

Future directions implied by the work include extending ABA to specialized domains (audio, embodied AI), addressing subjective evaluation benchmarks, and integrating automated auditing into the standard benchmark development lifecycle.