Summary of "Beyond Fabrication: Misattributed Numbers in Papers Written by AI Research Agents"

Summary (Overview)

• Core finding: In the Into the Unknown benchmark, integrity failures in AI research agent papers are predominantly misattribution (real numbers attached to wrong experiments, metrics, or runs) rather than fabrication (invented values with no source). Approximately 81% of unsupported numerical claims are misattributed, while only a minority are entirely absent from agent files.
• Key methodological contribution: A deterministic claim-to-evidence checker that traces numerical claims to execution artifacts using context scoring and per-claim decoy calibration, achieving over 90% precision against blinded human annotations (78 of 85 certified claims traceable; Cohen's κ = 0.89) while certifying only ~23% of claims.
• Critical insight on verification: Simple value matching (checking whether a number appears in agent outputs) accepts 87–95% of misattributed numbers, demonstrating that presence ≠ provenance.
• Judge robustness finding: A second LLM judge from a different vendor confirms the benchmark's agent ranking but passes twice as many submissions on the integrity gate, showing absolute pass rates are judge-dependent while relative rankings are stable.
• Practical guidance: Failure to verify a claim should not be treated as evidence of falsehood—most unverified claims are still traceable, so the checker serves as a conservative certifier, not a falsifier.

---

Introduction and Theoretical Foundation

Background and Motivation

• LLM agents now perform extended analytical work: writing code, running experiments, and producing finished-looking research papers (Lu et al., 2024; Chen et al., 2025a; Starace et al., 2025).
• Organizations increasingly rely on these agent-generated reports for decisions (funding, adoption, publication), yet readers rarely inspect the underlying code, logs, or intermediate files.
• Recent benchmarks (notably Into the Unknown) show that many agent-written papers fail integrity checks. However, existing verdicts are produced by a single LLM judge, identify failure without explaining its nature, and don't distinguish between:
  • Fabrication: reported number was never produced
  • Misattribution: number exists but is attached to the wrong experiment, metric, or run

Theoretical Foundation

The paper frames this as a data provenance and data quality question:

• Data quality (Wang and Strong, 1996): traceability is a dimension of fitness-for-use, distinct from accuracy
• Data provenance (Simmhan et al., 2005; Moreau and Missier, 2013): the derivation of a data item must be recorded to be trusted
• Trust calibration (Parasuraman and Manzey, 2010; Lee and See, 2004): users over-trust automation that appears competent; Dietvorst et al. (2015) show users abandon algorithms after errors

Four Research Questions

• RQ1: What kinds of integrity failures does the official judge record—fabrication or misalignment?
• RQ2: When a number is unsupported, is it absent from files or present but from the wrong source?
• RQ3: Can numerical claims be traced automatically with useful precision without treating unverifiable claims as false?
• RQ4: How robust are the benchmark's integrity conclusions to the choice of judge?

---

Methodology

Data: Into the Unknown Benchmark

• Nine agents from three vendor groups (OpenAI, Anthropic, Open-weight) each attempted fifteen open research problems
• 135 submissions total; each graded by an LLM judge (GPT-6-Astra) against a structured rubric of binary criteria (nodes) across three gates (G1: integrity, G2: completeness, G3: scientific merit)
• Only 21 of 135 submissions passed the integrity gate (G1)
• The release includes full working directories, judge outputs, and node-level justifications

Component 1: Re-Analysis of Official Judgments

• Extracted 2,376 counted G1 node verdicts at consensus level from 540 judge outputs
• Computed failure rates by rubric family and judge self-consistency across 4 repeats
• Tested run-length association using fixed-effects regression:

$$R_{int,i} = \beta_0 + \beta_1 \log h_i + \alpha_{a(i)} + \gamma_{t(i)} + \varepsilon_i \quad (1)$$

Component 2: Claim-to-Evidence Checker (Deterministic, Model-Free)

Claims extraction: Numeric values with decimal points, percentages, or scientific exponents from LaTeX source (expanding macros), with context (160 characters of prose, or row/column/caption for tables).

Evidence indexing: Machine-written result files (JSON, CSV, TSV, logs) with context strings (key paths, column headers, line content). Bulk entries flagged.

Three levels of verification:

Verdict function:

$$V(c) = \begin{cases} \text{certified} & \text{if } S(c) \geq \tau \text{ and } p(c) \leq \rho \\ \text{untraced} & \text{if } M^*(c) = \emptyset \\ \text{unattributed} & \text{otherwise} \end{cases} \quad (6)$$

Operating points (frozen before human labels): strict (τ = 0.12, ρ = 0.1), balanced (τ = 0, ρ = 0.1), broad (τ = 0, ρ = 0.2)

Component 3: Human Labeling Study

Four labels: Traceable, Wrong source, Not found, Not a claim

• Development sample: 10 submissions, 184 claims after practice exclusions; split into tuning and test halves
• Confirmation sample: 8 held-out submissions, 72 claims (one per agent, except one dropped); second annotator labeled 48 claims independently (κ = 0.89)
• Annotators were blind to the checker's verdicts and to each other's labels
• Population-weighted metrics with inverse sampling probabilities:

$$w_j = \frac{N_{mg}}{n_{mg}} \quad (7)$$

Component 4: Second Judge

• Re-graded a stratified sample of 30 submissions with Claude Opus 5.5 using the benchmark's own grading pipeline (identical prompt, rubric, evidence protocol)
• Tested judge leniency differences using a GEE logistic model with vendor interactions

---

Empirical Validation / Results

RQ1: What the Official Judge Fails

• Failures concentrate in write-up vs. work criteria:
  • Code–paper alignment: 52% failure rate
  • Number traceability: 38% failure rate
  • Held-fixed disclosure: 32%; Pre-stated method: 27%
• Criteria targeting overt misconduct almost never fire:
  • Hard-coded values: 1%; External code: 1%; Replication of published method: 0%
• Judge is self-consistent (90% unanimous agreement across repeats)

RQ2: Misattribution vs. Absence

• 89% of unsupported claims are wrong source (81% population-weighted)
• Recurring patterns: wrong statistic (std dev as mean), wrong model/arm, wrong aggregation (single seed as average), wrong condition, unrelated lines (timers), settings reported as results

RQ3: Can Provenance Be Checked Automatically?

Key performance metrics of the strict checker (against first annotator's labels):

Comparison with baselines (129 labeled claims not used for tuning):

Key findings:

• Level 1 value matching cannot discriminate: accepts 87–95% of unsupported claims
• The context score separates traceable from misattributed claims (raises precision from 0.77 to 0.92)
• The decoy test guards against coincidence (reduces false certification from 18.9% to 1.3%) but costs recall
• LLM verifier has high recall (0.97) but accepts 36% of misattributed claims (vs. 13% for strict checker)
• Non-verification ≠ falsehood: 57% (held-out) to 72% (development) of uncertified claims were traceable

RQ4: Judge Robustness

• Rankings are stable across judges (both rank OpenAI agents cleanest)
• Absolute pass rates are judge-dependent: second judge passed 12 submissions on G1 vs. official judge's 6
• No statistically significant vendor-bias effect (p = 0.26 Kruskal–Wallis; p = 0.075 GEE interaction), though the largest leniency is for Anthropic agents under an Anthropic judge

Benchmark-Wide Traceability

Notable anomaly: opus-5.5 scores highly on the official judge (R_int = 92.9) but only 13.8% of its claims are certified—a judge may verify headline numbers while missing untraceable content elsewhere.

---

Theoretical and Practical Implications

For Organizations Using AI-Produced Analyses

1. Ask "which run produced it?" rather than "does the number appear in outputs?"
2. Use verification asymmetrically: certified claims (~23% of claims at >90% precision) can be trusted; uncertified claims should be routed to human review, not treated as failures
3. This guards against both failure modes of trust:
   • Over-reliance on outputs that look verified
   • Wholesale rejection of useful work after visible errors

For Benchmark Designers

• Report which judge produced pass rates; absolute pass rates are judge-dependent
• Add a second judge from a different vendor or a deterministic check where feasible
• Claim-level checking complements node-level judging: judges may miss untraceable content beyond headline numbers

For Agent Developers

• Traceability depends on how results are recorded: descriptive keys are easy to tie to sentences; values computed only in prose cannot be traced
• Generate reports from recorded results (inserting values from result files rather than retyping) would eliminate most misattribution

For Research Integrity

• Misattribution is a record-keeping problem, not an intent problem: it calls for tooling (structured result files, report generation from files, provenance checks) rather than sanctions
• Findings echo human research reporting: statistical errors are common and mostly inadvertent (Bakker and Wicherts, 2011; Nuijten et al., 2016)

---

Conclusion

Main Takeaways

1. Misattribution, not fabrication, dominates integrity failures in this benchmark: ~80% of unsupported claims are real values attached to the wrong experiment, metric, or run
2. Presence ≠ provenance: value matching accepts 87–95% of misattributed claims; verification must establish which run produced the number
3. The deterministic checker provides high-precision certification (92% pooled precision) but with limited recall (~23% coverage), serving as a conservative certifier
4. Non-verification is not falsehood: most uncertified claims are traceable, so "not verified" should mean "needs human review"
5. Judge robustness: rankings are stable across judges, but absolute pass rates depend on judge choice; integrity verdicts should be reported with judge identity

Future Directions

• Extend to other benchmarks, domains, and agent types (the 81% misattribution share may vary)
• Adapt the checker for non-LaTeX papers and binary/database result storage
• Larger labeled samples and external annotator panels for stronger validation
• Adjudication of judge disagreements to determine which judge is closer to human experts
• Investigating the opus-5.5 phenomenon: an agent whose headline numbers pass the judge while most content is untraceable

Limitations

• Single benchmark; nine agents; fifteen computational/physical science tasks
• Two annotators, both research team members (though blinded and highly consistent, κ = 0.89)
• Checker certifies provenance, not scientific correctness
• 99 of 135 submissions processed (largest excluded due to resource limits)
• Second-judge sample limited to 30 submissions, mostly graded once