# Beyond fabrication: Misattributed numbers in papers written by AI research agents

> AI research agents mostly misattribute real numbers to wrong experiments rather than fabricating them, so verifying provenance requires checking which run produced each claim.

- **Source:** [arXiv](https://arxiv.org/abs/2610.38691)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/JrxI9I
- **Whiteboard:** https://picx.dev/p/JrxI9I/image

## Summary

# Summary of "Beyond Fabrication: Misattributed Numbers in Papers Written by AI Research Agents"

## Summary (Overview)

- **Core finding**: In the Into the Unknown benchmark, integrity failures in AI research agent papers are predominantly **misattribution** (real numbers attached to wrong experiments, metrics, or runs) rather than **fabrication** (invented values with no source). Approximately **81%** of unsupported numerical claims are misattributed, while only a minority are entirely absent from agent files.
- **Key methodological contribution**: A **deterministic claim-to-evidence checker** that traces numerical claims to execution artifacts using context scoring and per-claim decoy calibration, achieving **over 90% precision** against blinded human annotations (78 of 85 certified claims traceable; Cohen's κ = 0.89) while certifying only ~23% of claims.
- **Critical insight on verification**: Simple value matching (checking whether a number appears in agent outputs) accepts **87–95% of misattributed numbers**, demonstrating that presence ≠ provenance.
- **Judge robustness finding**: A second LLM judge from a different vendor confirms the benchmark's agent ranking but passes **twice as many submissions** on the integrity gate, showing absolute pass rates are judge-dependent while relative rankings are stable.
- **Practical guidance**: Failure to verify a claim should **not** be treated as evidence of falsehood—most unverified claims are still traceable, so the checker serves as a conservative certifier, not a falsifier.

---

## Introduction and Theoretical Foundation

### Background and Motivation

- LLM agents now perform extended analytical work: writing code, running experiments, and producing finished-looking research papers (Lu et al., 2024; Chen et al., 2025a; Starace et al., 2025).
- Organizations increasingly rely on these agent-generated reports for decisions (funding, adoption, publication), yet readers rarely inspect the underlying code, logs, or intermediate files.
- Recent benchmarks (notably *Into the Unknown*) show that many agent-written papers fail integrity checks. However, existing verdicts are produced by a **single LLM judge**, identify failure without explaining its nature, and don't distinguish between:
  - **Fabrication**: reported number was never produced
  - **Misattribution**: number exists but is attached to the wrong experiment, metric, or run

### Theoretical Foundation

The paper frames this as a **data provenance** and **data quality** question:

- **Data quality** (Wang and Strong, 1996): traceability is a dimension of fitness-for-use, distinct from accuracy
- **Data provenance** (Simmhan et al., 2005; Moreau and Missier, 2013): the derivation of a data item must be recorded to be trusted
- **Trust calibration** (Parasuraman and Manzey, 2010; Lee and See, 2004): users over-trust automation that appears competent; Dietvorst et al. (2015) show users abandon algorithms after errors

### Four Research Questions

- **RQ1**: What kinds of integrity failures does the official judge record—fabrication or misalignment?
- **RQ2**: When a number is unsupported, is it absent from files or present but from the wrong source?
- **RQ3**: Can numerical claims be traced automatically with useful precision without treating unverifiable claims as false?
- **RQ4**: How robust are the benchmark's integrity conclusions to the choice of judge?

---

## Methodology

### Data: Into the Unknown Benchmark

- **Nine agents** from three vendor groups (OpenAI, Anthropic, Open-weight) each attempted **fifteen open research problems**
- **135 submissions** total; each graded by an LLM judge (GPT-6-Astra) against a structured rubric of binary criteria (nodes) across three gates (G1: integrity, G2: completeness, G3: scientific merit)
- Only **21 of 135 submissions** passed the integrity gate (G1)
- The release includes full working directories, judge outputs, and node-level justifications

### Component 1: Re-Analysis of Official Judgments

- Extracted **2,376 counted G1 node verdicts** at consensus level from 540 judge outputs
- Computed failure rates by rubric family and judge self-consistency across 4 repeats
- Tested run-length association using fixed-effects regression:

$$R_{int,i} = \beta_0 + \beta_1 \log h_i + \alpha_{a(i)} + \gamma_{t(i)} + \varepsilon_i \quad (1)$$

### Component 2: Claim-to-Evidence Checker (Deterministic, Model-Free)

**Claims extraction**: Numeric values with decimal points, percentages, or scientific exponents from LaTeX source (expanding macros), with context (160 characters of prose, or row/column/caption for tables).

**Evidence indexing**: Machine-written result files (JSON, CSV, TSV, logs) with context strings (key paths, column headers, line content). Bulk entries flagged.

**Three levels of verification**:

| Level | Description | Formula |
|-------|-------------|---------|
| **Level 1 (Value match)** | Does the value appear anywhere? | $M(c,e) = \mathbb{1}\left[\min_{\lambda \in \Lambda_c} \|\lambda u_e - v_c\| \leq \delta_c\right]$ where $\delta_c = \frac{1}{2}10^{-d_c}(1+10^{-9}) \quad (2)$ |
| **Level 2 (Provenance score)** | Does the evidence context identify the claim? | $s(c,e) = \frac{\|C_c \cap E_e\|}{\sqrt{\|C_c \cup E_e\| \cdot \|C_c\|}}, \quad S(c) = \max_{e \in M^*(c)} s(c,e) \quad (3)$ |
| **Decoy calibration** | Does identification beat chance? | $v_{c}^{(k)} = v_c + k \cdot 10^{-d_c}, \; k \in \{\pm 2, \pm 4, \pm 6, \pm 8, \pm 10\} \quad (4)$; $p(c) = \frac{1 + \sum_k \mathbb{1}[S(c^{(k)}) \geq S(c)]}{1 + |K|} \quad (5)$ |

**Verdict function**:

$$V(c) = \begin{cases} \text{certified} & \text{if } S(c) \geq \tau \text{ and } p(c) \leq \rho \\ \text{untraced} & \text{if } M^*(c) = \emptyset \\ \text{unattributed} & \text{otherwise} \end{cases} \quad (6)$$

**Operating points** (frozen before human labels): strict (τ = 0.12, ρ = 0.1), balanced (τ = 0, ρ = 0.1), broad (τ = 0, ρ = 0.2)

### Component 3: Human Labeling Study

**Four labels**: Traceable, Wrong source, Not found, Not a claim

- **Development sample**: 10 submissions, 184 claims after practice exclusions; split into tuning and test halves
- **Confirmation sample**: 8 held-out submissions, 72 claims (one per agent, except one dropped); second annotator labeled 48 claims independently (κ = 0.89)
- Annotators were **blind** to the checker's verdicts and to each other's labels
- Population-weighted metrics with inverse sampling probabilities:

$$w_j = \frac{N_{mg}}{n_{mg}} \quad (7)$$

### Component 4: Second Judge

- Re-graded a stratified sample of **30 submissions** with Claude Opus 5.5 using the benchmark's own grading pipeline (identical prompt, rubric, evidence protocol)
- Tested judge leniency differences using a GEE logistic model with vendor interactions

---

## Empirical Validation / Results

### RQ1: What the Official Judge Fails

- Failures concentrate in **write-up vs. work** criteria:
  - Code–paper alignment: **52%** failure rate
  - Number traceability: **38%** failure rate
  - Held-fixed disclosure: **32%**; Pre-stated method: **27%**
- Criteria targeting overt misconduct **almost never fire**:
  - Hard-coded values: 1%; External code: 1%; Replication of published method: 0%
- Judge is self-consistent (90% unanimous agreement across repeats)

### RQ2: Misattribution vs. Absence

| Label | Development | Confirmation | Total |
|-------|-------------|--------------|-------|
| Traceable | 112 | 46 | **158** |
| Wrong source | 35 | 12 | **47** |
| Not found | 4 | 2 | **6** |
| Not a claim | 33 | 12 | **45** |
| **Total labeled** | **184** | **72** | **256** |

- **89% of unsupported claims are wrong source** (81% population-weighted)
- Recurring patterns: wrong statistic (std dev as mean), wrong model/arm, wrong aggregation (single seed as average), wrong condition, unrelated lines (timers), settings reported as results

### RQ3: Can Provenance Be Checked Automatically?

**Key performance metrics of the strict checker** (against first annotator's labels):

| Sample | Precision (weighted) | Recall | Coverage | Exact CI |
|--------|---------------------|--------|----------|----------|
| Development (n=151) | 0.91 | 0.39 | ~25% | [0.80, 0.97] |
| Confirmation (held-out, n=60) | 0.93–0.95 | 0.34 | ~23% | [0.77, 0.99] |
| **Pooled (n=211)** | **0.92** | 0.49 | — | **[0.84, 0.97]** |

**Comparison with baselines** (129 labeled claims not used for tuning):

| Method | Precision | Recall | Decoy acceptance |
|--------|-----------|--------|------------------|
| Value appears anywhere (Level 1) | 0.74 | 1.00 | 87–88% |
| Embedding score + decoy test | 0.74 | 0.43 | 3.3% |
| LLM verifier (Claude Opus 5.5) | 0.89 | 0.97 | — |
| **Checker (strict)** | **0.89** | **0.54** | **1.4%** |
| Checker and LLM agree | **0.98** | 0.54 | — |

**Key findings**:
- Level 1 value matching **cannot discriminate**: accepts 87–95% of unsupported claims
- The **context score** separates traceable from misattributed claims (raises precision from 0.77 to 0.92)
- The **decoy test** guards against coincidence (reduces false certification from 18.9% to 1.3%) but costs recall
- LLM verifier has high recall (0.97) but accepts **36% of misattributed claims** (vs. 13% for strict checker)
- **Non-verification ≠ falsehood**: 57% (held-out) to 72% (development) of uncertified claims were traceable

### RQ4: Judge Robustness

| Agent vendor group | Submissions | Official judge | Second judge | Difference |
|-------------------|-------------|----------------|--------------|------------|
| OpenAI | 8 | 3.5% | 1.3% | −2.2 pts |
| Anthropic | 13 | 16.7% | 8.1% | −8.6 pts |
| Open-weight | 9 | 27.6% | 23.9% | −3.8 pts |
| **All** | **30** | — | — | **−5.4 pts** (p = 0.014) |

- **Rankings are stable** across judges (both rank OpenAI agents cleanest)
- **Absolute pass rates are judge-dependent**: second judge passed 12 submissions on G1 vs. official judge's 6
- No statistically significant vendor-bias effect (p = 0.26 Kruskal–Wallis; p = 0.075 GEE interaction), though the largest leniency is for Anthropic agents under an Anthropic judge

### Benchmark-Wide Traceability

| Agent | Certified share | Untraced share | R_int | Judge NT fail |
|-------|----------------|----------------|-------|---------------|
| astra-6 | **75.6%** | 0.8% | 98.5 | 0.0% |
| sol-5.6 | **62.8%** | 0.8% | 92.4 | 8.3% |
| kimi-k3 | 17.3% | 6.1% | 57.4 | 58.3% |
| glm-5.3 | 16.7% | 5.1% | 61.0 | 38.9% |
| opus-5.5 | 13.8% | 1.0% | **92.9** | 13.9% |
| opus-5 | 11.5% | 4.5% | 59.9 | 58.3% |
| deepseek-v4-pro | 8.6% | 6.6% | 64.0 | 52.8% |
| opus-4.8 | 8.2% | 4.3% | 65.3 | 52.8% |
| fable-5 | 7.8% | 3.2% | 67.3 | 55.6% |

**Notable anomaly**: opus-5.5 scores highly on the official judge (R_int = 92.9) but only 13.8% of its claims are certified—a judge may verify headline numbers while missing untraceable content elsewhere.

---

## Theoretical and Practical Implications

### For Organizations Using AI-Produced Analyses

1. **Ask "which run produced it?"** rather than "does the number appear in outputs?"
2. **Use verification asymmetrically**: certified claims (~23% of claims at >90% precision) can be trusted; uncertified claims should be routed to human review, not treated as failures
3. This guards against both failure modes of trust:
   - Over-reliance on outputs that look verified
   - Wholesale rejection of useful work after visible errors

### For Benchmark Designers

- **Report which judge produced pass rates**; absolute pass rates are judge-dependent
- **Add a second judge from a different vendor** or a deterministic check where feasible
- Claim-level checking **complements** node-level judging: judges may miss untraceable content beyond headline numbers

### For Agent Developers

- Traceability depends on **how results are recorded**: descriptive keys are easy to tie to sentences; values computed only in prose cannot be traced
- **Generate reports from recorded results** (inserting values from result files rather than retyping) would eliminate most misattribution

### For Research Integrity

- Misattribution is a **record-keeping problem**, not an intent problem: it calls for tooling (structured result files, report generation from files, provenance checks) rather than sanctions
- Findings echo human research reporting: statistical errors are common and mostly inadvertent (Bakker and Wicherts, 2011; Nuijten et al., 2016)

---

## Conclusion

### Main Takeaways

1. **Misattribution, not fabrication, dominates** integrity failures in this benchmark: ~80% of unsupported claims are real values attached to the wrong experiment, metric, or run
2. **Presence ≠ provenance**: value matching accepts 87–95% of misattributed claims; verification must establish which run produced the number
3. **The deterministic checker provides high-precision certification** (92% pooled precision) but with limited recall (~23% coverage), serving as a conservative certifier
4. **Non-verification is not falsehood**: most uncertified claims are traceable, so "not verified" should mean "needs human review"
5. **Judge robustness**: rankings are stable across judges, but absolute pass rates depend on judge choice; integrity verdicts should be reported with judge identity

### Future Directions

- Extend to other benchmarks, domains, and agent types (the 81% misattribution share may vary)
- Adapt the checker for non-LaTeX papers and binary/database result storage
- Larger labeled samples and external annotator panels for stronger validation
- Adjudication of judge disagreements to determine which judge is closer to human experts
- Investigating the opus-5.5 phenomenon: an agent whose headline numbers pass the judge while most content is untraceable

### Limitations

- Single benchmark; nine agents; fifteen computational/physical science tasks
- Two annotators, both research team members (though blinded and highly consistent, κ = 0.89)
- Checker certifies provenance, **not scientific correctness**
- 99 of 135 submissions processed (largest excluded due to resource limits)
- Second-judge sample limited to 30 submissions, mostly graded once

---

_Markdown view of https://picx.dev/p/JrxI9I, served by PicX — AI-generated visual whiteboard summaries of research papers._