# Autonomous AI Agents Systematically Diverge From Human Preferences in Proof Formalization

> AI agents in proof formalization act as "sorry autofillers," targeting explicit gaps and peripheral regions while humans work the dense, surprising core.

- **Source:** [arXiv](https://arxiv.org/abs/2611.06995)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/JBh7bq
- **Whiteboard:** https://picx.dev/p/JBh7bq/image

## Summary

# Autonomous AI Agents Systematically Diverge From Human Preferences in Proof Formalization

**Authors:** Bálint Gyevnár (Carnegie Mellon University) & Simon DeDeo (Carnegie Mellon University, Santa Fe Institute)

---

## Summary (Overview)

- **Methodological innovation:** The paper introduces **"simulated collaboration"**, a methodology that replays real human proof-formalization histories (Git commits from Lean repositories) but delegates the "next step" to an AI agent, allowing researchers to compare human vs. AI preferences in *what to work on next* — rather than testing performance on pre-selected problems.
- **Key finding 1 (Syntactic):** AI agents are overwhelmingly drawn to **explicit `sorry` markers** (proof gaps). Definitions containing a `sorry` account for **59–92% of AI agent modifications** vs. only **19% of human modifications**. Agents act as "proof autofillers," treating sorries as implicit requests for assistance.
- **Key finding 2 (Semantic/Network):** AI agents preferentially modify definitions near the **periphery** of the dependency graph (near axioms or final theorems) and avoid the **densely connected intermediate regions** where humans concentrate their work.
- **Key finding 3 (Surprise):** AI agents are **attracted to high-surprise proof bodies** and act to **regularize them toward baseline** (LLM-standard) surprise levels, while humans leave aggregate surprise nearly unchanged. Conversely, agents avoid high-surprise type signatures.
- **Implications:** The results highlight potential **blind spots** in human–AI mathematical collaboration, including risks of "LLM homogenization," premature gap-filling, and the incentivization of "AI-friendly" proof styles that may frustrate human goals of synthesis, diversity, and innovation.

---

## Introduction and Theoretical Foundation

### Background

Proof assistants (e.g., Lean) have become central to AI-enabled mathematics. However, the fact that an AI system finds a proof reveals little about *how* it resolved the problem or *what subproblems it chose to visit*. Prior work shows AI-discovered proofs can be:

- Verbose, circuitous, and redundant [ABB26; Lu+26]
- Misformalized [BCH26]
- Exploitative of kernel holes [dMou26]
- Misaligned with human conceptual structures [DD26]

### The Core Problem

Existing evaluations of AI agents in formal mathematics test performance on **pre-selected problems** with human-chosen prescriptions for what to work on [Xin+26; Let+26; Zha+26; Zhu+26; XLZ26]. This leaves a critical gap: **we know little about what AI agents attend to when the choice of what to work on is left to them**.

### Research Questions

> **RQ1.** To what extent in mathematical collaboration do AI agents respond to and work with the surface syntax of a proof, or, conversely, attend to some underlying semantics?
>
> **RQ2.** To what extent do these preferences of AI agents either match or differ from those of human experts?

### Theoretical Foundations

The paper builds on two key theoretical tensions:

1. **Syntax vs. semantics:** Neural models exploit surface regularities (source code is repetitive and predictable [Hin+12]) as shortcuts [Gei+20; MPL19]. LLM-based provers for Lean succeed on one formulation of a theorem but fail on semantically equivalent ones, with statement surprise predicting solvability [Ole+26]. However, language models can also learn internal representations of program semantics beyond surface form [JR24; NLW23].

2. **Alignment in collaboration:** Effective human–AI collaboration requires understanding where agent preferences diverge from human ones. Agents that are misaligned can produce "costly mismatches between intuition and formal code," redirect human effort into "AI-friendly" styles, and frustrate the goal of establishing correspondences between formal reasoning and conceptual structure [DD26].

---

## Methodology

### Dataset

- **Nine major human collaborations in Lean** containing **4.7M definitions** across **more than 6K commits**
- Repositories include human formalization projects tracked via Git

### Simulated Collaboration Procedure

The experimental setup (Figure 1) works as follows:

1. Download a Lean formalization repository
2. Remove all merge commits and commits with no Lean file modifications
3. For each remaining human parent commit $H_{t-1}$, an agent is called to "take the next step," producing an artificial child state $A_t$
4. Compare the human-authored child state $H_t$ to $A_t$ using extracted features

### Models Compared

- **Claude Opus 4.6** (Anthropic)
- **Google Gemini 3.1 Pro** (Google DeepMind)
- **GPT-5.2** (OpenAI) — referred to as "Codex" in the paper

### Feature Extraction

**Syntactic predictors:**
- **Definition length** (number of characters)
- **Presence of `sorry` markers** (Lean keyword allowing compilation without a complete proof)
- **Distance to nearest sorry** in the dependency graph

**Semantic predictors:**

1. **Network position:**
   - **Max leaf distance:** maximum shortest-path distance from a definition to all accessible leaves (axioms) in the dependency graph
   - **Root distance:** shortest distance to the nearest significant root (final theorem)

2. **Surprise:** Per-token surprise of the definition's **type signature** (problem statement) and **body** (current proof attempt), computed using DeepSeek-Prover-V2-7B (with robustness checks using Goedel-Prover-V2-32B and Kimina-Prover-72B)

### Statistical Analysis

- **Logistic regression model** predicting whether an actor (human or agent) chooses a given definition to edit
- Predictors include syntactic and semantic features of definitions in the parent commit
- Fixed-effect coefficients with 95% Wald confidence intervals
- Gaussian mixed models for edit-size comparisons
- Holm-corrected p-values for multiple comparisons

---

## Empirical Validation / Results

### Summary Statistics: How Much Do Actors Edit?

**Table 1:** Number and per-actor percentage of definitions Added, Modified, or Removed, aggregated across all commits. Median chars. gives the median number of characters inserted plus deleted across the signature and body.

| Actor | Added Count | Added % | Median Chars. | Modified Count | Modified % | Median Chars. | Removed Count | Removed % | Median Chars. | Total Count |
|-------|-------------|---------|---------------|----------------|------------|---------------|---------------|-----------|---------------|-------------|
| Human | 10,618 | 0.90 | 133 | 6,975 | 0.59 | 20 | 4,913 | 0.41 | 129 | 1,185,143 |
| Claude | 4,640 | 0.39 | 171 | 6,178 | 0.52 | 149.5 | 408 | 0.03 | 81 | 1,176,797 |
| Gemini | 1,974 | 0.17 | 193 | 4,876 | 0.42 | 84 | 618 | 0.05 | 104 | 1,174,808 |
| Codex | 2,056 | 0.17 | 135 | 1,394 | 0.12 | 103 | 57 | 0.00 | 110 | 1,175,224 |
| **Overall** | **19,288** | **0.41** | **145** | **19,423** | **0.41** | **72** | **5,996** | **0.13** | **120** | **4,711,972** |

**Key observations:**
- Humans have the highest combined edit share: **1.90%** vs. 0.95% (Claude), 0.64% (Gemini), 0.30% (Codex) — significant overall $(\chi^2(3) = 31.7, p < 0.001)$
- **Agents almost never delete code:** humans are 8–86 times more likely to remove definitions (all $p < 0.001$)
- **Agent edits are larger:** median characters modified is 20 for humans vs. 84–150 for agents (Holm-corrected $p = 0.002$ for Gemini, $p < 0.001$ for Claude and Codex)

### Modification Preferences: Where Do Actors Change the Existing Proof?

#### Syntactic Preferences

**1. Agents focus on filling local gaps (`sorry` markers):**
- All three agents show stronger-than-human preferences for editing definitions with a `sorry`
- **Outcome statistics:** At least 59% of agent edits involved definitions with a `sorry` (Claude: 92%; Gemini: 80%; Codex: 59%), compared to just **19%** for humans
- The preference is for the *syntactic marker itself*, not the problem area: the decline in preference with distance from a `sorry` in the dependency graph is much steeper for agents than humans (Figure 3)

**2. Agents avoid long definitions:**
- Humans are strongly attracted to longer definitions $(\beta = +2.207 \pm 0.055)$
- Agent deviations are large and negative: −1.58 (Claude), −1.77 (Gemini), −2.56 (Codex)

**3. Agents make isolated edits:**
- 62% of human edits connect to another edited definition in the same commit
- Agents make isolated edits 77% (Claude), 69% (Gemini), and 82% (Codex) of the time

> "In short, and perhaps uncharitably, these AI agents, despite their sophistication, retain much of the flair of autocomplete systems of earlier eras."

#### Semantic Preferences

**1. Peripheral engagement (U-shaped preferences):**

The quadratic fit on **Max leaf distance** and **Root distance** reveals:

- **Humans:** Minimum edit probability at the leaves themselves (10th percentile), with probability rising steadily into the graph. Weak preference for editing near main roots.
- **AI agents:** Minimum edit probability at 1–3 dependency steps above a leaf (38th–52nd percentile), rising in *both* directions toward axioms and roots. Significantly stronger curvature (deviations on squared term: +0.94, +0.94, +4.84 for Claude, Gemini, Codex; all $p < 0.01$)

**Interpretation:** Agents prefer to work at the "logical boundaries" of a proof — either axiomward or working backward from the final theorem — while avoiding the densely connected intermediate regions where human mathematical work concentrates.

**2. Surprise regularization:**

- **High-surprise bodies:** All three agents are strongly attracted to definitions with high per-token surprise in their bodies, and their modifications **reduce surprise strongly toward baseline** (Figure 4, significant at $p < 0.001$)
- **High-surprise signatures:** All three agents show significant relative *avoidance* of high-surprise type signatures
- These results hold regardless of which model scores surprise (robustness check with three different models)

---

## Theoretical and Practical Implications

### 1. Agents as "Sorry Autofillers"

- **Human perspective:** A `sorry` is both a placeholder *and* a planning device [Tao23; Zhu+26] — it records a decision to defer a proof step for strategic reasons (routine labor, missing external results, undecided argument shape)
- **Agent perspective:** Each `sorry` is an open request for assistance — the most literal operationalization of "progress" available
- **Risk:** Autofilling with large, disconnected proof bodies produces "verbose and hard-to-understand artifacts" [Avi+26; Lu+26]. Evidence of prematurity: 25% (Codex) to 48% (Claude) of sorry-bearing definitions edited later evolve into definitions with *new signatures* — meaning the gap-filling was premature
- **Intervention:** Agents should reason about *why* a gap exists, not just *that* it exists

### 2. Peripheral Editing and the "Thick of the Action"

- The center of the dependency graph is where a proof's **core ideas** appear — the intermediate lemmas whose statement and proof choices make a formal proof correspond to a *particular human proof* rather than merely *some proof* [DD26]
- Mathematicians value proofs for the understanding they convey, not just certainty of conclusion [Thu06; Avi26]
- **Risk:** An agent that avoids the middle leaves the "work through which understanding is built" to human collaborators
- **Potential benefit:** A division of labor where agents handle foundational definitions and final theorem assembly while humans develop intermediate arguments could be complementary [Col+26]

### 3. Proof Regularization and "LLM Homogenization"

- Agents actively **seek out unusual proof bodies** and smooth them toward LLM-standard baselines
- This connects to concerns about "LLM homogenization" [Jai+25; Bur26] and the emergence of a "monocultural artificial hivemind" [Jia+26b]
- **Risk:** Increasing reliance on AI may drive human–AI collaboration products away from "unusual and idiosyncratic methods that may drive advances in both proof-making and Lean development"

### 4. The "AI-Friendly" Proof Style Problem

A critical second-order effect: "lazy" human mathematicians who delegate increasing work to agents will be **incentivized to make their proofs AI-friendly** — i.e., shorter, less densely connected, closer to leaves and roots, more predictable relative to training data. The resulting proof style would:
- Aid agents in finding next steps
- But **frustrate human goals of synthesis, diversity, and innovation**

### Limitations

1. **Dependency graphs** come from a heuristic lexical parser (validated with high reliability, but not exact) rather than the Lean compiler
2. **Training data contamination:** Repositories predate the models, so later states may appear in training data — this establishes a *conservative lower bound* on human–AI differences
3. **"The human" is treated as a single actor** — repositories have 1–61 contributors with likely different roles
4. **Merge commits excluded** — they combine work developed separately and don't correspond to single decisions
5. **Correlational analysis** — establishing causality would require non-natural interventions (e.g., inserting/removing sorries)

---

## Conclusion

The paper's "simulated collaboration" methodology reveals **strong and systematic deviations** between AI agents and human mathematicians in proof formalization, with high statistical significance and large effect sizes across all three tested models (Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.2).

The deviations cluster into three patterns:

1. **Syntactic autofilling:** Agents focus on explicit `sorry` markers, treating them as requests for assistance rather than strategic planning devices
2. **Peripheral engagement:** Agents avoid densely connected proof centers, preferring to work near axioms or final theorems
3. **Surprise regularization:** Agents seek out and smooth unusual proof bodies toward LLM-standard baselines

**Overall assessment:** The results paint a picture of *complementarity* — agents respond strongly to features humans consider less important. However, this complementarity carries risks: premature gap-filling, avoidance of the mathematical "core," and homogenization of proof style. The authors call for interventions that shift agents toward reasoning about *why* gaps exist, and for human–AI workflows that account for — rather than assume away — these asymmetries.

**Future directions:** Exploring how agents can recognize whether gaps ought to be filled at all, and how to preserve human agency and understanding in increasingly AI-assisted mathematical work.

---

**Data and code availability:** https://github.com/gyevnarb/proof-divergence

---

_Markdown view of https://picx.dev/p/JBh7bq, served by PicX — AI-generated visual whiteboard summaries of research papers._
