Autonomous AI Agents Systematically Diverge From Human Preferences in Proof Formalization

Authors: Bálint Gyevnár (Carnegie Mellon University) & Simon DeDeo (Carnegie Mellon University, Santa Fe Institute)


Summary (Overview)

  • Methodological innovation: The paper introduces "simulated collaboration", a methodology that replays real human proof-formalization histories (Git commits from Lean repositories) but delegates the "next step" to an AI agent, allowing researchers to compare human vs. AI preferences in what to work on next — rather than testing performance on pre-selected problems.
  • Key finding 1 (Syntactic): AI agents are overwhelmingly drawn to explicit sorry markers (proof gaps). Definitions containing a sorry account for 59–92% of AI agent modifications vs. only 19% of human modifications. Agents act as "proof autofillers," treating sorries as implicit requests for assistance.
  • Key finding 2 (Semantic/Network): AI agents preferentially modify definitions near the periphery of the dependency graph (near axioms or final theorems) and avoid the densely connected intermediate regions where humans concentrate their work.
  • Key finding 3 (Surprise): AI agents are attracted to high-surprise proof bodies and act to regularize them toward baseline (LLM-standard) surprise levels, while humans leave aggregate surprise nearly unchanged. Conversely, agents avoid high-surprise type signatures.
  • Implications: The results highlight potential blind spots in human–AI mathematical collaboration, including risks of "LLM homogenization," premature gap-filling, and the incentivization of "AI-friendly" proof styles that may frustrate human goals of synthesis, diversity, and innovation.

Introduction and Theoretical Foundation

Background

Proof assistants (e.g., Lean) have become central to AI-enabled mathematics. However, the fact that an AI system finds a proof reveals little about how it resolved the problem or what subproblems it chose to visit. Prior work shows AI-discovered proofs can be:

  • Verbose, circuitous, and redundant [ABB26; Lu+26]
  • Misformalized [BCH26]
  • Exploitative of kernel holes [dMou26]
  • Misaligned with human conceptual structures [DD26]

The Core Problem

Existing evaluations of AI agents in formal mathematics test performance on pre-selected problems with human-chosen prescriptions for what to work on [Xin+26; Let+26; Zha+26; Zhu+26; XLZ26]. This leaves a critical gap: we know little about what AI agents attend to when the choice of what to work on is left to them.

Research Questions

RQ1. To what extent in mathematical collaboration do AI agents respond to and work with the surface syntax of a proof, or, conversely, attend to some underlying semantics?

RQ2. To what extent do these preferences of AI agents either match or differ from those of human experts?

Theoretical Foundations

The paper builds on two key theoretical tensions:

  1. Syntax vs. semantics: Neural models exploit surface regularities (source code is repetitive and predictable [Hin+12]) as shortcuts [Gei+20; MPL19]. LLM-based provers for Lean succeed on one formulation of a theorem but fail on semantically equivalent ones, with statement surprise predicting solvability [Ole+26]. However, language models can also learn internal representations of program semantics beyond surface form [JR24; NLW23].

  2. Alignment in collaboration: Effective human–AI collaboration requires understanding where agent preferences diverge from human ones. Agents that are misaligned can produce "costly mismatches between intuition and formal code," redirect human effort into "AI-friendly" styles, and frustrate the goal of establishing correspondences between formal reasoning and conceptual structure [DD26].


Methodology

Dataset

  • Nine major human collaborations in Lean containing 4.7M definitions across more than 6K commits
  • Repositories include human formalization projects tracked via Git

Simulated Collaboration Procedure

The experimental setup (Figure 1) works as follows:

  1. Download a Lean formalization repository
  2. Remove all merge commits and commits with no Lean file modifications
  3. For each remaining human parent commit Ht−1H_{t-1}, an agent is called to "take the next step," producing an artificial child state AtA_t
  4. Compare the human-authored child state HtH_t to AtA_t using extracted features

Models Compared

  • Claude Opus 4.6 (Anthropic)
  • Google Gemini 3.1 Pro (Google DeepMind)
  • GPT-5.2 (OpenAI) — referred to as "Codex" in the paper

Feature Extraction

Syntactic predictors:

  • Definition length (number of characters)
  • Presence of sorry markers (Lean keyword allowing compilation without a complete proof)
  • Distance to nearest sorry in the dependency graph

Semantic predictors:

  1. Network position:

    • Max leaf distance: maximum shortest-path distance from a definition to all accessible leaves (axioms) in the dependency graph
    • Root distance: shortest distance to the nearest significant root (final theorem)
  2. Surprise: Per-token surprise of the definition's type signature (problem statement) and body (current proof attempt), computed using DeepSeek-Prover-V2-7B (with robustness checks using Goedel-Prover-V2-32B and Kimina-Prover-72B)

Statistical Analysis

  • Logistic regression model predicting whether an actor (human or agent) chooses a given definition to edit
  • Predictors include syntactic and semantic features of definitions in the parent commit
  • Fixed-effect coefficients with 95% Wald confidence intervals
  • Gaussian mixed models for edit-size comparisons
  • Holm-corrected p-values for multiple comparisons

Empirical Validation / Results

Summary Statistics: How Much Do Actors Edit?

Table 1: Number and per-actor percentage of definitions Added, Modified, or Removed, aggregated across all commits. Median chars. gives the median number of characters inserted plus deleted across the signature and body.

ActorAdded CountAdded %Median Chars.Modified CountModified %Median Chars.Removed CountRemoved %Median Chars.Total Count
Human10,6180.901336,9750.59204,9130.411291,185,143
Claude4,6400.391716,1780.52149.54080.03811,176,797
Gemini1,9740.171934,8760.42846180.051041,174,808
Codex2,0560.171351,3940.12103570.001101,175,224
Overall19,2880.4114519,4230.41725,9960.131204,711,972

Key observations:

  • Humans have the highest combined edit share: 1.90% vs. 0.95% (Claude), 0.64% (Gemini), 0.30% (Codex) — significant overall (χ2(3)=31.7,p<0.001)(\chi^2(3) = 31.7, p < 0.001)
  • Agents almost never delete code: humans are 8–86 times more likely to remove definitions (all p<0.001p < 0.001)
  • Agent edits are larger: median characters modified is 20 for humans vs. 84–150 for agents (Holm-corrected p=0.002p = 0.002 for Gemini, p<0.001p < 0.001 for Claude and Codex)

Modification Preferences: Where Do Actors Change the Existing Proof?

Syntactic Preferences

1. Agents focus on filling local gaps (sorry markers):

  • All three agents show stronger-than-human preferences for editing definitions with a sorry
  • Outcome statistics: At least 59% of agent edits involved definitions with a sorry (Claude: 92%; Gemini: 80%; Codex: 59%), compared to just 19% for humans
  • The preference is for the syntactic marker itself, not the problem area: the decline in preference with distance from a sorry in the dependency graph is much steeper for agents than humans (Figure 3)

2. Agents avoid long definitions:

  • Humans are strongly attracted to longer definitions (β=+2.207±0.055)(\beta = +2.207 \pm 0.055)
  • Agent deviations are large and negative: −1.58 (Claude), −1.77 (Gemini), −2.56 (Codex)

3. Agents make isolated edits:

  • 62% of human edits connect to another edited definition in the same commit
  • Agents make isolated edits 77% (Claude), 69% (Gemini), and 82% (Codex) of the time

"In short, and perhaps uncharitably, these AI agents, despite their sophistication, retain much of the flair of autocomplete systems of earlier eras."

Semantic Preferences

1. Peripheral engagement (U-shaped preferences):

The quadratic fit on Max leaf distance and Root distance reveals:

  • Humans: Minimum edit probability at the leaves themselves (10th percentile), with probability rising steadily into the graph. Weak preference for editing near main roots.
  • AI agents: Minimum edit probability at 1–3 dependency steps above a leaf (38th–52nd percentile), rising in both directions toward axioms and roots. Significantly stronger curvature (deviations on squared term: +0.94, +0.94, +4.84 for Claude, Gemini, Codex; all p<0.01p < 0.01)

Interpretation: Agents prefer to work at the "logical boundaries" of a proof — either axiomward or working backward from the final theorem — while avoiding the densely connected intermediate regions where human mathematical work concentrates.

2. Surprise regularization:

  • High-surprise bodies: All three agents are strongly attracted to definitions with high per-token surprise in their bodies, and their modifications reduce surprise strongly toward baseline (Figure 4, significant at p<0.001p < 0.001)
  • High-surprise signatures: All three agents show significant relative avoidance of high-surprise type signatures
  • These results hold regardless of which model scores surprise (robustness check with three different models)

Theoretical and Practical Implications

1. Agents as "Sorry Autofillers"

  • Human perspective: A sorry is both a placeholder and a planning device [Tao23; Zhu+26] — it records a decision to defer a proof step for strategic reasons (routine labor, missing external results, undecided argument shape)
  • Agent perspective: Each sorry is an open request for assistance — the most literal operationalization of "progress" available
  • Risk: Autofilling with large, disconnected proof bodies produces "verbose and hard-to-understand artifacts" [Avi+26; Lu+26]. Evidence of prematurity: 25% (Codex) to 48% (Claude) of sorry-bearing definitions edited later evolve into definitions with new signatures — meaning the gap-filling was premature
  • Intervention: Agents should reason about why a gap exists, not just that it exists

2. Peripheral Editing and the "Thick of the Action"

  • The center of the dependency graph is where a proof's core ideas appear — the intermediate lemmas whose statement and proof choices make a formal proof correspond to a particular human proof rather than merely some proof [DD26]
  • Mathematicians value proofs for the understanding they convey, not just certainty of conclusion [Thu06; Avi26]
  • Risk: An agent that avoids the middle leaves the "work through which understanding is built" to human collaborators
  • Potential benefit: A division of labor where agents handle foundational definitions and final theorem assembly while humans develop intermediate arguments could be complementary [Col+26]

3. Proof Regularization and "LLM Homogenization"

  • Agents actively seek out unusual proof bodies and smooth them toward LLM-standard baselines
  • This connects to concerns about "LLM homogenization" [Jai+25; Bur26] and the emergence of a "monocultural artificial hivemind" [Jia+26b]
  • Risk: Increasing reliance on AI may drive human–AI collaboration products away from "unusual and idiosyncratic methods that may drive advances in both proof-making and Lean development"

4. The "AI-Friendly" Proof Style Problem

A critical second-order effect: "lazy" human mathematicians who delegate increasing work to agents will be incentivized to make their proofs AI-friendly — i.e., shorter, less densely connected, closer to leaves and roots, more predictable relative to training data. The resulting proof style would:

  • Aid agents in finding next steps
  • But frustrate human goals of synthesis, diversity, and innovation

Limitations

  1. Dependency graphs come from a heuristic lexical parser (validated with high reliability, but not exact) rather than the Lean compiler
  2. Training data contamination: Repositories predate the models, so later states may appear in training data — this establishes a conservative lower bound on human–AI differences
  3. "The human" is treated as a single actor — repositories have 1–61 contributors with likely different roles
  4. Merge commits excluded — they combine work developed separately and don't correspond to single decisions
  5. Correlational analysis — establishing causality would require non-natural interventions (e.g., inserting/removing sorries)

Conclusion

The paper's "simulated collaboration" methodology reveals strong and systematic deviations between AI agents and human mathematicians in proof formalization, with high statistical significance and large effect sizes across all three tested models (Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.2).

The deviations cluster into three patterns:

  1. Syntactic autofilling: Agents focus on explicit sorry markers, treating them as requests for assistance rather than strategic planning devices
  2. Peripheral engagement: Agents avoid densely connected proof centers, preferring to work near axioms or final theorems
  3. Surprise regularization: Agents seek out and smooth unusual proof bodies toward LLM-standard baselines

Overall assessment: The results paint a picture of complementarity — agents respond strongly to features humans consider less important. However, this complementarity carries risks: premature gap-filling, avoidance of the mathematical "core," and homogenization of proof style. The authors call for interventions that shift agents toward reasoning about why gaps exist, and for human–AI workflows that account for — rather than assume away — these asymmetries.

Future directions: Exploring how agents can recognize whether gaps ought to be filled at all, and how to preserve human agency and understanding in increasingly AI-assisted mathematical work.


Data and code availability: https://github.com/gyevnarb/proof-divergence

Related papers