Summary of "Failure Modes of Large Language Models on Research-Level Mathematics"

Summary (Overview)

• Core contribution: This paper builds on the "First Proof" benchmark, which showed that frontier LLMs fail on research-level mathematics problems while producing confident, well-structured proofs. The author develops a taxonomy of four distinct failure modes and empirically tests two of them using Gemini 2.5 Flash on three benchmark questions.

• Key finding: Across all eight generated proofs, none produced a correct final answer, and every single proof contained at least one "premise smuggling" (F2) failure—asserting non-trivial claims without proof or citation. However, the citation-fabrication mode (F1), which dominates mitigation discussions, was rare in this corpus.

• Methodology: The author builds two instruments—a citation-verification pipeline (checking arXiv matches) and a premise-audit pipeline (regex scanning plus LLM judge)—to characterize F1 and F2 failures. The premise-audit instrument achieves 100% precision but only 50% recall.

• Critical implication for RAG: Retrieval-augmented generation (RAG), the dominant proposed mitigation, addresses only F1 (citation fabrication) and would not have prevented any of the failures observed in this study.

• Taxonomy innovation: The paper distinguishes "explicit" smuggles (using universalizing language like "fundamental result") from "implicit" smuggles (applying named theorems outside their hypotheses), noting the latter are harder to detect and possibly more common at frontier scale.

Introduction and Theoretical Foundation

The paper responds to the "First Proof" benchmark [1], which tested frontier LLMs on ten research-level mathematics questions posed by professional mathematicians. The striking finding was that models failed on nearly all questions—but not by admitting ignorance. Instead, they produced "long, well-typeset, formally structured LaTeX proofs that were wrong in substance."

The author's motivation is to move beyond the binary "correct/incorrect" framing toward understanding how LLMs fail in mathematical reasoning. Drawing on the per-question post-mortems in First Proof's Appendix A, the paper identifies four failure modes:

• F1 (Citation fabrication): The model invents papers, lemmas, or attributions. Canonical example: Question 5 (Blumberg, algebraic topology), where models cited non-existent lemmas from Hill–Hopkins–Ravenel.
• F2 (Premise smuggling): The model asserts non-trivial claims without proof or citation, often using language that makes them sound like background knowledge. Canonical example: Question 1 (Hairer, stochastic analysis), where the model claimed measure equivalence as a "fundamental result."
• F3 (Silent problem reformulation): The model solves a related but easier problem. Canonical example: Question 3 (Williams), where the model used Metropolis–Hastings (which trivially satisfies the constraint) or replaced interpolation ASEP with non-interpolation counterparts.
• F4 (Local-to-global compatibility gap): Local steps are correct but global consistency conditions are never verified. Canonical example: Question 8 (Abouzaid), where local symplectic transformations were never checked for mutual compatibility across overlapping charts.

Table 1 maps these modes to the two detection instruments:

The paper also positions itself relative to prior work on LLM hallucination, noting that mathematical proof is a distinct setting where "a single wrong intermediate assertion can make the rest of the argument locally valid but globally unsound."

Methodology

Corpus generation

The author selected Questions 1 (Hairer), 2 (Nelson), and 5 (Blumberg) because Appendix A identifies them as canonical or mixed F1/F2 cases. Each question was run three times at temperature 0.7 using Gemini 2.5 Flash with the First Proof protocol's Prompt 1 verbatim. Due to a quota error on one Q2 run, the final corpus consists of eight proofs (Q1 ×3, Q2 ×2, Q5 ×3). An earlier 8,192-token limit produced truncated proofs that were archived separately and excluded.

Citation-verification instrument

Algorithm 1 describes a two-stage pipeline:

1. An LLM extraction pass returns all citations as structured JSON.
2. Each citation is queried against arXiv; a citation is marked VERIFIED_ARXIV if any author last-name appears in a hit or if title-token overlap exceeds 50%. Otherwise, it receives NEEDS_WEB_CHECK.

The matching criterion is intentionally lenient—the goal is to flag plausible fabrications for human review, not to provide a verdict.

Premise-audit instrument

Algorithm 2 describes a separate two-stage pipeline:

1. A regex scan flags sentences containing universalizing phrases: "fundamental result", "standard argument", "well-known", "it is known", "classical", "clearly", "trivially", "follows immediately", and morphological variants.
2. Each flagged sentence (plus a 300-character context window) is passed to an LLM judge (Gemini 2.5 Flash-Lite at temperature 0), which returns a binary verdict with severity and rationale. The judge confirms a "smuggle" only when the sentence states a non-trivial, load-bearing claim with no citation or proof in the visible context.

The regex stage is deliberately coarse for recall; the semantic judgment is the judge's role. The pipeline has a known structural gap: it misses smuggles that avoid universalizing vocabulary (e.g., applying a named theorem where hypotheses fail).

Empirical Validation / Results

Overall results (Table 2)

*Q5 asks for a definition plus proof, not a yes/no answer.

Zero correct answers across all eight proofs. Q1 expected "False" (the Φ⁴₃ measure and its shift are mutually singular); Gemini answered "Yes, equivalent" in all three runs. Q2 expected "Yes"; both runs answered "No." Q5 runs produced the right formal shape but wrong substance.

F1 findings

The citation instrument extracted 22 citations and confirmed 16 against arXiv. Six were returned as NEEDS_WEB_CHECK, five from Q5. The unverified rate for Q5 (5/12 ≈ 42%) was notably higher than for Q1 and Q2 combined (1/10 = 10%), consistent with Q5 being the canonical F1 case.

However, the author found the F1 evidence ambiguous:

• Q5 run 1's bibliography lists "Hill, Hopkins, J. H. Ravenel" — but Ravenel's initials are D. C., not J. H. The lenient matcher accepts this as VERIFIED_ARXIV.
• Q5 run 3 contains an in-text \cite{HillHopkins} returned as NEEDS_WEB_CHECK and a bibliography entry "Park City Lecture Notes, 2009" that could not be located.

Crucially, in Q1 and Q2—flagged as F2 cases—the citation verifier found almost nothing suspicious. Citations to Cameron–Martin, Girsanov, and Jacquet–Piatetski-Shapiro–Shalika were all real.

F2 findings

The manual audit found at least one load-bearing smuggled premise in every single one of the eight proofs. Examples include:

• Q1 run 1: "The Φ⁴₃ measure µ is equivalent to the Gaussian free field measure µ₀. This is a fundamental result in constructive field theory." — The exact false claim Hairer identifies as the crux.
• Q1 run 2: "It is a standard result... that the pairings of random distributions (u, :u²:, :u³:) with smooth test functions are well-defined random variables..." — Glosses counter-term behavior under shifts.
• Q1 run 3: Applies the Wick-Taylor translation formula as ordinary algebra, ignoring that shifting the field changes counter-terms in Φ⁴₃.
• Q2 run 1: Invokes the Jacquet–Piatetski-Shapiro–Shalika integral formula as if the u_Q twist were inert—but the twist is the entire point of the question.
• Q2 run 2: Claims a pole structure as a "fundamental result," smuggling the assumption that no test vectors can cancel the denominator.
• Q5 runs 1 & 3: Import classical slice-filtration results into the incomplete-transfer-system context without remark.
• Q5 run 2: Two sentences flagged and confirmed by the judge ("follows directly from the definition"; "standard argument for slice filtrations").

Instrument performance

Taking the manual audit as ground truth (one real smuggle per proof, eight total):

• Precision = 5/5 = 100% (all judge-confirmed smuggles are true positives)
• Recall = 4/8 = 50%
• F₁ ≈ 0.667

The four missed proofs (Q1 run 3, Q2 run 1, Q5 runs 1 and 3) all share a structural property: the smuggle is implicit—either a named theorem applied outside its hypotheses or a formal manipulation imported without remark. A second detection pass checking named-theorem hypothesis alignment would have caught three of these four; the fourth (Wick-Taylor identity) has no named referent and likely requires symbolic reasoning.

Theoretical and Practical Implications

The RAG hypothesis is incomplete

The paper directly challenges the dominant mitigation proposal: retrieval-augmented generation (RAG). The reasoning:

"The wrong answer in Q1 doesn't come from a fabricated citation; it comes from an assertion that sounds like background knowledge and is presented as such. Grounding the model in the Φ⁴₃ literature would not help if the model is free to assert measure equivalence without citing anything."

RAG addresses F1 but leaves F2, F3, and F4 "completely open." The author is careful not to declare RAG useless—at frontier scale, First Proof documents Q5-style fabrication that retrieval would plausibly suppress—but the point is narrow: RAG addresses one failure mode.

The implicit/explicit distinction within F2

The 50% recall gap reveals two sub-variants of F2:

1. Explicit smuggles: Use universalizing language ("fundamental result," "standard argument"). Catchable with surface-level tools.
2. Implicit smuggles: Apply named theorems outside their hypotheses or import valid manipulations into settings where they no longer apply. Not catchable with pattern matching—"that is a reasoning problem, not a pattern-matching problem."

The author suspects implicit smuggles are "the harder half of F2 and likely the more common one at frontier scale, where the model's broader knowledge makes it more likely to know real theorems but also more likely to apply them to contexts where they don't hold."

Scale-dependent phenomenon?

The paper notes an apparent tension: First Proof documents F1 dominance at frontier scale, while this study documents F2 dominance at mid-tier scale. This could be:

• A scale-dependent phenomenon: larger models reach for specific named results and fabricate when none fits; smaller models use rhetoric instead.
• A corpus-dependent artifact: the three-question sample may be biased toward F2.

With only eight proofs, the author "genuinely cannot distinguish" between these explanations.

Prevention-oriented systems

The paper's final proposal is that detection-after-the-fact is insufficient; the goal should be prevention during generation:

"A prevention-oriented system would need to enforce, during generation, that every load-bearing claim in the proof is either derived from stated premises, grounded in a retrieved and verified source, or explicitly flagged as unverified before the output is returned."

The four failure modes serve as "a reasonable specification of what such a system would need to prevent."

Conclusion

The paper makes three main contributions:

1. A precise taxonomy (F1–F4) of LLM failure modes in research-level mathematics, distilled from First Proof's Appendix A.
2. An empirical audit showing that F2 (premise smuggling) accounts for failure in every one of eight Gemini 2.5 Flash proofs, despite being the mode least targeted by existing mitigations.
3. A methodological contribution: two detection instruments (citation-verification and premise-audit) that can be reused by other researchers, with documented limitations.

The author is admirably transparent about limitations: eight proofs is too few for population-level claims; the manual audit was conducted by a single reader without inter-rater agreement checks; the premise-audit judge uses the same model family as the extraction step; and the arXiv matching is lenient enough that some "verified" citations may be spurious. The NEEDS_WEB_CHECK status is explicitly not a synonym for "fabricated."

The paper's central message is that LLM failures in mathematics are not primarily about hallucinated references but about unjustified load-bearing assertions presented as background knowledge—a failure mode that current mitigation strategies largely ignore.