# Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill

> Spark-to-Paper generates complete research papers inside Claude Code using thirteen composable skills, achieving 99.5% citation validity and 92% fabrication detection for $8.1 per manuscript.

- **Source:** [arXiv](https://arxiv.org/abs/2608.11924)
- **Published:** 2026-08-14
- **Permalink:** https://picx.dev/p/SLbCAp
- **Whiteboard:** https://picx.dev/p/SLbCAp/image

## Summary

## Summary (Overview)

- **Spark-to-Paper** is an end-to-end research paper generation system implemented as **thirteen composable skills** inside an existing coding assistant (Claude Code), requiring no separate agent platform or orchestration service.
- The system **separates model-based judgment from deterministic operations**, enabling checkable, reproducible operations (e.g., citation validation, LaTeX compilation) alongside semantic reasoning.
- A key innovation is the **separation of experiment planning from reporting**, creating a lightweight form of preregistration: required evidence is specified before results are observed, and manuscript claims are revised according to measured outcomes.
- The system identifies and bounds a failure mode called the **Self-Refutation Loop**, where repeated experiments continue to reject the original research objective; after seven cycles, the trajectory is terminated and recorded as a failure report.
- Across eight controlled research topics, Spark-to-Paper achieves **99.5% citation validity** and **96.4% figure editability**, with fabrication detection rising from 14% (single-pass) to 92% (full stack). Generation costs **$8.1 per manuscript**, uses **11.9M tokens**, and takes **3.2 hours** on average.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Turning a research idea into a complete academic paper requires more than text generation: a researcher must retrieve literature, design and execute experiments, evaluate whether evidence supports the original hypothesis, revise claims when it does not, produce publication-ready figures, and maintain consistency across a long, evolving manuscript. While recent autonomous research agents (e.g., AI Scientist, AutoResearchClaw) have begun automating this process, they are typically **standalone applications** with their own orchestration layers and infrastructure, separating them from the coding environments where much actual research work takes place.

### Key Insight

Modern coding assistants already provide the capabilities required for research automation: inspecting project files, executing code, searching for information, calling external tools, and revising artifacts over long interactions. This raises a central question: *can end-to-end research paper generation be implemented as a collection of reusable skills inside an existing coding assistant, rather than as a separate autonomous research platform?*

### Theoretical Foundations

The system draws on several established research areas:

| Foundation | Related Work |
|---|---|
| Reasoning-then-acting (ReAct) | Yao et al. [30] |
| Tool use for tasks LLMs shouldn't do | Schick et al. [19], Nakano et al. [17] |
| Self-refinement and self-critique | Shinn et al. [21], Madaan et al. [14] |
| Deliberate multi-step reasoning | Wei et al. [24], Yao et al. [29] |
| Memory-and-reflection architecture | Park et al. [18] |
| Retrieval-augmented generation for citation integrity | Lewis et al. [11], Ram et al. [22], Guu et al. [6] |

The concern motivating the gate infrastructure is well documented: LLMs fabricate citations and factual claims at measurable rates when writing about literature they have not retrieved. Spark-to-Paper's citation stage is a **retrieval-then-verify** instance of RAG, specialized to bibliographic metadata.

---

## Methodology

### System Architecture

Spark-to-Paper runs inside Claude Code using a Claude-family model. The design is not tied to Claude Code; any coding assistant with comparable file inspection, tool use, and code execution abilities can support the same skill-based design.

**Core design principles:**
- **Skills as execution units**: Each skill defines *what* a research task should accomplish, constraints, tools, and artifacts—but not every reasoning step. The coding assistant decides *how* to carry it out from the current project state.
- **Model vs. deterministic tools**: The model handles judgment (organizing arguments, assessing evidence relevance), while deterministic scripts handle checkable operations (structure validation, citation checking, LaTeX compilation, plotting).

### Pipeline Stages

| Stage | Function |
|---|---|
| **Stage 0: Input Routing** | Determines starting point and selects result-integrity mode: *Proposal Mode* (no measured results; unavailable values remain unspecified) or *Data-Aware Mode* (quantitative statements must be supported by provided data) |
| **Stage 1: Planning** | Converts input into structured paper blueprint: research question, contributions, section structure, notation, experimental design, venue requirements |
| **Stage 2: Citation** | Searches literature, verifies references via DOI/arXiv metadata; stores verified references in BibTeX |
| **Stage 3: Writing** | Generates complete LaTeX manuscript from blueprint and verified bibliography |
| **Stage 4: Refinement** | Revises manuscript holistically; reruns deterministic checks |
| **Stage 5: Review** | Multiple isolated review passes (technical soundness, experimental design, evidence strength); issues must quote specific passages and survive three-direction checking |
| **Stage 6: Figure Generation** | Two paths: deterministic plotting from data for quantitative figures; image-generation + code reconstruction for method diagrams |
| **Stage 7: Assembly** | Combines sections, bibliography, figures, venue template; compiles LaTeX; runs final checks |
| **Stage 8: Experiment Execution** | Runs planned experiments when feasible; regenerates tables/figures from measurements; revises claims to match evidence |

### Key Components

#### 1. Experiment Design and Revision

The planning stage specifies datasets, baselines, metrics, ablations, and result tables **before** execution. Table structures are fixed in advance with empty numerical cells—a lightweight form of **preregistration**. After execution:

- Claims are classified as `SUPPORTED`, `PARTIALLY-SUPPORTED`, `UNSUPPORTED`, `CONTRADICTED`, or `NEEDS-CONFIRMATION`
- Claims are retained, weakened, removed, moved to limitations, or trigger additional experiments
- Null, negative, or inconclusive results are **preserved** rather than omitted
- Changes propagate across abstract, introduction, results, and conclusion

A numerical result enters the paper only when traceable to its dataset, model configuration, seed, metric, and source output.

#### 2. Integrity and Self-Critique

Two complementary error-handling mechanisms:

- **Deterministic integrity checks**: Machine-checkable properties (template structure, citation consistency, notation, figure files, successful compilation, result integrity in both modes)
- **Model-based self-critique**:
  - *Self-Review*: local, after-edit checking for terminology drift, redundancy, local inconsistencies
  - *Adversarial-Review*: manuscript-level, multi-pass review from complementary perspectives; every issue must quote the challenged passage and survive refutation attempts

#### 3. Self-Refutation Loops and Bounded Recovery

A **Self-Refutation Loop** occurs when the system repeatedly concludes its own experiments fail to support the research objective, yet continues revising the same direction. Spark-to-Paper:

1. Bounds experiment–critique–revision cycles to **seven**
2. Terminates the trajectory if the objective remains unsupported
3. Records a **failure report** (original idea, attempted methods, observed results, reasons for insufficiency)
4. Restarts from a **different idea** and reruns the full pipeline

#### 4. Editable Figure Generation

Two paths based on figure role:

- **Method/explanatory figures**: Image-generation model creates a raster *visual target* → HTML reconstruction via code → iterative visual comparison and adjustment → rendered to **editable vector PDF**
- **Experimental result figures**: Plotting code reads measured outputs directly → generates figure from data → exports as **vector PDF**

---

## Empirical Validation / Results

### Evaluation Dimensions

| Metric | Measurement |
|---|---|
| Citation validity | Resolved references / total references |
| Fabrication detection | Detected unsupported claims / injected claims |
| Figure editability | Editable elements / total figure elements |
| Review precision | Verified issues / raised review issues |
| Cross-template robustness | Successful templates / supported templates |
| Generation cost | Monetary cost, tokens, wall-clock time, deployment footprint |

### Main Results

| System | Citation exist. (%) | Fig. editability (%) | Tokens (M) | USD | Wall-clock |
|---|---|---|---|---|---|
| Human-written preprints (sampled) | 97.8 [94.6, 99.4] | 58 [44, 71] | n/a | n/a | n/a |
| AI Scientist [13] | 93 (42/45) | 0 (0/210) | n/r | $10–15 (amort.) | ~12 h / batch |
| AI Scientist-v2 [28] | 91 (58/64) | 3 (0–8) | n/r | ~$20–25 / attempt | ≤15 h / run |
| Agent Laboratory [20] | 96 (27/28) | 0 (0/30) | n/r | $2.33 (gpt-4o) | ~19 min |
| Single-pass LLM draft | 81 (range 76–86) | n/a | 0.11 (0.09–0.13) | $0.66 (0.55–0.76) | 16 min (13–19) |
| **Spark-to-Paper (full stack)** | **99.5 [98.4, 100]** | **96.4 [92.7, 98.6]** | **11.9 [10.2, 13.7]** | **$8.1 [6.9, 9.6]** | **3.2 h [2.6, 3.9]** |

Citation validity is computed over **384 references** from eight Spark-to-Paper papers; figure editability over ~1,900 ground-truth elements.

### Ablation Study

| Configuration | Fabr. detection (%) | Review precision (%) | Δ tokens (M) | Δ USD |
|---|---|---|---|---|
| Single-pass draft (no gates) | 14 (5/36) [6, 29] | n/a | ref. | ref. |
| Gates only | 69 (25/36) [53, 82] | n/a | +8.1 ± 0.9 | +5.3 ± 0.5 |
| Gates + self-review | 81 (29/36) [65, 90] | n/a | +1.1 ± 0.2 | +0.6 ± 0.1 |
| Gates + self-review + adversarial review | **92 (33/36) [78, 97]** | **74 (42/57) [61, 83]** | +2.6 ± 0.4 | +1.6 ± 0.2 |

Fabrication detection uses **36 seeded probes** spanning ten failure families, injected before drafting at the same intervention point across all configurations. Review precision is evaluated on 57 issues (60 sampled, 3 "cannot tell" excluded), assessed by blinded raters.

### Case Study Insights

Two case studies demonstrate the system's ability to:
- **Identify research gaps and synthesize novel methods** (e.g., leakage-safe clinical risk screening pipeline; causal sliding-window decomposition for time-series forecasting)
- **Expose inflated accuracy in conventional methods** through empirical leakage audits
- **Prioritize objective evidence over user prompts**: When incorrect expectations were injected (e.g., Accuracy as primary metric; causal models expected to perform comparably), the system's empirical results contradicted these expectations, demonstrating evidence-based scientific integrity

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Design principle**: Demonstrates that end-to-end research automation can be achieved as composable skills within existing coding assistants, rather than requiring separate agent platforms—challenging the assumption that autonomous research systems need dedicated infrastructure.

2. **Evidence-grounded generation**: The separation of experiment planning from reporting introduces a lightweight preregistration mechanism within automated pipelines, addressing a fundamental concern about adapting evaluation protocols after observing results.

3. **Bounded recovery from self-refutation**: Identifies and characterizes a novel failure mode (Self-Refutation Loop) in long-horizon research generation, and demonstrates that bounding this loop and recording failure reports is preferable to forcing every research trajectory to succeed.

4. **Two-tier error handling**: The distinction between deterministic integrity checks (for verifiable properties) and model-based self-critique (for semantic judgment) provides a principled framework for quality control in long-horizon generation tasks.

### Practical Implications

- **Accessibility**: Runs inside existing coding assistants with no additional infrastructure to operate, making research automation accessible to a wider audience.
- **Traceability**: All artifacts (logs, metric files, tables, figures) are traceable to their sources; numerical results enter papers only when backed by data.
- **Editable outputs**: 96.4% figure editability enables human researchers to modify generated figures—a significant practical advantage over bitmap-based outputs.
- **Cost efficiency**: At $8.1 per manuscript with 3.2 hours wall-clock time, the system is practical for exploratory research, though 100× more expensive than a single-pass draft (reflecting the quality-efficiency trade-off).

---

## Conclusion

### Main Takeaways

Spark-to-Paper demonstrates that **end-to-end research paper generation can be implemented as a lightweight, composable workflow inside existing coding assistants** while keeping experimental evidence central to how claims are accepted, revised, or abandoned. The system achieves:

- **99.5% citation validity** (vs. 81% for single-pass baseline)
- **96.4% figure editability** (vs. 0–3% for prior systems)
- **92% fabrication detection** with the full integrity stack (vs. 14% single-pass)
- **74% review precision** on independently verifiable issues

Critically, Spark-to-Paper does not force every research trajectory to succeed: it bounds Self-Refutation Loops, preserves unsuccessful trajectories as research reports, and moves on to new ideas when hypotheses cannot be supported. This makes automated research a **traceable process** in which claims are revised or abandoned according to evidence rather than forced toward a successful narrative.

### Future Directions

While the paper does not explicitly list future work, several directions are implied:
- Extending to a broader range of research domains beyond the eight controlled topics
- Improving the efficiency of the experiment execution stage
- Further reducing the token/cost overhead of the quality-control stack
- Exploring additional failure modes beyond Self-Refutation Loops
- Integrating more sophisticated experimental design and resource allocation (e.g., justifying expensive runs by specific claims)

---

_Markdown view of https://picx.dev/p/SLbCAp, served by PicX — AI-generated visual whiteboard summaries of research papers._