Summary (Overview)

• Spark-to-Paper is an end-to-end research paper generation system implemented as thirteen composable skills inside an existing coding assistant (Claude Code), requiring no separate agent platform or orchestration service.
• The system separates model-based judgment from deterministic operations, enabling checkable, reproducible operations (e.g., citation validation, LaTeX compilation) alongside semantic reasoning.
• A key innovation is the separation of experiment planning from reporting, creating a lightweight form of preregistration: required evidence is specified before results are observed, and manuscript claims are revised according to measured outcomes.
• The system identifies and bounds a failure mode called the Self-Refutation Loop, where repeated experiments continue to reject the original research objective; after seven cycles, the trajectory is terminated and recorded as a failure report.
• Across eight controlled research topics, Spark-to-Paper achieves 99.5% citation validity and 96.4% figure editability, with fabrication detection rising from 14% (single-pass) to 92% (full stack). Generation costs $8.1 per manuscript, uses 11.9M tokens, and takes 3.2 hours on average.

---

Introduction and Theoretical Foundation

Background and Motivation

Turning a research idea into a complete academic paper requires more than text generation: a researcher must retrieve literature, design and execute experiments, evaluate whether evidence supports the original hypothesis, revise claims when it does not, produce publication-ready figures, and maintain consistency across a long, evolving manuscript. While recent autonomous research agents (e.g., AI Scientist, AutoResearchClaw) have begun automating this process, they are typically standalone applications with their own orchestration layers and infrastructure, separating them from the coding environments where much actual research work takes place.

Key Insight

Modern coding assistants already provide the capabilities required for research automation: inspecting project files, executing code, searching for information, calling external tools, and revising artifacts over long interactions. This raises a central question: can end-to-end research paper generation be implemented as a collection of reusable skills inside an existing coding assistant, rather than as a separate autonomous research platform?

Theoretical Foundations

The system draws on several established research areas:

The concern motivating the gate infrastructure is well documented: LLMs fabricate citations and factual claims at measurable rates when writing about literature they have not retrieved. Spark-to-Paper's citation stage is a retrieval-then-verify instance of RAG, specialized to bibliographic metadata.

---

Methodology

System Architecture

Spark-to-Paper runs inside Claude Code using a Claude-family model. The design is not tied to Claude Code; any coding assistant with comparable file inspection, tool use, and code execution abilities can support the same skill-based design.

Core design principles:

• Skills as execution units: Each skill defines what a research task should accomplish, constraints, tools, and artifacts—but not every reasoning step. The coding assistant decides how to carry it out from the current project state.
• Model vs. deterministic tools: The model handles judgment (organizing arguments, assessing evidence relevance), while deterministic scripts handle checkable operations (structure validation, citation checking, LaTeX compilation, plotting).

Pipeline Stages

Key Components

1. Experiment Design and Revision

The planning stage specifies datasets, baselines, metrics, ablations, and result tables before execution. Table structures are fixed in advance with empty numerical cells—a lightweight form of preregistration. After execution:

• Claims are classified as SUPPORTED, PARTIALLY-SUPPORTED, UNSUPPORTED, CONTRADICTED, or NEEDS-CONFIRMATION
• Claims are retained, weakened, removed, moved to limitations, or trigger additional experiments
• Null, negative, or inconclusive results are preserved rather than omitted
• Changes propagate across abstract, introduction, results, and conclusion

A numerical result enters the paper only when traceable to its dataset, model configuration, seed, metric, and source output.

2. Integrity and Self-Critique

Two complementary error-handling mechanisms:

• Deterministic integrity checks: Machine-checkable properties (template structure, citation consistency, notation, figure files, successful compilation, result integrity in both modes)
• Model-based self-critique:
  • Self-Review: local, after-edit checking for terminology drift, redundancy, local inconsistencies
  • Adversarial-Review: manuscript-level, multi-pass review from complementary perspectives; every issue must quote the challenged passage and survive refutation attempts

3. Self-Refutation Loops and Bounded Recovery

A Self-Refutation Loop occurs when the system repeatedly concludes its own experiments fail to support the research objective, yet continues revising the same direction. Spark-to-Paper:

1. Bounds experiment–critique–revision cycles to seven
2. Terminates the trajectory if the objective remains unsupported
3. Records a failure report (original idea, attempted methods, observed results, reasons for insufficiency)
4. Restarts from a different idea and reruns the full pipeline

4. Editable Figure Generation

Two paths based on figure role:

• Method/explanatory figures: Image-generation model creates a raster visual target → HTML reconstruction via code → iterative visual comparison and adjustment → rendered to editable vector PDF
• Experimental result figures: Plotting code reads measured outputs directly → generates figure from data → exports as vector PDF

---

Empirical Validation / Results

Evaluation Dimensions

Main Results

Citation validity is computed over 384 references from eight Spark-to-Paper papers; figure editability over ~1,900 ground-truth elements.

Ablation Study

Fabrication detection uses 36 seeded probes spanning ten failure families, injected before drafting at the same intervention point across all configurations. Review precision is evaluated on 57 issues (60 sampled, 3 "cannot tell" excluded), assessed by blinded raters.

Case Study Insights

Two case studies demonstrate the system's ability to:

• Identify research gaps and synthesize novel methods (e.g., leakage-safe clinical risk screening pipeline; causal sliding-window decomposition for time-series forecasting)
• Expose inflated accuracy in conventional methods through empirical leakage audits
• Prioritize objective evidence over user prompts: When incorrect expectations were injected (e.g., Accuracy as primary metric; causal models expected to perform comparably), the system's empirical results contradicted these expectations, demonstrating evidence-based scientific integrity

---

Theoretical and Practical Implications

Theoretical Contributions

1. Design principle: Demonstrates that end-to-end research automation can be achieved as composable skills within existing coding assistants, rather than requiring separate agent platforms—challenging the assumption that autonomous research systems need dedicated infrastructure.

2. Evidence-grounded generation: The separation of experiment planning from reporting introduces a lightweight preregistration mechanism within automated pipelines, addressing a fundamental concern about adapting evaluation protocols after observing results.

3. Bounded recovery from self-refutation: Identifies and characterizes a novel failure mode (Self-Refutation Loop) in long-horizon research generation, and demonstrates that bounding this loop and recording failure reports is preferable to forcing every research trajectory to succeed.

4. Two-tier error handling: The distinction between deterministic integrity checks (for verifiable properties) and model-based self-critique (for semantic judgment) provides a principled framework for quality control in long-horizon generation tasks.

Practical Implications

• Accessibility: Runs inside existing coding assistants with no additional infrastructure to operate, making research automation accessible to a wider audience.
• Traceability: All artifacts (logs, metric files, tables, figures) are traceable to their sources; numerical results enter papers only when backed by data.
• Editable outputs: 96.4% figure editability enables human researchers to modify generated figures—a significant practical advantage over bitmap-based outputs.
• Cost efficiency: At $8.1 per manuscript with 3.2 hours wall-clock time, the system is practical for exploratory research, though 100× more expensive than a single-pass draft (reflecting the quality-efficiency trade-off).

---

Conclusion

Main Takeaways

Spark-to-Paper demonstrates that end-to-end research paper generation can be implemented as a lightweight, composable workflow inside existing coding assistants while keeping experimental evidence central to how claims are accepted, revised, or abandoned. The system achieves:

• 99.5% citation validity (vs. 81% for single-pass baseline)
• 96.4% figure editability (vs. 0–3% for prior systems)
• 92% fabrication detection with the full integrity stack (vs. 14% single-pass)
• 74% review precision on independently verifiable issues

Critically, Spark-to-Paper does not force every research trajectory to succeed: it bounds Self-Refutation Loops, preserves unsuccessful trajectories as research reports, and moves on to new ideas when hypotheses cannot be supported. This makes automated research a traceable process in which claims are revised or abandoned according to evidence rather than forced toward a successful narrative.

Future Directions

While the paper does not explicitly list future work, several directions are implied:

• Extending to a broader range of research domains beyond the eight controlled topics
• Improving the efficiency of the experiment execution stage
• Further reducing the token/cost overhead of the quality-control stack
• Exploring additional failure modes beyond Self-Refutation Loops
• Integrating more sophisticated experimental design and resource allocation (e.g., justifying expensive runs by specific claims)