Summary (Overview)

• Core contribution: The paper introduces "compile by training," a method that transforms natural-language specifications into reusable neural functions (programs) by using teacher models to synthesize training examples and fine-tuning a small LoRA adapter for a shared compact interpreter.

• Key result: On FuzzyBench-Hard, a challenging subset where the fast Program-as-Weights (PAW) compiler produced no exact matches, compile by training achieves 83.6% semantic accuracy (mean LLM Exact Match), a substantial improvement over the fast compiler's 22.4% accuracy.

• Trade-off: This accuracy gain comes at a higher compile-time cost—approximately 50.9 seconds versus 3.5 secondsfor the fast compiler—but the resulting programs run without teacher models, enabling local, reusable, and composable deployment

• Deployment: The authors deployed the compiler in a public interactive service and demonstrated three applications: a multi-site website helper (paw-helper), a language-controlled 3D avatar, and a bidirectional English–Claudish translator that handled over 100,000 requests in ten days

• System design: The compilation process overlaps teacher synthesis with training, coordinates jobs across shared GPU workers, and treats compilation as a persistent background job, making minute-scale builds usable interactively.

Introduction and Theoretical Foundation

The paper addresses a fundamental gap in text processing: many recurring text functions are "easy to describe but difficult to implement with rules" (e.g., email triage, extracting identifiers, style translation). While large remote LLMs can perform these tasks, calling them for every input incurs repeated network latency, provider cost, and dependence on an external service. These functions fall "between these two options: they are too fuzzy for conventional code, but too narrow and frequent to justify a large-model call at every invocation."

The theoretical foundation builds on Program-as-Weights (PAW) (Zhang et al.,, 2026), a framework where a neural program specializes a shared local interpreter. PAW introduced an amortized compiler that predicts a task-specific program (LoRA adapter + prompt scaffold) in a single forward pass, making compilation fast but spending "the same fixed amount of computation on every function."

Compile by training extends this paradigm by treating adaptation as a software build step: it uses the amortized prediction as a starting point, then invests additional computation in teacher synthesis and specification-specific optimization. This reframes large language models as "tool builders rather than run-time dependencies"—teachers are used once at compile time, not at every invocation.

The formal interface is:

$$p _ {s} = \operatorname{Compile} (s), \quad \hat {y} = \operatorname{Run} (p _ {s}, x),$$

where $s$ is a natural-language function specification, $p _ {s}$ is the compiled program, $x$ is a new input, and $\hat {y}$ is the program's output. A shared frozen language model (Qwen3-0.6B) serves as the interpreter, while each compiled program supplies the adapter and prompt that specialize it for one function.

Methodology

The compile-by-training process consists of two main stages:

1. From Specification to Supervision

Teacher models synthesize a task-specific dataset from the natural-language specification:

$$D _ {s} = \{(x _ {i}, y _ {i}) \} _ {i = ​1} ^ {n} \sim T (s),$$

where $s$ is the specification, $T$ represents teacher models, and each pair $(x _ i, y _ i)$ illustrates the desired input-output mapping. Teacher requests usea structured JSON format for automatic validation, and the public service combinesa lower-cost teacher (GPT-5.4-mini) witha larger teacher (GPT-5.5)to supply complementary supervision.

2. Specialization and Packaging

Rather than traininga full model per spec, all programs sharea frozen Qwen3-0.6B interpreter,and each function is represented bya lightweight LoRA adapter (Hu et al.,, 2022)and a run-time scaffold. The scaffold isa compiler-generated prompt template encoding the user's spec as structured instructions and examples, witha placeholder for run-time input. The amortized PAW compiler provides initial adapter parameters $\theta _ { s } ^ { ( 0 ) }$ and scaffold $r _ { s }$; training then minimizes:

$$\mathcal {L} \left(\theta_ {s}\right) = \sum_ {(x, y) \in D _ {s}} - \log p _ {\theta_ {s}} \left(y \mid r _ {s} (x)\right).$$

The final program $p _ {s}$ packages the adapter $\theta _ {s}$, scaffold $r _ {s}$, original specification, and interpreter metadata. The public configuration usesa rank-64 LoRA adapter with alpha 16, an amortized-compiler warm start, and a 100-step cosine schedule.

3. System Architecture for Interactive Use

To make minute-scale compilation usable interactively, the system:

• Overlaps synthesis with training: Teacher requests, model loading, and training start concurrently; training begins as soon as first-batch examples are availableand blocks only if it catches up with synthesis.

• Coordinates jobs and reuses work: A persistent API record tracks each job;a shared queue dispatches to GPU workers;workers checka cache before requesting new teacher outputs;completed programs are stored centrally.

• Presents compilation asa background job: Users can continue browsing while the interface reports queue position and training progress, which persist across page reloads.

Empirical Validation / Results

Metric: LLM Exact Match (LEM)

Because multiple outputs can be semantically valid (e.g., JSON with different key orders), the authors use an LLM judge (GPT-5.5)to assess semantic correctness. The judge achieves 0.977 accuracy and Cohen's $\kappa = 0 . 9 4 6$ against 128 author labels.

Main Result: Correctness Improvement

On FuzzyBench-Hard (specifications where PAW's fast compiler produced no exact matches):

• Compile by training: 0.836 mean LEM (83.6% semantic accuracy)
• PAW fast compiler: 0.224 mean LEM
• Compile time: 50.9 seconds vs. 3.5 seconds

This representsa 0.612 absolute improvement in mean LEM at the cost of ~47 seconds additional compile time.

Supervision Choices

Table 1 reports controlled sweeps on development specifications:

Table 1: Mean LEM across controlled teacher-mixture and data-scaling sweeps. These results show that mixinga larger teacher improves accuracy (0.746→0.851), and that more unique training pairs monotonically improve performance up to 0.866 at 7200 pairs.

Latency and Scalability

• A cold compile ofa representative specification took 50.9 s on a B300, 68.2 s on an H200, and 99.2 s onan RTX GPU.
• Ina load test with four concurrent compile jobs, all completed witha mean queue wait of 1.01 s and even utilization across workers.
• Teacher synthesis dominates end-to-end latency, so overlapping synthesis with training directly shortens the critical path.

Application Validation

• Paw-helper (website helper): A deployed pack contains 30 compiled programs serving four websites; a program tree routes questions through classifiers, answerers, selectors, and validators alongside deterministic retrieval (BM25)and link handling.

• Avatar Director (3D avatar): A finetuned PAW program translates natural-language commands intoa small action DSL (supporting sequences, durations, repetition, parallel motions). On 44 hand-authored validation instructions, the program produced the expected action structure in 43 cases (97.7% accuracy).

• English–Claudish translator: Two separately compiled programs (one per direction)powera live bidirectional translation service. Between August 22 and September 2, 2026, the web demo completed 100,747 successful translation requests.

Theoretical and Practical Implications

Theoretical implications: The work introducesa new point in PAW's speed–accuracy tradeoff frontier. The fast amortized compiler generatesa LoRA adapter in a single forward pass (seconds), while compile by training spends roughlya minute to obtain substantially higher accuracy. This demonstrates that "additional compile-time investment produces better functions than one-shot weight prediction," validating the core hypothesis that adaptation can be treated asa software build step rather thana runtime dependency.

Practical implications: The system shows that independently compiled neural functions can serve as components in larger applications, composing with ordinary code. The authors articulatea clear division of labor: "compiled functions make fuzzy decisions, while ordinary code handles exact operations such as retrieval, caching, and branch control." This enables:

• Reusability: Downloaded .paw artifacts package the spec, adapter, and scaffold, enabling versioning, caching, and composition like ordinary software components.

• Privacy and cost efficiency: Local SDK execution does not send future inputs to PAW or teacher models, eliminating per-invocation API costs and network latency.

• Interactive usability: The minute-scale compile time is made tolerable through background job processing, progress persistence, and overlapping synthesis with training, demonstrating that "a minute-scale build remain usable in an interactive service."

Conclusion

The paper introduces compile by training, a method that compilesa neural program from a user-provided natural-language function description. It uses large language models to synthesize examples, then fine-tunesa LoRA adapter to specializea smaller shared interpreter model. The key findings are:

1. Higher accuracy: On FuzzyBench-Hard, compile by training reaches 83.6% semantic accuracy versus 22.4% for the fast compiler, demonstrating the value of additional compile-time computation.

2. Practical trade-off: The ~1-minute compile time is acceptable for interactive use, especially given the zero per-invocation cost and local execution at run time.

3. Composability: The resulting programs work as components in real applications—a multi-site website helper, a 3D avatar controller, and a bidirectional translator with over 100k requests—showing that independently compiled neural functions can integrate with ordinary software architectures.

Future directions and limitations include: synthetic supervision may inherit teacher errors; applications requiring guaranteed correctness should validate outputs or retain deterministic control paths; and systematic user studies of the compiled functions remain as future work. The authors also note that the public service combines multiple teacher models and that data scaling (up to 7200 pairs)continues to improve accuracy, suggesting further gains from larger synthesized datasets are possible.