Summary (Overview)

• This paper presents the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from $3 \times 10^{17}$ to $1 \times 10^{19}$ FLOPs on FineWeb-EDU.
• The compute-optimal frontier is found to be $N_{opt} \propto C^{0.56}$, close to the 0.50 reported by Hoffmann et al. [2022] for dense Transformers, indicating spiking does not fundamentally alter the optimal parameter-data tradeoff.
• The loss gap against non-spiking baselines splits into a constant backbone cost (~0.16 nats) and a spiking cost that falls from 0.34 to ~0.23 nats before leveling off.
• The apparent departure from the Chinchilla exponent is shown to be largely a parameter-counting artifact—the exponent varies from 0.75 (strict non-embedding) to 0.56 (total parameters) depending on counting convention.
• The frontier extrapolates out-of-sample to a hero run at $C = 10^{20}$ FLOPs (predicted 2.78 nats vs. measured 2.76 nats).

---

Introduction and Theoretical Foundation

Background and Motivation

Scaling laws address a fundamental allocation question: given a fixed compute budget $C$, how should it be split between model size $N$ and data $D$ to minimize loss? Kaplan et al. [2020] established power-law scaling for Transformers, while Hoffmann et al. [2022] (Chinchilla) introduced the IsoFLOP methodology and revised the optimal allocation toward roughly equal scaling of $N$ and $D$.

The key open question this paper addresses: How do these laws transfer to spiking neural networks (SNNs)? SpikeGPT [Zhu et al., 2023] binarizes activations into spikes on an RWKV backbone [Peng et al., 2023] for energy-efficient neuromorphic inference, but was trained far below the regime where compute-optimal behavior is characterized.

Theoretical Foundation

• Spiking neurons follow the standard leaky integrate-and-fire (LIF) model [Gerstner and Kistler, 2002, Maass, 1997], with the arctangent (arctan) surrogate gradient [Neftci et al., 2019] enabling backpropagation through the non-differentiable spiking threshold.
• RWKV [Peng et al., 2023] replaces quadratic self-attention with a linear-time recurrence, removing the sequence-length term that complicates transformer FLOP accounting.
• Compute estimation follows the Kaplan et al. [2020] approximation: $C \approx 6ND$. Since RWKV has no attention term, matrix multiplies dominate.

Parameter Counting Conventions

With a 50,257-token vocabulary, the embedding is 11–50% of all parameters, making the counting convention critical. Three conventions are distinguished:

where $l$ = depth, $d$ = width, $v$ = vocabulary size. Inverting a parameter budget to a width $d$ requires solving a cubic (Appendix E).

---

Methodology

Experimental Settings

• Optimizer: AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.999$)
• Batch size: 16
• Context length: 1024 tokens
• Tokenizer: EleutherAI/gpt-neox-20b [Black et al., 2022]
• Data: FineWeb-EDU sample-100BT split [Penedo et al., 2024]
• Learning rate: Power law $LR_{opt}(d) = k d^{-p}$ fit to a pilot sweep, with cosine decay to one-tenth—ensuring loss differences reflect the $N/D$ allocation rather than LR tuning.

IsoFLOP Methodology

Following Hoffmann et al. [2022] Approach 2, the authors:

1. Define a grid of compute budgets $C$ and model widths $d$
2. Train models single-epoch to the token budget implied by their compute tier and width
3. Fit parabolas to loss vs. log $N$ profiles at each compute tier
4. Recover compute-optimal $(N_{opt}, D_{opt})$ at each budget from parabola vertices

Baseline Comparisons

To measure the cost of spiking, the authors repeat the IsoFLOP grid for:

• Continuous-valued RWKV (non-spiking)
• Standard softmax-attention Transformer

All baselines use the same data, batch size, learning-rate schedule, and evaluation, tuning only the learning rate per architecture.

---

Empirical Validation / Results

IsoFLOP Profiles

Every compute tier produces the U-shaped profile in log $N$ that the IsoFLOP method assumes, with an interior optimum well inside the sampled widths at all four compute budgets. The parabolas flatten as compute grows (leading quadratic coefficient $0.148 \rightarrow 0.057$ from $3 \times 10^{17}$ to $1 \times 10^{19}$), so the vertex is less tightly determined at high compute.

Compute-Optimal Frontier

Log-log regression over the four compute budgets ($R^2 = 0.996$) yields:

$$N_{T, opt}(C) = 1.82 \times 10^{-2} C^{0.556}, \quad D_{opt}(C) \propto C^{0.444}$$

Since the grid is defined by $C \equiv 6ND$, $\beta = 1 - \alpha$ is not estimated separately.

Out-of-sample extrapolation: A compute-optimal model trained at $C = 10^{20}$ FLOPs ($N_T \approx 2.44$B, width 2048/depth 41, $D \approx 7.1$B tokens) reaches $L \approx 2.76$ nats vs. the predicted $L^* \approx 2.78$ nats—the frontier holds out of sample.

Parameter-Counting Artifact

Table 1: Compute-optimal exponent $\alpha$ ($N_{opt} \propto C^\alpha$) under three parameter-counting conventions

The exponent $\alpha$ moves from 0.75 (near Kaplan's 0.73) through 0.62 to 0.56 (near Chinchilla's 0.5), spanning essentially the entire Kaplan–Chinchilla gap on a single dataset and architecture at matched fit quality ($R^2 \approx 0.996$).

Sensitivity Analysis

• Jackknife (leave-one-group-out): $\alpha_{total} = 0.556$ with range $[0.532, 0.615]$ (mean $0.565 \pm 0.053$)
• Seed variance: ≤ 0.003 nats, shifting the exponent by < 0.01
• Tier count sensitivity: 2, 3, and 4 tiers give 0.454, 0.537, and 0.556—more tiers are the main way to tighten the estimate
• Independent cross-check: Chinchilla Approach 3 parametric fit ($L = E + A/N^a + B/D^b$) reproduces per-convention exponents to within 0.04 and estimates irreducible loss $E \approx 2.7$ nats

Comparison to Non-Spiking Baselines

Table 2: Loss decomposition at each compute budget

• Backbone cost stays between 0.15 and 0.17 nats at every budget—it does not change with compute.
• Spiking cost falls from 0.338 to 0.227 nats between $3 \times 10^{17}$ and $3 \times 10^{18}$ FLOPs, then holds at 0.225 at $10^{19}$. With four budgets, the authors cannot determine whether it has stopped falling or is only falling more slowly.
• Spike rates depend on training progress rather than model size, rising toward a ~0.36 firing fraction as tokens-per-parameter grows (Appendix H).

---

Theoretical and Practical Implications

Theoretical Implications

1. Spiking does not alter scaling behavior: The compute-optimal exponent $\alpha \approx 0.56$ (total parameters) is close to Chinchilla's 0.5 for dense Transformers, suggesting the fundamental scaling dynamics of language modeling are preserved under spiking constraints.

2. Parameter counting is critical: The finding that $\alpha$ spans the entire Kaplan–Chinchilla gap (0.75 to 0.56) purely from counting convention reinforces Pearce and Song [2024]'s argument and implies that small-scale IsoFLOP exponents should not be compared across studies without matching the counting convention.

3. Decomposable loss gap: The separation of the total loss gap into a constant backbone cost and a decreasing-then-leveling spiking cost provides a principled framework for understanding where accuracy is lost in spiking models.

Practical Implications

1. Compute-optimal recipe for spiking LMs: Practitioners can now allocate compute between parameters and data for spiking language models following $N_{T, opt} \propto C^{0.556}$.

2. Quantified sparsity cost: The ~0.23 nats spiking cost at $10^{19}$ FLOPs provides a concrete accuracy budget for energy-efficiency decisions—practitioners can now make informed tradeoffs between accuracy and the energy savings of neuromorphic inference.

3. Backbone choice matters: The constant 0.16 nats backbone cost suggests that architectural improvements to RWKV (rather than spiking mechanisms) may be a more fruitful direction for closing the gap to Transformers at scale.

---

Conclusion

Under total-parameter counting, the compute-optimal frontier of a spiking language model is:

$$N_{T, opt} \propto C^{0.556}$$

close to dense Transformers' 0.5; larger non-embedding exponents come from parameter counting, not the architecture. About 0.16 nats of the gap to a Transformer comes from the RWKV backbone and does not change with compute, while the cost of spiking falls to about 0.23 nats by $10^{19}$ FLOPs.

Limitations and Future Directions

• Results come from four tiers between $3 \times 10^{17}$ and $10^{19}$ FLOPs, far below Chinchilla's $10^{21}$–$10^{24}$ range.
• Single dataset (FineWeb-EDU only).
• Aspect ratio $d/l$ drifts from 38 to 53 across widths.
• Baselines use one seed per configuration—whether the spiking cost stays near 0.23 nats at larger scale is untested.
• More tiers beyond the $10^{20}$-FLOP point would test both the power law and whether the spiking cost continues to decrease.