# Compute-Optimal Scaling Laws for Spiking Language Models

> Compute-optimal scaling for spiking language models matches dense Transformers (exponent 0.56 vs 0.50), with the apparent deviation explained by parameter-counting conventions.

- **Source:** [arXiv](https://arxiv.org/abs/2610.15146)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/5jKM4N
- **Whiteboard:** https://picx.dev/p/5jKM4N/image

## Summary

## Summary (Overview)

- This paper presents the **first Chinchilla-style IsoFLOP scaling study of a spiking language model**, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from $3 \times 10^{17}$ to $1 \times 10^{19}$ FLOPs on FineWeb-EDU.
- The compute-optimal frontier is found to be $N_{opt} \propto C^{0.56}$, **close to the 0.50 reported by Hoffmann et al. [2022] for dense Transformers**, indicating spiking does not fundamentally alter the optimal parameter-data tradeoff.
- The loss gap against non-spiking baselines splits into a **constant backbone cost (~0.16 nats)** and a **spiking cost that falls from 0.34 to ~0.23 nats** before leveling off.
- The apparent departure from the Chinchilla exponent is shown to be **largely a parameter-counting artifact**—the exponent varies from 0.75 (strict non-embedding) to 0.56 (total parameters) depending on counting convention.
- The frontier **extrapolates out-of-sample** to a hero run at $C = 10^{20}$ FLOPs (predicted 2.78 nats vs. measured 2.76 nats).

---

## Introduction and Theoretical Foundation

### Background and Motivation

Scaling laws address a fundamental allocation question: given a fixed compute budget $C$, how should it be split between model size $N$ and data $D$ to minimize loss? Kaplan et al. [2020] established power-law scaling for Transformers, while Hoffmann et al. [2022] (Chinchilla) introduced the IsoFLOP methodology and revised the optimal allocation toward roughly equal scaling of $N$ and $D$.

The key open question this paper addresses: **How do these laws transfer to spiking neural networks (SNNs)?** SpikeGPT [Zhu et al., 2023] binarizes activations into spikes on an RWKV backbone [Peng et al., 2023] for energy-efficient neuromorphic inference, but was trained far below the regime where compute-optimal behavior is characterized.

### Theoretical Foundation

- **Spiking neurons** follow the standard **leaky integrate-and-fire (LIF) model** [Gerstner and Kistler, 2002, Maass, 1997], with the **arctangent (arctan) surrogate gradient** [Neftci et al., 2019] enabling backpropagation through the non-differentiable spiking threshold.
- **RWKV** [Peng et al., 2023] replaces quadratic self-attention with a linear-time recurrence, removing the sequence-length term that complicates transformer FLOP accounting.
- **Compute estimation** follows the Kaplan et al. [2020] approximation: $C \approx 6ND$. Since RWKV has no attention term, matrix multiplies dominate.

### Parameter Counting Conventions

With a 50,257-token vocabulary, the embedding is 11–50% of all parameters, making the counting convention critical. Three conventions are distinguished:

| Convention | Formula | Description |
|---|---|---|
| Strict non-embedding (Kaplan) | $13ld^2$ | Excludes all embedding parameters |
| FLOP-matmul (grid build) | $13ld^2 + vd$ | Includes output projection (enters $C = 6ND$) |
| Total (Chinchilla, primary) | $13ld^2 + 2vd$ | Includes both input and output embeddings |

where $l$ = depth, $d$ = width, $v$ = vocabulary size. Inverting a parameter budget to a width $d$ requires solving a cubic (Appendix E).

---

## Methodology

### Experimental Settings

- **Optimizer**: AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.999$)
- **Batch size**: 16
- **Context length**: 1024 tokens
- **Tokenizer**: EleutherAI/gpt-neox-20b [Black et al., 2022]
- **Data**: FineWeb-EDU sample-100BT split [Penedo et al., 2024]
- **Learning rate**: Power law $LR_{opt}(d) = k d^{-p}$ fit to a pilot sweep, with cosine decay to one-tenth—ensuring loss differences reflect the $N/D$ allocation rather than LR tuning.

### IsoFLOP Methodology

Following Hoffmann et al. [2022] Approach 2, the authors:
1. Define a grid of compute budgets $C$ and model widths $d$
2. Train models single-epoch to the token budget implied by their compute tier and width
3. Fit parabolas to loss vs. log $N$ profiles at each compute tier
4. Recover compute-optimal $(N_{opt}, D_{opt})$ at each budget from parabola vertices

### Baseline Comparisons

To measure the cost of spiking, the authors repeat the IsoFLOP grid for:
- **Continuous-valued RWKV** (non-spiking)
- **Standard softmax-attention Transformer**

All baselines use the same data, batch size, learning-rate schedule, and evaluation, tuning only the learning rate per architecture.

---

## Empirical Validation / Results

### IsoFLOP Profiles

Every compute tier produces the **U-shaped profile in log $N$** that the IsoFLOP method assumes, with an interior optimum well inside the sampled widths at all four compute budgets. The parabolas flatten as compute grows (leading quadratic coefficient $0.148 \rightarrow 0.057$ from $3 \times 10^{17}$ to $1 \times 10^{19}$), so the vertex is less tightly determined at high compute.

### Compute-Optimal Frontier

Log-log regression over the four compute budgets ($R^2 = 0.996$) yields:

$$N_{T, opt}(C) = 1.82 \times 10^{-2} C^{0.556}, \quad D_{opt}(C) \propto C^{0.444}$$

Since the grid is defined by $C \equiv 6ND$, $\beta = 1 - \alpha$ is not estimated separately.

**Out-of-sample extrapolation**: A compute-optimal model trained at $C = 10^{20}$ FLOPs ($N_T \approx 2.44$B, width 2048/depth 41, $D \approx 7.1$B tokens) reaches $L \approx 2.76$ nats vs. the predicted $L^* \approx 2.78$ nats—the frontier holds out of sample.

### Parameter-Counting Artifact

**Table 1: Compute-optimal exponent $\alpha$ ($N_{opt} \propto C^\alpha$) under three parameter-counting conventions**

| Convention | $N$ | $\alpha$ (IsoFLOP) | $\alpha$ (Approach 3) | $D/N$ ($3 \times 10^{17} \rightarrow 10^{19}$) |
|---|---|---|---|---|
| Strict non-embedding (Kaplan) | $13ld^2$ | 0.748 | 0.745 | $25.1 \rightarrow 4.7$ |
| FLOP-matmul (grid build) | $13ld^2 + vd$ | 0.624 | 0.644 | $9.2 \rightarrow 4.1$ |
| Total (Chinchilla, primary) | $13ld^2 + 2vd$ | 0.556 | 0.595 | $4.9 \rightarrow 3.5$ |

The exponent $\alpha$ moves from 0.75 (near Kaplan's 0.73) through 0.62 to 0.56 (near Chinchilla's 0.5), **spanning essentially the entire Kaplan–Chinchilla gap on a single dataset and architecture** at matched fit quality ($R^2 \approx 0.996$).

### Sensitivity Analysis

- **Jackknife** (leave-one-group-out): $\alpha_{total} = 0.556$ with range $[0.532, 0.615]$ (mean $0.565 \pm 0.053$)
- **Seed variance**: ≤ 0.003 nats, shifting the exponent by < 0.01
- **Tier count sensitivity**: 2, 3, and 4 tiers give 0.454, 0.537, and 0.556—more tiers are the main way to tighten the estimate
- **Independent cross-check**: Chinchilla Approach 3 parametric fit ($L = E + A/N^a + B/D^b$) reproduces per-convention exponents to within 0.04 and estimates irreducible loss $E \approx 2.7$ nats

### Comparison to Non-Spiking Baselines

**Table 2: Loss decomposition at each compute budget**

| Compute Budget | Backbone Cost (dense RWKV − Transformer) | Spiking Cost (spiking − dense RWKV) |
|---|---|---|
| $3 \times 10^{17}$ | 0.15–0.17 nats | 0.338 nats |
| $1 \times 10^{18}$ | 0.15–0.17 nats | 0.263 nats |
| $3 \times 10^{18}$ | 0.15–0.17 nats | 0.227 nats |
| $1 \times 10^{19}$ | 0.15–0.17 nats | 0.225 nats |

- **Backbone cost** stays between 0.15 and 0.17 nats at every budget—it does not change with compute.
- **Spiking cost** falls from 0.338 to 0.227 nats between $3 \times 10^{17}$ and $3 \times 10^{18}$ FLOPs, then holds at 0.225 at $10^{19}$. With four budgets, the authors cannot determine whether it has stopped falling or is only falling more slowly.
- **Spike rates** depend on training progress rather than model size, rising toward a ~0.36 firing fraction as tokens-per-parameter grows (Appendix H).

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Spiking does not alter scaling behavior**: The compute-optimal exponent $\alpha \approx 0.56$ (total parameters) is close to Chinchilla's 0.5 for dense Transformers, suggesting the fundamental scaling dynamics of language modeling are preserved under spiking constraints.

2. **Parameter counting is critical**: The finding that $\alpha$ spans the entire Kaplan–Chinchilla gap (0.75 to 0.56) purely from counting convention reinforces Pearce and Song [2024]'s argument and implies that **small-scale IsoFLOP exponents should not be compared across studies without matching the counting convention**.

3. **Decomposable loss gap**: The separation of the total loss gap into a constant backbone cost and a decreasing-then-leveling spiking cost provides a principled framework for understanding where accuracy is lost in spiking models.

### Practical Implications

1. **Compute-optimal recipe for spiking LMs**: Practitioners can now allocate compute between parameters and data for spiking language models following $N_{T, opt} \propto C^{0.556}$.

2. **Quantified sparsity cost**: The ~0.23 nats spiking cost at $10^{19}$ FLOPs provides a concrete accuracy budget for energy-efficiency decisions—practitioners can now make informed tradeoffs between accuracy and the energy savings of neuromorphic inference.

3. **Backbone choice matters**: The constant 0.16 nats backbone cost suggests that architectural improvements to RWKV (rather than spiking mechanisms) may be a more fruitful direction for closing the gap to Transformers at scale.

---

## Conclusion

Under total-parameter counting, the compute-optimal frontier of a spiking language model is:

$$N_{T, opt} \propto C^{0.556}$$

close to dense Transformers' 0.5; larger non-embedding exponents come from parameter counting, not the architecture. About 0.16 nats of the gap to a Transformer comes from the RWKV backbone and does not change with compute, while the cost of spiking falls to about 0.23 nats by $10^{19}$ FLOPs.

### Limitations and Future Directions

- Results come from **four tiers between $3 \times 10^{17}$ and $10^{19}$ FLOPs**, far below Chinchilla's $10^{21}$–$10^{24}$ range.
- **Single dataset** (FineWeb-EDU only).
- **Aspect ratio** $d/l$ drifts from 38 to 53 across widths.
- Baselines use **one seed per configuration**—whether the spiking cost stays near 0.23 nats at larger scale is untested.
- More tiers beyond the $10^{20}$-FLOP point would test both the power law and whether the spiking cost continues to decrease.

---

_Markdown view of https://picx.dev/p/5jKM4N, served by PicX — AI-generated visual whiteboard summaries of research papers._