Research
Compute-Optimal Scaling Laws for Spiking Language Models
arXiv:2610.15146v1 Announce Type: new Abstract: Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking la
arXiv:2610.15146v1 Announce Type: new Abstract: Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale. We present the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from 3imes10^{17} to 10^{19} FLOPs on FineWeb-EDU. We report two findings. First, the compute-optimal frontier is N_{opt} propto C^{0.56}, close to the 0.50 that Hoffmann et al. report for dense Transformers, so spiking does not change how parameters and data should be traded off under a compute budget. Second, against non-spiking RWKV and Transformer baselines trained the same way, the loss gap splits into a backbone cost of about 0.16 nats that does not change with compute and a spiking cost that falls from 0.34 to about 0.23 nats and then levels off. These results give the first compute-optimal recipe for spiking language models and quantify the accuracy cost of their sparsity.
Related
- The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy
- Multi-Timescale Conductance Spiking Networks: A Sparse, Gradient-Trainable Framework with Rich Firing Dynamics for Enhanced Temporal Processing
- Criticality-Constrained Iterative Pruning for Energy-Efficient Spiking Neural Networks via Combined Importance Scoring
- Large Language Models with At Most One Spike per Neuron
Source: arXiv cs.LG | 2026-10-14