I scaled a pure Spiking Neural Network (SNN) to 1.088B parameters from scratch. Ran out of budget, but here is what I found [R]

A researcher on r/MachineLearning documented an independent attempt to scale a pure Spiking Neural Network (SNN) to 1.088 billion parameters, built from scratch, exploring whether SNNs can achieve performance competitive with conventional large language models at a comparable parameter count. The post shares empirical findings from the training run despite being cut short due to budget constraints, offering rare real-world data on the challenges of scaling SNNs — including their well-known difficulties with training stability, surrogate gradient methods, and computational cost. This represents a notable community-driven contribution to the open question of whether biologically inspired, spike-based architectures can be viably scaled to the billion-parameter regime.

'There's a new generation of empirical deep learning researchers, hacking away at whatever seems trendy, blowing with the wind' [D]
AI/ML Algorithm Simulation & Visualization Tool [Project]
[[n-ama-announcement-max-welling-vaes-gnns-ai4science-cuspai|[N] AMA Announcement: Max Welling (VAEs, GNNs, AI4Science & CuspAI)]]
FlashAttention (FA1–FA4) in PyTorch - educational implementations focused on algorithmic differences [P]

Source: r/MachineLearning | 2026-04-13

Related