Model Releases
KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches
arXiv:2505.14777v2 Announce Type: replace-cross Abstract: The design of effective optimization algorithms for neural networks remains a fundamental challenge, and most existing methods rely on heurist
arXiv:2505.14777v2 Announce Type: replace-cross Abstract: The design of effective optimization algorithms for neural networks remains a fundamental challenge, and most existing methods rely on heuristic extensions of gradient-based updates. We introduce KO (Kinetics-inspired Optimizer), a plug-and-play optimization module grounded in kinetic theory and partial differential equations. KO models parameter dynamics as a particle system, augmenting standard gradient updates with stochastic interactions induced by a discretization of the Boltzmann transport equation. This mechanism naturally promotes parameter diversity and mitigates weight condensation, the tendency of parameters to collapse into low-dimensional subspaces, a phenomenon closely associated with degraded generalization. We provide both a rigorous theoretical analysis and a physical interpretation, showing that KO provably increases parameter diversity while preserving convergence guarantees. Extensive experiments on image classification benchmarks (CIFAR-10/100, ImageNet) and large-scale language model pretraining demonstrate that KO consistently improves accuracy over competitive baselines with negligible additional computational cost.
Related
- TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classification
- naPINN: Noise-Adaptive Physics-Informed Neural Networks for Recovering Physics from Corrupted Measurement
- A Physics-Inspired Optimizer: Velocity Regularized Adam
Source: arXiv cs.AI | 2026-10-07