A Formal Kinetic Theory for Zeroth-Order Newton Dynamics:Stein-Corrected Hessian Estimation and Curvature--Variance Trade-offs

arXiv:2607.22567v1 Announce Type: cross Abstract: Zeroth-order Newton-type methods are useful when gradients and Hessians are unavailable, but they behave quite differently from first-order gradient-free methods. We develop a kinetic framework for algorithms that estimate both gradient and Hessian from black-box function values. The naive random-direction Hessian estimator turns out to be biased even on quadratics; a Gaussian--Stein correction is needed to estimate the Hessian of the Gaussian-smoothed objective. Linearizing the inverse Hessian exposes two noise channels: gradient noise preconditioned by the inverse Hessian, and Hessian noise transmitted through an inverse-Hessian sandwich. Under a noisy oracle the second channel carries the second-difference factor mu_H^{-4}. A small-mass kinetic lift links the finite-step Newton update to an underdamped phase-space model; the overdamped spatial limit yields a Lyapunov bound that exposes the curvature--variance trade-off between step size, batch sizes, smoothing radii, and regularization. Numerical experiments confirm estimator identities, the gradient and Hessian variance laws, dimension scaling, inverse-perturbation accuracy, and optimization behavior under query-budget and regularization ablations.

Related

• Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network
• AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
• Wall-Clock Complexity for Zeroth-Order Optimization with Tunable Oracle Fidelity
• New Insight of Variance reduce in Zero-Order Hard-Thresholding: Mitigating Gradient Error and Expansivity Contradictions

Source: arXiv cs.AI | 2026-07-28