Research
Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers
arXiv:2604.23046v1 Announce Type: new Abstract: We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners
arXiv:2604.23046v1 Announce Type: new Abstract: We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners for their ability to handle the data deletion task with varying degrees of eigendecomposition to mimic the loss model memory. While both first and second-order methods realign with the ideal counterfactul in terms of performance and gradient, the second-order optimizer shows significant volatility in the optimizer state. This indicates residual information, supposedly deleted, that isn't detectable by first-order analysis. Various eigendecay treatments show that stability and information loss is regained only under controlled state pertubation where geometric information (or memory) is erased.
Related
- The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
- Mitigating Privacy Risk via Forget Set-Free Unlearning
- Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning
- Towards Reliable Testing of Machine Unlearning
- WIN-U: Woodbury-Informed Newton-Unlearning as a retain-free Machine Unlearning Framework
Source: arXiv cs.LG | 2026-04-28