Toward a First-Principles Update Geometry for the Language-Model Head

arXiv:2608.22253v1 Announce Type: new Abstract: We study the language-model head and softmax as a single module, deriving an update geometry from their composition rather than from the weight matrix in isolation. Under Hilbert's projective distance, the maximum change caused by an update S over left|left|{h}right|right|2le H is Hmax{i<j}left|left|{s_i-s_j}right|right|_2, which is H times the Euclidean diameter of its token rows. Motivated by Muon's singular-value conditioning, we propose maximizing the smallest row separation while constraining this diameter, producing an approximate-equidistance problem when Vgg d.

Source: arXiv cs.LG | 2026-08-25