The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction

arXiv:2610.06653v1 Announce Type: new Abstract: Hyper-connections widen the residual stream of a Transformer to n parallel streams. Their manifold-constrained version (mHC) mixes the streams at each layer with a doubly stochastic matrix, which it computes by Sinkhorn normalization of exponentiated logits. We give a geometric theory of this design on the Birkhoff polytope. First, a doubly stochastic mixer splits the stream into a mean channel, on which mHC is exactly a residual network, and a difference channel, which each layer contracts by its second singular value sigma_2 le 1 - n min_{ij} H_{ij}. Thus the extra width is a fading memory with a horizon of 1/(1-sigma_2) layers, and among nonnegative mixers only the permutations do not collapse. Second, the Sinkhorn-logit map is a global chart, and its logit gradient is exactly the Fisher-Rao gradient. Thus logit gradient flow follows a squared Fisher-Rao metric, and the straight-through update is exactly entropic mirror descent. Third, under logit gradient flow the logarithm of each entry moves at a rate of at most 4n^3|nabla f|_infty arepsilon, where arepsilon is the distance to the nearest permutation. Thus gradient flow approaches and leaves the vertices only at rate 1/t, but mirror descent moves at an exponential rate. Fourth, the local convergence factor of Sinkhorn is sigma_2^2, so a fixed iteration budget limits the horizon. Experiments confirm the predicted rates.

Related

• TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes
• Spectral-Sphere-Constrained Hyper-Connections
• Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation

Source: arXiv cs.LG | 2026-10-06