The EDelta-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality

arXiv:2605.06729v1 Announce Type: cross Abstract: We present the EDelta-MHC-Geo Transformer, a novel architecture that unifies Manifold-Constrained Hyper-Connections (mHC), Deep Delta Learning (DDL), and the Cayley transform to obtain input-adaptive, unconditionally orthogonal residual connections. Unlike DDL, whose Householder operator is orthogonal only at eta in {0,2}, our Data-Dependent Cayley rotation Q(x)=(I+(eta/2)A(x))^{-1}(I-(eta/2)A(x)) preserves orthogonality for all eta and all inputs. To handle negation, an eigenvalue -1 case that Cayley provably excludes, we introduce the EDelta-MHC-Geo Hybrid, which combines Cayley rotation with Householder reflection via a learned operator-selection gate X'=gamma(X)Q(X)X+(1-gamma(X))H_2(X)X. A midpoint-collapse regularizer, 4gamma(1-gamma), encourages boundary gate decisions, where each selected component is orthogonal. In matched-parameter comparisons, with approximately 1.79M parameters per model and mean +/- standard deviation over 3 seeds, against four baselines including the concurrent JPmHC, EDelta-MHC-Geo achieves the best long-horizon stability, 1.9x over JPmHC and 3.8x over GPT; the best near-pi rotation loss, 4.5x over JPmHC on single-plane; strong norm preservation, with 0.001 mean deviation; and 0.96 negation cosine alignment in a diagnostic reflection probe, all with 33% fewer layers. While JPmHC's wider representation excels on pure rotation, its finite Cayley residual mixer excludes an exact lambda=-1 operator and has no reflection branch, motivating our hybrid approach for accessing both connected components of O(n).

Source: arXiv cs.AI | 2026-05-11