The Automaton Underneath: The Additive Input Pathway Is a Parasitic Attractor for State Tracking in Householder Linear RNN

arXiv:2609.18966v1 Announce Type: new Abstract: Linear RNNs with input-dependent Householder-product transitions (DeltaNet/DeltaProduct-class) can provably represent hard state-tracking automata, yet trained models fail to length-generalize -- a gap recent work attributes to optimization, without a causal account. We give one, in a pre-registered, within-architecture causal ablation: the same model with one term deleted -- the additive input injection b_t = W_b e_t. With b_t, models fit length 32 and collapse out-of-distribution on parity, S_4, A_5, and non-solvable S_5 word problems (0.20 at position 512 on S_5). Without it -- input acting only through the orthogonal transitions -- the same architecture learns the exact automaton: median accuracy 1.00 at 16x the training length, at every width admitted by a representation law we state and test: the minimal number of Householder factors per token equals the maximal reflection length of the task's generators in the format-pinned representation (parity 1, S_4 3, A_5 and S_5 4); below it, nothing fits. The contrast with DeltaProduct's S_4/A_5 at n_h{=}2 (group-element classification, SO(3) realization) shows the law is representation-relative: task format is a hidden variable in state-tracking benchmarks. Two pre-registered arms locate the mechanism. (i) Initialized at a verified-exact solution with W_b{=}0, Adam grows the additive path and pulls the model off the exact solution; -b controls stay at 1.00. (ii) Our registered prediction that the fit routes through b_t fired its kill criterion: all 49 fitting seeds retain in-domain fit under W_b{:=}0 -- and at law-minimal width, zeroing W_b at inference restores exact generalization (parity 5/5, S_5 5/5, S_4 4/5, A_5 4/5). The additive pathway is parasitic: it destabilizes, then conceals, a correctly learned automaton. All 202 runs pre-registered; all numbers regenerate from artifacts.

Source: arXiv cs.LG | 2026-09-17