SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

arXiv:2608.22354v1 Announce Type: cross Abstract: Delta-Rule recurrent models maintain a fixed-size state, enabling O(1) inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: extbf{localized norm explosion atop a relatively sparse substrate}, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose extbf{State Anomaly Neutralization (SANE)}, which applies adaptive anh compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range (3 le alpha le 5), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over 24{,}000imes, SANE retains functional reasoning (33.46--35.56) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds (alpha ge 8) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.

Source: arXiv cs.AI | 2026-08-25