How much of the weight-space perception gap is actually symmetry? Evidence from ~1.8M fitted SIRENs [R]

I’ve been looking at a fairly basic question in weight-space learning that I don’t think gets separated cleanly enough: Why does reading semantics directly from neural network weights work pretty well when the networks share an initialization, but collapse when the networks are fitted independently? The usual explanation is parameter symmetry. Permute hidden units, flip equivalent signs, etc., and two parameter vectors can represent the same function while looking completely different to a downstream model. But there are actually several different claims hiding in that explanation: the parameterization has a symmetry group, accounting for that symmetry improves weight-space prediction, the symmetry is actually sufficient to explain the observed degradation between shared-init and independently fitted networks. Those aren’t equivalent, so I tried to measure them separately. The setting is SIREN-style implicit neural representations. For a hidden sine neuron, the relevant function-preserving transformations generate the infinite dihedral group D_inf = Z semidirect_product Z_2 and including neuron permutations gives the layer action D_inf wr S_n. For one hidden layer, I prove generic identifiability modulo this group using the distributional Fourier transform of the realized function. Roughly, the Fourier transform becomes an atomic measure supported at the incoming frequencies +/- w_i, which lets you recover the parameters up to exactly the D_inf wr S_n action under explicit genericity conditions. One consequence is that this isn’t just the usual permutation/sign story. Integer-pi phase transformations are affine rather than linear, so they aren’t captured by symmetry descriptions restricted to monomial matrix actions. At depth two things get more annoying because a neuron’s outgoing weights are simultaneously acted on by the next layer. I ended up constructing exact cross-layer invariants by coupling the layers through the second-layer Gram matrix instead of treating neurons independently. The empirical part then uses roughly 1.8 million fitted INRs across MNIST, FashionMNIST, and CIFAR-10, with controlled protocols separating shared initialization, optimization stochasticity, and independent initialization. The result I found most interesting: Randomizing only the exact symmetry group, while keeping each network’s represented function fixed, destroys 79.1 of the 80.4 accuracy points in the MNIST shared-init vs. random-init gap. I want to be careful about the interpretation here. This establishes sufficiency: symmetry scatter alone can reproduce almost the entire degradation. It does not establish that 79.1 / 80.4 of the naturally occurring gap is causally mediated by symmetry. Those are different estimands. Breaking the group apart, sign flips account for roughly 63 points of that induced loss, neuron relabeling about 15, and integer phase shifts about 1. There was another result that changed my interpretation of the problem quite a bit. A reader that directly quotients the D_inf wr S_n structure on the raw parameters reaches 0.917, compared with: 0.628 for the best orbit-valued reframing, 0.526 for the same reader family over a fixed invariant encoding, 0.265 for a permutation-equivariant baseline. But when I FLOPs-match weight-space inference against simply querying the INR as a function, the function-space route is still much better: 95.3% at 1.6 MFLOP using 64 learned query coordinates versus 64.4% at 5.5 MFLOP for the best weight-space rung on that frontier. That leads to what I think is the more interesting conceptual question: If a complete invariant is informationally equivalent to access to the realized function, then the strongest justification for operating directly in weight space may ultimately have to be computational rather than informational. Everything is public here: https://github.com/ITheClixs/project-siren-gap The repo includes the paper, implementation, tests, pre-registrations, lab notebook, prediction ledger, claims ledger, and experimental results. I’d particularly appreciate criticism on three things: whether the sufficiency/mediation distinction is being drawn correctly, whether anyone sees a counterexample or missing assumption in the one-hidden-layer maximality argument, whether there is related work on affine symmetry groups of periodic-activation networks that I’m missing. Also very interested in attempts to break the invariants or reproduce the group-randomization result. If something here is wrong, I’d rather find out from someone trying to kill it. submitted by /u/ITheClixs [link] [comments]

Source: r/MachineLearning | 2026-08-19