How Shared Representations Become Specialized: Formation, Maintenance, and Latent Susceptibility

arXiv:2610.15830v1 Announce Type: new Abstract: When several predictions use the same internal representation, information about some targets can remain easier to recover than information about others. We study how this specialization forms, persists, and changes in minimal artificial learners. With reconstruction errors weighted equally during learning, changing which target is observed precisely selects the favored target and redirects an already specialized learner. Removing current support can reduce a formed bias, whereas state-following update feedback can maintain it. We then compare learners formed through different observation and update conditions but matched on specified reconstruction measurements. Both change under the same brief opposing intervention, yet their responses differ. Resetting Adam largely removes the ten-update difference; exchanging its second moment reverses which route is more susceptible. New interventions show that equalizing the mean second-moment scaling reduces but does not remove the difference: its coordinate-wise pattern also matters. With fresh Adam or calibrated SGD, smaller route differences emerge later, whereas 500 common opposing updates bring the selected allocation endpoints close. A further control produces response differences with identical acquisition and equal mean loss weights but jointly different weight variability and realized sequence. Acquisition-selected formation also extends to a symmetric three-target setting. These interventions distinguish specialization's formation, present support, and time-dependent latent susceptibility.

Related

• Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
• The Trade-off Between Covariate Dependence and Latent Structure in Representation Learning
• Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning
• Making Tensor Decoupling More Robust and Less Parametric
• The Platonic Universe: Do Foundation Models See the Same Sky?

Source: arXiv cs.LG | 2026-10-14