Mechanistic Evidence for Preserved-but-Misaligned Representations in Non-IID FedAvg

arXiv:2512.23043v2 Announce Type: replace Abstract: Federated Averaging (FedAvg) often degrades under non-IID client data, but it remains unclear whether this degradation reflects the loss of client-learned representations or a failure to use representations that are still present. We study this question mechanistically in sparse client-trained vision models, using dense-model controls to test whether the observed effects depend on sparsity. Our analysis combines class-specific circuit discovery, linear probing of frozen representations, head-only finetuning, and sparse feature dictionaries. Across CNN and ResNet models on CIFAR-10 and Fashion-MNIST, severe label skew can drive some per-class accuracies near zero even when class-specific internal structure remains recoverable. Linear probes substantially outperform the aggregated classifier, head-only finetuning partially restores accuracy, and USAE transfer reveals a largely shared feature basis between IID and non-IID models. Together, these diagnostics suggest that, in our setting, non-IID FedAvg degradation is not fully explained by representational erasure; it also reflects misalignment between preserved internal structure and the final prediction pathway. We release code, trained checkpoints, extracted circuits, and experiment logs at the following repository: href{https://github.com/ha405/FedMI/tree/icml}{FedMI}.

Related

• Client-Conditional Federated Learning via Local Training Data Statistics
• Mechanistic Independence: A Principle for Identifiable Disentangled Representations
• From Circuit Evidence to Mechanistic Theory: An Inductive Logic Approach

Source: arXiv cs.LG | 2026-07-15