Sharp Rates and a One-Line Correction for Spectral Representation Learning

arXiv:2609.15825v1 Announce Type: new Abstract: A self-supervised encoder is trained once, frozen, and reused through lightweight probes on tasks nobody named at training time; the practitioner's question is when the off-the-shelf features are good enough and when they need fixing. Canonical correlation analysis, HGR maximal correlation, and the population optimum of the spectral contrastive loss all return the top-k singular subspace of a cross-view dependence operator, justified by isotropy: if the task prior has no directional preference, that subspace is universally optimal. We show isotropy is the wrong hypothesis. The prior enters the transfer risk only through the task covariance Lambda=E[DeltaDelta^op], and only through its compression onto the operator's leading singular directions; what matters is not whether Lambda is isotropic but whether its preferred directions are ordered consistently with the operator's spectrum. We prove matching two-sided rates---worst-case regret is exactly 1-1/kappa(Lambda), refines to 1-A_k for an alignment coefficient A_k, localizes to the top-2k subspace, becomes second order under a spectral gap, and is improvable by no task-agnostic representation---and show why alignment is generic: incoherent preferences cancel in high dimension, and T diverse tasks force alpha=widetilde O(sqrt{d_x/T}), a quantitative account of why task diversity, not symmetry, makes self-supervised features transfer. The governing statistics cost O(kd_x^2), and when they signal misalignment a one-line reweighting of the positive-pair term provably restores exact optimality. The result is a diagnostic that answers the practitioner's question from a small labelled budget and refuses when the task bank cannot support the width requested; on controlled data it takes a regret of 0.86 down to 0.003, and on a CIFAR-100 encoder it correctly predicts that no correction is needed.

Source: arXiv cs.LG | 2026-09-15