Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning

arXiv:2610.17453v1 Announce Type: new Abstract: We propose a continuous learning formulation for T non-forgetting tasks with the spherical parameter space S^{d-1}(sqrt{d}) in asymptotic regime drightarrow infty. Each task i has an optimum m{Theta}i, minimizing its corresponding loss function R_i(m{w}) on sphere. We reduce each task i to an acceptable loss region modeled by a geodesic ball B{m{Theta}i}(r_i(d)) on S^{d-1}(sqrt{d}) centered at m{Theta}i with radius r_i(d). To model task dependencies (similarities), task optima m{Theta}i are distributed by a Neyman-Scott clustered point process Phi. Cluster centers {m{Y}j}{j=1}^{N_p} form a homogeneous Poisson point process on sphere, while intra-cluster task optima follow a von Mises-Fisher distribution with mean m{Y}j and concentration kappa. Each task tin {1, ..., T} learning algorithm is considered to be a channel Q{t}(dm{w}t | m{w}{t-1}, K{t-1}(m{heta}{1:t-1}), m{heta}t), where m{W}t is the learned parameter of task t and K{t-1}(m{Theta}{1:t-1})=igcap{i=1}^{t-1}B_{m{Theta}i}(r_i(d)). Since in continual learning, the learner does not need to reach m{Theta}t exactly to successfully learn task t, to avoid forgetting as well as reliable learning, it only needs for the learned parameter vector m{W}{t} to enter B{m{Theta}t}(r_t(d)) ap K{t-1}(m{Theta}{1:t-1}). Using this model, we derive an upper bound on T ensuring P[K{T}(m{Theta}_{1:T}) neq emptyset] rightarrow 1 as drightarrow infty and kappa/d rightarrow alpha, where alpha>0 is a constant. Finally, for T tasks sampled uniformly from Phi, we derive an upper bound for total information theoretic capacity under the non-forgetting constraint for each task.

Source: arXiv cs.LG | 2026-10-14