Research
Measuring the stability and plasticity of recommender systems
arXiv:2508.03941v3 Announce Type: replace-cross Abstract: The typical offline protocol to evaluate recommendation algorithms is to collect a dataset of user-item interactions and then use a part of th
arXiv:2508.03941v3 Announce Type: replace-cross Abstract: The typical offline protocol to evaluate recommendation algorithms is to collect a dataset of user-item interactions and then use a part of this dataset to train a model, and the remaining data to measure how closely the model recommendations match the observed user interactions. This protocol is straightforward, useful and practical, but it only provides snapshot performance. We know, however, that online systems evolve over time. In general, it is a good idea that models are frequently retrained with recent data. But if this is the case, to what extent can we trust previous evaluations? How will a model perform when a different pattern (re)emerges? In this paper we propose a methodology to study how recommendation models behave when they are retrained. The idea is to profile algorithms according to their ability to, on the one hand, retain past patterns - stability - and, on the other hand, (quickly) adapt to changes - plasticity. We devise an offline evaluation protocol that provides detail on the long-term behavior of models, and that is agnostic to datasets, algorithms and metrics. To illustrate the potential of this framework, we present preliminary results of three different types of algorithms on the GoodReads dataset that suggest different stability and plasticity profiles depending on the algorithmic technique, and a possible trade-off between stability and plasticity. We further discuss the potential and limitations of the proposal and advance some possible improvements.
Related
- Collaborative Filtering Through Weighted Similarities of User and Item Embeddings
- A single algorithm for both restless and rested rotting bandits
- Is Sliding Window All You Need? An Open Framework for Long-Sequence Recommendation
- Learning Behaviorally Grounded Item Embeddings via Personalized Temporal Contexts
- ItemRAG: Item-Based Retrieval-Augmented Generation for LLM-Based Recommendation
Source: arXiv cs.LG | 2026-04-29