Tuning Language Models for Robust Prediction of Diverse User Behaviors

arXiv:2505.17682v2 Announce Type: replace-cross Abstract: Predicting user behavior is essential for intelligent assistant services, yet deep learning models often struggle to capture long-tailed behaviors. Large language models (LLMs), with their pretraining on vast corpora containing rich behavioral knowledge, offer promise. However, existing fine-tuning approaches tend to overfit to frequent anchor'' behaviors, reducing their ability to predict less common tail'' behaviors. In this paper, we introduce BehaviorLM, a progressive fine-tuning approach that addresses this issue. In the first stage, LLMs are fine-tuned on anchor behaviors while preserving general behavioral knowledge. In the second stage, fine-tuning uses a balanced subset of all behaviors based on sample difficulty to improve tail behavior predictions without sacrificing anchor performance. Experimental results on two real-world datasets demonstrate that BehaviorLM robustly predicts both anchor and tail behaviors and effectively leverages LLM behavioral knowledge to master tail behavior prediction with few-shot examples.

Related

• A Comparative Study of Demonstration Selection for Practical Large Language Models-based Next POI Prediction
• Nirvana: A Specialized Generalist Model With Task-Aware Memory Mechanism
• Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition
• Understanding Generalization in Role-Playing Models via Information Theory

Source: arXiv cs.AI | 2026-04-14