BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes

arXiv:2509.15974v2 Announce Type: replace Abstract: Fine-tuning the bias terms of large language models (LLMs) has the potential to achieve unprecedented parameter efficiency while maintaining competitive performance, particularly in low-data regimes. However, the link between fine-tuning different bias terms (i.e., oldsymbol{b}_q, oldsymbol{b}_k, and oldsymbol{b}_v in the query, key, or value projections) and downstream performance remains largely unclear to date. In this paper, we investigate the link between fine-tuning oldsymbol{b}_q, oldsymbol{b}_k, and oldsymbol{b}_v with the performance of the downstream task. Our key finding is that directly fine-tuning oldsymbol{b}_v generally leads to higher downstream performance in low-data regimes, in comparison to oldsymbol{b}_q and oldsymbol{b}_k. We extensively evaluate this unique property across a wide range of LLMs spanning encoder-only and decoder-only architectures up to 6.7B parameters (including bias-free LLMs). Our results provide strong evidence for the effectiveness of directly fine-tuning oldsymbol{b}_v across various downstream tasks. The implementation code is available at https://github.com/whubaichuan/BEFT.

Source: arXiv cs.CL | 2026-04-21