Ultra-Low-Dimensional Prompt Tuning via Random Projection

arXiv:2502.04501v3 Announce Type: replace Abstract: Large language models achieve state-of-the-art performance but are increasingly costly to fine-tune. Prompt tuning is a parameter-efficient fine-tuning method that addresses parameter-efficiency by learning prompt embeddings, but these embeddings are typically tied to the model's hidden dimensionality, limiting parameter saving. In this paper, we propose Ultra-Low-dimensional Prompt Tuning (ULPT), a simple yet effective method that optimizes prompts in a low-dimensional space (e.g., 2D) and uses a frozen random matrix for up-projection. ULPT can achieve 98% reduction in the training parameters compared to vanilla prompt tuning while preserving performance. Our extensive experiments across over 20 NLP tasks demonstrate that ULPT consistently outperforms recent parameter-efficient tuning methods using significantly fewer parameters, making it well-suited as a storage-efficient framework for massive LLM customization.

Related

• LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
• PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention
• LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
• TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models
• PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection

Source: arXiv cs.CL | 2026-04-14