Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases

arXiv:2609.09462v1 Announce Type: new Abstract: Prompt learning adapts CLIP to downstream recognition by replacing hand-written templates with learned continuous context vectors, which in Context Optimization (CoOp) form a dense prompt matrix mathbf{P}inR^{mimes d} trained from only a few examples per class. We study whether this matrix is over-parameterized by factorizing it as mathbf{P}=mathbf{B}mathbf{A}, which cuts the trainable prompt parameters from md to r(m+d), and to rd once the token-side factor mathbf{B} is fixed. Across seven few-shot benchmarks and two CLIP backbones, low-rank prompts match or improve dense CoOp at far fewer parameters, with the clearest gains on low-shot base-to-new generalization. We then find that the token-side factor need not be learned at all: fixing mathbf{B} to a Gaussian, orthogonal, SVD-derived, or even random basis and training only the embedding-side factor mathbf{A} stays on par with the fully trainable factorization, and a source-trained mathbf{B} offers no advantage over a random one. A prompt-factor asymmetry and a local update-space dimension gap show why fixing mathbf{B} is far less restrictive than fixing mathbf{A}, and a smoothness-only guarantee certifies that optimizing mathbf{A} over a fixed mathbf{B} converges. In the CLIP prompt setting, the embedding-side coefficients carry the adaptation while the token basis can simply be fixed.

Related

• Beyond Low-Rank: Low-Rank Sparse Prompting via Spiking Neural Network and Prompt Factorization
• ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
• Near OOD Detection for Vision-Language Prompt Learning with Contrastive Logit Score
• Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models

Source: arXiv cs.CV | 2026-09-10