Model Releases
HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts
arXiv:2608.29790v1 Announce Type: new Abstract: As large language models (LLMs) continue to scale, parameter-efficient fine-tuning (PEFT) has become a practical alternative to full-parameter adaptatio
arXiv:2608.29790v1 Announce Type: new Abstract: As large language models (LLMs) continue to scale, parameter-efficient fine-tuning (PEFT) has become a practical alternative to full-parameter adaptation. Prompt tuning is effective, but existing approaches either use flat prompt structures or hierarchical structures with fixed prompt composition, limiting adaptive prompt specialization. To address this limitation, we propose HiVe, a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization. HiVe constructs a prompt hierarchy by leveraging inter-task relationships during training, and employs a vertical mixture-of-experts (V-MoE) mechanism at inference time to compose prompts up to the level of specialization required for each input. Experiments show that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.
Related
- MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation
- CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging
- MoE^2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Source: arXiv cs.CL | 2026-09-01