From Transfer Probes to Expert Grouping: Evidence from LoRA Post-Training

arXiv:2610.14437v1 Announce Type: new Abstract: A platform that post-trains one base model for many business lines often has more lines than experts it can maintain, so lines must share experts, yet lines seldom come with a taxonomy saying which should share. We ask whether transfer probes (train on one source, measure paired changes on all sources) can discover a grouping when the standard one is unknown. To test probes against a reference, we use seven public sources that do have a label tree (five mathematics subfields and two kinds of programming, standing in for business lines) and treat the tree's grouping as a label reference that the evaluator holds and the method does not. With four LoRA experts under fixed per-source exposure and per-expert budgets, we compare a purely probe-derived grouping and an uninformed uniform default (whose tied routing serves every line from one expert) against this reference. Trained once for one interval from a shared warm-up state and routed by a rule fixed by each allocation, neither reached the prespecified practical gain of 2 percentage points, and both had lower overall utility than the reference (-2.43 and -2.54 pp; 95% intervals below zero). In a post hoc comparison, the probe-derived grouping was indistinguishable from the uniform default (+0.11 pp, [-1.25,+1.54]). In fixed-budget LoRA post-training, a grouping built from initial-state single-source probes thus did not deliver the advantage predicted from the probes once carried to a later training state, a finite budget, and specified routing.

Related

• Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation
• GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
• Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization

Source: arXiv cs.LG | 2026-10-14