Ortho-Hydra: Orthogonalized Experts for DiT LoRA

arXiv:2605.03252v1 Announce Type: cross Abstract: LoRA fine-tuning of diffusion transformers (DiT) on multi-style data suffers from style bleed: a single low-rank residual cannot represent several distinct artist fingerprints, and the optimizer converges to their average. Mixture-of-experts LoRA in the HydraLoRA style replaces the up-projection with E heads under a router, but when every expert is zero-initialized the router receives identical gradient from each head and remains at the uniform prior. The experts then evolve permutation-symmetrically, and the network trains as a single rank-r LoRA at E{imes} the cost. We present extbf{Ortho-Hydra}, a re-parameterisation that combines an OFT-style Cayley-orthogonal shared basis with per-expert disjoint output subspaces carved from the top-(Er) left singular vectors of the pretrained weight. Disjointness makes the router's per-expert score non-degenerate at step~0, so specialization receives gradient signal before any expert has trained. We test the predicted deadlock on a DiT pipeline by comparing two HydraLoRA baselines, a zero-initialized shared-basis variant and the original sigma{=}0.1 Gaussian-jitter mitigation, against Ortho-Hydra under a matched optimiser, dataset, and step budget. Neither baseline leaves the uniform prior within the first 1ext{k} steps; Ortho-Hydra begins de-uniformising within the first few hundred. End-task generation quality on multi-style data is out of scope; we report the construction, the cold-start mechanism, and the routing dynamics it changes. Code: https://github.com/sorryhyun/anima_lora.

Source: arXiv cs.CV | 2026-05-06