QLoRA on ExLlamaV3 Quants - Train @ 2-8bpw

EXL3-QLoRA I've built this over the last few months and I think it's stable enough to share now. A fork of ExLlamaV3, this is a native ExLlamaV3 training implementation with plain PyTorch, without transformers. exl3-qlora on github Why? With it, you can train 2-8bpw exl3 quantized models, rather than the old BNB 4 bit lock-in. EXL3 models are currently the highest precision models for their size, which leads to lower loss. Adapters are saved in standard PEFT format and load in TabbyAPI, PEFT, and can be merged back to the base model. Model Support Llama3, Mistral Small/Medium, Qwen3 dense and MoE, Qwen3.5/3.6/3.8 hybrid models, Gemma3/4 including MoE, Arcee Trinity. Each family is checked against the inference forward before training. Want more? I'm open to requests. Features: SFT, DPO, KTO rsLoRA, PiSSA Optional LoRA on routed MoE experts, embedding, and LM head Single GPU, multi-GPU layer split, and DDP Sample packing, chat rendering through jinja, including tool calls and reasoning Gradient checkpointing, activation offload, chunked cross entropy, 8-bit and paged optimizers, Liger kernels Live run monitor in the browser (wandb at home) Real-time training, where the loaded model serves chat and trains its adapter in place Benchmarks: Qwen3.5-4B Base Eval loss Peak VRAM EXL3 6.0 bpw 2.128 12.7 GiB EXL3 4.0 bpw 2.152 11.9 GiB EXL3 3.0 bpw 2.165 11.5 GiB bnb NF4 (axolotl) 2.183 12.8 GiB All on the same dataset and hardware. bnb is about twice as fast, but exl3 is more flexible, vram efficient, and produces lower loss in my tests. This is an unofficial community project, and a work in progress. Any feedback or feature requests are welcome, you can post them here, the github repo, or join us in the ExLlama Discord, where we have a thread in the #community-projects channel. I have already produced several successful models with this training pipeline and released them, you can test them here. submitted by /u/Unstable_Llama [link] [comments]

Source: r/LocalLLaMA | 2026-10-12