[Help] Gemma 4 26B LoRA Training on 16GB VRAM: Loss decreases, but inference degenerates into loops (Masking vs. MoE?)

This Reddit thread discusses a user's experience attempting LoRA fine-tuning of the Gemma 4 26B-A4B model on a 16GB VRAM GPU, where training loss decreases normally but the resulting model degenerates into repetitive loops during inference. The core issue stems from Gemma 4's MoE 26B-A4B using 3D fused expert tensors that current quantization tooling cannot handle properly , and using QLoRA on the 26B-A4B is problematic because the MoE architecture and 4-bit quantization interact poorly — 16-bit LoRA is the recommended approach instead. The thread likely explores whether the inference loops are caused by incorrect attention masking, chat template mismatches, or the fundamental incompatibility between the MoE architecture and constrained 16GB VRAM training configurations, since using a wrong chat template or EOS token at inference time (different from the one used during training) is a known common cause of degraded output.

Source: r/ollama | 2026-04-14