Model Releases
Am I just hallucinating
Or is there any reason why I feel like model output quality seems to be better when I use higher micro-batch values (ub) in llama-cpp? I don't really have any hard numbers or anything (just running th
Or is there any reason why I feel like model output quality seems to be better when I use higher micro-batch values (ub) in llama-cpp? I don't really have any hard numbers or anything (just running the same prompts), it's all just vibes. Some context, I'm running the latest build of llama-cpp, vulkan, 6900xt 16gb, 64giggles of system ram. I run gemma and qwens models; q8_0 for the moe's, q4_k_m for dense. KV at bf16. I lock a seed in to try to reduce the differences. Any theoretical reason for the difference or am I just seeing ghosts. submitted by /u/Xyklone [link] [comments]
Related
- Honest take on Laguna S2.1 and its uses (from actual use)
- 4x 3090, 96gb vram what Model to drive Hermes?
- GLM-5.2 UD-IQ1_M on llama.cpp — 5090 + 3090 Ti speed test (~ 579 t/s prefill @ 8k ctx, ~324 t/s prefill @ 57k ctx, ~10.6 t/s decode)
Source: r/LocalLLaMA | 2026-08-07