Unsloth's Gemma 4 mmproj silently broke vision & audio on newer llama.cpp builds — anyone else hit this?

So I had been building ScreenMind, kinda like local ai desktop assistant that uses Gemma 4 for screen analysis, voice memo transcription, and meeting transcription — all through llama-server. Everything runs locally. A weeks ago, all multimodal features just stopped working. I wanted to share what happened and if anyone else also ran into this , because it was genuinely hard to catch. Screenshot analysis started returning tokens instead of actual descriptions Voice memo transcription was producing garbage or empty strings Text-only chat worked perfectly fine The model loaded without errors, server started normally, no crashes — just garbage multimodal output The annoying part was nothing in my code changed. It broke between llama-server updates. I spent a whole day thinking it was my code. Everything looked right tho. Then I noticed when sending a ~5 second audio clip, the model was only receiving 87 input tokens. way too low. A 5-second clip should produce hundreds of audio tokens. The mmproj was clearly not encoding the audio properly — the model was basically getting nothing and filling the output with garbage tokens. I wrote a minimal test script — just llama-server + a single image + a single audio file, completely outside of ScreenMind. Same garbage. So it wasn't my app. The root cause(thats what i speculate feel free to correct me) I was using models from unsloth/gemma-4-E2B-it-GGUF. Their mmproj file (mmproj-BF16.gguf, 941 MB) became incompatible with newer llama.cpp builds (confirmed broken on b10244, regression reportedly starts around b9318). The key realization: ggml-org maintains both llama.cpp AND the official GGUF models. When they update how multimodal tokens are processed in the server code, their mmproj files get updated to match. Third-party quantizers like Unsloth produce their mmproj files independently using their own conversion pipeline. So when llama.cpp changes the internal mmproj format, ggml-org's files stay in sync but Unsloth's may break. I switched to ggml-org/gemma-4-E2B-it-GGUF with their mmproj and everything worked immediately: Same llama-server build (b10244), same quantization level, only difference was which repo the model + mmproj came from. What I did to fix it in my project Switched E2B and E4B model sources from Unsloth → ggml-org Added a regex safety net to strip tokens from output Made voice memos save to DB even if transcription fails (previously they were silently lost) Built a Model Hub so users can pick their own quantization variant and re-download easily Shipped as v0.2.0(https://github.com/ayushh0110/ScreenMind/releases/tag/v0.2.0). I found some related upstream issues on the repo but couldnt figure out cleanly Questions for the community Has anyone else hit this? Specifically the output when using Unsloth or other third-party GGUFs with Gemma 4 multimodal features. Is mmproj incompatibility between llama.cpp versions and third-party quantizers a known recurring thing? Or is this specific to the Gemma 4 architecture? I've only been using Gemma 4 so I don't have a baseline with other multimodal models. How do you handle this in your projects? I'm thinking about pinning my llama-server setup script to a specific tested build instead of always pulling latest. Is that what everyone does, or is there a better approach? Are ggml-org's official GGUFs generally the safest choice for production-ish use? The tradeoff is fewer quantization options (they only offer Q4_0, Q8_0, BF16 for E2B) compared to Unsloth/bartowski who have many more variants. Env: Windows 11, Python 3.12, llama-server 9193, Gemma 4 E2B/E4B submitted by /u/Top_Speaker_7785 [link] [comments]

Source: r/LocalLLaMA | 2026-08-06

Related