EXL3 seems to be fading from the r/LocalLLaMa consciousness, and while I suspected it, I'm surprised at this point in time.

EXL3 is an alternative to llama.cpp. And while there is extensive tooling for llama.cpp, EXL3's primary deployment (TabbyAPI), has a OpenAI compatible API so it shouldn't matter. Why won't this tool matter to you? If you have a GPU with under 24 GB of VRAM, the value kind of plummets as it is VRAM only. 16 gb might be worth it for a 12b model... idk... even then... EXL3 has decimal quants... like 2.5B and more. Why should this tool matter to you? I consistently see all the coding and agentic bro's flexing their tokens-per-second command line settings for llama.cpp, but EXL3 has always performed very strongly with tokens-per-second, and it has beat llama.cpp with model support from time-to-time. Besides speed, it also has some interesting compression behind it, which might be better than GGUFs (not very technically minded and I haven't dug into it)... GGUFs don't do decimal bitrates. Qwen 3.8 27B is the hotness right now so if you are a coding or agentic bro/gal... then you should at least take a peak at TabbyAPI as most who are using Qwen have at least 24GB of VRAM if not more and this tool lives in VRAM. There are lots of other ~30b models like Glimmer that have shown up as of late... so even if Qwen isn't your jam, we have peanut butter, honey, cream cheese ... okay enough with that. I'm a pretty strong regular, and commentator, so I'll be surprised if I've missed a recent post concerning EXL3. That said, even if I missed A POST, the point is they're not all over the place. Perhaps this does matter to some of you. Love to hear what quant models you're using. Considering picking up EXL3 again. Why don't I care so much? I don't use it much because my interest with LLM is not currently coding or agentic stuff, and I tend to favor monolith MoE's that spill into RAM (not supported by EXL3). I questioned EXL3's doom a while back: Is exl3 doomed - Reddit for this very reason. Why the post then? My hope is if interest surges the creator won't lose interest, and eventually it might get RAM spill over... perhaps a jack of all trades is not ideal for all tools (since we already have GGUF), but I wouldn't mind another quantization library to rely on outside of llama.cpp when it comes to model support. Turboderp has had model support at least weeks sooner than llama.cpp for some models. Have you used EXL3? Why do you think it doesn't get much attention, especially with the re-awakening of 30B models? I'm curious if someone more technically minded could confirm if EXL3 still wins on some benchmark... if not speed then compression. ADDITION: Apparently I missed the fact RAM spill over is now supported per a comment below. Also, here is a graph for some comparison to GGUF from the same comment. https://preview.redd.it/eg7cw5x86yjh1.png?width=2240&format=png&auto=webp&s=e98551738dcdce212c9626c34c7f1d429cdda45d submitted by /u/silenceimpaired [link] [comments]

Source: r/LocalLLaMA | 2026-08-17

Related