Model Releases
Gemma 4 12B Q3: +8.55% Coding Performance From Tensor-Level Quantization Allocation
Ive been experimenting with task-aware GGUF quants for months, taking inspiration from TASA and TAQO but pushing the allocation lower to to the tensor level. The basic idea is to generate a custom ima
Ive been experimenting with task-aware GGUF quants for months, taking inspiration from TASA and TAQO but pushing the allocation lower to to the tensor level. The basic idea is to generate a custom imatrix from a category-specific corpus, measure where quantization causes damage, then redistribute a fixed bit budget toward tensors where additional precision recovers the best performance in that category. Today I got my first real win. The pipeline is live. On Gemma 4 12b, the hand tuned imatrix q3_k_s scored 45.974. After tensor level allocation, the same q3_k_s based scored 49.905. Thats +3.931 points or an 8.55% relative improvement from the allocation on top of the imatrix. The models are effectively the same size: 5,528,230,848 bytes for the comparator versus 5,534,804,928 bytes for the allocated model, a difference of only +0.119%. I chose q3 deliberately. It had the largest amount of recoverable headroom while sitting above the quant cliff. This model is intentionally category specialized. Degredation in categories that werent selected is expected. There is still a lot of tweaking to do in order to maximize these results but the performance of this imatrix + allocation at q3 when compared to no imatrix and source is pretty staggering when considering the amount of space required to run it. I'd love some community testing and feedback if anyone is interested. https://huggingface.co/ByteOtter/Gemma-4-12B-it-CADA-Q3_K_S My ULTIMATE goal is to create a pipeline that takes an full precision .gguf file and auto spits out an optimized model within a selected category at whatever optimal size for recovery. This is a solid start. Charts were provided from my data by ChatGPT. TL;DR: I built a task-aware GGUF quantization pipeline that combines a category-specific imatrix with tensor-level damage measurement and bit allocation. On Gemma 4 12B Q3_K_S, allocation improved coding performance from 45.974 to 49.905, an 8.55% relative gain OVER handtuned imatrix, while increasing model size by only 0.119%. The model is intentionally specialized, so some out-of-category regression is expected. This is the first result where the allocation stage itself produced a clear win. submitted by /u/devildip [link] [comments]
Related
- Appreciation for Gemma 4 26b A4b
- KV cache quantization benchmarks: 413 pairs tested on Qwen 3.6 27B, Gemma 4 31B. KLD with BeeLlama.cpp v0.4.0: KVarN 6-bit beats q8_0, precision tail 1024 dominates
- DeepSeek v4 Flash vs. Qwen3.6-27B, 3.5-122B, and Gemma 4 31B Benchmark
- Current smallest usable coding model
Source: r/LocalLLaMA | 2026-08-13