Model Releases
[Release] SOTA GGUFs for Qwen3.8-27B: GSQ-RCO at 2.5 to 3.0 bpw
We're releasing Qwen3.8-27B quantized with our newest methods, GSQ + RCO. Higher-quality models, same file size, now with the search and the quantizer both learned. What's inside: GSQ (Gumbel-Softmax
We're releasing Qwen3.8-27B quantized with our newest methods, GSQ + RCO. Higher-quality models, same file size, now with the search and the quantizer both learned. What's inside: GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and scales, closing most of the scalar-vs-vector gap at 2 to 3 bits while staying fully GGUF-deployable RCO (Riemannian Constrained Optimization): assigns a quantization type to every tensor under a strict size budget by gradient descent directly on the task loss, no per-constraint tuning Three GGUFs (2.50 / 2.75 / 3.00 bpw, 8.4 to 10.1 GB) plus the vision projector Further Details: Uniform quants give every tensor the same precision. RCO finds where the precision actually matters and GSQ quantizes every tensor at its assigned type. The result runs unmodified in llama.cpp, Ollama, and LM Studio. Results (vs. the BF16 base and Unsloth Dynamic quants) To the best of our knowledge, these are the best size-to-accuracy GGUFs available for Qwen3.8-27B at these file sizes: at every size we evaluated, they match or beat the strongest published quantizations. 3.00 bpw (10.1 GB): matches the base model on AIME25 (100.00), within about one point on GPQA-Diamond (88.89 vs 89.90) and LiveCodeBench v6 (84.57 vs 85.71) 2.75 bpw (9.3 GB): AIME25 100.00, and its zero-shot average actually exceeds BF16 (75.70 vs 74.34) At matched ~8.4 GB size: +10.0 AIME25, +8.6 GPQA-Diamond, +4.6 LiveCodeBench over UD-IQ2_S Full tables and per-benchmark plots are on the model card. Links: Models: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF GSQ: paper https://arxiv.org/abs/2604.18556 | code https://github.com/IST-DASLab/GSQ RCO: paper https://arxiv.org/abs/2605.00649 | code https://github.com/IST-DASLab/RCO This is the first of a series: we plan to release more SOTA GGUFs across model families as a lab (ISTA Deep Algorithms and Systems Lab). We are happy for feedback, requests for models you want quantized, and experiments! submitted by /u/Loginhe [link] [comments]
Related
- We could really use Qwen3.8 in 27B, 35B, 122B and 397B sizes
- [[benchmark-optimal-dflash2-quants-for-speed-and-context-size-|[Benchmark] Optimal DFlash2 quants for speed and context size, 5090 RTX, llama.cpp, Qwen 3.8 27B Dynamic3 Unsloth. Comparison with MTP]]
Source: r/LocalLLaMA | 2026-08-28