Model Releases
Improved TPS of Gemma 4 31B : the journey and also creating custom patches with VLLM fork
I improved the TPS of Gemma 4 31B. Improving TPS and performing optimisations requires understanding of the model architecture, and I had to fork VLLM and apply my own patch to break into making a con
I improved the TPS of Gemma 4 31B. Improving TPS and performing optimisations requires understanding of the model architecture, and I had to fork VLLM and apply my own patch to break into making a configuration work as per my idea. I wrote a full article so that even beginners can understand and improve TPS for any model, this will serve as a mental model for anyone who start with TPS optimisations. Also I strongly believe being just a mecha pilot wont be enough for inference optimisations. I had to go deep dive into the codebase and architecture for the solution to work. P.S: Wrote in my own words. https://medium.com/@abhijithneilabraham/learning-inference-how-to-host-and-improve-the-token-speed-of-an-llm-cff5623ab505?postPublishedType=initial submitted by /u/metalvendetta [link] [comments]
Related
- Gemma 4 QAT could be improved further by Google aligning the QAT model to modern q4_k instead of q4_0
- If you don't like how DeepSeek V4 Flash 0731 talks, give Motif 3 NVFP4 a try (custom vLLM only) -- probably the most underrated model right now
- KV cache quantization benchmarks: 413 pairs tested on Qwen 3.6 27B, Gemma 4 31B. KLD with BeeLlama.cpp v0.4.0: KVarN 6-bit beats q8_0, precision tail 1024 dominates
Source: r/LocalLLaMA | 2026-09-19