Model Releases
List of suggestions for Gemma 5.
Recently Omar Sanseviero asked for suggestions on Twitter for future Gemma models. Here is my take on this: Gemma 5 should be a looped transformer. Being a looped transformer with configurable loop de
Recently Omar Sanseviero asked for suggestions on Twitter for future Gemma models. Here is my take on this: Gemma 5 should be a looped transformer. Being a looped transformer with configurable loop depth will address one of the biggest Gemma 4 weaknesses - reasoning ability. Don't get me wrong, I love how concise Gemma 4 is with its reasoning, but it actively hinders its performance in certain scenarios, being looped will allow the model to perform better in such tasks, while still being configurable to reason shallower and being significantly more efficient than a traditional transformer without needing additional expensive pre-training. Gemma 5 should have DeepSeek's Engram. It will widen the world knowledge of the model while costing a few extra gigabytes of RAM or SSD space. This will allow the model to perform better in real world tasks, while costing next to nothing. Keep using 4 bit QAT. It genuinely allowed more people to run your models with great quality, it had been proven to work very well in the real world and I am all for it. Implement more efficient attention mechanisms to get cheaper KV cache. Implement something along the lines of Compressed Sparse Attention 2 from DeepSeek, this will allow the model to have significantly cheaper KV cache. This was one of the main downsides of the Gemma 4 31b on a 24gb VRAM setup when compared to Qwen 3.5 and onwards. Gemma 5 ~120b MOE model? I know that this is probably not possible due to it competing with your API offerings, but if you release a 120b MOE model in the Gemma 5 series the community will be very happy. Drafter model improvements. A small update to the drafter models would be nice as since the release of Gemma 4 many new speculative decoding techniques have been published: DSpark, DFlash 2, EAGLE-4, etc. Linear projection into the model layers like in Gemma 4 12b. Instead of the traditional architecture for vision and audio capabilities. Linear projection seems like an interesting approach that is worth looking into, but of course don't force it if it doesn't work out. Forget about 2 bit QAT for small models. Forget that 2 bit QAT exists for small models. I am currently talking about gemma-4-E2B-it-QAT-mobile and gemma-4-E4B-it-QAT-mobile. In practice quantizing the weights to 2 bits even with QAT just completely lobotomizes these small models and it is not worth pursuing in practice. Don't overfit on coding at the expense of everything else. Almost every lab is currently chasing coding benchmarks while regressing on general knowledge, creative writing, and nuance. A small model can't be best-in-class at everything simultaneously, and coding-focused models go obsolete fast in such a crowded space. Gemma is one of the few model families that still feels well-rounded for non-coding use cases - please keep that identity. What do you all think? Feel free to comment your opinion down below. submitted by /u/Paramecium_caudatum_ [link] [comments]
Related
- [[paper-latentloop-scaling-looped-moe-models-with-asymmetric-c|[Paper] LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation]]
- mini-AGI: Continual-learning dynamically looped transformer with evolutionary grown (on a laptop)
- GPT-6.1 Sol looped 'leak' hints at nested models serving architecture
Source: r/LocalLLaMA | 2026-10-28