Model Releases
Qwen 4 architecture: What do we know?
My best bet is the embedding-offloaded linear where the 51b n-grams track semantics and context, like 3.8 with its loss of real world knowledge bolted back on. Otherwise? Sparse full-attention with de
My best bet is the embedding-offloaded linear where the 51b n-grams track semantics and context, like 3.8 with its loss of real world knowledge bolted back on. Otherwise? Sparse full-attention with dense routing? 6b active route to 'heavy' layers when the n-gram gets stuck. Scenario 3: multi-head latent attention (mla) hybrid (reverse engineering Deepseek) compressing KV on the fly with the embeddings 'decompressing' on demand submitted by /u/challis88ocarina [link] [comments]
Related
- Long Review: Qwen 3.8 27B is VERY good at tapping into it's real-world knowledge. It's 'overthinking' brings it to Sonnet level performance with the potential for Opus level results.
- Kimi Linear 48B A3B?
- I built a new attention mechanism (wave field) — runs 128K context where standard attention OOMs, 80+ tok/s on laptop CPU
Source: r/LocalLLaMA | 2026-08-25