Model Releases
Running Qwen 3.6 35B MoE (Q4_K_M) on a Zeus (Xiaomi 12 Pro, 12GB RAM)
Shoutout to this awesome guy - https://www.reddit.com/r/LLM/s/IDUyU3v9ap Thanks to his project, BigMoeOnEdge https://github.com/Helldez/BigMoeOnEdge, I managed to successfully run a 35B MoE model on j
Shoutout to this awesome guy - https://www.reddit.com/r/LLM/s/IDUyU3v9ap Thanks to his project, BigMoeOnEdge https://github.com/Helldez/BigMoeOnEdge, I managed to successfully run a 35B MoE model on just 12GB of RAM! My setup is a modified Xiaomi 12 Pro (12GB RAM) that I call "Zeus". https://www.reddit.com/r/LocalLLaMA/s/5zBUl15jd6 There is a bottleneck, of course—the maximum context is currently limited to 8192 tokens due to RAM constraints—but it’s still absolutely mind-blowing to see a model this size running locally on an edge device. I haven't tested the Image-to-Text (vision) capabilities yet, but I'm really hoping to get that working next. Check out the video ! It's completely unedited and recorded in real-time so you can see the actual, raw generation speed. Also, here is stats in text: generation: 107 tokens, 0.412 s/token (2.428 tok/s) compute: 88.1% CPU occupancy (1.4508 cpu-s/token over 4 threads), 51.93 major faults/token prefill: 24 tokens, 5.499 s (4.4 tok/s) | model load 14.421 s | TTFT 19.920 s moe-stream: read 14589.9 MiB (136.35 MiB/token), decode 0.412 s/token (compute 0.314 + cache mgmt 0.014 + flash I/O 0.382 s/token, 357 MiB/s) moe-cache: 70.8% hit, resident 2998.5 MiB moe-overlap: stall 0.084 s/token (flash reads overlapped with FFN compute) submitted by /u/Aromatic_Ad_7557 [link] [comments]
Related
- Running Qwen 3.6 35b MoE With Zoo Code On M1 Max is Amazing! Fully local, battery-powered coding powerhouse!
- Hello from 10KM high! - Thanks to Qwen 3.6 35b a3b!
- What it feels like to have to have Qwen 3.6 or Gemma 4 running locally
- Qwen 3.6 27B BF16 vs Q4_K_M vs Q8_0 GGUF evaluation
Source: r/LocalLLaMA | 2026-07-23