Ornith-1.5-35B-A3B Q4 running 60tk/s on 4070Ti.

Not everyone has the disposable income to build a small data center, so making this post for the underdogs as I was very surprised by the performance/results of this 35B MOE model. Context is admittedly tight and will get laughed at by the big boys. I tried to think of something inspirational to say here but failed, so you just get laughed at. Sorry. The thought here is to push as many active experts into Vram and offload the rest into system ram. At 27 it leaves about 1gig of overhead for KV cache, which hasnt overflowed on me yet. Hardware Component Spec GPU NVIDIA RTX 4070 Ti — 12 GB GDDR6X CPU Intel i9-13900KF (8 P-cores + 16 E-cores, 24C / 32T) RAM 32 GB DDR5-6000 OS Windows 11 Runtime llama.cpp build b10470, CUDA 12.4 (prebuilt Windows binaries) Driver 566.xx (Dec 2025) Model Model ornith-ai/Ornith-1.5-35B-A3B-GGUF Quant Q4_K_M (~20 GB total weights) Architecture qwen3_5_moe — Mixture-of-Experts, ~3B active of 36B total Type Reasoning model (emits blocks), multimodal Context 32,768 (native 256K, YaRN-extendable) Launch command (Windows .bat) bat llama-server -hf ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M --no-mmproj -ngl 99 --n-cpu-moe 28 --load-mode none -c 32768 --parallel 1 -fa on --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp --spec-draft-n-max 2 -t 8 --jinja --reasoning-format auto --temp 0.6 --top-p 0.95 --top-k 20 Aquarium Prompt Example Metric Value Prefill / prompt processing ~650–700 tok/s Generation, sustained ~50–56 tok/s MTP draft acceptance ~42–48% (mean draft len ~1.9) llama-bench (isolated, tiny context, no KV growth) n_cpu_moe tg128 (gen) pp2048 (prefill) 26 64.9 ± 0.2 t/s 571 ± 23 t/s 27 64.3 ± 0.4 t/s 482 ± 26 t/s 28 62.6 ± 0.2 t/s 461 ± 31 t/s submitted by /u/Seraphym87 [link] [comments]

Source: r/LocalLLaMA | 2026-08-20