Ninfer and a 5090 with 3.8 27B is making me cry tears of joy it's so good.

Built the latest and I'm getting as much as 220 tokens per second and averaging in the 170s, I can't get over it. If anyone on here is on that project, fuckkkin' chapeau man, really incredible job. I can't believe I was able to like double or more my throughput from llama.cpp This is what I set up: command: > ninfer-serve /models/qwen3_8_27b_nvfp4.ninfer --model-id qwen3.8-27b-nvfp4 --host 0.0.0.0 --max-context 240000 --kv-capacity 240000 --max-concurrency 2 --kv-dtype fp8 --host-kv-mib 16384 --spec mtp --draft-tokens 3 --lm-head-draft --vision --media-live-mib 2048 submitted by /u/Rollingsound514 [link] [comments]

Source: r/LocalLLaMA | 2026-08-28