Model Releases
vllm + p2p driver hack + qwen 3.8 27B vs llamacpp + qwen flash next ?
Hi everyone I'm running four rtx 4090, 64GB ram, on a threadripper pro motherboard so all PCIe x16 ports, as a homelab machine for coding. I was migrating from vllm + qwen 3.8 27B (fp8+256k kv cache)
Hi everyone I'm running four rtx 4090, 64GB ram, on a threadripper pro motherboard so all PCIe x16 ports, as a homelab machine for coding. I was migrating from vllm + qwen 3.8 27B (fp8+256k kv cache) to llamacpp + qwen flash next iq4xs + 8 bit cache 200k kv cache... until someone had to ruin my migration with this awesome patch to enable p2p between cards on the open source nvidia drivers. On my hardware configuration apparently this would matter a lot. But only VLLM would benefit from it, not llamacpp - and no 4 bit like quant available on vllm and not enough vram for qwen flash next fp8. I hereby ask the grand experts, what should I do amd what are your opinions? vllm + qwen 3.8 27B dense + p2p driver patch + 256k kv cache llamacpp + qwen flash next MoE iq4xs + 200k kv cache Is qwen flash next really worth the extra intelligence versus higher throughput with vllm and sticking to qwen 3.8 27B ? submitted by /u/dowitex [link] [comments]
Related
- My Qwen3.8-Flash-Next recipe for single GB10/DGX Spark, uses Intel AutoRound int4 quant and vLLM, fp8 ngram table offloaded to local SSD or external RDMA server. At mtp=3 c=1, code is ~47.5t/s, json is ~60t/s. Prefix cache is ON.
- 2x R9700, 64 GB DDR5 is an absolute beast machine with vLLM Radiance / R9V and Qwen 3.8 27b and Flash next
- Updated my benchmark with a new vLLM based recipe for Qwen 3.8 Flash Next : now up to 98/100 (instead of 91 previously)
- Qwen3.8-Flash-Next + MTP on Strix Halo: Vulkan Runtime Notes
Source: r/LocalLLaMA | 2026-09-06