Model Releases
Basalt: NVFP4 + faster prefill for Qwen3.8 Flash-Next on Blackwell (single 5090)
Basalt: Qwen 3.8 FN NVFP4, single 5090 Hi guys, so I finally got a RAM upgrade and have been working on the top two pain points from Basalt: NVFP4 support and faster prefill. NVFP4: Basalt runs NVFP4
Basalt: Qwen 3.8 FN NVFP4, single 5090 Hi guys, so I finally got a RAM upgrade and have been working on the top two pain points from Basalt: NVFP4 support and faster prefill. NVFP4: Basalt runs NVFP4 natively now, a new .basalt pack keeps stock NVIDIA experts and stores the dense layers as Q6_K. Numbers for a 5090 (400w) + 128 GB of DDR5 RAM, single stream: - struct: 304 tok/s - prose: 183 tok/s - w/ peer gpu (5060 Ti): struct 479 / prose 239 Quality vs Q8_0 (100 chunks): PPL 3.308 (Q8_0: 3.286), KLD 0.068, 91.5% same top-1. Prefill: 8,582 tok/s on a 64K prompt (7.6s to first token). Against Infernix on the same NVFP4 weights and same box it's up to 1.3x on decode and 1.6x on prefill. Also merged a bunch of stability and performance fixes over the last week, coming up next Unsloth Q4 + Q8 (near lossless) tuning. Give it a try: Repo: github.com/jesdga95/basalt HF: huggingface.co/jesdga/Qwen3.8-Flash-Next-Basalt submitted by /u/jesdga95 [link] [comments]
Related
- Basalt: Flash-Next at 665 tok/s structured, 354 prose on a 5090 + 5060 Ti (2.6x Strata)
- Basalt: Fast Blackwell inference engine for Qwen 3.8 Flash Next (and how we got vision for free)
- Qwen3.8-Flash-Next on 2x3090 + DDR4, part 4: 2.2-2.5x faster prefill by kicking the expert cache off the GPU while the prompt runs
- Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and Infernix
- Fastest NVFP4 quant of Qwen3.8 27B out there
Source: r/LocalLLaMA | 2026-10-18