Qwen3.8 27B C8 at 972 TG / 5,680 PP on 4x MI100 rig ($6.5k) using my new INT8 vLLM fork

Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2. Its not just another vibed autoresearch loop. No, vLLM ships with very little int8 support, and this stack adds INT8 into every crevice of Qwen3.8 including in dependent libraries and new fused kernels. So no longer are your old INT8-centric cards relegated to second rate algos and suboptimal dtypes. My system runs at 15 tok/s TG on stock vLLM and is now running at 972 tok/s TG on this branch. This stack adds thorough and complete tensor parallel INT8 support across the entire Qwen3 vLLM arch stack, plus significant and thorough tuning for a 4x MI100 system. My system is benchmarking 972 TG / 5,680 PP on this model at near-reference quality. Itemized work includes: Complete W8A8 INT8 GEMM library tuned for MI100 and usage everywhere INT8 DFlash2 (INT8 MTP also, but MTP is slower for most usecases -- note that the quant I provide has an MTP sidecar included so you can choose either MTP or DFlash2 for your usecase) INT8 KV cache INT8 AITER Unified Attention (and fallback Triton impl) -- faster than Flash Attention INT8 GDN attention INT8 custom allreduce / allgather optimized for XGMI interlinks INT8 Embedding Many new fused INT8 kernels More INT8s than you can count The stack's performance has been vetted for near-reference quality/accuracy by measuring accuracy not just for every token, but for every GEMM, every attention block, every KV lookup, and every layer. I've left all of the diagnostic scripts for this accuracy vetting process in the vLLM branch so you can verify it yourself. This fork has an extreme level of testing that isn't present even in stock vLLM. Check your card's specs for whether its INT8 TOPS exceeds its FP8 FLOPS, and if it does, this project is for you. Much of the work is generalizable to all older GPUs (AMD and Nvidia) that don't have native FP8. In stock vLLM they will mostly use BF16/FP16 which is very slow! While I specifically tuned for MI100, this fork has many INT8 triton fallbacks, which are hardware agnostic, so those should run less optimally, but still much faster than stock vLLM for any card. So if you are technically proficient and have some of these older cards, you should be able to blow away any other high-concurrency serving options with this fork. This project should also generalize to other model architectures, but none of them have been polished up with love like Qwen3.8. This stack should work almost out of box and especially well for older AMD cards like MI50 and MI210 (you just need to build aiter for your arches). Here are the projects and models of this stack: https://github.com/curvedinf/int8-vllm https://github.com/curvedinf/int8-aiter https://huggingface.co/curvedinf/Qwen3.8-27B-GPTQ-INT8-W8A8-GS128 https://huggingface.co/curvedinf/Qwen3.8-27B-DFlash2-GPTQ-INT8-W8A8-GS128 Use this serving recipe for the most optimal experience: https://github.com/curvedinf/int8-vllm/blob/main/docs/recipes/README.md Please submit feedback here or in the github issues of vllm. Tuning for other cards/model arches is welcome in PRs. submitted by /u/1ncehost [link] [comments]

Source: r/LocalLLaMA | 2026-08-26

Related