Hardware
NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments
Qwen3.8-27B is finally here, and NInfer already has Day-0 support! Weights: https://huggingface.co/neroued/Qwen3.8-27B-NInfer Just update to the latest source and give it a try. On a single RTX 5090,
Qwen3.8-27B is finally here, and NInfer already has Day-0 support! Weights: https://huggingface.co/neroued/Qwen3.8-27B-NInfer Just update to the latest source and give it a try. On a single RTX 5090, NInfer can still reach around 200 tok/s generation with speculative decoding. NInfer has also improved a lot since my last post. Now NInfer supports up to 8 concurrent requests, with a shared paged KV cache pool, each request can still use the full context length. I aslo implemented ReplaySSM for GDN + speculative decoding, which greatly reduces the recurrent-state memory overhead under concurrency, and this is also something vllm does not yet fully supported. There have also been many cuda kernel optimizations, as well as usage of PDL to further reduce latency. Feedback and bug reports are always welcome, I'll try to fix issues as soon as possible! submitted by /u/FormOne2615 [link] [comments]
Related
- unsloth/Qwen3.6-27B-NVFP4 vs. Intel/Qwen3.6-27B-int4-AutoRound vs. nvidia/Qwen3.6-27B-NVFP4 -- which one to choose?
- What do you guys do for GPU Kernels?
- Has anyone been working on a solid setup for DSV4F on x2+ R9700s?
- NVIDIA AI Releases Star Elastic: One Checkpoint that Contains 30B, 23B, and 12B Reasoning Models with Zero-Shot Slicing
Source: r/LocalLLaMA | 2026-08-14