Model Releases
Best current R9700 inference engine?
There are way too many forks to keep track of, so I've gotten lost. As far as I can tell, Radiance VLLM is best for models that fit in GPUs while some form of llama.cpp is probably best for MOE RAM-sp
There are way too many forks to keep track of, so I've gotten lost. As far as I can tell, Radiance VLLM is best for models that fit in GPUs while some form of llama.cpp is probably best for MOE RAM-spill? My specific current goal is to run GLM5.3-Flash over 6 R9700s + system RAM but also looking to try Q-FN / DSv4-vision (or any other big models that I can fit, so not DSv4.1) submitted by /u/KingCpzombie [link] [comments]
Related
- Question: Why is prefill unbelievably faster in vLLM than other inference engines?
- Multi-hour llama.cpp optimization experiments on Qwen MoE models, patches, benchmarks, and reproduction guides
- Qwen3.8 flash next ISTA-DASLab GGUF 50t/s TG and 1500t/s PP with 12GB VRAM and 64GB RAM Laptop on 'Strata' engine
- Ling-3.0-flash weights: SGLang says day-0, vLLM says when they land, llama.cpp closed the 2.6 request as not_planned
Source: r/LocalLLaMA | 2026-10-08