Benchmark results: what is the best and fastest engine to run Qwen3.8-27B on macOS

WARNING: I think I made a mistake with omlx, and its results below seem to be without any kind of acceleration. I am rerunning them with Lightning MTP and ANE support for prefill. Will update the post once done. The new Qwen 3.8 27B is fantastic for local agentic use. The problem is, what makes it so good, being a dense model, also makes it slow. Many engines and versions of the model claim various speed increase. How true are those claim? And does a proven huge boost claim on a short synthetic test actually holds on real agentic work? I spent five days and over 100 GPU hours running rigorous repeatable agentic coding tests, collecting and comparing the data. 200 GB of models downloaded. Ended up with 20 Bubble Bobble clones on my disk, including the ones from failed tests. None of them good, but that wasn't the point. My recommendation: MTPLX or llama.cpp+MTP I ran the tests on 8-bit quants (GGUF or MLX), unquantised KV cache. 100k max response size, full 262k context, official Qwen coding sampler, official Qwen Jinja chat template. I first tested each engine with a short synthetic benchmark, to verify the installation, usage, results collection, and to verify the speed claims Then came the real test, with an agentic coding benchmark consisting of 4 phases/prompts; each engine was tested twice, at medium and xhigh reasoning effort: 2.1. capturing requirements and writing the specification plan 2.2. writing the implementation plan 2.3. implementing the core functionality 2.4. finishing the implementation with advanced functionality Last test, for prefill speed, was with a cold 128k token prompt test, with reasoning turned off and deterministic sample, repeated twice Test hardware: Apple Mac Studio, M2 Max, 96 GB unified memory. Engine Effort Decode Prefill Wall Tokens Score MTPLX xhigh 20–22 tok/s 109 t/s 2h05 152K 93 MTPLX medium 21–24 tok/s 109 t/s 1h35 117K 91 llama.cpp + MTP xhigh 17–19 tok/s 103 t/s 2h52 179K 86 llama.cpp + MTP medium 18–20 tok/s 103 t/s 1h04 70K 84 mlx-dspark DFlash2 medium ~16–18 tok/s 145 t/s 2h09 125K 82 mlx-dspark DSpark medium ~12–17 tok/s 145 t/s 2h05 96K 77 llama.cpp + DFlash2 medium 14–15 tok/s ~103 t/s 2h09 111K 74 llama.cpp + DFlash2 xhigh 13–14 tok/s ~103 t/s 3h31 174K 72 oMLX xhigh 10–11 tok/s 85 t/s 3h57 146K 67 llama.cpp baseline xhigh 10–11 tok/s 103 t/s 3h43 140K 66 llama.cpp baseline medium 10–12 tok/s 103 t/s 1h44 66K 66 oMLX medium 10–12 tok/s 85 t/s 1h44 64K 65 vllm-mlx medium 9–11 tok/s 134 t/s 3h30 115K 51 vllm-mlx xhigh 9–12 tok/s 134 t/s 4h08 136K 50 mlx-dspark DFlash2 xhigh ~9–15 tok/s 145 t/s 3h58 226K 47 vllm-mlx: completed the task, but it does not separate thinking from output, so the raw chain of thought leaks into the final output. mlx-dspark DFlash2 xhigh: thought for 226 thousand tokens and delivered nothing. What about xhigh vs medium reasoning effort? MTPLX xhigh costs only half an hour more than medium (2h05 vs 1h35) and the game quality went significantly up on my scorecard. Worth it. llama.cpp + MTP xhigh costs 1h48 more, with again a significant increase in quality. Worth it if you have the time. oMLX xhigh took more than twice as long as medium. The slower token speed has a huge impact on the time difference between both modes. Not worth it. Get a faster engine instead. dspark DFlash2 xhigh is the cautionary tale at the bottom of the table. See the anomaly note above. xhigh is worth it when the engine is fast enough and uncapped. The model thinks 30 to 100 percent longer, at the same speed, and the result is better. On slow engines you can really feel the slower performance compounded by the extra thinking; instead it could be better to use medium, with more of your own time spent planning a better process and workflow. Commands and configuration notes Exact commands I used for each engine: # llama.cpp baseline (homebrew, b10470) llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf --jinja -c 262144 --parallel 1 -ngl 99 --port 8030 # llama.cpp + MTP, the recommended GGUF setup llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99 # llama.cpp + DFlash2 (for now, needs PR #27342 build) llama-server -m Qwen3.8-27B-Q8_0.gguf -md Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-type draft-dflash --spec-draft-n-max 7 --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99 # MTPLX (pip install mtplx). Note --max-tokens mtplx start web --model Qwen3.8-27B-MTPLX-Optimized-Quality --max-tokens 100000 # oMLX (brew install jundot/omlx/omlx), model dir contains the checkpoint omlx serve --model-dir /path/to/models # mlx-dspark (pip install mlx-dspark). Note --max-tokens-cap mlx-dspark serve --model Qwen3.8-27B-MLX-8bit --mode dflash --context-window 262144 --max-tokens-cap 262144 --default-max-tokens 100000 # vllm-mlx (pip install vllm-mlx). Note --timeout vllm-mlx serve Qwen3.8-27B-MLX-8bit --max-request-tokens 262144 --timeout 7200 The MLX engines all take the plain mlx-community 8-bit weights, except MTPLX which needs its own checkpoint format with the MTP tensors included (I used Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality). For llama.cpp GGUF, I used the official MTP draft from ggml-org. MTPLX has a default low response cap. Use --max-tokens 100000 dspark also has a low default generation cap of 32K vllm-mlx kills any request that spends more than 300 seconds on prefill. Raise it with --timeout. One measurement warning for anyone reproducing this: engines do not all stream one token per event. Some pack several tokens per chunk, MTPLX about 1.7, oMLX about 3. submitted by /u/ex-arman68 [link] [comments]

Source: r/LocalLLaMA | 2026-08-23

Related