Qwen3.5 0.8B on CPU

Since the Qwen3.5 0.8B model is an interesting one for small specialized fine tunes, I was curious how fast it can run on CPUs. Why CPUs? Mainly because I want to use it as a local dictation cleanup model when I'm using the GPU for something else. Over the weekend, I let Codex build a small C++ engine and a custom 4-bit format, H128/Q4-G32-DOT4, with activation-based calibration and blockwise error compensation. The resulting model has a 425 MB weight payload, roughly 71 MB smaller than Unsloth’s mixed-precision Q4_0, with similar measured perplexity and KL divergence. Comparison on my Ryzen 9 9955HX3D, Linux, eight physical V-Cache cores: Engine / quant Weight MB Prefill Decode B1 Decode B16 PPL ↓ KL ↓ ik_llama.cpp IQ4_KS_R4 401 2,060 129.2 455 15.90 0.0906 qwen35-cpu, H128/Q4-G32-DOT4 425 2,758 122.9 642 15.80 0.0602 ik_llama.cpp Unsloth Q4_0 496 2,731 104.0 420 15.58 0.0684 llama.cpp Unsloth Q4_0 496 954 93.7 373 15.58 0.0684 Compared with llama.cpp running Unsloth Q4_0, that’s roughly 2.9× prefill, 1.3× single-request decode and 1.7× batch-16 throughput. Against IK running the same GGUF, prefill is essentially tied, single-request decode is 18% faster, and batch-16 throughput is 53% higher. IK’s smaller IQ4_KS_R4 wins single-request decode, but my engine delivers 41% higher batch-16 throughput. https://github.com/Danmoreng/qwen35-cpu submitted by /u/Danmoreng [link] [comments]

Related

• I fine-tuned a 0.8B local model for dictation cleanup. It matched a hosted frontier model on this narrow task
• A collection of small domain-specific benchmarks for local models (30+ and growing)
• SOTA Apple Silicon Inference (August 15, 2026)
• endless-frontier/BigBang-v1 - qwen 3.5 finetunes

Source: r/LocalLLaMA | 2026-09-07