Apple M5 isn't making full use of its matmul cores yet

At the moment MLX (and Llama.cpp for Macs) run 16bit activations everywhere. Despite this, the M5 generation silicon actually does support INT8 activations - it actually allows w4a8 d_type. It's just that no inference backends are using them yet I built some w8a8 kernels and have managed to get 1.4x speed up on Gemma4 prefill tasks; on my M5 MacBook Air it brings baseline prefill for the E2B from 2193 tps stock to 3,029 tps for 130,173 tokens of input* *Even faster at small context lengths; it approaches nearly 10k tps submitted by /u/maddie-lovelace [link] [comments]

Source: r/LocalLLaMA | 2026-07-23