I built BitRun: A standalone runner for 1-bit LLMs (100–127 t/s on Apple Silicon & 40–45 t/s on PC CPUs with zero compile)

I built BitRun because I wanted to run 1-bit ternary LLMs on ordinary laptops without melting my battery, spinning up a dedicated GPU, or wrestling with complex build environments. Right now, BitRun gives you 98 to 127 tokens/sec on an Apple M4 Pro CPU and 40 to 45 tokens/sec on an Intel Core i5 PC CPU, pulling minimal power with 0% GPU usage. To make this actually work on modern hardware, I had to fix several low-level C++ kernel math bugs in Microsoft's research codebase (bitnet.cpp): Memory stride clobbering: In ggml-bitnet-mad.cpp, batch stride was ignored in the write-back stage, causing evaluated batch rows to silently overwrite each other in memory. The infinite @ @ @ loop on ARM NEON: Block size was hardcoded to 64 instead of the GGUF standard 128 elements in ggml-cpu-i2s.c. That 50% pointer arithmetic shift fed scrambled bytes into the NEON DotProd adders. Activation mismatch: The model graph initialized the FFN gate with SiLU, but the 2.4B model was trained with squared ReLU. Using SiLU distorted the activation range and produced gibberish. Multi-turn BLAS crashes: BLAS was claiming matrix multiplications past 32 context tokens and jumping into unmapped memory with unaligned byte offsets. Windows MSVC lockout: Removed artificial CMake checks that blocked native Visual Studio compilation. I submitted the fixes upstream to Microsoft: PR #642: https://github.com/microsoft/BitNet/pull/642 PR #644: https://github.com/microsoft/BitNet/pull/644 Then I packaged the patched engine into BitRun so anyone can test ternary models in 60 seconds with an Ollama-style workflow. No CMake, no Clang toolchains, no Python virtualenvs, and no heavy Docker images. HOW TO RUN IT (ZERO COMPILE) Mac / Linux: curl -fsSL https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh | sh Windows (PowerShell): irm https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1 | iex Pull and chat on pure CPU: bitrun pull falcon3-1b bitrun run falcon3-1b You can also run "bitrun serve falcon3-1b" to get an OpenAI-compatible REST server on port 11435 (point Cursor, Continue.dev, or Open WebUI to it). Custom personas can be created using standard Modelfiles (bitrun create engineer -f Modelfile). VERIFIED CPU SPEEDS (PURE CPU SIMD, 0% GPU) Apple M4 Pro CPU (12 Cores): Falcon3 1B Instruct (~1.30 GB): 98 to 127 tokens/sec (~1.9 GB RAM) BitNet b1.58 2.4B (~1.13 GB): ~40 tokens/sec (~1.8 GB RAM) BitNet 70M (~79 MB): 1,350+ tokens/sec (< 150 MB RAM) Windows x86_64 (Intel Core i5-12600K, AVX2): Falcon3 1B Instruct: 39 to 45 tokens/sec (209.6 t/s prompt processing) BitNet 70M: 492 tokens/sec Linux (WSL2 / 4 cores, AVX2): Falcon3 1B Instruct: 40 to 45 tokens/sec REALISTIC SCOPE Ternary weights require Quantization-Aware Training (QAT) from scratch. You cannot just take an existing 70B FP16 model and post-quantize it down to 1.58-bit without destroying it. Because of that, the public model catalog right now is limited to models under 3B parameters (falcon3-1b, bitnet-2b, falcon3-3b, bitnet-70m). They will not replace Claude or GPT-4 for heavy coding. However, for low-power edge devices, battery-conscious laptop use, and background automations where you want 100+ tokens/sec on pure CPU, it is ready to run today. GitHub Repo: https://github.com/IshekKhal/bitrun Upstream PRs: https://github.com/microsoft/BitNet/pull/642 and https://github.com/microsoft/BitNet/pull/644 Pre-compiled Releases: https://github.com/IshekKhal/bitrun/releases/tag/v0.1.0-alpha Drop your CPU benchmark numbers if you try it! submitted by /u/The_VibePreneur [link] [comments]

Source: r/LocalLLaMA | 2026-10-14

Related