I built a DwarfStar-inspired Vulkan/Metal inference engine for Qwen3.6-35B-A3B on 16 GB machines

Disclosure: I’m the author and maintainer of QuarkStar. I built QuarkStar, a small native inference engine inspired by Antirez’s DwarfStar. QuarkStar currently supports: Qwen3.6-35B-A3B, using the same Antirez-inspired Q2 and Q2/Q4 quantization recipes KAT-Coder-V2.5-Dev, the coding-focused post-training of Qwen3.6-35B-A3B, using the same recipes Native Vulkan on Linux Native Metal on Apple Silicon Fully resident inference on 16 GB machines Bounded SSD expert streaming when the model does not fit in memory DwarfStar is built around much larger models and primarily targets 96/128 GB-class machines. I wanted to explore the other end of the spectrum: useful local models on 16 GB machines and 24/32 GB workstations, with an SSD-streaming path designed for even smaller 8 GB systems. Not everyone can spend 3,000–5,000 on local AI hardware. This project was born with the intent of improving my skills in LLMs. It's useful for me for inference and for learning, and I hope it will be useful for you too. My primary development machine is an AMD BC-250: a roughly $150 board with 16 GB of unified GDDR6. The current Vulkan fast path was developed using RADV on this device. I also developed and tested the native Metal backend on a M2 Pro 16 GB. BC-250 Q2 prefill and decode t/s Some current Q2 resident results: Device Context Prefill Generation BC-250 16 GB 2K 639.85 tok/s 81.85 tok/s BC-250 16 GB 8K 501.50 tok/s 74.72 tok/s BC-250 16 GB 32K 244.06 tok/s 51.26 tok/s M2 Pro 16 GB 2K 448.75 tok/s 37.78 tok/s M2 Pro 16 GB 8K 270.02 tok/s 31.08 tok/s M2 Pro 16 GB 16K 177.21 tok/s 25.64 tok/s I think the 35B size class is going to become increasingly interesting. DeepSeek V4 Flash-0731 recently showed once again how quickly the intelligence-to-active-parameter ratio can improve. Model support in QuarkStar is therefore intentionally opportunistic: the project will follow whichever open checkpoints are most useful on ordinary local machines. With yesterday's news of the release of Qwen3.8 27b and probably other lines of the family as well, I also created a branch for the dense model but for now it's experimental. Whether it will merge will depend on the power of the new model and when and if a MoE on the 35B will also be released. I still see the future of this project on MoE of that size order. I think we'll have some fun with Qwen 3.8 and Quarkstar. The project is still young, and Vulkan hardware varies a lot. I would especially appreciate testing and feedback from: Vulkan users with GPUs other than the BC-250 Apple Silicon users, particularly those with older or 8 GB Macs Anyone interested in improving kernels, quantization quality, or SSD caching Repository: https://github.com/Ninnix/q36 Licence: MIT Special thanks to Salvatore, he is a continuous source of inspiration for me, and his content on YouTube has greatly improved me as a software engineer and as a person. Demo: Edit: Reddit’s mobile app may show a black frame. Working demo video: https://youtu.be/3y2rkLUg1ug Demo Prompt: Create a single self-contained HTML file using Three.js from a CDN that opens into a cinematic neon wormhole with hundreds of glowing particles, rotating torus rings, fog, and a slow automatic camera flight through the tunnel. Add mouse parallax and make each click launch a visible energy pulse down the tunnel. Use only procedural geometry and materials, with no external assets or build step, and keep it smooth and responsive. Work in /tmp folder. submitted by /u/Nicolodeva [link] [comments]

Source: r/LocalLLaMA | 2026-08-04