SOTA Apple Silicon Inference (August 15, 2026)
This is a HANDWRITTEN post. I spent way too much time trying to get fast inference on Apple Silicon. This post is for people who want to know what's the latest on running local models on their mac, an
Knowledge catalogue
This is a HANDWRITTEN post. I spent way too much time trying to get fast inference on Apple Silicon. This post is for people who want to know what's the latest on running local models on their mac, an
I released KaozKit this week, a Swift library for running LLM agents written in JavaScript inside a Mac app. Built from the start to keep everything local. Providers: MLX with Hugging Face model manag
Apple Silicon Macs on macOS 26+ come with a small LLM built in. No download, no API key, and nothing leaves your Mac. Why I built it I was making a tool that writes API docs from code, and I didn't wa
https://preview.redd.it/nulsv53o8vqh1.png?width=4500&format=png&auto=webp&s=74765dbd409f4c221640f9f6000a685f6fdbb242 Spent weekend benchmarking the Splash engine (by Incoai) and extending its architec
Apple is considering offering an AI server built around 'M8' series chips and has discussed incorporating Nvidia networking hardware, The Information reports. The system would be sold to outside custo
I am planning to buy hardware to run local models. I'm not sure what kind of hardware is best and what will I be able to run. I'm torn between getting a used M1 Max 64gb Macbook for €1500, M3 Max 128g
Hey experts! I genuinely have this question and would love a general consensus from people actually using these models. LLMs have advanced a lot in benchmarks and in practical usage. Even GPT-4o and G
LTX Desktop v1.1.0 just shipped with some big updates. Apple Silicon Macs can generate video locally now, there's a built-in LoRA/IC-LoRA library you can browse and apply from inside the app with per-
At the moment MLX (and Llama.cpp for Macs) run 16bit activations everywhere. Despite this, the M5 generation silicon actually does support INT8 activations - it actually allows w4a8 d_type. It's just
A seasoned developer seeking advice on selecting an Apple product to locally host large language models in the 27-36 billion parameter range, discussing the trade-offs and specifications of different
Apple has removed its 256GB M3 Ultra Mac Studio from sale, limiting the machine to 96GB of unified memory , following the removal of the 512GB configuration in March . The removal is likely due to a g
An iOS application that enables large language models to run directly on-device using llama.cpp technology, allowing users to integrate their own Ollama instances for processing Apple HealthKit data t
This post benchmarks three Qwen3.6 model variants against gpt-oss:120b when running on Apple Silicon hardware, evaluating their performance characteristics and practical usability. It documents both t
Ollama consistently fails to run any model on Apple M5 hardware running macOS 26, with the Metal backend failing to initialize and the runner process terminating with a 500 Internal Server Error. The
I was unable to retrieve the specific Reddit thread or locate reliable sourced details about Apple's head of cloud making a statement that open source models will address 90% of use cases. The sear...
The Mac Mini M4 Pro with 48GB unified memory is a popular choice in the local AI community for running large language models via Ollama, as its Apple Silicon architecture makes all 48GB of RAM dire...
I built Image Studio, a local web app for Qwen-Image 2.1 on Apple Silicon. I had the model running on my Mac, but the only way to use it was the ComfyUI graph, and chaining edits meant wiring loaders,
Apple Foundation Model 3 through Private Cloud Compute is directly available in macOS 27.2 through familiar AI clients. For Mac users running local models such as Qwen 3.x or Gemma 4, AFM 3 PCC is a c
https://preview.redd.it/xs01ds930nsh1.png?width=4800&format=png&auto=webp&s=939762bcf3fbd93e137d4f97e927dda38aaf30fd Back in July I shared LokalBot here. It's a free, open-source Mac app that records
Image-to-3D models redraw your picture, and fine detail - text, logos, faces - comes back garbled or scuffed. My local pipeline that fixes it. (repo link at the end of the post) The latest update adds
It seems to use a 96-bit LPDDR5X memory bus, instead of the previous 64-bit wide busses. Considering it's on 2nm, that's expensive silicon. That should result in around 115 GB/s memory bandwidth. A20
Cosmos3 INT4 T2I + I2V on Apple Silicon — code, weights and a Grok comparison GitHub - https://github.com/gtrg55/cosmos3-quant-mlx-cuda HF weights - https://huggingface.co/JuliaML/Cosmos3-Super-Text2I
A few weeks ago, I saw this post where the author trained a SIREN MLP to implicitly memorize Bad Apple as a coordinate function: (t, y, x) to pixel. That got me curious about a slightly different form
Its a custom Swift/Metal inference engine that runs Gemma 4 26B-A4B-IT on M-series Macs with very low RAM. It uses ~2GB instead of ~14 GB. The result is reportedly 5–6 tok/s on an 8 GB M2 MacBook Air
At Bitwize, we've been building Logue, a native macOS app for AI meeting notes and writing, and we just open-sourced it (MIT). We're sharing it here because the whole point is that it runs 100% on-dev
I ran a few benchmarks on my MacBook Pro M5 Max with oMLX: Running Qwen3.8-27B with thinking off has a big effect on output quality. Running it with thinking on xhigh burns 5.5x the tokens and runs 6x
'A 12-core GPU, also with two more cores than before, now includes Neural Accelerators in each core for the first time on Mac mini, resulting in up to 4x faster AI performance and 2x faster graphics t
mlx-dspark is an MLX port of DeepSeek's DSpark speculative-decoding drafters (the DeepSpec release), plus z-lab's DFlash, with one lossless verify loop. v0.10.0 adds Qwen3.8-27B via RadixArk's drafter
Another new model dropped in the course of this week that is well deployable on consumer hardware: Nex N2.5 Mini I went with the recommended settings for the best generation quality and ran a few benc
https://huggingface.co/bartowski/LensVLM-9B-GGUF LensVLM-9B LensVLM is a 9B Vision Language Model (VLM) that scans compressed images of text, then selectively expands only the relevant pages to their
I've been experimenting with running larger models locally with Ollama, mainly because I'm interested in using them for agentic coding. I kept seeing benchmarks expressed as something like: 40 tokens/
Hi folks, I'm a long-time lurker and big fan of this subreddit and a massive self-hosting fan (also outside of AI). I doubt many people will disagree with me here because I see the same arguments bein
I believe I may currently hold the record for memory constrained inference for Qwen3.8–Flash-Next on Apple Silicon — needing only about 21GB of allocations. Introducing Cherenkov, an inference engine
Meet Inco Splash, open-source inference engine, built around the model and around Apple silicon. Up to 3× the decode speed of Ollama, 2× oMLX, and almost 4× when an agent fans out into sub-agents. Req
We have SWE-bench, Terminal-Bench, OSWorld, BrowseComp, etc. But I haven’t seen a good apples-to-apples benchmark for platforms like OpenAI Agents, Anthropic’s agent stack, AWS AgentCore, Google’s age
This may be somewhat specific to Qwen3.8 27b and the Apple M5 series, perhaps, but enough of us are running this combo that it's worth tossing out there. GGUF models with MTP have been the fastest way
I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth. Qwen3.8 Flash-Next gives a lot of detai
Given OpenRouter.ai was snapped up by Stripe, who do we think would go after the 'GitHib' of AI models? It is a big chunk of change they are looking ($13B). Apple may be a contender to give them a rea
It's a small experiment of mine to check thinking effort on Qwen and I do have to say xhigh does overthink but I'm not sure if it's bad because the result is rather amazing. Although the prompt was ve
TL;DR: The result looks TOO GOOD TO BE TRUE — that's exactly why I'm posting it. I ran Ox alpha on the entire SWE-bench Verified-Mini set with the official mini-swe-agent scaffold — the same agent use
Everything runs through WebGPU, in-browser or in electron/tauri apps. It's fully portable and supports either Nvidia and Apple Silicon (Metal). The actual kernels are optimized for the specific hardwa
Disclosure: I'm the developer. The app is called Citenook. I wanted to ask questions across a few hundred PDFs and notes with Ollama, but every setup I tried either needed a lot of glue code or gave a
Ok, so the Mac M5 Ultra (256GB) hit the market, but the only publicly available benchmarks material are flashy YouTube 'clown influencers' videos. We need serious numbers to evaluate whether Apple’s s
Hello Everyone! Good news for anyone who can compile code on apple, you can now experience using a AMD Radeon AI Pro R9700 on a MacBook Pro or Mac mini, Mac Studio or any OSX variant as long as it has
I'm about to buy a Mac Studio mainly for running LLMs locally and I'm stuck between two configs: M5 Ultra (30/64) with 96GB: 1.2 TB/s bandwidth, roughly 1.7x faster generation and much faster prefill
Hey everyone, Jovan from UkisAI here! Today, we are introducing Swift, a family of efficient reasoning LLMs based on Qwen, trained by penalizing tokens related to pathological overthinking patterns an
We are considering to buy computer, server to run decent model on premises. I would like to run big open source model with more that 70b parameters roughly. I read the people used to run on Apple Stud
Hey guys, Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFl
Spec-dec has been a thing for a while, in fact, it's wasn't an idea that was born for LLM inference. E.g. Uber's https://github.com/uber/submitqueue applied it to a merge queue. Apple & GDM had been r
TL;DR: I spent 9 days developing a new quantization method for MLX models and measured 18 variants against each other on a single M5 Max MacBook Pro (128 GB). The result is the best-measuring MLX quan
This r/ollama community guide provides a practical breakdown of which AI language models (and combinations of models) can realistically fit within the unified memory constraints of common Apple Silico
I've made a few posts recently, just talking about the hardware I've got and the model I use. Pair of P100s, Unsloth Qwen3.6 35b A3B UD Q4 K XL. There's been a ton of talk of model and hardware specif
Currently contemplating adding either M5 Ultra 256GB or dual DGX Spark in addition to existing 5090. I have been researching for past few days and have zeroed in on dual sparks because higher throughp
Been seeing quite a few posts about Strata lately, so I figured I'd give it a shot on my RTX PRO 6000 and I am very impressed. Most of the benchmarks I've seen are using IQ2 or IQ3 quants, so I wanted
I've been tinkering with GLM-5.3-Flash on two 64GB CMP 170HX cards for a while, and the setup is finally stable enough that I figured I'd share it. I also compared it against the Qwen3.8-Flash-Next se
Early this year when I was first looking at building up my inference capability you could get the 16GB Tesla P100s for between 60 and 80. Asked claude about it, told me absolutely not worth it. No ten
Good day data nerds! I am trying to get more into running models, learning about agentic workflows, and creating my own tools. But as you do (right?) I had to fine tune my current setup. With the way
Hi! I'm the author of MoEspresso, which is my way of putting my own ideas about inference engines to the test. A lot of the fun has been trying different design choices, measuring what happens, and fi
For those who have V100 cards, I wanted to point you to 1Cat-vLLM, a vLLM fork that enables optimized serving for these cards. Showing stats for Qwen3.6-35b comparing a Strix Halo with a hughly optimi
FYI the engine's default kernel rules were measured on smaller chips (16/20-core M5s and a 32-core M4 Max), so a 40-core M5 Max runs guesses. Splash's repo includes a developer tool, “make tune-kernel