I benchmarked Qwen 3.8 27B on my M4 Max — here's what I learned about local models, context, and coding performance

I've been experimenting with running larger models locally with Ollama, mainly because I'm interested in using them for agentic coding. I kept seeing benchmarks expressed as something like: 40 tokens/sec That's useful, but I eventually realized it doesn't answer the question I actually care about: How long am I going to sit there waiting when the coding agent has 8K, 16K, 32K or more project context? So I spent some time testing this properly on my machine. This post isn't meant to say that one model or runtime is "best." I mainly wanted to understand what the numbers actually mean in day-to-day use. And some of the results surprised me. My machine Everything here was tested on: MacBook Pro Apple M4 Max 36 GB unified memory macOS 26.6.2 Ollama 0.34.4 The main model was: Qwen 3.8 27B NVFP4 / MLX 128K configured context medium thinking I also tested the regular Q4_K_M/GGUF version of the same 27B model so I could compare it with MLX. My Ollama server was configured roughly like this: OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_NUM_PARALLEL=1 OLLAMA_KEEP_ALIVE=5m A quick explanation if you're new to this: Flash Attention reduces the cost of attention, which becomes particularly important as context gets larger. q8_0 KV cache reduces the memory used by the model's working context compared with keeping that cache at full precision. I chose q8_0 as a compromise between memory use and precision. NUM_PARALLEL=1 means I'm testing one request at a time rather than allowing concurrent inference to muddy the measurements. And KEEP_ALIVE=5m keeps a loaded model resident for a while, which lets me separately look at cold starts and warm performance. These settings matter. Local-model benchmarks aren't just "model + computer." Runtime configuration can change the result too. First: what does "tokens/sec" actually mean? Models don't really read and write words. They operate on tokens, which are small pieces of text. When someone says: 40 tok/s they usually mean the model is generating around 40 output tokens every second. That's decode speed. And my MLX model was indeed around there. Across repeated warm measurements I was seeing roughly: Decode: ~40–41 tok/s TTFT: ~3.5 seconds Prompt: ~180–190 tok/s Those are nice numbers for a 27B local model. But there's another half of the equation. Before the model can write its answer, it first needs to read/process your input. That's usually called prefill or prompt processing. For coding, that input might contain: your instructions system prompts conversation history source files tool results build errors documentation previous edits As an agent works, that context can get large. And that's where things become interesting. MLX made a huge difference on my Mac Before looking at context scaling, I compared the regular Q4_K_M version with the NVFP4/MLX version. My measured decode performance was roughly: Model format Decode speed Q4_K_M / GGUF ~10.7 tok/s NVFP4 / MLX ~40–41 tok/s That's roughly a 3.8x difference in decode throughput in my tests. The Q4_K_M model wasn't broken. It worked. But the user experience was completely different. At around 10 tok/s, a long coding response can take minutes just to generate. I actually asked the non-MLX model to create a Next.js/Tailwind/Framer Motion landing page during my experiments and ended up waiting somewhere around 18 minutes for the overall task. That's what made the benchmark numbers start feeling less academic. On this particular Apple Silicon machine, MLX was a very substantial improvement. For anyone unfamiliar with it: MLX is Apple's machine-learning framework designed around Apple Silicon and its unified-memory architecture. It isn't another Qwen architecture. It's part of how the model is represented/executed on the machine. Likewise, NVFP4 and Q4_K_M describe weight formats/quantization, not different Qwen architectures. Then I tested what happens as context grows This was the part I actually wanted to understand for coding. Instead of only testing a tiny prompt, I tested approximately: 4K 8K 16K 32K of active input. The result: Active input Time before generation Prompt processing Decode ~4K 23.6 sec 198 tok/s 46.6 tok/s ~8K 46.8 sec 178 tok/s 41.4 tok/s ~16K 1m 30s 176 tok/s 30.8 tok/s ~30K 3m 15s 157 tok/s 31.9 tok/s This table taught me more than the original 40 tok/s number. At ~8K, waiting around 47 seconds before generation begins may be perfectly reasonable depending on what the agent is doing. At ~16K, I'm already around a minute and a half. At ~30K, I'm waiting more than three minutes. And notice something important: The model is still generating at ~32 tok/s at 30K. If I only reported decode speed, that would sound pretty good. But as a user, I've already waited three minutes before that generation even begins. That's why I think TTFT — time to first token — is particularly important for local coding models. What seems usable for agentic coding? This part is subjective, so I don't think there should be some universal rule saying "16K is good" or "32K is bad." For me, though, the measurements give a useful mental model. Around 4K, the interaction still feels relatively responsive for a large local model. Around 8K, I'm waiting close to a minute, but that could still be reasonable if the agent is about to perform meaningful work. Around 16K, the wait becomes much more noticeable. By 32K, we're talking several minutes before generation. That doesn't mean you should configure your model with a tiny context window. A 128K configured context gives the model room when it needs it. The important distinction is: Configured context capacity is not the same thing as currently occupied context. You can configure 128K and only use 8K. Think of the context window as the size of the desk available to the model. A 128K desk doesn't mean you've covered the entire desk with documents. But if you actually put 100K tokens worth of documents on it, the model has a lot more material to process. So what does 128K actually mean? Another thing that confused me initially was context size itself. A 128K context window is approximately the amount of token space available for the model's working conversation/context. That space has to accommodate the information involved in the interaction — input/history and the generation budget within the runtime/model's context handling. For an agentic coding workflow, a larger context can be valuable because the agent may need to keep track of: architecture source files previous changes requirements errors test results tool outputs But having the capacity doesn't make processing 128K free. My stress test made that extremely obvious. Then I tried to break it After testing the more practical sizes, I pushed the same configuration toward its 128K limit. I tested approximately: 32K 64K 96K 115K At each level I also embedded four deterministic markers throughout the context and asked the model to retrieve them. That gave me a basic integrity check: Did the runtime merely accept this giant prompt, or can the model still access information distributed throughout it? Here are the results: Context Integrity Time before generation Prompt Decode ~32K 4/4 PASS 3m 29s 152 tok/s 29 tok/s ~64K 4/4 PASS 7m 55s 134 tok/s 24 tok/s ~96K 4/4 PASS 14m 55s 107 tok/s 20 tok/s ~115K 4/4 PASS 21m 01s 87 tok/s 14 tok/s This was both impressive and slightly ridiculous to sit through. :) The interesting part is that all four integrity checks still passed at ~115K. So the model/runtime really was handling the long context. But technically working and being pleasant to use are clearly two different things. At ~115K, I waited roughly 21 minutes before generation and decode had fallen to around 14 tok/s. That's stress-test territory on this machine, not something I'd want every coding-agent interaction to look like. Memory tells the other half of the story The larger contexts also started putting substantially more pressure on my 36 GB of unified memory. Around the 64K stage, I saw approximately: Available memory: 89% → 27% Swap: 529 MB → 2.5 GB Around 96K: Available memory: 89% → 24% Swap: 529 MB → 2.9 GB That's another reason I don't think context-window specifications should be read as: "My model supports 128K, therefore 128K should be comfortable." It may support it. Your hardware still has to pay for it. On Apple Silicon, CPU and GPU share unified memory, so model weights, context/KV state, applications and the rest of the system are competing within that memory architecture. Once macOS starts leaning harder on compression and swap, that's useful context for understanding the benchmark rather than just staring at tok/s. My main takeaway For local agentic coding, I've stopped thinking about performance as one number. I now think about at least three: Prompt processing How quickly can the model digest my context? TTFT How long before generation actually begins? Decode How quickly does it generate once it starts? And memory pressure matters too. A model saying: 40 tok/s can be completely true while the actual user waits three minutes for a large coding context to be processed. Likewise, a model advertising: 128K context can genuinely handle close to that amount while taking 20+ minutes to start responding on a particular machine. Neither specification is wrong. They're just describing different parts of the experience. I ended up building a small benchmark tool for this All of this started with me manually sending Ollama API requests. That got annoying quickly, so somewhere between staring at terminal output and wondering why my MacBook had turned into a very expensive space heater, I ended up turning the experiments into a CLI: Devbits Ollama Bench https://github.com/devbitsxyz/ollama-bench It's an open-source CLI for benchmarking local Ollama models, with a particular focus on the things I wanted to understand during this experiment: context scaling, TTFT, prompt processing, decode performance, memory pressure and cold/warm behaviour. The idea isn't to replace lower-level benchmarking tools. I wanted something I could run interactively and use to answer a more practical question: What is this model actually going to feel like on my machine? It has four benchmark modes: Quick — a fast, repeatable baseline. Practical — tests isolated 4K → 8K → 16K → 32K workloads to show how everyday context scaling affects performance. Stress — pushes toward the configured context limit with integrity checks and memory-pressure warnings. Custom — lets you choose the context workloads yourself. It detects the machine and locally installed Ollama models, supports thinking levels, measures prompt/decode throughput and TTFT, watches memory pressure, and produces Markdown and JSON reports. It also deliberately avoids silently downloading models or silently reducing the context you asked it to test. Large-context runs can get expensive, so it warns before starting them. Getting started is intentionally boring: git clone https://github.com/devbitsxyz/ollama-bench.git cd ollama-bench chmod +x devbits-ollama-bench ./devbits-ollama-bench There's also a demo mode for exploring the terminal UI without actually running inference, which became rather useful while developing the Stress mode. My M4 Max had already contributed enough heat to the project. The longer-term idea is to optionally let people submit benchmark reports to devbits.xyz so we can compare real hardware + model + runtime configurations rather than relying on isolated screenshots. That would be opt-in, and I'd want the raw configuration and protocol information alongside the numbers so we're comparing like with like. The project is now on GitHub under the MIT license: Devbits Ollama Bench: https://github.com/devbitsxyz/ollama-bench If you try it on different hardware, I'm particularly interested in how context scaling behaves. A 32K or 64K run on another Apple Silicon generation — or completely different hardware — is much more interesting to me than another isolated "X tokens/sec" screenshot. What about your machine? This experiment changed how I think about local-model performance. I'm much less interested now in asking only: How many tokens/sec does it get? and much more interested in: How long does it take to digest the context I actually use? So for people running local models for coding: what context size do you actually use most of the time? And at what TTFT does a local model start feeling too slow for you? If you run Devbits Ollama Bench, I'd also love to see what you get. submitted by /u/rodrigodevbits [link] [comments]

Source: r/ollama | 2026-10-02

Related