OLLAMA_KV_CACHE_TYPE is global, not per-model. That is why one model got noticeably worse when you quantised the cache and the others did not.

Quantising the K/V cache is the highest-leverage memory setting most people never touch, and it has one behaviour in the docs that catches almost everyone. Every number below is out of Ollama's own FAQ, re-read today rather than recalled, because these defaults have changed before.The setting is global. It is not per-model.OLLAMA_KV_CACHE_TYPE applies to every model the server loads. The docs say so directly: it is currently a global option, meaning all models run with the specified quantisation type. There is no per-model override for it.So the usual sequence is: you set it once to fix the one model that was spilling, everything keeps working, and then weeks later a different model feels dumber and you have no idea why. Nothing about that model changed. You quantised its cache too.What each setting actually costs you:f16 is the default. High precision, high memory.q8_0 uses roughly half the memory of f16, with a very small loss the docs describe as usually having no noticeable impact on quality. This is the recommended one if you are not staying on f16.q4_0 uses roughly a quarter, with a small-to-medium loss that gets more noticeable at higher context sizes.Read that last clause carefully, because it is the opposite of the intuition. Cache quantisation is not a fixed quality tax. It gets worse the longer your context is, which is exactly the situation that made you want the memory back in the first place.Why the damage is uneven across your models:The docs are explicit that the impact depends on the model, and they name the mechanism: models with a high GQA count may see a larger precision hit than models with a low GQA count. Their own example of a high-GQA family is Qwen2.That is the answer to "why did only one of my models get worse". It is not random and it is not placebo. Two models under one global setting genuinely do not pay the same price for it, and the docs tell you which direction to expect.The Flash Attention advice you have read is probably stale:A lot of guides still say you must set OLLAMA_FLASH_ATTENTION=1 before cache quantisation will do anything. As the FAQ reads today, Ollama uses Flash Attention automatically when the selected backend and devices support it. The env var is now a manual override in both directions: 1 forces it on, 0 turns it off. Worth checking yours rather than trusting me or a guide from a year ago.A sane order, cheapest first:1. ollama ps. Confirm you are actually spilling before changing anything. If the PROCESSOR column does not say 100% GPU, that is your problem and the model is not what is wrong.2. ollama stop anything you are not using. Default keep-alive is 5 minutes and default OLLAMA_MAX_LOADED_MODELS is 3 per GPU, so something you tried an hour ago can still be holding VRAM.3. Drop the context to what you actually need. Default is 4096 whatever the model card advertises.4. Then q8_0 on the cache. Not q4_0 as a first move.5. Only after all of that, drop the model quant.Most people run that list backwards, give up real model quality first, and never touch the thing that was actually eating the card.If the sizing side is useful, weights versus context memory per model and what the quant suffixes actually mean is written up here: https://noizz.io/local-ai submitted by /u/blossend [link] [comments]

Source: r/ollama | 2026-08-27

Related