Model Releases
Your model fits in VRAM. Your context doesn't. The KV cache math nobody shows you.
Spent way too long confused about why an 8B at Q4_K_M, about 4.9GB on disk, would OOM on a 12GB card. The weights fit twice over. The answer is the KV cache, and the math is simple enough that you can
Spent way too long confused about why an 8B at Q4_K_M, about 4.9GB on disk, would OOM on a 12GB card. The weights fit twice over. The answer is the KV cache, and the math is simple enough that you can work out your own ceiling in about a minute. The formula KV bytes = 2 x layers x kv_heads x head_dim x context_tokens x bytes_per_value The 2 is K and V. bytes_per_value is 2 at fp16. Llama 3.1 8B, worked out 32 layers, 8 KV heads (it is GQA, not 32), head_dim 128. 2 x 32 x 8 x 128 = 65,536 values per token x 2 bytes = 128 KB per token So: 2k context: 256 MB 8k context: 1 GB 32k context: 4 GB 128k context: 16 GB Weights are 4.9GB. At 128k the cache is more than three times the model. Why you might not have hit this yet Ollama does not hand you the model's full advertised context by default. If you never touched num_ctx you are running a small window and paying almost nothing for cache. The moment you raise it to actually use that 128k you read about on the model card, the cache shows up, and it shows up fast, because it is linear in tokens. /set parameter num_ctx 32768 The part that actually matters when picking a model Two models with the same parameter count can have wildly different cache costs, because kv_heads and layers differ. A model with 8 KV heads costs a quarter of an equivalent one with 32. Old multi-head-attention models are brutal at long context. GQA models are cheap. This is not in the parameter count and it is not in the quant name. You have to open config.json and read num_key_value_heads and num_hidden_layers yourself. Which is the annoying part, because those two numbers are almost never on the same page as the download button. I keep my model comparisons on noizz.io so I stop re-deriving this every time I swap, but honestly a config.json and a calculator gets you the same answer. Rule of thumb Budget VRAM as weights plus cache, never weights alone. On consumer hardware a smaller GQA model at a big window usually beats a bigger model you are forced to keep at 4k. submitted by /u/blossend [link] [comments]
Related
- fraQtl D1 for Mistral-7B-v0.3: long-context KV that survives 128K where q4/q8 KV collapse (NIAH receipts)
- Mistral-7B v0.3 at 128K in llama.cpp: 22,657 → 13,235 MiB live VRAM with ≤0.004 PPL drift
- Running Gemma 4 QAT 12B on an 8GB GPU at 16k context — measured the KV-cache tradeoffs
Source: r/ollama | 2026-08-23