24 GB of VRAM is not really 24 GB for a local LLM. Here is the worksheet I use

I kept seeing model file size compared directly with the number printed on the GPU box. That misses several memory buckets. A simple planning model is: usable capacity = advertised VRAM x 0.90 total target = model weights + KV cache + runtime headroom The 90% value is only a conservative starting point. Measure your own system once it is running. For exact four-bit weight arithmetic, the floor is: 7B: 3.26 GiB 13B: 6.05 GiB 32B: 14.90 GiB 70B: 32.60 GiB That is only the clean parameter floor. Real GGUF, GPTQ, AWQ, and other files can be larger because scales, metadata, mixed precision tensors, and format choices also take space. Using the 90% planning rule, the card labels become: 8 GB: 7.2 GiB 12 GB: 10.8 GiB 16 GB: 14.4 GiB 24 GB: 21.6 GiB 32 GB: 28.8 GiB Then add KV cache. A useful starting equation is: 2 x layers x KV heads x head dimension x cached tokens x bytes per element x concurrent sequences Double the context and the cache roughly doubles. Double concurrent full-context requests and it doubles again. My order of operations is: Start with the exact checkpoint, not only the parameter count. Add KV cache for the context and concurrency you will really use. Add runtime workspace and headroom. Compare the total with usable capacity, not the box label. Benchmark peak memory, time to first token, and tokens per second. A model fitting in memory does not mean it will be fast. CPU offload can make it load while making generation much slower. I run ResearchAudio and built a browser calculator for this. It is free and does not upload your inputs: https://tools.researchaudio.io/llm-gpu-memory-calculator/?utm_source=reddit&utm_medium=community_referral&utm_campaign=ai_evidence_lab&utm_content=ollama_vram_worksheet If you have measured peak memory for a specific model, quant, context, and GPU, please share it. I want to compare the worksheet with real setups. submitted by /u/dever121 [link] [comments]

Source: r/ollama | 2026-08-09