Model Releases
TIL VRAM disappears before the model even loads 🫠what we measured on 6 GB and 8 GB laptops with Ollama
tbh we sized models for small cards by weight file size and kept getting it wrong. These are the hidden costs we measured on real 6 GB and 8 GB Windows laptops running Ollama: - Vision projector: with
tbh we sized models for small cards by weight file size and kept getting it wrong. These are the hidden costs we measured on real 6 GB and 8 GB Windows laptops running Ollama: - Vision projector: with a projector attached, Ollama holds back about the projector's size plus 1 GiB. Removing the projector was the difference between fitting and not fitting at 32K. - The display: when the same card drives the screen, 0.8 to 1.8 GB is already gone before the model loads. It moved with whatever else was open. - System RAM, not VRAM: llama.cpp keeps a prompt cache in system RAM (8 GB by default). On the 6 GB laptop the runner crashed three times in one night. Part of that was this cache, part was a memory leak in our own process (our bad). - Spilling past the card: on a 9B model, spilling 12% to system RAM cost about a third of the speed, and 27% cost almost two thirds. So imo "fits in 8 GB" doesn't mean much without the machine around it. We now load the model at a few context sizes and read /api/ps (size_vram vs size) on each PC instead of trusting the file size. YMMV on other setups for sure. What did you lose before the model loaded on yours? 👀 Esp. curious about AMD and Intel cards, which we haven't measured. submitted by /u/Stunning-Sherbet1853 [link] [comments]
Related
- I built a desktop benchmark for local Ollama models — TTFT, tok/s, VRAM, with a HuggingFace GGUF catalogue
- I built a thing that benchmarks Ollama models on your own hardware and ranks them
- 24 GB of VRAM is not really 24 GB for a local LLM. Here is the worksheet I use
Source: r/ollama | 2026-10-17