The 27B FP8 uncensored weights are ~31GB, so 'just run it on a 24GB card' doesn't hold up

The uncensored FP8 build that showed up on HuggingFace this week is about 31GB in block-FP8, roughly half the BF16 weights. People keep reading "27B" and assuming it drops onto a 3060 or any single 24GB card. The weights alone already don't fit, and that's before the KV cache. That's the part the "does 27B fit in 24GB" threads skip. The card lists a 262K context window, and long context is what actually eats VRAM once the weights are loaded. Fill any real amount of it and you're realistically on an 80GB card (a single H100 or H200) to serve it without quanting further, at which point it stops being the exact-FP8 drop-in that made this build interesting. So for most people the honest options are quant it down harder and lose that property, or just don't run this specific build locally. If you mainly want the model's output and not this specific local build, the plain 27B is also up as a free hosted version. It's $0 usage because the weights are open and self-run, so there's no per-token vendor cost to pass on. It happens to sit on OrcaRouter, which fronts 200+ models behind one endpoint at 0% token nmarkup, so it's the same base_url whether you're hitting the free 27B or something frontier. I haven't seen an independent throughput number for that free tier though, and it's rate-limited, so it's not a straight swap for a local box. For people actually running 27B-class locally: what's your real floor? Is 2x24GB the practical entry, or is anyone getting usable speeds on a single 24GB with aggressive quant and a short context? submitted by /u/Crafty-Fennel1252 [link] [comments]

Source: r/ollama | 2026-08-15