Model Releases
Running decision models: Picking quant, VRAM usage
Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them. I want to try them out. Are decision models sensitive to quantization? I know that for autocom
Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them. I want to try them out. Are decision models sensitive to quantization? I know that for autocomplete (FIM) models like qwen-2.5-coder, it is recommended to use Q8 instead of Q4 to preserve quality. Does this also apply to decision models? Also, do decision models use more VRAM than the model size itself? If yes, what do they use VRAM for? Normal LLM use VRAM for context, but I think decision models do not have "context" (I can be wrong). For reference, I have a 6GB GPU and want to use ggml-org/vllm-d3-mini-GGUF Q8 (5.16 GB). Edit: Should I keep the decision model in VRAM only? For MoE models it is generally okay to split the weights to both VRAM and RAM, given that your VRAM can fit the active experts. I think they should be VRAM only, as the base model used for creating these decision models (i.e. qwen) is dense. submitted by /u/regunakyle [link] [comments]
Related
- Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards
- I turned Qwen3.8-27B Q2_64 + llama.cpp into a fully TypeSafe AI-compatible Jev-like system. OpenAI API still intact! World’s first Vision-enabled Jev-like model! <10 GB VRAM, 170 ms on an RTX 3090 and ~140 tok/s in chat. 76% vs. 88% Jev-1.13 Acc. on a diverse 22,000-request typed-decision benchmark
- Laguna-S-2.1 'thinking forever' loops seem to be a quantization artifact
Source: r/LocalLLaMA | 2026-10-13