Local Ai
Ornith-397B running at Q4 on a single RTX PRO 6000 Blackwell 96GB - 2,354 tok/s prefill, ~20–24 tok/s decode
I've been building Krasis, an MoE-focused runtime for streaming big models through limited VRAM on NVIDIA consumer/workstation GPUs, and I think this is the most interesting result so far: Ornith-1.0-
I've been building Krasis, an MoE-focused runtime for streaming big models through limited VRAM on NVIDIA consumer/workstation GPUs, and I think this is the most interesting result so far: Ornith-1.0-397B running interactively on one GPU. Hardware: 1× RTX PRO 6000 Blackwell 96GB + AMD EPYC 7742 (64c although the CPU isn't really relevant to the run) with enough DDR4 system RAM to hold the model in RAM at Q4. The model obviously doesn't fit in 96GB, Krasis keeps experts in CPU RAM and dynamically manages which ones stay resident in VRAM (~43% of routed experts resident for this run). Peak process RAM was ~202GB, so you need beyond that amount in order to run it but 256GB ram is doable with a consumer DDR5 motherboard. Measured numbers (INT4 experts, HQQ4 attention, 4-bit KV): 1,346.3 tok/s prefill at 10,000 tokens: 7.43 seconds 2,354.5 tok/s prefill at 39,920 tokens: 16.95 seconds 23.58 tok/s decode over 50 tokens 21.85 tok/s decode over 100 tokens 20.40 tok/s decode sustained over 250 tokens 25.73 tok/s over 50 tokens (+9.1%) using Krasis Adaptive Cold Mass Pruning, which skipped low-ranked, non-resident expert routes while omitting just ~1.8% of routed probability mass on average during the measured run. The same runtime also runs smaller MoEs much faster when everything fits in system RAM (e.g. 35B-class models at ~117 tok/s decode on a 5090), and Ornith-397B even runs on a single RTX 5090 32GB at ~7.9 tok/s decode if you're patient. Github repo Speed benchmarks Quality benchmarks Happy to answer questions or run specific prompts/configs people are curious about. submitted by /u/mrstoatey [link] [comments]
Related
- Do people building local LLM rigs track RTX Ada/workstation card prices, or just consumer cards like the 5090?
- Qwen3.6-35B-A3B APEX on a Single RTX 3090 - Getting the Most Out of It
- What options exist for running the largest local models at full precision?
Source: r/LocalLLaMA | 2026-07-27