Model Releases
Solving local MoE PCIe offloading stalls (Mixtral / Qwen-MoE / DeepSeek) via Lookahead Pinned-Memory Prefetching (+46.5% tok/s on 12GB GPUs)
When running sparse Mixture-of-Experts (MoE) models locally via Ollama (`mixtral:8x7b`, `qwen-moe`, `deepseek-moe`) on 12GB–16GB consumer GPUs, expert weights overflow VRAM into system DDR4/DDR5 RAM.
When running sparse Mixture-of-Experts (MoE) models locally via Ollama (mixtral:8x7b, qwen-moe, deepseek-moe) on 12GB–16GB consumer GPUs, expert weights overflow VRAM into system DDR4/DDR5 RAM. Every token step incurs a blocking PCIe Gen4 x16 transfer stall, dropping generation speed to ~20 tok/s and pushing GPU temperatures above 82°C. I built an open-source PyTorch orchestration layer (NPOL Engine) with a Local Ollama PCIe Bridge that solves this using two mechanisms: Lookahead Low-Rank Speculative Probe (L-k): A tiny rank-32 probe (<0.0004% parameter overhead) reads hidden states 3 layers ahead and asynchronously pre-stages predicted experts from Pinned Host RAM (pin_memory=True) into GPU VRAM on a non-blocking CUDA stream while upstream layers compute (67.0% VRAM cache hit rate, saving 35.8% PCIe stall time). Local Thermal & VRAM Guard: Monitors GPU junction temperature (Tj < 80°C) and sheds borderline tokens (p1 - p2 <= 0.18) to already-resident VRAM experts instead of triggering a cold PCIe fetch, keeping peak temps around 58°C while preserving 99.55% output cosine similarity. Verified Consumer GPU Benchmarks: RTX 4070 (12GB):** 20.2 tok/s -> 29.6 tok/s (+46.5% speedup, 35.8% PCIe stall saved) RTX 4080 Laptop (12GB):** 19.6 tok/s -> 28.9 tok/s (Prevents laptop thermal throttling) RTX 3060 (12GB): 19.2 tok/s -> 28.5 tok/s (82.5°C -> 58.0°C) RTX 4090 (24GB): 27.6 tok/s -> 37.2 tok/s (74.2% VRAM cache hit rate) Includes a CLI benchmark suite + dual-mode FastAPI telemetry dashboard (Datacenter 8x A100 mode + Local Ollama PCIe mode). Would love feedback from anyone running local MoEs! (Open-source MIT repo link is in the first comment below) submitted by /u/luckyraj_16 [link] [comments]
Related
- Built an SQLite-based memory system for Ollama that doesn't eat all your VRAM
- Detecting hallucinations in local models without eating VRAM: What we learned testing 1.5B to 120B models
- M3 16GB running Ollama (Qwen 9B) is extremely slow (10-12 mins per task). Am I doing something wrong?
Source: r/ollama | 2026-10-18