Model Releases
NInfer fork: 555k context@fp4 for 5090 with YARN, reliable kv host cacheing, monitoring, jinja, opened model support
Hiya, NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code sessio
Hiya, NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code session until it didn't break any longer. Hope you like it. NVFP4 KV cache (from scratch) I implemented a 4-bit KV cache for QIn3.8-27B from the ground up. Upstream has since added their own NVFP4 path, but ours differs architecturally: Custom MMA kernel (mma_nvfp4_e4m3, m16n8k64) with hardware E4M3 block scales for the QK matmul. Both Q and K are quantized to NVFP4; V is dequantized to BF16 for the PV matmul via a dedicated decode kernel. Hadamard rotation applied to K (and Q) pre-quantization for outlier suppression, with V left unrotated. Upstream uses fp16 V storage instead — no outlier suppression. Fused append: the decode kernel quantizes current K/V to NVFP4 in-place during generation — no separate quantization pass. Custom scale layout: natural row-major for KV scales (not the M128x4 swizzle used for weight MMA), because KV access patterns differ from weight access patterns. Result: 144 bytes/token/KV-head (vs 264 for int8, 512 for bf16) — 45% VRAM reduction with no quality loss (LongBench 45% matching int8, AIME 96.7%, needle-in-haystack 100%). YaRN context extension QIn3.8-27B's RoPE config (theta=1e7, 25% rotary dims, 48/64 GDN layers with no RoPE) makes linear scaling sufficient — full NTK-by-parts is unnecessary. I extend native 262k to 555k (c=3+vision) or 600k (c=1) on a 5090. Quality verified at 600k: LongBench matches int8 baseline, coherent 592k-token output. Also projected 8M token context on 96GB+ GPUs (untested, I only have a 5090). Multi-level prefix reuse with host-KV safety net Upstream implements a budget-bounded HostKvProvider with LRU park/restore. I replaced it with a substantially different system: HostKVSafetyNet: pinned host arena with scatter-gather multi-extent allocation, arena compaction, and a pin/take protocol for safe concurrent restore. Two-level prefix matching: full execution frontier first, then rewrite checkpoint fallback. Each entry carries a ResidentPrefixIdentity (per-token type/position/vision metadata), rolling FNV digests for shortlist, and a compact_prefix (reasoning-stripped token prefix) for thinking-mode consistency. Session-key fallback: when prefix matching fails (e.g. Claude Code drops reasoning betIen turns), a session-key fallback matches by conversation identity instead of token content. Spill-before-evict at every release path: pressure planner eviction, normal continuation release, start_sequence slot takeover, and fail-all cleanup all route through the safety net. Token stability: reasoning is dropped from ALL assistant messages when preserve_thinking=off, keeping the prompt token stream stable across turns. Checkpoint capture is anchored at the turn boundary, not the execution frontier. Verified across 260+ requests with 3 concurrent 330k-470k sessions — zero re-prefills on cached turns, H2D restore cost ~0.4s, D2H spill at 67K pages/s. Performance (3 concurrent sessions, 400k+ ctx) Metric Value Decode at 400k+ ctx 117 tok/s (MTP 4.62 tok/round, 92% acceptance) Cached turn turnaround 2-16s (414k cached, 1-14k delta) Cold start prefill 260s (414k tokens at 1600 tok/s) H2D restore cost 0.4s per evicted turn Host KV 30 GB (96% utilized, 181 evictions managed) Tool calling --tolerant-tool-calls: recovers complete Qwen calls when the model emits malformed wrapper/suffix tokens — instead of dropping the call. Depth-matching close scan: handles balanced/nested markers in parameter values that would break naive parsers. Responses API accepts text/reasoning after tool calls (upstream rejects this ordering). Froggeric v22 template: C++ renderer with no-dangling-intent rule, XML think tags, correct function tag delimiters. Some further modifications for reliability. Also included Dynamic chat template loading (--chat-template) — supports any .ninfer image without artifact patching Explicit weights profile override (--weights-profile) — handles Ostfralla, QUASAR, and other converter layouts with per-layer tensor format auto-detection OOM recovery: catches std::bad_alloc, clears state, preserves pending requests Stream sync fix: synchronize CUDA stream before workspace reset in prefill (prevents use-after-free) Request-log rotation (--request-log-max-mib, --request-log-keep) for bounded disk usage Admission pressure fix: un-suppress demote-to-host when candidate needs host KV budget Monitoring dashboard with live KV occupancy, decode/prefill graphs, 12VHPWR sensor E2E test suite for KV eviction, device pressure, and slot pressure scenarios Removed hash check of models, use any NInfer you like as long as there is a supported path. Tested with Ostfralla and QUASAR. Fork: https://github.com/gzenz/ninfer (master) Research: https://github.com/gzenz/ninfer/blob/master/docs/maintainer/kv-nvfp4-yarn.md I'll keep rebasing from upstream what seems useful and experimenting with new papers in order to improve speed and context. submitted by /u/Lumpy-Comedian-1027 [link] [comments]
Related
Source: r/LocalLLaMA | 2026-09-05