AI Wiki — Knowledge Base

I turned Qwen3.8-27B Q2_64 + llama.cpp into a fully TypeSafe AI-compatible Jev-like system. OpenAI API still intact! World’s first Vision-enabled Jev-like model! <10 GB VRAM, 170 ms on an RTX 3090 and ~140 tok/s in chat. 76% vs. 88% Jev-1.13 Acc. on a diverse 22,000-request typed-decision benchmark

Yesterday I released Bonsai-Llama-Jev - a Jev-like Typed Decision inference system for locally operated, sovereign AI. In my new typed-decision-bench, covering more than 22,000 decisions across 175 us

model-releasesr-localllama

23 Sept 2026

Search: “engineering”

Question: Why is prefill unbelievably faster in vLLM than other inference engines?

Mreaia v1.0.0: Universal Zero-Token Engine & Hardware-Locked Context Distillation for LLMs (>99.9% Reduction)

LLM Inference Dashboard

poorman inference engine for 16GB GPU and 35B moe Qwen 3.6for coding

Flyweight: open-source C++/CUDA engine for running MoE models bigger than your VRAM on one GPU + system RAM. First PyPI release, looking for contributors.

52 knobs, 72 hours, one compiled Rust model: Catalyst told an AI agent what actually mattered in 1.37s and verified every answer

1-bit 27B in the browser: 25–30 tok/s on a 6 GB RTX 3060 Laptop (WebGPU, no install)

Built a local memory engine for Ollama that runs in <1.2GB VRAM without vector DBs (Hillock v0.5)

Local agentic coding Benchmark : Qwen 3.8 27B (in many weights quants / cache quants / engine / reasoning effort) vs others.

NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments

How Kimi K3 Engineered Its Way to the Frontier [R]

I made an Anima AI Character & Artist search engine with 49,000 sample images

Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and Infernix

Cactus Needle 3: A Sliceable 8-29MB Automation Foundation Model That Matches DeepSeek v4 Flash

NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090

R9V: A designer set of kernels I've been working on for R9700s/RDNA4. Qwen3.8-Flash-Next Unsloth IQ4_XS (w/ TP on 2 R9700s, MTP, SSD n-gram, 128k ctx, vision): TG256 of *78 tok/s* (~3x increase), PP8192 of *1510 tok/s* (~30x increase).

Benchmarks: TensorSharp vs. llama.cpp

Tencent HY-World-2.0 is now public

Java vllm-like framework claims 90% of perfomance of llama.cpp on local inference on NVIDIA GPUs by compiling Java to CUDA and cuTile

Running an LLM-driven town with 800+ persistent agents: concurrency, context caching, and inference costs

I built Ninfer 4080 for 16GB class GPUs

The models are not trying to escape. They are trying to finish the task.

For the longest time I’ve felt this sub should have a pinned section where a detailed post about each model should get featured.

Systems for Machine Learning[D]

Finally got Qwen 3.8 Next running on my v100 6gpu setup (TP2 PP3)

Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro

Faster than Light in Air: 8-22 tg/s Qwen3.8-Flash-Next (Q4/Q4ish) on a 32GB M4 MacBook Air

Validate your local LLM advertised KV cache against real pressure; see exactly how old contexts get evicted from cache

I ported vLLM's serving stack to C++20: 66 MiB binary, no Python at inference, output checked token-for-token against vLLM

Has anyone been working on a solid setup for DSV4F on x2+ R9700s?

I tried running a 1.56TB MoE model on a 6GB RTX 4050 Laptop, Here’s the result

Spent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.

Final revision of zzpanic vllm-radiance for 1 x R9700 GPU, qwen3.8-27b

Basalt: Flash-Next at 665 tok/s structured, 354 prose on a 5090 + 5060 Ti (2.6x Strata)

BigMoeOnEdge+ – an Android chat app fork of BigMoeOnEdge (llama.cpp / GGUF)

Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB + 16GB DDR4 RAM(No gate pruning, 100% bit-exact)

I made my iPhone a second GPU for my 24 GB MacBook: Qwen 3.8 27B prefills 29–44% faster & my holds part of the CTX window.

FreeToken vs llama.cpp on one RTX 3090: llama.cpp is 2–3× faster when the MoE fits in VRAM. On gpt-oss-120b (63 GB), FreeToken gets the first token out 7× faster at 32 concurrent users.

Qwen3.8-Flash-Next (125B) at 12-15 tok/s on a 2021 32GB M1 Max

Splash fork optimised for M5 Max: ~1.5× faster (1.25× single request)

Splash on a 40-core M5 Max: +20% decode by tuning the kernels for your own chip

The bear can dance: Qwen 3.8 27B on one 3090 for 3 weeks

[P] Built a 100% Client-Side Vision Pipeline for Real-Time Chessboard & Multi-Board Detection (Chrome/Firefox Extension) [P]

Qwen3.8-Flash-Next (UD-IQ4_XS) on 2x RTX 3060 + 7800X3D, from initial 36 tps prefill to 400 tps and other benchmarks (-sm tensor trap) + VRAM/RAM usage

Qwen3.8-Flash-Next: Time to Update Those Benchmarks

NVFP4 on VOLTA! Despite being built for Blackwell, I made four 2017 V100s run Qwen 3.8 NVFP4 natively and match my $6000 RTX 5090.

DeepSeek V4-Flash (284B MoE) at 33 tok/s single / 68 tok/s aggregate on 2× RTX 3090 + a used quad-Xeon DDR4 server — full config

R9V: A designer set of kernels I've been working on for R9700s/RDNA4. Qwen3.8-Flash-Next Unsloth IQ4_XS (w/ TP on 2 R9700s, MTP, SSD n-gram, 128k ctx, vision): TG256 of 78 tok/s (~3x increase), PP8192 of 1510 tok/s (~30x increase).